2023-11-16T18:00:11.838 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-16T18:00:11.838 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_choffman@teuthology --archive /home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950 --name choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi --verbose --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} -- /home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950/orig.config.yaml 2023-11-16T18:00:11.897 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-16T18:00:11.990 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950 branch: wip-choffman-62793 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} email: null first_in_suite: false job_id: '7460950' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-choffman-62793 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-62793-testing sha1: 1c9565146c8647d4e00eb6c7ae626dc192680e7b owner: scheduled_choffman@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8994 sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 sleep_before_teardown: 0 suite: fs suite_branch: wip-62793-testing suite_path: /home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa suite_relpath: qa suite_repo: https://github.com/chrisphoffman/ceph.git suite_sha1: 1c9565146c8647d4e00eb6c7ae626dc192680e7b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/misc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-16_17:54:03 tube: smithi user: choffman verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1224587 2023-11-16T18:00:11.991 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa; will attempt to use it 2023-11-16T18:00:11.991 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks 2023-11-16T18:00:11.992 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-16T18:00:11.993 INFO:teuthology.task.internal:Checking packages... 2023-11-16T18:00:12.015 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '5e4a743ad7fd1422ebd6c4dc950a21a303279207' 2023-11-16T18:00:12.015 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-16T18:00:12.016 INFO:teuthology.packaging:ref: None 2023-11-16T18:00:12.016 INFO:teuthology.packaging:tag: None 2023-11-16T18:00:12.016 INFO:teuthology.packaging:branch: wip-choffman-62793 2023-11-16T18:00:12.016 INFO:teuthology.packaging:sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T18:00:12.016 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-choffman-62793 2023-11-16T18:00:12.236 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-3905.g5e4a743a 2023-11-16T18:00:12.236 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-16T18:00:12.242 INFO:teuthology.task.internal:no buildpackages task found 2023-11-16T18:00:12.242 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-16T18:00:12.281 INFO:teuthology.task.internal:Saving configuration 2023-11-16T18:00:12.301 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-16T18:00:12.354 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T18:00:12.380 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 17:55:14.998896', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T18:00:12.405 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 17:55:14.996199', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T18:00:12.425 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 17:55:15.000300', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T18:00:12.425 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-16T18:00:12.431 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-11-16T18:00:12.432 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-11-16T18:00:12.432 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-11-16T18:00:12.432 INFO:teuthology.run_tasks:Running task console_log... 2023-11-16T18:00:12.594 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5851efbf70>, signals=[15]) 2023-11-16T18:00:12.594 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-16T18:00:12.600 INFO:teuthology.task.internal:Opening connections... 2023-11-16T18:00:12.601 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T18:00:12.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T18:00:12.671 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2023-11-16T18:00:12.673 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T18:00:12.740 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2023-11-16T18:00:12.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T18:00:12.810 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-16T18:00:12.817 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2023-11-16T18:00:12.839 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2023-11-16T18:00:12.840 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2023-11-16T18:00:12.899 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="Red Hat Enterprise Linux" 2023-11-16T18:00:12.899 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="8.6 (Ootpa)" 2023-11-16T18:00:12.899 INFO:teuthology.orchestra.run.smithi002.stdout:ID="rhel" 2023-11-16T18:00:12.900 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="fedora" 2023-11-16T18:00:12.900 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="8.6" 2023-11-16T18:00:12.900 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el8" 2023-11-16T18:00:12.900 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-16T18:00:12.900 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2023-11-16T18:00:12.900 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-16T18:00:12.900 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://www.redhat.com/" 2023-11-16T18:00:12.900 INFO:teuthology.orchestra.run.smithi002.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-16T18:00:12.900 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T18:00:12.900 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:00:12.900 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T18:00:12.900 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-16T18:00:12.900 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-16T18:00:12.901 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-16T18:00:12.901 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2023-11-16T18:00:12.926 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2023-11-16T18:00:12.947 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2023-11-16T18:00:12.948 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2023-11-16T18:00:13.007 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="Red Hat Enterprise Linux" 2023-11-16T18:00:13.007 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="8.6 (Ootpa)" 2023-11-16T18:00:13.007 INFO:teuthology.orchestra.run.smithi040.stdout:ID="rhel" 2023-11-16T18:00:13.007 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="fedora" 2023-11-16T18:00:13.008 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="8.6" 2023-11-16T18:00:13.008 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el8" 2023-11-16T18:00:13.008 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-16T18:00:13.008 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2023-11-16T18:00:13.008 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-16T18:00:13.008 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://www.redhat.com/" 2023-11-16T18:00:13.008 INFO:teuthology.orchestra.run.smithi040.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-16T18:00:13.008 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T18:00:13.008 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-11-16T18:00:13.008 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T18:00:13.009 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-16T18:00:13.009 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-16T18:00:13.009 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-16T18:00:13.009 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2023-11-16T18:00:13.034 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2023-11-16T18:00:13.056 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2023-11-16T18:00:13.057 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2023-11-16T18:00:13.116 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="Red Hat Enterprise Linux" 2023-11-16T18:00:13.116 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8.6 (Ootpa)" 2023-11-16T18:00:13.116 INFO:teuthology.orchestra.run.smithi188.stdout:ID="rhel" 2023-11-16T18:00:13.116 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="fedora" 2023-11-16T18:00:13.116 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8.6" 2023-11-16T18:00:13.116 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2023-11-16T18:00:13.117 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-16T18:00:13.117 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2023-11-16T18:00:13.117 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-16T18:00:13.117 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://www.redhat.com/" 2023-11-16T18:00:13.117 INFO:teuthology.orchestra.run.smithi188.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-16T18:00:13.117 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T18:00:13.117 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:00:13.117 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T18:00:13.117 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-16T18:00:13.117 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-16T18:00:13.117 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-16T18:00:13.117 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2023-11-16T18:00:13.134 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-16T18:00:13.143 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-16T18:00:13.149 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-16T18:00:13.149 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2023-11-16T18:00:13.152 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2023-11-16T18:00:13.154 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2023-11-16T18:00:13.175 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-16T18:00:13.180 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-16T18:00:13.181 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2023-11-16T18:00:13.212 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2023-11-16T18:00:13.214 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2023-11-16T18:00:13.238 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-16T18:00:13.313 INFO:teuthology.run_tasks:Running task kernel... 2023-11-16T18:00:13.327 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-11-16T18:00:13.327 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-11-16T18:00:13.327 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-11-16T18:00:13.328 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T18:00:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:00:13.346 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2023-11-16T18:00:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:4.18.0-372.9.1.el8.x86_64 2023-11-16T18:00:13.406 INFO:teuthology.task.kernel:Running kernel on smithi002: 4.18.0-372.9.1.el8.x86_64 2023-11-16T18:00:13.406 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum install -y kernel 2023-11-16T18:00:14.661 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2023-11-16T18:00:14.661 INFO:teuthology.orchestra.run.smithi002.stdout:Unable to read consumer identity 2023-11-16T18:00:16.179 INFO:teuthology.orchestra.run.smithi002.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2023-11-16T18:00:16.930 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 22 MB/s | 16 MB 00:00 2023-11-16T18:00:20.613 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 588 kB/s | 24 kB 00:00 2023-11-16T18:00:21.349 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-11-16T18:00:21.397 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-11-16T18:00:21.399 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2023-11-16T18:00:21.399 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-11-16T18:00:21.523 DEBUG:teuthology.orchestra.run.smithi002:> echo no | sudo yum reinstall kernel || true 2023-11-16T18:00:21.872 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2023-11-16T18:00:21.872 INFO:teuthology.orchestra.run.smithi002.stdout:Unable to read consumer identity 2023-11-16T18:00:22.148 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:02 ago on Thu 16 Nov 2023 06:00:20 PM UTC. 2023-11-16T18:00:22.204 INFO:teuthology.orchestra.run.smithi002.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-16T18:00:22.245 INFO:teuthology.orchestra.run.smithi002.stderr:Error: No packages marked for reinstall. 2023-11-16T18:00:22.292 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum reinstall -y kernel || true 2023-11-16T18:00:22.644 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2023-11-16T18:00:22.644 INFO:teuthology.orchestra.run.smithi002.stdout:Unable to read consumer identity 2023-11-16T18:00:22.915 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:02 ago on Thu 16 Nov 2023 06:00:20 PM UTC. 2023-11-16T18:00:22.959 INFO:teuthology.orchestra.run.smithi002.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-16T18:00:22.989 INFO:teuthology.orchestra.run.smithi002.stderr:Error: No packages marked for reinstall. 2023-11-16T18:00:23.035 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel | sort -rV | head -n 1 2023-11-16T18:00:23.118 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-11-16T18:00:23.119 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-11-16T18:00:23.119 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-11-16T18:00:23.120 DEBUG:teuthology.parallel:result is None 2023-11-16T18:00:23.120 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-16T18:00:23.127 INFO:teuthology.task.internal:Creating test directory... 2023-11-16T18:00:23.127 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T18:00:23.129 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T18:00:23.132 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T18:00:23.153 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-16T18:00:23.159 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-16T18:00:23.170 INFO:teuthology.task.internal:Creating archive directory... 2023-11-16T18:00:23.170 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T18:00:23.186 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T18:00:23.193 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T18:00:23.233 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-16T18:00:23.240 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-16T18:00:23.240 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T18:00:23.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:00:23.277 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T18:00:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:00:23.295 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T18:00:23.297 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T18:00:23.320 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T18:00:23.333 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T18:00:23.350 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T18:00:23.370 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T18:00:23.384 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T18:00:23.390 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T18:00:23.399 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T18:00:23.401 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-16T18:00:23.408 INFO:teuthology.task.internal:Configuring sudo... 2023-11-16T18:00:23.409 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T18:00:23.411 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T18:00:23.413 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T18:00:23.474 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-16T18:00:23.482 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-16T18:00:23.483 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T18:00:23.485 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T18:00:23.487 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T18:00:23.535 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T18:00:23.582 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T18:00:23.635 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T18:00:23.719 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T18:00:23.761 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:00:23.761 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T18:00:23.829 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T18:00:23.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T18:00:23.933 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T18:00:24.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T18:00:24.064 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-11-16T18:00:24.064 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T18:00:24.147 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T18:00:24.194 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T18:00:24.251 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T18:00:24.336 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T18:00:24.381 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-16T18:00:24.382 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T18:00:24.450 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2023-11-16T18:00:24.453 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2023-11-16T18:00:24.455 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2023-11-16T18:00:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T18:00:24.515 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T18:00:24.546 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T18:00:24.868 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-16T18:00:24.877 INFO:teuthology.task.internal:Starting timer... 2023-11-16T18:00:24.877 INFO:teuthology.run_tasks:Running task pcp... 2023-11-16T18:00:24.886 INFO:teuthology.run_tasks:Running task selinux... 2023-11-16T18:00:24.895 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-11-16T18:00:24.895 DEBUG:teuthology.orchestra.run.smithi002:> sudo service auditd rotate 2023-11-16T18:00:24.957 INFO:teuthology.orchestra.run.smithi002.stdout:Rotating logs: 2023-11-16T18:00:24.959 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2023-11-16T18:00:25.017 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2023-11-16T18:00:25.019 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2023-11-16T18:00:25.073 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2023-11-16T18:00:25.074 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-16T18:00:25.075 DEBUG:teuthology.orchestra.run.smithi002:> /usr/sbin/getenforce 2023-11-16T18:00:25.108 INFO:teuthology.orchestra.run.smithi002.stdout:Permissive 2023-11-16T18:00:25.109 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2023-11-16T18:00:25.139 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2023-11-16T18:00:25.139 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2023-11-16T18:00:25.162 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2023-11-16T18:00:25.162 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi002.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2023-11-16T18:00:25.162 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T18:00:25.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:00:25.190 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T18:00:25.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:00:25.219 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T18:00:25.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:00:25.246 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-16T18:00:25.246 DEBUG:teuthology.orchestra.run.smithi002:> sudo /usr/sbin/setenforce permissive 2023-11-16T18:00:25.273 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2023-11-16T18:00:25.300 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2023-11-16T18:00:25.326 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-16T18:00:25.337 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-16T18:00:25.338 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-16T18:00:25.353 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-16T18:00:25.354 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi040.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-16T18:11:43.826 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com'), Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2023-11-16T18:11:43.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-16T18:11:43.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T18:11:43.947 DEBUG:teuthology.orchestra.run.smithi002:> true 2023-11-16T18:11:44.022 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-16T18:11:44.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-11-16T18:11:44.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T18:11:44.107 DEBUG:teuthology.orchestra.run.smithi040:> true 2023-11-16T18:11:44.189 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-11-16T18:11:44.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-11-16T18:11:44.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T18:11:44.261 DEBUG:teuthology.orchestra.run.smithi188:> true 2023-11-16T18:11:44.334 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-11-16T18:11:44.334 INFO:teuthology.run_tasks:Running task clock... 2023-11-16T18:11:44.358 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-16T18:11:44.358 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T18:11:44.364 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T18:11:44.380 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T18:11:44.380 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T18:11:44.388 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T18:11:44.389 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T18:11:44.408 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T18:11:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T18:11:44.425 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T18:11:44.426 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T18:11:44.434 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T18:11:44.442 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T18:11:44.499 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: ntpd: command not found 2023-11-16T18:11:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2023-11-16T18:11:44.512 INFO:teuthology.orchestra.run.smithi002.stdout:506 Cannot talk to daemon 2023-11-16T18:11:44.512 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2023-11-16T18:11:44.526 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2023-11-16T18:11:44.527 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T18:11:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T18:11:44.538 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2023-11-16T18:11:44.557 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T18:11:44.558 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T18:11:44.558 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T18:11:44.567 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T18:11:44.611 INFO:teuthology.orchestra.run.smithi002.stderr:bash: ntpq: command not found 2023-11-16T18:11:44.612 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T18:11:44.612 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2023-11-16T18:11:44.612 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:11:44.612 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:11:44.612 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:11:44.612 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:11:44.624 INFO:teuthology.orchestra.run.smithi040.stderr:bash: ntpq: command not found 2023-11-16T18:11:44.625 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T18:11:44.625 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2023-11-16T18:11:44.625 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:11:44.625 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:11:44.626 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:11:44.626 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:11:44.653 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2023-11-16T18:11:44.657 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T18:11:44.657 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2023-11-16T18:11:44.657 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:11:44.657 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:11:44.657 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:11:44.657 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:11:44.659 INFO:teuthology.run_tasks:Running task pexec... 2023-11-16T18:11:44.669 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-16T18:11:44.669 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T18:11:44.669 DEBUG:teuthology.orchestra.run.smithi002:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T18:11:44.670 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi040.front.sepia.ceph.com 2023-11-16T18:11:44.670 DEBUG:teuthology.orchestra.run.smithi040:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T18:11:44.670 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi188.front.sepia.ceph.com 2023-11-16T18:11:44.671 DEBUG:teuthology.orchestra.run.smithi188:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T18:11:45.086 INFO:teuthology.orchestra.run.smithi040.stdout:Updating Subscription Management repositories. 2023-11-16T18:11:45.088 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2023-11-16T18:11:45.089 INFO:teuthology.orchestra.run.smithi188.stdout:Updating Subscription Management repositories. 2023-11-16T18:11:46.695 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:52 ago on Thu 16 Nov 2023 06:08:54 PM UTC. 2023-11-16T18:11:46.736 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:03:16 ago on Thu 16 Nov 2023 06:08:30 PM UTC. 2023-11-16T18:11:46.799 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:03:14 ago on Thu 16 Nov 2023 06:08:32 PM UTC. 2023-11-16T18:11:48.479 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-16T18:11:48.479 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-16T18:11:48.479 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2023-11-16T18:11:48.480 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-16T18:11:48.480 INFO:teuthology.orchestra.run.smithi188.stdout:Resetting modules: 2023-11-16T18:11:48.480 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools 2023-11-16T18:11:48.480 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:11:48.480 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-16T18:11:48.480 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-16T18:11:48.480 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:11:48.588 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-11-16T18:11:48.589 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-11-16T18:11:48.589 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2023-11-16T18:11:48.589 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-11-16T18:11:48.589 INFO:teuthology.orchestra.run.smithi040.stdout:Resetting modules: 2023-11-16T18:11:48.589 INFO:teuthology.orchestra.run.smithi040.stdout: container-tools 2023-11-16T18:11:48.590 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-11-16T18:11:48.590 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-11-16T18:11:48.590 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-11-16T18:11:48.590 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-11-16T18:11:48.817 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-11-16T18:11:48.818 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-11-16T18:11:48.818 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2023-11-16T18:11:48.818 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-11-16T18:11:48.818 INFO:teuthology.orchestra.run.smithi002.stdout:Resetting modules: 2023-11-16T18:11:48.818 INFO:teuthology.orchestra.run.smithi002.stdout: container-tools 2023-11-16T18:11:48.818 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:11:48.819 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-11-16T18:11:48.819 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-11-16T18:11:48.819 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:11:49.447 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:03:19 ago on Thu 16 Nov 2023 06:08:30 PM UTC. 2023-11-16T18:11:49.623 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:55 ago on Thu 16 Nov 2023 06:08:54 PM UTC. 2023-11-16T18:11:49.828 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:03:17 ago on Thu 16 Nov 2023 06:08:32 PM UTC. 2023-11-16T18:11:58.593 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-16T18:11:59.131 INFO:teuthology.orchestra.run.smithi188.stdout:Updating Subscription Management repositories. 2023-11-16T18:12:00.262 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-11-16T18:12:00.750 INFO:teuthology.orchestra.run.smithi040.stdout:Updating Subscription Management repositories. 2023-11-16T18:12:00.891 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:03:30 ago on Thu 16 Nov 2023 06:08:30 PM UTC. 2023-11-16T18:12:00.933 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-11-16T18:12:01.411 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2023-11-16T18:12:02.356 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:03:07 ago on Thu 16 Nov 2023 06:08:54 PM UTC. 2023-11-16T18:12:02.730 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-16T18:12:02.824 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================================ 2023-11-16T18:12:02.824 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2023-11-16T18:12:02.824 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================================ 2023-11-16T18:12:02.824 INFO:teuthology.orchestra.run.smithi188.stdout:Installing group/module packages: 2023-11-16T18:12:02.824 INFO:teuthology.orchestra.run.smithi188.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-11-16T18:12:02.824 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-11-16T18:12:02.824 INFO:teuthology.orchestra.run.smithi188.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-11-16T18:12:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-11-16T18:12:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-11-16T18:12:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-16T18:12:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-11-16T18:12:02.825 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-11-16T18:12:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-11-16T18:12:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-11-16T18:12:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-16T18:12:02.825 INFO:teuthology.orchestra.run.smithi188.stdout:Installing module profiles: 2023-11-16T18:12:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools/common 2023-11-16T18:12:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:12:02.825 INFO:teuthology.orchestra.run.smithi188.stdout:Enabling module streams: 2023-11-16T18:12:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools rhel8 2023-11-16T18:12:02.826 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:12:02.826 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-16T18:12:02.826 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================================ 2023-11-16T18:12:02.826 INFO:teuthology.orchestra.run.smithi188.stdout:Install 10 Packages 2023-11-16T18:12:02.826 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:12:02.827 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 18 M 2023-11-16T18:12:02.827 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 65 M 2023-11-16T18:12:02.827 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-16T18:12:02.966 INFO:teuthology.orchestra.run.smithi188.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 287 kB/s | 41 kB 00:00 2023-11-16T18:12:02.992 INFO:teuthology.orchestra.run.smithi188.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 149 kB/s | 25 kB 00:00 2023-11-16T18:12:03.034 INFO:teuthology.orchestra.run.smithi188.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 449 kB/s | 94 kB 00:00 2023-11-16T18:12:03.092 INFO:teuthology.orchestra.run.smithi188.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 393 kB/s | 49 kB 00:00 2023-11-16T18:12:03.126 INFO:teuthology.orchestra.run.smithi188.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.8 MB/s | 524 kB 00:00 2023-11-16T18:12:03.180 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:03:30 ago on Thu 16 Nov 2023 06:08:32 PM UTC. 2023-11-16T18:12:03.218 INFO:teuthology.orchestra.run.smithi188.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.2 MB/s | 149 kB 00:00 2023-11-16T18:12:03.271 INFO:teuthology.orchestra.run.smithi188.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.5 MB/s | 213 kB 00:00 2023-11-16T18:12:03.368 INFO:teuthology.orchestra.run.smithi188.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 24 MB/s | 8.0 MB 00:00 2023-11-16T18:12:03.436 INFO:teuthology.orchestra.run.smithi188.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 13 MB/s | 2.2 MB 00:00 2023-11-16T18:12:03.535 INFO:teuthology.orchestra.run.smithi188.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 22 MB/s | 7.0 MB 00:00 2023-11-16T18:12:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-16T18:12:03.536 INFO:teuthology.orchestra.run.smithi188.stdout:Total 26 MB/s | 18 MB 00:00 2023-11-16T18:12:03.737 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-16T18:12:03.756 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-16T18:12:03.756 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-16T18:12:03.939 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-16T18:12:03.943 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-16T18:12:04.339 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-11-16T18:12:04.441 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-16T18:12:04.442 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================================ 2023-11-16T18:12:04.442 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-11-16T18:12:04.442 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================================ 2023-11-16T18:12:04.442 INFO:teuthology.orchestra.run.smithi040.stdout:Installing group/module packages: 2023-11-16T18:12:04.442 INFO:teuthology.orchestra.run.smithi040.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-11-16T18:12:04.442 INFO:teuthology.orchestra.run.smithi040.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-11-16T18:12:04.442 INFO:teuthology.orchestra.run.smithi040.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-11-16T18:12:04.442 INFO:teuthology.orchestra.run.smithi040.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-11-16T18:12:04.443 INFO:teuthology.orchestra.run.smithi040.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-11-16T18:12:04.443 INFO:teuthology.orchestra.run.smithi040.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-16T18:12:04.443 INFO:teuthology.orchestra.run.smithi040.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-11-16T18:12:04.443 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-11-16T18:12:04.443 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-11-16T18:12:04.443 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-11-16T18:12:04.443 INFO:teuthology.orchestra.run.smithi040.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-16T18:12:04.443 INFO:teuthology.orchestra.run.smithi040.stdout:Installing module profiles: 2023-11-16T18:12:04.443 INFO:teuthology.orchestra.run.smithi040.stdout: container-tools/common 2023-11-16T18:12:04.443 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-11-16T18:12:04.443 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling module streams: 2023-11-16T18:12:04.443 INFO:teuthology.orchestra.run.smithi040.stdout: container-tools rhel8 2023-11-16T18:12:04.444 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-11-16T18:12:04.444 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-11-16T18:12:04.444 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================================ 2023-11-16T18:12:04.444 INFO:teuthology.orchestra.run.smithi040.stdout:Install 10 Packages 2023-11-16T18:12:04.444 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-11-16T18:12:04.444 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 18 M 2023-11-16T18:12:04.444 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 65 M 2023-11-16T18:12:04.444 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-11-16T18:12:04.617 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-11-16T18:12:04.618 INFO:teuthology.orchestra.run.smithi040.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 141 kB/s | 25 kB 00:00 2023-11-16T18:12:04.657 INFO:teuthology.orchestra.run.smithi040.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 201 kB/s | 41 kB 00:00 2023-11-16T18:12:04.661 INFO:teuthology.orchestra.run.smithi040.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 427 kB/s | 94 kB 00:00 2023-11-16T18:12:04.762 INFO:teuthology.orchestra.run.smithi040.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 344 kB/s | 49 kB 00:00 2023-11-16T18:12:04.780 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T18:12:04.798 INFO:teuthology.orchestra.run.smithi040.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.4 MB/s | 524 kB 00:00 2023-11-16T18:12:04.957 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-11-16T18:12:04.988 INFO:teuthology.orchestra.run.smithi040.stdout:(6/10): buildah-1.26.2-1.module+el8.6.0+16771+2 24 MB/s | 8.0 MB 00:00 2023-11-16T18:12:05.004 INFO:teuthology.orchestra.run.smithi040.stdout:(7/10): python3-podman-4.0.1-1.module+el8.6.0+1 617 kB/s | 149 kB 00:00 2023-11-16T18:12:05.030 INFO:teuthology.orchestra.run.smithi040.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 911 kB/s | 213 kB 00:00 2023-11-16T18:12:05.146 INFO:teuthology.orchestra.run.smithi040.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 15 MB/s | 2.2 MB 00:00 2023-11-16T18:12:05.151 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-11-16T18:12:05.175 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-11-16T18:12:05.275 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================================================ 2023-11-16T18:12:05.275 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-11-16T18:12:05.275 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================================================ 2023-11-16T18:12:05.275 INFO:teuthology.orchestra.run.smithi002.stdout:Installing group/module packages: 2023-11-16T18:12:05.275 INFO:teuthology.orchestra.run.smithi002.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-11-16T18:12:05.275 INFO:teuthology.orchestra.run.smithi002.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-11-16T18:12:05.275 INFO:teuthology.orchestra.run.smithi002.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-11-16T18:12:05.275 INFO:teuthology.orchestra.run.smithi002.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-11-16T18:12:05.275 INFO:teuthology.orchestra.run.smithi002.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-11-16T18:12:05.275 INFO:teuthology.orchestra.run.smithi002.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-16T18:12:05.275 INFO:teuthology.orchestra.run.smithi002.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-11-16T18:12:05.276 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2023-11-16T18:12:05.276 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-11-16T18:12:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-11-16T18:12:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-16T18:12:05.280 INFO:teuthology.orchestra.run.smithi002.stdout:Installing module profiles: 2023-11-16T18:12:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: container-tools/common 2023-11-16T18:12:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:12:05.280 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling module streams: 2023-11-16T18:12:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: container-tools rhel8 2023-11-16T18:12:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:12:05.281 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-11-16T18:12:05.282 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================================================ 2023-11-16T18:12:05.284 INFO:teuthology.orchestra.run.smithi002.stdout:Install 10 Packages 2023-11-16T18:12:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:12:05.285 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 18 M 2023-11-16T18:12:05.286 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 65 M 2023-11-16T18:12:05.286 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-11-16T18:12:05.296 INFO:teuthology.orchestra.run.smithi040.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 23 MB/s | 7.0 MB 00:00 2023-11-16T18:12:05.297 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-11-16T18:12:05.298 INFO:teuthology.orchestra.run.smithi040.stdout:Total 21 MB/s | 18 MB 00:00 2023-11-16T18:12:05.398 INFO:teuthology.orchestra.run.smithi002.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 774 kB/s | 94 kB 00:00 2023-11-16T18:12:05.480 INFO:teuthology.orchestra.run.smithi002.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 126 kB/s | 25 kB 00:00 2023-11-16T18:12:05.502 INFO:teuthology.orchestra.run.smithi002.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 184 kB/s | 41 kB 00:00 2023-11-16T18:12:05.503 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-11-16T18:12:05.526 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-11-16T18:12:05.532 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-11-16T18:12:05.534 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-11-16T18:12:05.541 INFO:teuthology.orchestra.run.smithi002.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 347 kB/s | 49 kB 00:00 2023-11-16T18:12:05.674 INFO:teuthology.orchestra.run.smithi002.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.6 MB/s | 524 kB 00:00 2023-11-16T18:12:05.717 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-11-16T18:12:05.726 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-11-16T18:12:05.766 INFO:teuthology.orchestra.run.smithi002.stdout:(6/10): buildah-1.26.2-1.module+el8.6.0+16771+2 30 MB/s | 8.0 MB 00:00 2023-11-16T18:12:05.792 INFO:teuthology.orchestra.run.smithi002.stdout:(7/10): python3-podman-4.0.1-1.module+el8.6.0+1 596 kB/s | 149 kB 00:00 2023-11-16T18:12:05.817 INFO:teuthology.orchestra.run.smithi002.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.5 MB/s | 213 kB 00:00 2023-11-16T18:12:05.917 INFO:teuthology.orchestra.run.smithi002.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 18 MB/s | 2.2 MB 00:00 2023-11-16T18:12:06.002 INFO:teuthology.orchestra.run.smithi002.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 30 MB/s | 7.0 MB 00:00 2023-11-16T18:12:06.002 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-11-16T18:12:06.002 INFO:teuthology.orchestra.run.smithi002.stdout:Total 25 MB/s | 18 MB 00:00 2023-11-16T18:12:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-11-16T18:12:06.228 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-11-16T18:12:06.229 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-11-16T18:12:06.230 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-11-16T18:12:06.287 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-11-16T18:12:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-11-16T18:12:06.421 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-11-16T18:12:06.426 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-11-16T18:12:06.599 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T18:12:06.794 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-11-16T18:12:06.942 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-11-16T18:12:07.008 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-11-16T18:12:07.134 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-11-16T18:12:07.153 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-11-16T18:12:07.301 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T18:12:07.374 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-11-16T18:12:07.497 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-11-16T18:12:07.545 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-11-16T18:12:07.545 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-11-16T18:12:07.676 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-11-16T18:12:07.723 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T18:12:07.998 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T18:12:07.999 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-11-16T18:12:08.000 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T18:12:08.000 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-11-16T18:12:08.000 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-11-16T18:12:08.000 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-11-16T18:12:08.001 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-11-16T18:12:08.001 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-11-16T18:12:08.001 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-11-16T18:12:08.002 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-11-16T18:12:08.033 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-11-16T18:12:08.346 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-11-16T18:12:08.809 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-11-16T18:12:09.135 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-11-16T18:12:09.142 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:03:38 ago on Thu 16 Nov 2023 06:08:30 PM UTC. 2023-11-16T18:12:09.273 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-11-16T18:12:09.469 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-11-16T18:12:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-11-16T18:12:09.723 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-11-16T18:12:09.828 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T18:12:09.938 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-11-16T18:12:10.105 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T18:12:10.106 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-11-16T18:12:10.106 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T18:12:10.107 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-11-16T18:12:10.107 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-11-16T18:12:10.107 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-11-16T18:12:10.107 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-11-16T18:12:10.108 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-11-16T18:12:10.108 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-11-16T18:12:10.109 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-11-16T18:12:10.240 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-11-16T18:12:10.407 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T18:12:10.678 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T18:12:10.678 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-11-16T18:12:10.678 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T18:12:10.678 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-11-16T18:12:10.678 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-11-16T18:12:10.678 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-11-16T18:12:10.678 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-11-16T18:12:10.679 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-11-16T18:12:10.679 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-11-16T18:12:10.679 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-11-16T18:12:11.286 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-11-16T18:12:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:03:16 ago on Thu 16 Nov 2023 06:08:54 PM UTC. 2023-11-16T18:12:11.790 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-11-16T18:12:11.790 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:03:39 ago on Thu 16 Nov 2023 06:08:32 PM UTC. 2023-11-16T18:12:18.771 INFO:teuthology.orchestra.run.smithi188.stdout:Installed products updated. 2023-11-16T18:12:18.959 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:12:18.960 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-16T18:12:18.960 INFO:teuthology.orchestra.run.smithi188.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T18:12:18.960 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T18:12:18.961 INFO:teuthology.orchestra.run.smithi188.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T18:12:18.961 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-11-16T18:12:18.961 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T18:12:18.961 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T18:12:18.962 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T18:12:18.962 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T18:12:18.962 INFO:teuthology.orchestra.run.smithi188.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T18:12:18.966 INFO:teuthology.orchestra.run.smithi188.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-16T18:12:18.967 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:12:18.967 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-16T18:12:19.093 DEBUG:teuthology.parallel:result is None 2023-11-16T18:12:21.895 INFO:teuthology.orchestra.run.smithi040.stdout:Installed products updated. 2023-11-16T18:12:21.967 INFO:teuthology.orchestra.run.smithi002.stdout:Installed products updated. 2023-11-16T18:12:22.099 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-11-16T18:12:22.099 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-11-16T18:12:22.099 INFO:teuthology.orchestra.run.smithi040.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T18:12:22.099 INFO:teuthology.orchestra.run.smithi040.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T18:12:22.099 INFO:teuthology.orchestra.run.smithi040.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T18:12:22.099 INFO:teuthology.orchestra.run.smithi040.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-11-16T18:12:22.100 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T18:12:22.100 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T18:12:22.100 INFO:teuthology.orchestra.run.smithi040.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T18:12:22.100 INFO:teuthology.orchestra.run.smithi040.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T18:12:22.100 INFO:teuthology.orchestra.run.smithi040.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T18:12:22.100 INFO:teuthology.orchestra.run.smithi040.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-16T18:12:22.100 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-11-16T18:12:22.101 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-11-16T18:12:22.173 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:12:22.174 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-11-16T18:12:22.174 INFO:teuthology.orchestra.run.smithi002.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T18:12:22.174 INFO:teuthology.orchestra.run.smithi002.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T18:12:22.174 INFO:teuthology.orchestra.run.smithi002.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T18:12:22.174 INFO:teuthology.orchestra.run.smithi002.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-11-16T18:12:22.174 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T18:12:22.174 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T18:12:22.174 INFO:teuthology.orchestra.run.smithi002.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T18:12:22.174 INFO:teuthology.orchestra.run.smithi002.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T18:12:22.175 INFO:teuthology.orchestra.run.smithi002.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T18:12:22.175 INFO:teuthology.orchestra.run.smithi002.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-16T18:12:22.175 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:12:22.175 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-11-16T18:12:22.240 DEBUG:teuthology.parallel:result is None 2023-11-16T18:12:22.325 DEBUG:teuthology.parallel:result is None 2023-11-16T18:12:22.325 INFO:teuthology.run_tasks:Running task install... 2023-11-16T18:12:22.348 DEBUG:teuthology.task.install:project ceph 2023-11-16T18:12:22.348 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207'}} 2023-11-16T18:12:22.348 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207'} 2023-11-16T18:12:22.348 INFO:teuthology.task.install:Using flavor: default 2023-11-16T18:12:22.367 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-16T18:12:22.367 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-11-16T18:12:22.367 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'tag': None, 'wait_for_package': False} 2023-11-16T18:12:22.368 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T18:12:22.405 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'tag': None, 'wait_for_package': False} 2023-11-16T18:12:22.407 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T18:12:22.421 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'tag': None, 'wait_for_package': False} 2023-11-16T18:12:22.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T18:12:22.963 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/ 2023-11-16T18:12:22.964 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3905.g5e4a743a 2023-11-16T18:12:22.993 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/ 2023-11-16T18:12:22.994 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3905.g5e4a743a 2023-11-16T18:12:23.102 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/ 2023-11-16T18:12:23.102 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3905.g5e4a743a 2023-11-16T18:12:23.386 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T18:12:23.387 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-11-16T18:12:23.387 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T18:12:23.422 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T18:12:23.422 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:12:23.422 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T18:12:23.423 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-16T18:12:23.424 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e4a743ad7fd1422ebd6c4dc950a21a303279207/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T18:12:23.498 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T18:12:23.498 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-16T18:12:23.498 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T18:12:23.510 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-16T18:12:23.510 DEBUG:teuthology.orchestra.run.smithi002:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e4a743ad7fd1422ebd6c4dc950a21a303279207/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T18:12:23.532 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T18:12:23.533 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-16T18:12:23.533 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e4a743ad7fd1422ebd6c4dc950a21a303279207/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T18:12:23.588 DEBUG:teuthology.orchestra.run.smithi002:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T18:12:23.607 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T18:12:23.618 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T18:12:23.694 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T18:12:23.696 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2023-11-16T18:12:23.697 DEBUG:teuthology.orchestra.run.smithi002:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T18:12:23.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2023-11-16T18:12:23.766 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2023-11-16T18:12:23.767 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2023-11-16T18:12:23.768 INFO:teuthology.orchestra.run.smithi002.stdout:check_obsoletes = 1 2023-11-16T18:12:23.776 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum clean all 2023-11-16T18:12:24.093 INFO:teuthology.orchestra.run.smithi040.stdout:Updating Subscription Management repositories. 2023-11-16T18:12:24.141 INFO:teuthology.orchestra.run.smithi188.stdout:Updating Subscription Management repositories. 2023-11-16T18:12:24.176 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2023-11-16T18:12:25.396 INFO:teuthology.orchestra.run.smithi188.stdout:44 files removed 2023-11-16T18:12:25.406 INFO:teuthology.orchestra.run.smithi040.stdout:44 files removed 2023-11-16T18:12:25.414 INFO:teuthology.orchestra.run.smithi002.stdout:44 files removed 2023-11-16T18:12:25.450 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-16T18:12:25.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-16T18:12:25.469 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-16T18:12:25.785 INFO:teuthology.orchestra.run.smithi188.stdout:Updating Subscription Management repositories. 2023-11-16T18:12:25.817 INFO:teuthology.orchestra.run.smithi040.stdout:Updating Subscription Management repositories. 2023-11-16T18:12:25.823 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2023-11-16T18:12:27.636 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 98 kB/s | 80 kB 00:00 2023-11-16T18:12:27.865 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 86 kB/s | 80 kB 00:00 2023-11-16T18:12:28.010 INFO:teuthology.orchestra.run.smithi002.stdout:ceph packages for x86_64 87 kB/s | 80 kB 00:00 2023-11-16T18:12:28.344 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-11-16T18:12:28.691 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 12 kB/s | 9.4 kB 00:00 2023-11-16T18:12:28.711 INFO:teuthology.orchestra.run.smithi002.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-11-16T18:12:29.053 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 2.2 kB/s | 1.5 kB 00:00 2023-11-16T18:12:29.204 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-11-16T18:12:29.491 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 1.9 kB/s | 1.5 kB 00:00 2023-11-16T18:12:29.502 INFO:teuthology.orchestra.run.smithi002.stdout:ceph source packages 1.9 kB/s | 1.5 kB 00:00 2023-11-16T18:12:29.625 INFO:teuthology.orchestra.run.smithi040.stdout:Copr repo for python3-asyncssh owned by ceph 31 kB/s | 3.5 kB 00:00 2023-11-16T18:12:29.645 INFO:teuthology.orchestra.run.smithi002.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-11-16T18:12:30.323 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2023-11-16T18:12:31.160 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-11-16T18:12:33.289 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 16 MB 00:03 2023-11-16T18:12:33.829 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 500 kB/s | 24 kB 00:00 2023-11-16T18:12:34.964 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 214 kB/s | 24 kB 00:00 2023-11-16T18:12:35.184 INFO:teuthology.orchestra.run.smithi188.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 41 MB/s | 53 MB 00:01 2023-11-16T18:12:36.128 INFO:teuthology.orchestra.run.smithi040.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 MB/s | 47 MB 00:01 2023-11-16T18:12:37.054 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 402 kB/s | 24 kB 00:00 2023-11-16T18:12:38.255 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 MB/s | 47 MB 00:01 2023-11-16T18:12:43.374 INFO:teuthology.orchestra.run.smithi188.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 50 MB/s | 47 MB 00:00 2023-11-16T18:12:44.779 INFO:teuthology.orchestra.run.smithi040.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 45 MB/s | 53 MB 00:01 2023-11-16T18:12:47.040 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 45 MB/s | 53 MB 00:01 2023-11-16T18:12:54.248 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T18:12:54.248 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T18:12:54.262 INFO:teuthology.orchestra.run.smithi188.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-16T18:12:54.262 INFO:teuthology.orchestra.run.smithi188.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-16T18:12:54.264 INFO:teuthology.orchestra.run.smithi188.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-16T18:12:54.265 INFO:teuthology.orchestra.run.smithi188.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-16T18:12:54.267 INFO:teuthology.orchestra.run.smithi188.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-16T18:12:54.270 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-16T18:12:54.271 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-16T18:12:54.402 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-16T18:12:54.420 INFO:teuthology.orchestra.run.smithi188.stdout:=========================================================================================================================== 2023-11-16T18:12:54.420 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2023-11-16T18:12:54.421 INFO:teuthology.orchestra.run.smithi188.stdout:=========================================================================================================================== 2023-11-16T18:12:54.421 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-16T18:12:54.421 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-16T18:12:54.421 INFO:teuthology.orchestra.run.smithi188.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-16T18:12:54.421 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.4 k 2023-11-16T18:12:54.421 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 5.3 M 2023-11-16T18:12:54.421 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 847 k 2023-11-16T18:12:54.421 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 165 k 2023-11-16T18:12:54.421 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 1.5 M 2023-11-16T18:12:54.421 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 127 k 2023-11-16T18:12:54.421 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 1.7 M 2023-11-16T18:12:54.421 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 7.4 M 2023-11-16T18:12:54.421 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 47 k 2023-11-16T18:12:54.422 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 13 M 2023-11-16T18:12:54.422 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 41 M 2023-11-16T18:12:54.422 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 265 k 2023-11-16T18:12:54.422 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 83 k 2023-11-16T18:12:54.422 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 258 k 2023-11-16T18:12:54.422 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 18 k 2023-11-16T18:12:54.422 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-16T18:12:54.422 INFO:teuthology.orchestra.run.smithi188.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-16T18:12:54.431 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-16T18:12:54.432 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 31 k 2023-11-16T18:12:54.432 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 771 k 2023-11-16T18:12:54.432 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 139 k 2023-11-16T18:12:54.432 INFO:teuthology.orchestra.run.smithi188.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-16T18:12:54.432 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-16T18:12:54.432 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-16T18:12:54.432 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-16T18:12:54.432 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-16T18:12:54.432 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-16T18:12:54.432 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 197 k 2023-11-16T18:12:54.432 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 371 k 2023-11-16T18:12:54.433 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 365 k 2023-11-16T18:12:54.433 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 107 k 2023-11-16T18:12:54.433 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 94 k 2023-11-16T18:12:54.433 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.4 M 2023-11-16T18:12:54.433 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 180 k 2023-11-16T18:12:54.433 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-16T18:12:54.433 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2023-11-16T18:12:54.433 INFO:teuthology.orchestra.run.smithi188.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-16T18:12:54.433 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.7 M 2023-11-16T18:12:54.433 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.6 M 2023-11-16T18:12:54.433 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-11-16T18:12:54.433 INFO:teuthology.orchestra.run.smithi188.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-16T18:12:54.434 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 22 M 2023-11-16T18:12:54.434 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 23 k 2023-11-16T18:12:54.434 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 2.3 M 2023-11-16T18:12:54.434 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 240 k 2023-11-16T18:12:54.434 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 4.6 M 2023-11-16T18:12:54.434 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 18 M 2023-11-16T18:12:54.434 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 15 k 2023-11-16T18:12:54.434 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 24 k 2023-11-16T18:12:54.434 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-16T18:12:54.434 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T18:12:54.434 INFO:teuthology.orchestra.run.smithi188.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-16T18:12:54.434 INFO:teuthology.orchestra.run.smithi188.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-16T18:12:54.434 INFO:teuthology.orchestra.run.smithi188.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-16T18:12:54.435 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 137 k 2023-11-16T18:12:54.435 INFO:teuthology.orchestra.run.smithi188.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-16T18:12:54.435 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-16T18:12:54.435 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T18:12:54.435 INFO:teuthology.orchestra.run.smithi188.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-16T18:12:54.435 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-16T18:12:54.435 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-16T18:12:54.435 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 473 k 2023-11-16T18:12:54.435 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-16T18:12:54.435 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.5 M 2023-11-16T18:12:54.435 INFO:teuthology.orchestra.run.smithi188.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-16T18:12:54.435 INFO:teuthology.orchestra.run.smithi188.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-16T18:12:54.436 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T18:12:54.436 INFO:teuthology.orchestra.run.smithi188.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-16T18:12:54.436 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-16T18:12:54.436 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-16T18:12:54.436 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-16T18:12:54.436 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-16T18:12:54.436 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-16T18:12:54.436 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-16T18:12:54.436 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-16T18:12:54.436 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-16T18:12:54.436 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-16T18:12:54.436 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T18:12:54.436 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T18:12:54.437 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T18:12:54.437 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 45 k 2023-11-16T18:12:54.437 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 119 k 2023-11-16T18:12:54.437 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T18:12:54.437 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-16T18:12:54.437 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T18:12:54.437 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T18:12:54.437 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-16T18:12:54.437 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T18:12:54.437 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T18:12:54.437 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T18:12:54.437 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-16T18:12:54.437 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-16T18:12:54.438 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-16T18:12:54.438 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T18:12:54.438 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T18:12:54.438 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T18:12:54.461 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-16T18:12:54.461 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-16T18:12:54.462 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T18:12:54.462 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-16T18:12:54.463 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-16T18:12:54.463 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-16T18:12:54.463 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-16T18:12:54.463 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T18:12:54.463 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T18:12:54.463 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-16T18:12:54.463 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-16T18:12:54.463 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-16T18:12:54.463 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-16T18:12:54.463 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-16T18:12:54.463 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-16T18:12:54.463 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T18:12:54.463 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T18:12:54.464 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T18:12:54.464 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T18:12:54.464 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-16T18:12:54.464 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T18:12:54.464 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T18:12:54.464 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T18:12:54.464 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T18:12:54.464 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T18:12:54.464 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T18:12:54.464 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T18:12:54.464 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T18:12:54.464 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-16T18:12:54.465 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T18:12:54.465 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-16T18:12:54.465 INFO:teuthology.orchestra.run.smithi188.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-16T18:12:54.465 INFO:teuthology.orchestra.run.smithi188.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-16T18:12:54.465 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-16T18:12:54.495 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-16T18:12:54.495 INFO:teuthology.orchestra.run.smithi188.stdout:Enabling module streams: 2023-11-16T18:12:54.495 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql 10 2023-11-16T18:12:54.495 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:12:54.495 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-16T18:12:54.496 INFO:teuthology.orchestra.run.smithi188.stdout:=========================================================================================================================== 2023-11-16T18:12:54.496 INFO:teuthology.orchestra.run.smithi188.stdout:Install 128 Packages 2023-11-16T18:12:54.503 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 3 Packages 2023-11-16T18:12:54.506 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:12:54.508 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 197 M 2023-11-16T18:12:54.509 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-16T18:12:54.598 INFO:teuthology.orchestra.run.smithi188.stdout:(1/131): ceph-18.0.0-3905.g5e4a743a.el8.x86_64. 47 kB/s | 6.4 kB 00:00 2023-11-16T18:12:54.794 INFO:teuthology.orchestra.run.smithi188.stdout:(2/131): ceph-fuse-18.0.0-3905.g5e4a743a.el8.x8 4.3 MB/s | 847 kB 00:00 2023-11-16T18:12:54.842 INFO:teuthology.orchestra.run.smithi188.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 3.2 MB/s | 165 kB 00:00 2023-11-16T18:12:55.154 INFO:teuthology.orchestra.run.smithi188.stdout:(4/131): ceph-mds-18.0.0-3905.g5e4a743a.el8.x86 7.4 MB/s | 2.3 MB 00:00 2023-11-16T18:12:55.350 INFO:teuthology.orchestra.run.smithi188.stdout:(5/131): ceph-mgr-18.0.0-3905.g5e4a743a.el8.x86 7.7 MB/s | 1.5 MB 00:00 2023-11-16T18:12:55.591 INFO:teuthology.orchestra.run.smithi188.stdout:(6/131): ceph-base-18.0.0-3905.g5e4a743a.el8.x8 4.7 MB/s | 5.3 MB 00:01 2023-11-16T18:12:55.944 INFO:teuthology.orchestra.run.smithi188.stdout:(7/131): ceph-mon-18.0.0-3905.g5e4a743a.el8.x86 7.8 MB/s | 4.6 MB 00:00 2023-11-16T18:12:56.392 INFO:teuthology.orchestra.run.smithi188.stdout:(8/131): ceph-common-18.0.0-3905.g5e4a743a.el8. 11 MB/s | 22 MB 00:01 2023-11-16T18:12:56.434 INFO:teuthology.orchestra.run.smithi188.stdout:(9/131): ceph-selinux-18.0.0-3905.g5e4a743a.el8 586 kB/s | 24 kB 00:00 2023-11-16T18:12:56.720 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T18:12:56.721 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T18:12:56.736 INFO:teuthology.orchestra.run.smithi040.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-16T18:12:56.737 INFO:teuthology.orchestra.run.smithi040.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-16T18:12:56.738 INFO:teuthology.orchestra.run.smithi040.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-16T18:12:56.739 INFO:teuthology.orchestra.run.smithi040.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-16T18:12:56.741 INFO:teuthology.orchestra.run.smithi040.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-16T18:12:56.744 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-16T18:12:56.745 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-16T18:12:56.890 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-11-16T18:12:56.902 INFO:teuthology.orchestra.run.smithi040.stdout:=========================================================================================================================== 2023-11-16T18:12:56.903 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-11-16T18:12:56.903 INFO:teuthology.orchestra.run.smithi040.stdout:=========================================================================================================================== 2023-11-16T18:12:56.910 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-11-16T18:12:56.911 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-16T18:12:56.911 INFO:teuthology.orchestra.run.smithi040.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-16T18:12:56.911 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.4 k 2023-11-16T18:12:56.911 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 5.3 M 2023-11-16T18:12:56.912 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 847 k 2023-11-16T18:12:56.912 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 165 k 2023-11-16T18:12:56.912 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 1.5 M 2023-11-16T18:12:56.912 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 127 k 2023-11-16T18:12:56.913 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 1.7 M 2023-11-16T18:12:56.913 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 7.4 M 2023-11-16T18:12:56.913 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 47 k 2023-11-16T18:12:56.914 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 13 M 2023-11-16T18:12:56.914 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 41 M 2023-11-16T18:12:56.914 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 265 k 2023-11-16T18:12:56.914 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 83 k 2023-11-16T18:12:56.914 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 258 k 2023-11-16T18:12:56.924 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 18 k 2023-11-16T18:12:56.924 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-16T18:12:56.925 INFO:teuthology.orchestra.run.smithi040.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-16T18:12:56.925 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-16T18:12:56.925 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 31 k 2023-11-16T18:12:56.925 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 771 k 2023-11-16T18:12:56.926 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 139 k 2023-11-16T18:12:56.926 INFO:teuthology.orchestra.run.smithi040.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-16T18:12:56.927 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-16T18:12:56.927 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-16T18:12:56.928 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-16T18:12:56.928 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-16T18:12:56.928 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-16T18:12:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 197 k 2023-11-16T18:12:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 371 k 2023-11-16T18:12:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 365 k 2023-11-16T18:12:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 107 k 2023-11-16T18:12:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 94 k 2023-11-16T18:12:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.4 M 2023-11-16T18:12:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 180 k 2023-11-16T18:12:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-16T18:12:56.932 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2023-11-16T18:12:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-16T18:12:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.7 M 2023-11-16T18:12:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.6 M 2023-11-16T18:12:56.937 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-11-16T18:12:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-16T18:12:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 22 M 2023-11-16T18:12:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 23 k 2023-11-16T18:12:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 2.3 M 2023-11-16T18:12:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 240 k 2023-11-16T18:12:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 4.6 M 2023-11-16T18:12:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 18 M 2023-11-16T18:12:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 15 k 2023-11-16T18:12:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 24 k 2023-11-16T18:12:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-16T18:12:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T18:12:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-16T18:12:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-16T18:12:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-16T18:12:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 137 k 2023-11-16T18:12:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-16T18:12:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-16T18:12:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T18:12:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-16T18:12:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-16T18:12:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-16T18:12:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 473 k 2023-11-16T18:12:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-16T18:12:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.5 M 2023-11-16T18:12:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-16T18:12:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-16T18:12:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T18:12:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-16T18:12:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-16T18:12:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-16T18:12:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-16T18:12:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-16T18:12:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-16T18:12:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-16T18:12:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-16T18:12:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-16T18:12:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-16T18:12:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T18:12:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T18:12:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T18:12:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 45 k 2023-11-16T18:12:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 119 k 2023-11-16T18:12:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T18:12:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-16T18:12:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T18:12:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T18:12:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-16T18:12:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T18:12:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T18:12:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T18:12:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-16T18:12:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-16T18:12:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-16T18:12:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T18:12:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T18:12:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T18:12:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-16T18:12:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-16T18:12:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T18:12:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-16T18:12:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-16T18:12:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-16T18:12:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-16T18:12:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T18:12:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T18:12:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-16T18:12:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-16T18:12:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-16T18:12:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-16T18:12:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-16T18:12:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-16T18:12:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T18:12:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T18:12:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T18:12:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T18:12:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-16T18:12:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T18:12:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T18:12:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T18:12:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T18:12:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T18:12:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T18:12:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T18:12:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T18:12:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-16T18:12:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T18:12:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-16T18:12:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-16T18:12:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-16T18:12:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-16T18:12:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-16T18:12:57.019 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling module streams: 2023-11-16T18:12:57.020 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql 10 2023-11-16T18:12:57.020 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-11-16T18:12:57.020 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-11-16T18:12:57.020 INFO:teuthology.orchestra.run.smithi040.stdout:=========================================================================================================================== 2023-11-16T18:12:57.020 INFO:teuthology.orchestra.run.smithi040.stdout:Install 128 Packages 2023-11-16T18:12:57.020 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 3 Packages 2023-11-16T18:12:57.020 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-11-16T18:12:57.023 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 197 M 2023-11-16T18:12:57.023 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-11-16T18:12:57.355 INFO:teuthology.orchestra.run.smithi040.stdout:(1/131): ceph-18.0.0-3905.g5e4a743a.el8.x86_64. 23 kB/s | 6.4 kB 00:00 2023-11-16T18:12:57.401 INFO:teuthology.orchestra.run.smithi188.stdout:(10/131): ceph-radosgw-18.0.0-3905.g5e4a743a.el 8.9 MB/s | 13 MB 00:01 2023-11-16T18:12:57.484 INFO:teuthology.orchestra.run.smithi188.stdout:(11/131): cephfs-mirror-18.0.0-3905.g5e4a743a.e 3.0 MB/s | 258 kB 00:00 2023-11-16T18:12:57.560 INFO:teuthology.orchestra.run.smithi188.stdout:(12/131): libcephfs-devel-18.0.0-3905.g5e4a743a 414 kB/s | 31 kB 00:00 2023-11-16T18:12:57.727 INFO:teuthology.orchestra.run.smithi188.stdout:(13/131): libcephfs2-18.0.0-3905.g5e4a743a.el8. 4.5 MB/s | 771 kB 00:00 2023-11-16T18:12:57.810 INFO:teuthology.orchestra.run.smithi188.stdout:(14/131): libcephsqlite-18.0.0-3905.g5e4a743a.e 1.6 MB/s | 137 kB 00:00 2023-11-16T18:12:57.894 INFO:teuthology.orchestra.run.smithi188.stdout:(15/131): librados-devel-18.0.0-3905.g5e4a743a. 1.6 MB/s | 139 kB 00:00 2023-11-16T18:12:58.002 INFO:teuthology.orchestra.run.smithi188.stdout:(16/131): libradosstriper1-18.0.0-3905.g5e4a743 4.3 MB/s | 473 kB 00:00 2023-11-16T18:12:58.322 INFO:teuthology.orchestra.run.smithi188.stdout:(17/131): ceph-osd-18.0.0-3905.g5e4a743a.el8.x8 6.5 MB/s | 18 MB 00:02 2023-11-16T18:12:58.338 INFO:teuthology.orchestra.run.smithi040.stdout:(2/131): ceph-fuse-18.0.0-3905.g5e4a743a.el8.x8 862 kB/s | 847 kB 00:00 2023-11-16T18:12:58.414 INFO:teuthology.orchestra.run.smithi188.stdout:(18/131): python3-ceph-argparse-18.0.0-3905.g5e 489 kB/s | 45 kB 00:00 2023-11-16T18:12:58.520 INFO:teuthology.orchestra.run.smithi188.stdout:(19/131): python3-ceph-common-18.0.0-3905.g5e4a 1.1 MB/s | 119 kB 00:00 2023-11-16T18:12:58.526 INFO:teuthology.orchestra.run.smithi040.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 903 kB/s | 165 kB 00:00 2023-11-16T18:12:58.620 INFO:teuthology.orchestra.run.smithi188.stdout:(20/131): python3-cephfs-18.0.0-3905.g5e4a743a. 1.9 MB/s | 197 kB 00:00 2023-11-16T18:12:58.706 INFO:teuthology.orchestra.run.smithi040.stdout:(4/131): ceph-base-18.0.0-3905.g5e4a743a.el8.x8 3.3 MB/s | 5.3 MB 00:01 2023-11-16T18:12:58.737 INFO:teuthology.orchestra.run.smithi188.stdout:(21/131): python3-rados-18.0.0-3905.g5e4a743a.e 3.1 MB/s | 371 kB 00:00 2023-11-16T18:12:58.855 INFO:teuthology.orchestra.run.smithi188.stdout:(22/131): librgw2-18.0.0-3905.g5e4a743a.el8.x86 7.7 MB/s | 6.5 MB 00:00 2023-11-16T18:12:58.882 INFO:teuthology.orchestra.run.smithi188.stdout:(23/131): python3-rbd-18.0.0-3905.g5e4a743a.el8 2.5 MB/s | 365 kB 00:00 2023-11-16T18:12:58.939 INFO:teuthology.orchestra.run.smithi188.stdout:(24/131): python3-rgw-18.0.0-3905.g5e4a743a.el8 1.3 MB/s | 107 kB 00:00 2023-11-16T18:12:58.965 INFO:teuthology.orchestra.run.smithi188.stdout:(25/131): rbd-fuse-18.0.0-3905.g5e4a743a.el8.x8 1.1 MB/s | 94 kB 00:00 2023-11-16T18:12:58.998 INFO:teuthology.orchestra.run.smithi040.stdout:(5/131): ceph-mgr-18.0.0-3905.g5e4a743a.el8.x86 5.3 MB/s | 1.5 MB 00:00 2023-11-16T18:12:59.057 INFO:teuthology.orchestra.run.smithi188.stdout:(26/131): rbd-nbd-18.0.0-3905.g5e4a743a.el8.x86 1.9 MB/s | 180 kB 00:00 2023-11-16T18:12:59.156 INFO:teuthology.orchestra.run.smithi188.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3905.g 230 kB/s | 23 kB 00:00 2023-11-16T18:12:59.224 INFO:teuthology.orchestra.run.smithi002.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T18:12:59.225 INFO:teuthology.orchestra.run.smithi002.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T18:12:59.248 INFO:teuthology.orchestra.run.smithi002.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-16T18:12:59.250 INFO:teuthology.orchestra.run.smithi002.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-16T18:12:59.251 INFO:teuthology.orchestra.run.smithi002.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-16T18:12:59.251 INFO:teuthology.orchestra.run.smithi002.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-16T18:12:59.252 INFO:teuthology.orchestra.run.smithi002.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-16T18:12:59.253 INFO:teuthology.orchestra.run.smithi002.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-16T18:12:59.254 INFO:teuthology.orchestra.run.smithi002.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-16T18:12:59.270 INFO:teuthology.orchestra.run.smithi188.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3905.g5e4a743 1.1 MB/s | 127 kB 00:00 2023-11-16T18:12:59.396 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-11-16T18:12:59.410 INFO:teuthology.orchestra.run.smithi002.stdout:=========================================================================================================================== 2023-11-16T18:12:59.412 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-11-16T18:12:59.412 INFO:teuthology.orchestra.run.smithi002.stdout:=========================================================================================================================== 2023-11-16T18:12:59.413 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-11-16T18:12:59.413 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-16T18:12:59.414 INFO:teuthology.orchestra.run.smithi002.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-16T18:12:59.414 INFO:teuthology.orchestra.run.smithi002.stdout: ceph x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.4 k 2023-11-16T18:12:59.415 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 5.3 M 2023-11-16T18:12:59.416 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 847 k 2023-11-16T18:12:59.416 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 165 k 2023-11-16T18:12:59.416 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 1.5 M 2023-11-16T18:12:59.417 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 127 k 2023-11-16T18:12:59.418 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 1.7 M 2023-11-16T18:12:59.418 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 7.4 M 2023-11-16T18:12:59.419 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 47 k 2023-11-16T18:12:59.419 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 13 M 2023-11-16T18:12:59.420 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 41 M 2023-11-16T18:12:59.420 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 265 k 2023-11-16T18:12:59.421 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 83 k 2023-11-16T18:12:59.421 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 258 k 2023-11-16T18:12:59.422 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-top noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 18 k 2023-11-16T18:12:59.422 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-16T18:12:59.422 INFO:teuthology.orchestra.run.smithi002.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-16T18:12:59.423 INFO:teuthology.orchestra.run.smithi002.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-16T18:12:59.423 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 31 k 2023-11-16T18:12:59.424 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 771 k 2023-11-16T18:12:59.424 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 139 k 2023-11-16T18:12:59.424 INFO:teuthology.orchestra.run.smithi002.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-16T18:12:59.425 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-16T18:12:59.425 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-16T18:12:59.426 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-16T18:12:59.426 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-16T18:12:59.426 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-16T18:12:59.427 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 197 k 2023-11-16T18:12:59.427 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 371 k 2023-11-16T18:12:59.435 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 365 k 2023-11-16T18:12:59.447 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 107 k 2023-11-16T18:12:59.447 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 94 k 2023-11-16T18:12:59.448 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.4 M 2023-11-16T18:12:59.448 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 180 k 2023-11-16T18:12:59.448 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-16T18:12:59.449 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrading: 2023-11-16T18:12:59.449 INFO:teuthology.orchestra.run.smithi002.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-16T18:12:59.450 INFO:teuthology.orchestra.run.smithi002.stdout: librados2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.7 M 2023-11-16T18:12:59.450 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.6 M 2023-11-16T18:12:59.451 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2023-11-16T18:12:59.451 INFO:teuthology.orchestra.run.smithi002.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-16T18:12:59.451 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 22 M 2023-11-16T18:12:59.452 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 23 k 2023-11-16T18:12:59.453 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 2.3 M 2023-11-16T18:12:59.453 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 240 k 2023-11-16T18:12:59.453 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 4.6 M 2023-11-16T18:12:59.454 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 18 M 2023-11-16T18:12:59.454 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 15 k 2023-11-16T18:12:59.455 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 24 k 2023-11-16T18:12:59.455 INFO:teuthology.orchestra.run.smithi002.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-16T18:12:59.457 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T18:12:59.458 INFO:teuthology.orchestra.run.smithi002.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-16T18:12:59.458 INFO:teuthology.orchestra.run.smithi002.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-16T18:12:59.459 INFO:teuthology.orchestra.run.smithi002.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-16T18:12:59.459 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 137 k 2023-11-16T18:12:59.459 INFO:teuthology.orchestra.run.smithi002.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-16T18:12:59.460 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-16T18:12:59.461 INFO:teuthology.orchestra.run.smithi002.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T18:12:59.461 INFO:teuthology.orchestra.run.smithi002.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-16T18:12:59.461 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-16T18:12:59.462 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-16T18:12:59.462 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 473 k 2023-11-16T18:12:59.486 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-16T18:12:59.494 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.5 M 2023-11-16T18:12:59.508 INFO:teuthology.orchestra.run.smithi002.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-16T18:12:59.522 INFO:teuthology.orchestra.run.smithi002.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-16T18:12:59.523 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T18:12:59.527 INFO:teuthology.orchestra.run.smithi002.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-16T18:12:59.530 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-16T18:12:59.530 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-16T18:12:59.531 INFO:teuthology.orchestra.run.smithi002.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-16T18:12:59.531 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-16T18:12:59.532 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-16T18:12:59.532 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-16T18:12:59.532 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-16T18:12:59.533 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-16T18:12:59.533 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-16T18:12:59.534 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T18:12:59.534 INFO:teuthology.orchestra.run.smithi002.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T18:12:59.534 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T18:12:59.535 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 45 k 2023-11-16T18:12:59.535 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 119 k 2023-11-16T18:12:59.537 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T18:12:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-16T18:12:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T18:12:59.553 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T18:12:59.554 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-16T18:12:59.556 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T18:12:59.557 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T18:12:59.557 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T18:12:59.564 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-16T18:12:59.566 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-16T18:12:59.566 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-16T18:12:59.568 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T18:12:59.569 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T18:12:59.570 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T18:12:59.571 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-16T18:12:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-16T18:12:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T18:12:59.582 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-16T18:12:59.582 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-16T18:12:59.582 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-16T18:12:59.584 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-16T18:12:59.585 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T18:12:59.585 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T18:12:59.586 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-16T18:12:59.586 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-16T18:12:59.587 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-16T18:12:59.587 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-16T18:12:59.587 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-16T18:12:59.588 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-16T18:12:59.589 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T18:12:59.589 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T18:12:59.655 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T18:12:59.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T18:12:59.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-16T18:12:59.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T18:12:59.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T18:12:59.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T18:12:59.675 INFO:teuthology.orchestra.run.smithi002.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T18:12:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T18:12:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T18:12:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T18:12:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T18:12:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-16T18:12:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T18:12:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-16T18:12:59.678 INFO:teuthology.orchestra.run.smithi002.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-16T18:12:59.678 INFO:teuthology.orchestra.run.smithi002.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-16T18:12:59.678 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-16T18:12:59.678 INFO:teuthology.orchestra.run.smithi002.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-16T18:12:59.678 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling module streams: 2023-11-16T18:12:59.678 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql 10 2023-11-16T18:12:59.679 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:12:59.679 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-11-16T18:12:59.679 INFO:teuthology.orchestra.run.smithi002.stdout:=========================================================================================================================== 2023-11-16T18:12:59.679 INFO:teuthology.orchestra.run.smithi002.stdout:Install 128 Packages 2023-11-16T18:12:59.687 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrade 3 Packages 2023-11-16T18:12:59.687 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:12:59.728 INFO:teuthology.orchestra.run.smithi188.stdout:(29/131): rbd-mirror-18.0.0-3905.g5e4a743a.el8. 5.9 MB/s | 3.4 MB 00:00 2023-11-16T18:12:59.728 INFO:teuthology.orchestra.run.smithi188.stdout:(30/131): ceph-mgr-dashboard-18.0.0-3905.g5e4a7 3.9 MB/s | 1.7 MB 00:00 2023-11-16T18:12:59.728 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 197 M 2023-11-16T18:12:59.730 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-11-16T18:12:59.735 INFO:teuthology.orchestra.run.smithi002.stdout:(1/131): ceph-18.0.0-3905.g5e4a743a.el8.x86_64. 26 kB/s | 6.4 kB 00:00 2023-11-16T18:12:59.777 INFO:teuthology.orchestra.run.smithi040.stdout:(6/131): ceph-mon-18.0.0-3905.g5e4a743a.el8.x86 5.9 MB/s | 4.6 MB 00:00 2023-11-16T18:12:59.796 INFO:teuthology.orchestra.run.smithi188.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3905.g5e 2.6 MB/s | 240 kB 00:00 2023-11-16T18:12:59.890 INFO:teuthology.orchestra.run.smithi188.stdout:(32/131): ceph-mgr-rook-18.0.0-3905.g5e4a743a.e 475 kB/s | 47 kB 00:00 2023-11-16T18:12:59.974 INFO:teuthology.orchestra.run.smithi188.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3905.g5 175 kB/s | 15 kB 00:00 2023-11-16T18:13:00.056 INFO:teuthology.orchestra.run.smithi040.stdout:(7/131): ceph-mds-18.0.0-3905.g5e4a743a.el8.x86 1.5 MB/s | 2.3 MB 00:01 2023-11-16T18:13:00.091 INFO:teuthology.orchestra.run.smithi188.stdout:(34/131): ceph-volume-18.0.0-3905.g5e4a743a.el8 2.2 MB/s | 265 kB 00:00 2023-11-16T18:13:00.200 INFO:teuthology.orchestra.run.smithi188.stdout:(35/131): cephadm-18.0.0-3905.g5e4a743a.el8.noa 769 kB/s | 83 kB 00:00 2023-11-16T18:13:00.292 INFO:teuthology.orchestra.run.smithi188.stdout:(36/131): cephfs-top-18.0.0-3905.g5e4a743a.el8. 196 kB/s | 18 kB 00:00 2023-11-16T18:13:00.384 INFO:teuthology.orchestra.run.smithi188.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 3.3 MB/s | 306 kB 00:00 2023-11-16T18:13:00.419 INFO:teuthology.orchestra.run.smithi188.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.7 MB/s | 59 kB 00:00 2023-11-16T18:13:00.451 INFO:teuthology.orchestra.run.smithi188.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 2.2 MB/s | 75 kB 00:00 2023-11-16T18:13:00.616 INFO:teuthology.orchestra.run.smithi002.stdout:(2/131): ceph-fuse-18.0.0-3905.g5e4a743a.el8.x8 959 kB/s | 847 kB 00:00 2023-11-16T18:13:00.757 INFO:teuthology.orchestra.run.smithi002.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 1.1 MB/s | 165 kB 00:00 2023-11-16T18:13:00.818 INFO:teuthology.orchestra.run.smithi188.stdout:(40/131): ceph-test-18.0.0-3905.g5e4a743a.el8.x 9.3 MB/s | 41 MB 00:04 2023-11-16T18:13:00.852 INFO:teuthology.orchestra.run.smithi188.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.2 MB/s | 476 kB 00:00 2023-11-16T18:13:00.885 INFO:teuthology.orchestra.run.smithi188.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 5.5 MB/s | 185 kB 00:00 2023-11-16T18:13:00.910 INFO:teuthology.orchestra.run.smithi188.stdout:(43/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 475 kB/s | 44 kB 00:00 2023-11-16T18:13:00.936 INFO:teuthology.orchestra.run.smithi188.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 657 kB/s | 33 kB 00:00 2023-11-16T18:13:00.963 INFO:teuthology.orchestra.run.smithi188.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 324 kB/s | 16 kB 00:00 2023-11-16T18:13:00.987 INFO:teuthology.orchestra.run.smithi188.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 3.4 MB/s | 173 kB 00:00 2023-11-16T18:13:01.022 INFO:teuthology.orchestra.run.smithi188.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 2.4 MB/s | 81 kB 00:00 2023-11-16T18:13:01.040 INFO:teuthology.orchestra.run.smithi040.stdout:(8/131): ceph-common-18.0.0-3905.g5e4a743a.el8. 5.5 MB/s | 22 MB 00:03 2023-11-16T18:13:01.054 INFO:teuthology.orchestra.run.smithi188.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 336 kB/s | 11 kB 00:00 2023-11-16T18:13:01.079 INFO:teuthology.orchestra.run.smithi188.stdout:(49/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.2 MB/s | 384 kB 00:00 2023-11-16T18:13:01.105 INFO:teuthology.orchestra.run.smithi188.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 349 kB/s | 18 kB 00:00 2023-11-16T18:13:01.119 INFO:teuthology.orchestra.run.smithi040.stdout:(9/131): ceph-selinux-18.0.0-3905.g5e4a743a.el8 325 kB/s | 24 kB 00:00 2023-11-16T18:13:01.167 INFO:teuthology.orchestra.run.smithi188.stdout:(51/131): python3-kubernetes-11.0.0-6.el8.noarc 14 MB/s | 1.2 MB 00:00 2023-11-16T18:13:01.197 INFO:teuthology.orchestra.run.smithi188.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 531 kB/s | 49 kB 00:00 2023-11-16T18:13:01.222 INFO:teuthology.orchestra.run.smithi188.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2023-11-16T18:13:01.247 INFO:teuthology.orchestra.run.smithi188.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2023-11-16T18:13:01.277 INFO:teuthology.orchestra.run.smithi188.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2023-11-16T18:13:01.298 INFO:teuthology.orchestra.run.smithi188.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2023-11-16T18:13:01.331 INFO:teuthology.orchestra.run.smithi188.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 669 kB/s | 34 kB 00:00 2023-11-16T18:13:01.348 INFO:teuthology.orchestra.run.smithi188.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2023-11-16T18:13:01.374 INFO:teuthology.orchestra.run.smithi002.stdout:(4/131): ceph-mds-18.0.0-3905.g5e4a743a.el8.x86 3.7 MB/s | 2.3 MB 00:00 2023-11-16T18:13:01.374 INFO:teuthology.orchestra.run.smithi188.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 1.2 MB/s | 64 kB 00:00 2023-11-16T18:13:01.399 INFO:teuthology.orchestra.run.smithi188.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 373 kB/s | 19 kB 00:00 2023-11-16T18:13:01.424 INFO:teuthology.orchestra.run.smithi188.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 478 kB/s | 24 kB 00:00 2023-11-16T18:13:01.450 INFO:teuthology.orchestra.run.smithi188.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 578 kB/s | 29 kB 00:00 2023-11-16T18:13:01.475 INFO:teuthology.orchestra.run.smithi188.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 546 kB/s | 27 kB 00:00 2023-11-16T18:13:01.501 INFO:teuthology.orchestra.run.smithi188.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 5.0 MB/s | 255 kB 00:00 2023-11-16T18:13:01.526 INFO:teuthology.orchestra.run.smithi188.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 4.9 MB/s | 251 kB 00:00 2023-11-16T18:13:01.552 INFO:teuthology.orchestra.run.smithi188.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2023-11-16T18:13:01.577 INFO:teuthology.orchestra.run.smithi188.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2023-11-16T18:13:01.602 INFO:teuthology.orchestra.run.smithi188.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 457 kB/s | 23 kB 00:00 2023-11-16T18:13:01.627 INFO:teuthology.orchestra.run.smithi188.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-11-16T18:13:01.677 INFO:teuthology.orchestra.run.smithi188.stdout:(70/131): thrift-0.13.0-2.el8.x86_64.rpm 17 MB/s | 1.7 MB 00:00 2023-11-16T18:13:01.827 INFO:teuthology.orchestra.run.smithi188.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 234 kB/s | 47 kB 00:00 2023-11-16T18:13:01.878 INFO:teuthology.orchestra.run.smithi188.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.0 MB/s | 200 kB 00:00 2023-11-16T18:13:02.003 INFO:teuthology.orchestra.run.smithi188.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 243 kB/s | 43 kB 00:00 2023-11-16T18:13:02.029 INFO:teuthology.orchestra.run.smithi188.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.4 MB/s | 220 kB 00:00 2023-11-16T18:13:02.049 INFO:teuthology.orchestra.run.smithi002.stdout:(5/131): ceph-mgr-18.0.0-3905.g5e4a743a.el8.x86 2.3 MB/s | 1.5 MB 00:00 2023-11-16T18:13:02.179 INFO:teuthology.orchestra.run.smithi188.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 275 kB/s | 48 kB 00:00 2023-11-16T18:13:02.205 INFO:teuthology.orchestra.run.smithi188.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 497 kB/s | 87 kB 00:00 2023-11-16T18:13:02.338 INFO:teuthology.orchestra.run.smithi188.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 269 kB/s | 43 kB 00:00 2023-11-16T18:13:02.363 INFO:teuthology.orchestra.run.smithi188.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 688 kB/s | 109 kB 00:00 2023-11-16T18:13:02.488 INFO:teuthology.orchestra.run.smithi188.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.5 MB/s | 229 kB 00:00 2023-11-16T18:13:02.532 INFO:teuthology.orchestra.run.smithi188.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.3 MB/s | 560 kB 00:00 2023-11-16T18:13:02.680 INFO:teuthology.orchestra.run.smithi188.stdout:(81/131): ceph-mgr-diskprediction-local-18.0.0- 2.3 MB/s | 7.4 MB 00:03 2023-11-16T18:13:02.699 INFO:teuthology.orchestra.run.smithi040.stdout:(10/131): ceph-osd-18.0.0-3905.g5e4a743a.el8.x8 6.1 MB/s | 18 MB 00:02 2023-11-16T18:13:02.706 INFO:teuthology.orchestra.run.smithi188.stdout:(82/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 86 kB/s | 19 kB 00:00 2023-11-16T18:13:02.763 INFO:teuthology.orchestra.run.smithi188.stdout:(83/131): python3-oauthlib-2.1.0-1.el8.noarch.r 667 kB/s | 155 kB 00:00 2023-11-16T18:13:02.782 INFO:teuthology.orchestra.run.smithi040.stdout:(11/131): cephfs-mirror-18.0.0-3905.g5e4a743a.e 3.0 MB/s | 258 kB 00:00 2023-11-16T18:13:02.874 INFO:teuthology.orchestra.run.smithi040.stdout:(12/131): libcephfs-devel-18.0.0-3905.g5e4a743a 339 kB/s | 31 kB 00:00 2023-11-16T18:13:02.883 INFO:teuthology.orchestra.run.smithi188.stdout:(84/131): python3-cryptography-3.2.1-5.el8.x86_ 2.7 MB/s | 559 kB 00:00 2023-11-16T18:13:02.948 INFO:teuthology.orchestra.run.smithi188.stdout:(85/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 75 kB/s | 18 kB 00:00 2023-11-16T18:13:03.024 INFO:teuthology.orchestra.run.smithi188.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.6 MB/s | 238 kB 00:00 2023-11-16T18:13:03.057 INFO:teuthology.orchestra.run.smithi188.stdout:(87/131): pcre2-devel-10.32-2.el8.x86_64.rpm 2.0 MB/s | 605 kB 00:00 2023-11-16T18:13:03.126 INFO:teuthology.orchestra.run.smithi188.stdout:(88/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 13 MB/s | 2.3 MB 00:00 2023-11-16T18:13:03.133 INFO:teuthology.orchestra.run.smithi040.stdout:(13/131): libcephfs2-18.0.0-3905.g5e4a743a.el8. 2.9 MB/s | 771 kB 00:00 2023-11-16T18:13:03.150 INFO:teuthology.orchestra.run.smithi188.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 900 kB/s | 83 kB 00:00 2023-11-16T18:13:03.200 INFO:teuthology.orchestra.run.smithi002.stdout:(6/131): ceph-base-18.0.0-3905.g5e4a743a.el8.x8 1.4 MB/s | 5.3 MB 00:03 2023-11-16T18:13:03.216 INFO:teuthology.orchestra.run.smithi040.stdout:(14/131): libcephsqlite-18.0.0-3905.g5e4a743a.e 1.6 MB/s | 137 kB 00:00 2023-11-16T18:13:03.218 INFO:teuthology.orchestra.run.smithi188.stdout:(90/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 375 kB/s | 72 kB 00:00 2023-11-16T18:13:03.242 INFO:teuthology.orchestra.run.smithi188.stdout:(91/131): elfutils-libelf-devel-0.186-1.el8.x86 510 kB/s | 60 kB 00:00 2023-11-16T18:13:03.268 INFO:teuthology.orchestra.run.smithi188.stdout:(92/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 838 kB/s | 98 kB 00:00 2023-11-16T18:13:03.301 INFO:teuthology.orchestra.run.smithi040.stdout:(15/131): librados-devel-18.0.0-3905.g5e4a743a. 1.6 MB/s | 139 kB 00:00 2023-11-16T18:13:03.309 INFO:teuthology.orchestra.run.smithi188.stdout:(93/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 626 kB/s | 58 kB 00:00 2023-11-16T18:13:03.384 INFO:teuthology.orchestra.run.smithi188.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 5.4 MB/s | 644 kB 00:00 2023-11-16T18:13:03.410 INFO:teuthology.orchestra.run.smithi188.stdout:(95/131): libcom_err-devel-1.45.6-4.el8.x86_64. 230 kB/s | 39 kB 00:00 2023-11-16T18:13:03.460 INFO:teuthology.orchestra.run.smithi188.stdout:(96/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.1 MB/s | 171 kB 00:00 2023-11-16T18:13:03.475 INFO:teuthology.orchestra.run.smithi040.stdout:(16/131): libradosstriper1-18.0.0-3905.g5e4a743 2.6 MB/s | 473 kB 00:00 2023-11-16T18:13:03.502 INFO:teuthology.orchestra.run.smithi188.stdout:(97/131): NetworkManager-initscripts-updown-1.3 1.1 MB/s | 138 kB 00:00 2023-11-16T18:13:03.585 INFO:teuthology.orchestra.run.smithi188.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 650 kB/s | 54 kB 00:00 2023-11-16T18:13:03.661 INFO:teuthology.orchestra.run.smithi188.stdout:(99/131): python3-pyasn1-modules-0.3.7-6.el8.no 549 kB/s | 110 kB 00:00 2023-11-16T18:13:03.821 INFO:teuthology.orchestra.run.smithi002.stdout:(7/131): ceph-mon-18.0.0-3905.g5e4a743a.el8.x86 2.6 MB/s | 4.6 MB 00:01 2023-11-16T18:13:03.834 INFO:teuthology.orchestra.run.smithi188.stdout:(100/131): python3-scipy-1.0.0-21.module+el8.5. 33 MB/s | 14 MB 00:00 2023-11-16T18:13:03.854 INFO:teuthology.orchestra.run.smithi188.stdout:(101/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 970 kB/s | 259 kB 00:00 2023-11-16T18:13:03.882 INFO:teuthology.orchestra.run.smithi188.stdout:(102/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.5 MB/s | 323 kB 00:00 2023-11-16T18:13:03.953 INFO:teuthology.orchestra.run.smithi188.stdout:(103/131): python3-markupsafe-0.23-19.el8.x86_6 310 kB/s | 39 kB 00:00 2023-11-16T18:13:04.038 INFO:teuthology.orchestra.run.smithi188.stdout:(104/131): bison-3.0.4-10.el8.x86_64.rpm 3.7 MB/s | 688 kB 00:00 2023-11-16T18:13:04.063 INFO:teuthology.orchestra.run.smithi188.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2023-11-16T18:13:04.087 INFO:teuthology.orchestra.run.smithi188.stdout:(106/131): python3-jinja2-2.10.1-3.el8.noarch.r 2.5 MB/s | 538 kB 00:00 2023-11-16T18:13:04.115 INFO:teuthology.orchestra.run.smithi188.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2023-11-16T18:13:04.118 INFO:teuthology.orchestra.run.smithi040.stdout:(17/131): ceph-test-18.0.0-3905.g5e4a743a.el8.x 14 MB/s | 41 MB 00:03 2023-11-16T18:13:04.176 INFO:teuthology.orchestra.run.smithi040.stdout:(18/131): python3-ceph-argparse-18.0.0-3905.g5e 772 kB/s | 45 kB 00:00 2023-11-16T18:13:04.196 INFO:teuthology.orchestra.run.smithi188.stdout:(108/131): automake-1.16.1-7.el8.noarch.rpm 5.2 MB/s | 713 kB 00:00 2023-11-16T18:13:04.243 INFO:teuthology.orchestra.run.smithi040.stdout:(19/131): python3-ceph-common-18.0.0-3905.g5e4a 1.7 MB/s | 119 kB 00:00 2023-11-16T18:13:04.249 INFO:teuthology.orchestra.run.smithi188.stdout:(109/131): python3-jsonpatch-1.21-2.el8.noarch. 199 kB/s | 27 kB 00:00 2023-11-16T18:13:04.314 INFO:teuthology.orchestra.run.smithi188.stdout:(110/131): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2023-11-16T18:13:04.315 INFO:teuthology.orchestra.run.smithi040.stdout:(20/131): python3-cephfs-18.0.0-3905.g5e4a743a. 2.9 MB/s | 197 kB 00:00 2023-11-16T18:13:04.365 INFO:teuthology.orchestra.run.smithi188.stdout:(111/131): flex-2.6.1-9.el8.x86_64.rpm 1.9 MB/s | 320 kB 00:00 2023-11-16T18:13:04.386 INFO:teuthology.orchestra.run.smithi040.stdout:(21/131): python3-rados-18.0.0-3905.g5e4a743a.e 4.8 MB/s | 371 kB 00:00 2023-11-16T18:13:04.390 INFO:teuthology.orchestra.run.smithi188.stdout:(112/131): python3-prettytable-0.7.2-14.el8.noa 309 kB/s | 44 kB 00:00 2023-11-16T18:13:04.468 INFO:teuthology.orchestra.run.smithi040.stdout:(22/131): python3-rbd-18.0.0-3905.g5e4a743a.el8 4.3 MB/s | 365 kB 00:00 2023-11-16T18:13:04.481 INFO:teuthology.orchestra.run.smithi188.stdout:(113/131): python3-werkzeug-0.12.2-4.el8.noarch 3.9 MB/s | 457 kB 00:00 2023-11-16T18:13:04.523 INFO:teuthology.orchestra.run.smithi188.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.6 MB/s | 225 kB 00:00 2023-11-16T18:13:04.527 INFO:teuthology.orchestra.run.smithi040.stdout:(23/131): python3-rgw-18.0.0-3905.g5e4a743a.el8 1.8 MB/s | 107 kB 00:00 2023-11-16T18:13:04.590 INFO:teuthology.orchestra.run.smithi188.stdout:(115/131): python3-numpy-1.14.3-10.el8.x86_64.r 14 MB/s | 3.7 MB 00:00 2023-11-16T18:13:04.603 INFO:teuthology.orchestra.run.smithi040.stdout:(24/131): rbd-fuse-18.0.0-3905.g5e4a743a.el8.x8 1.2 MB/s | 94 kB 00:00 2023-11-16T18:13:04.615 INFO:teuthology.orchestra.run.smithi188.stdout:(116/131): uuid-1.6.2-43.el8.x86_64.rpm 478 kB/s | 64 kB 00:00 2023-11-16T18:13:04.657 INFO:teuthology.orchestra.run.smithi188.stdout:(117/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 945 kB/s | 126 kB 00:00 2023-11-16T18:13:04.716 INFO:teuthology.orchestra.run.smithi188.stdout:(118/131): python3-jsonpointer-1.10-11.el8.noar 159 kB/s | 20 kB 00:00 2023-11-16T18:13:04.730 INFO:teuthology.orchestra.run.smithi040.stdout:(25/131): librgw2-18.0.0-3905.g5e4a743a.el8.x86 5.2 MB/s | 6.5 MB 00:01 2023-11-16T18:13:04.750 INFO:teuthology.orchestra.run.smithi188.stdout:(119/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 771 kB/s | 103 kB 00:00 2023-11-16T18:13:04.795 INFO:teuthology.orchestra.run.smithi188.stdout:(120/131): libtool-2.4.6-25.el8.x86_64.rpm 5.2 MB/s | 709 kB 00:00 2023-11-16T18:13:04.811 INFO:teuthology.orchestra.run.smithi040.stdout:(26/131): rbd-nbd-18.0.0-3905.g5e4a743a.el8.x86 2.1 MB/s | 180 kB 00:00 2023-11-16T18:13:04.901 INFO:teuthology.orchestra.run.smithi188.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 26 MB/s | 4.9 MB 00:00 2023-11-16T18:13:04.929 INFO:teuthology.orchestra.run.smithi188.stdout:(122/131): libpq-13.5-1.el8.x86_64.rpm 1.1 MB/s | 198 kB 00:00 2023-11-16T18:13:04.951 INFO:teuthology.orchestra.run.smithi040.stdout:(27/131): rbd-mirror-18.0.0-3905.g5e4a743a.el8. 9.9 MB/s | 3.4 MB 00:00 2023-11-16T18:13:04.979 INFO:teuthology.orchestra.run.smithi040.stdout:(28/131): ceph-grafana-dashboards-18.0.0-3905.g 138 kB/s | 23 kB 00:00 2023-11-16T18:13:05.012 INFO:teuthology.orchestra.run.smithi040.stdout:(29/131): ceph-mgr-cephadm-18.0.0-3905.g5e4a743 2.1 MB/s | 127 kB 00:00 2023-11-16T18:13:05.043 INFO:teuthology.orchestra.run.smithi188.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2023-11-16T18:13:05.062 INFO:teuthology.orchestra.run.smithi188.stdout:(124/131): jq-1.6-3.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2023-11-16T18:13:05.085 INFO:teuthology.orchestra.run.smithi188.stdout:(125/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.2 MB/s | 354 kB 00:00 2023-11-16T18:13:05.227 INFO:teuthology.orchestra.run.smithi188.stdout:(126/131): postgresql-server-10.21-2.module+el8 26 MB/s | 5.1 MB 00:00 2023-11-16T18:13:05.261 INFO:teuthology.orchestra.run.smithi188.stdout:(127/131): postgresql-contrib-10.21-2.module+el 3.9 MB/s | 808 kB 00:00 2023-11-16T18:13:05.302 INFO:teuthology.orchestra.run.smithi188.stdout:(128/131): postgresql-10.21-2.module+el8.6.0+15 6.9 MB/s | 1.5 MB 00:00 2023-11-16T18:13:05.463 INFO:teuthology.orchestra.run.smithi040.stdout:(30/131): ceph-mgr-dashboard-18.0.0-3905.g5e4a7 3.5 MB/s | 1.7 MB 00:00 2023-11-16T18:13:05.502 INFO:teuthology.orchestra.run.smithi188.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 3.9 MB/s | 798 kB 00:00 2023-11-16T18:13:05.565 INFO:teuthology.orchestra.run.smithi040.stdout:(31/131): ceph-mgr-diskprediction-local-18.0.0- 13 MB/s | 7.4 MB 00:00 2023-11-16T18:13:05.589 INFO:teuthology.orchestra.run.smithi040.stdout:(32/131): ceph-mgr-modules-core-18.0.0-3905.g5e 1.9 MB/s | 240 kB 00:00 2023-11-16T18:13:05.623 INFO:teuthology.orchestra.run.smithi040.stdout:(33/131): ceph-mgr-rook-18.0.0-3905.g5e4a743a.e 810 kB/s | 47 kB 00:00 2023-11-16T18:13:05.660 INFO:teuthology.orchestra.run.smithi040.stdout:(34/131): ceph-prometheus-alerts-18.0.0-3905.g5 217 kB/s | 15 kB 00:00 2023-11-16T18:13:05.707 INFO:teuthology.orchestra.run.smithi040.stdout:(35/131): ceph-volume-18.0.0-3905.g5e4a743a.el8 3.1 MB/s | 265 kB 00:00 2023-11-16T18:13:05.739 INFO:teuthology.orchestra.run.smithi040.stdout:(36/131): cephadm-18.0.0-3905.g5e4a743a.el8.noa 1.0 MB/s | 83 kB 00:00 2023-11-16T18:13:05.790 INFO:teuthology.orchestra.run.smithi040.stdout:(37/131): cephfs-top-18.0.0-3905.g5e4a743a.el8. 215 kB/s | 18 kB 00:00 2023-11-16T18:13:06.028 INFO:teuthology.orchestra.run.smithi188.stdout:(130/131): librbd1-18.0.0-3905.g5e4a743a.el8.x8 4.6 MB/s | 3.6 MB 00:00 2023-11-16T18:13:06.142 INFO:teuthology.orchestra.run.smithi002.stdout:(8/131): ceph-radosgw-18.0.0-3905.g5e4a743a.el8 5.6 MB/s | 13 MB 00:02 2023-11-16T18:13:06.183 INFO:teuthology.orchestra.run.smithi002.stdout:(9/131): ceph-selinux-18.0.0-3905.g5e4a743a.el8 589 kB/s | 24 kB 00:00 2023-11-16T18:13:06.192 INFO:teuthology.orchestra.run.smithi040.stdout:(38/131): ceph-radosgw-18.0.0-3905.g5e4a743a.el 2.1 MB/s | 13 MB 00:06 2023-11-16T18:13:06.217 INFO:teuthology.orchestra.run.smithi040.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 138 kB/s | 59 kB 00:00 2023-11-16T18:13:06.219 INFO:teuthology.orchestra.run.smithi188.stdout:(131/131): librados2-18.0.0-3905.g5e4a743a.el8. 3.7 MB/s | 3.7 MB 00:00 2023-11-16T18:13:06.222 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-16T18:13:06.222 INFO:teuthology.orchestra.run.smithi188.stdout:Total 17 MB/s | 197 MB 00:11 2023-11-16T18:13:06.332 INFO:teuthology.orchestra.run.smithi040.stdout:(40/131): gperftools-libs-2.7-9.el8.x86_64.rpm 517 kB/s | 306 kB 00:00 2023-11-16T18:13:06.449 INFO:teuthology.orchestra.run.smithi040.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 374 kB/s | 44 kB 00:00 2023-11-16T18:13:06.567 INFO:teuthology.orchestra.run.smithi040.stdout:(42/131): libunwind-1.3.1-3.el8.x86_64.rpm 200 kB/s | 75 kB 00:00 2023-11-16T18:13:06.617 INFO:teuthology.orchestra.run.smithi040.stdout:(43/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.2 MB/s | 476 kB 00:00 2023-11-16T18:13:06.642 INFO:teuthology.orchestra.run.smithi040.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 964 kB/s | 185 kB 00:00 2023-11-16T18:13:06.709 INFO:teuthology.orchestra.run.smithi040.stdout:(45/131): python3-cachetools-3.1.1-4.el8.noarch 232 kB/s | 33 kB 00:00 2023-11-16T18:13:06.735 INFO:teuthology.orchestra.run.smithi040.stdout:(46/131): python3-certifi-2018.10.15-7.el8.noar 138 kB/s | 16 kB 00:00 2023-11-16T18:13:06.776 INFO:teuthology.orchestra.run.smithi040.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.3 MB/s | 173 kB 00:00 2023-11-16T18:13:06.851 INFO:teuthology.orchestra.run.smithi040.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 694 kB/s | 81 kB 00:00 2023-11-16T18:13:06.885 INFO:teuthology.orchestra.run.smithi040.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2023-11-16T18:13:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 140 kB/s | 18 kB 00:00 2023-11-16T18:13:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:(51/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.2 MB/s | 384 kB 00:00 2023-11-16T18:13:07.074 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-16T18:13:07.136 INFO:teuthology.orchestra.run.smithi040.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 307 kB/s | 49 kB 00:00 2023-11-16T18:13:07.195 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-16T18:13:07.196 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-16T18:13:07.246 INFO:teuthology.orchestra.run.smithi040.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 260 kB/s | 59 kB 00:00 2023-11-16T18:13:07.371 INFO:teuthology.orchestra.run.smithi040.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 258 kB/s | 60 kB 00:00 2023-11-16T18:13:07.378 INFO:teuthology.orchestra.run.smithi002.stdout:(10/131): ceph-common-18.0.0-3905.g5e4a743a.el8 2.7 MB/s | 22 MB 00:07 2023-11-16T18:13:07.407 INFO:teuthology.orchestra.run.smithi040.stdout:(55/131): python3-kubernetes-11.0.0-6.el8.noarc 2.2 MB/s | 1.2 MB 00:00 2023-11-16T18:13:07.462 INFO:teuthology.orchestra.run.smithi040.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.5 MB/s | 283 kB 00:00 2023-11-16T18:13:07.486 INFO:teuthology.orchestra.run.smithi002.stdout:(11/131): cephfs-mirror-18.0.0-3905.g5e4a743a.e 2.4 MB/s | 258 kB 00:00 2023-11-16T18:13:07.488 INFO:teuthology.orchestra.run.smithi040.stdout:(57/131): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2023-11-16T18:13:07.515 INFO:teuthology.orchestra.run.smithi040.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 308 kB/s | 34 kB 00:00 2023-11-16T18:13:07.543 INFO:teuthology.orchestra.run.smithi002.stdout:(12/131): libcephfs-devel-18.0.0-3905.g5e4a743a 533 kB/s | 31 kB 00:00 2023-11-16T18:13:07.580 INFO:teuthology.orchestra.run.smithi040.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.6 MB/s | 196 kB 00:00 2023-11-16T18:13:07.605 INFO:teuthology.orchestra.run.smithi040.stdout:(60/131): python3-rsa-4.9-2.el8.noarch.rpm 548 kB/s | 64 kB 00:00 2023-11-16T18:13:07.630 INFO:teuthology.orchestra.run.smithi040.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 160 kB/s | 19 kB 00:00 2023-11-16T18:13:07.685 INFO:teuthology.orchestra.run.smithi040.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 238 kB/s | 24 kB 00:00 2023-11-16T18:13:07.714 INFO:teuthology.orchestra.run.smithi040.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 269 kB/s | 29 kB 00:00 2023-11-16T18:13:07.718 INFO:teuthology.orchestra.run.smithi002.stdout:(13/131): libcephfs2-18.0.0-3905.g5e4a743a.el8. 4.3 MB/s | 771 kB 00:00 2023-11-16T18:13:07.739 INFO:teuthology.orchestra.run.smithi040.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 253 kB/s | 27 kB 00:00 2023-11-16T18:13:07.785 INFO:teuthology.orchestra.run.smithi002.stdout:(14/131): libcephsqlite-18.0.0-3905.g5e4a743a.e 2.0 MB/s | 137 kB 00:00 2023-11-16T18:13:07.810 INFO:teuthology.orchestra.run.smithi040.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 2.0 MB/s | 255 kB 00:00 2023-11-16T18:13:07.840 INFO:teuthology.orchestra.run.smithi040.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 610 kB/s | 61 kB 00:00 2023-11-16T18:13:07.844 INFO:teuthology.orchestra.run.smithi002.stdout:(15/131): librados-devel-18.0.0-3905.g5e4a743a. 2.3 MB/s | 139 kB 00:00 2023-11-16T18:13:07.865 INFO:teuthology.orchestra.run.smithi040.stdout:(67/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.6 MB/s | 251 kB 00:00 2023-11-16T18:13:07.921 INFO:teuthology.orchestra.run.smithi040.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 784 kB/s | 85 kB 00:00 2023-11-16T18:13:07.932 INFO:teuthology.orchestra.run.smithi040.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 4.1 MB/s | 70 kB 00:00 2023-11-16T18:13:07.949 INFO:teuthology.orchestra.run.smithi040.stdout:(70/131): python3-zc-lockfile-2.0-2.el8.noarch. 210 kB/s | 23 kB 00:00 2023-11-16T18:13:07.985 INFO:teuthology.orchestra.run.smithi002.stdout:(16/131): libradosstriper1-18.0.0-3905.g5e4a743 3.3 MB/s | 473 kB 00:00 2023-11-16T18:13:08.091 INFO:teuthology.orchestra.run.smithi040.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 777 kB/s | 110 kB 00:00 2023-11-16T18:13:08.175 INFO:teuthology.orchestra.run.smithi040.stdout:(72/131): thrift-0.13.0-2.el8.x86_64.rpm 5.6 MB/s | 1.7 MB 00:00 2023-11-16T18:13:08.280 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-16T18:13:08.287 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-16T18:13:08.288 INFO:teuthology.orchestra.run.smithi040.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 282 kB/s | 54 kB 00:00 2023-11-16T18:13:08.351 INFO:teuthology.orchestra.run.smithi040.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2023-11-16T18:13:08.454 INFO:teuthology.orchestra.run.smithi040.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.9 MB/s | 323 kB 00:00 2023-11-16T18:13:08.476 INFO:teuthology.orchestra.run.smithi040.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 310 kB/s | 39 kB 00:00 2023-11-16T18:13:08.585 INFO:teuthology.orchestra.run.smithi002.stdout:(17/131): ceph-osd-18.0.0-3905.g5e4a743a.el8.x8 3.3 MB/s | 18 MB 00:05 2023-11-16T18:13:08.635 INFO:teuthology.orchestra.run.smithi040.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 3.7 MB/s | 688 kB 00:00 2023-11-16T18:13:08.652 INFO:teuthology.orchestra.run.smithi002.stdout:(18/131): python3-ceph-argparse-18.0.0-3905.g5e 674 kB/s | 45 kB 00:00 2023-11-16T18:13:08.720 INFO:teuthology.orchestra.run.smithi002.stdout:(19/131): python3-ceph-common-18.0.0-3905.g5e4a 1.7 MB/s | 119 kB 00:00 2023-11-16T18:13:08.803 INFO:teuthology.orchestra.run.smithi040.stdout:(78/131): python3-scipy-1.0.0-21.module+el8.5.0 16 MB/s | 14 MB 00:00 2023-11-16T18:13:08.828 INFO:teuthology.orchestra.run.smithi002.stdout:(20/131): python3-cephfs-18.0.0-3905.g5e4a743a. 1.8 MB/s | 197 kB 00:00 2023-11-16T18:13:08.839 INFO:teuthology.orchestra.run.smithi040.stdout:(79/131): python3-jinja2-2.10.1-3.el8.noarch.rp 1.5 MB/s | 538 kB 00:00 2023-11-16T18:13:08.861 INFO:teuthology.orchestra.run.smithi040.stdout:(80/131): python3-mako-1.0.6-13.el8.noarch.rpm 696 kB/s | 157 kB 00:00 2023-11-16T18:13:08.927 INFO:teuthology.orchestra.run.smithi040.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2023-11-16T18:13:08.937 INFO:teuthology.orchestra.run.smithi002.stdout:(21/131): python3-rados-18.0.0-3905.g5e4a743a.e 3.3 MB/s | 371 kB 00:00 2023-11-16T18:13:08.994 INFO:teuthology.orchestra.run.smithi040.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 4.4 MB/s | 713 kB 00:00 2023-11-16T18:13:09.053 INFO:teuthology.orchestra.run.smithi002.stdout:(22/131): python3-rbd-18.0.0-3905.g5e4a743a.el8 3.1 MB/s | 365 kB 00:00 2023-11-16T18:13:09.065 INFO:teuthology.orchestra.run.smithi040.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 199 kB/s | 27 kB 00:00 2023-11-16T18:13:09.122 INFO:teuthology.orchestra.run.smithi002.stdout:(23/131): python3-rgw-18.0.0-3905.g5e4a743a.el8 1.6 MB/s | 107 kB 00:00 2023-11-16T18:13:09.178 INFO:teuthology.orchestra.run.smithi040.stdout:(84/131): python3-babel-2.5.1-7.el8.noarch.rpm 15 MB/s | 4.8 MB 00:00 2023-11-16T18:13:09.179 INFO:teuthology.orchestra.run.smithi002.stdout:(24/131): rbd-fuse-18.0.0-3905.g5e4a743a.el8.x8 1.6 MB/s | 94 kB 00:00 2023-11-16T18:13:09.204 INFO:teuthology.orchestra.run.smithi040.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 309 kB/s | 44 kB 00:00 2023-11-16T18:13:09.229 INFO:teuthology.orchestra.run.smithi040.stdout:(86/131): flex-2.6.1-9.el8.x86_64.rpm 1.3 MB/s | 320 kB 00:00 2023-11-16T18:13:09.305 INFO:teuthology.orchestra.run.smithi040.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 4.5 MB/s | 457 kB 00:00 2023-11-16T18:13:09.362 INFO:teuthology.orchestra.run.smithi002.stdout:(25/131): librgw2-18.0.0-3905.g5e4a743a.el8.x86 4.7 MB/s | 6.5 MB 00:01 2023-11-16T18:13:09.396 INFO:teuthology.orchestra.run.smithi040.stdout:(88/131): python3-numpy-1.14.3-10.el8.x86_64.rp 17 MB/s | 3.7 MB 00:00 2023-11-16T18:13:09.422 INFO:teuthology.orchestra.run.smithi040.stdout:(89/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.1 MB/s | 225 kB 00:00 2023-11-16T18:13:09.429 INFO:teuthology.orchestra.run.smithi002.stdout:(26/131): rbd-nbd-18.0.0-3905.g5e4a743a.el8.x86 2.6 MB/s | 180 kB 00:00 2023-11-16T18:13:09.447 INFO:teuthology.orchestra.run.smithi040.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 450 kB/s | 64 kB 00:00 2023-11-16T18:13:09.496 INFO:teuthology.orchestra.run.smithi002.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3905.g 346 kB/s | 23 kB 00:00 2023-11-16T18:13:09.550 INFO:teuthology.orchestra.run.smithi040.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 160 kB/s | 20 kB 00:00 2023-11-16T18:13:09.566 INFO:teuthology.orchestra.run.smithi002.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3905.g5e4a743 1.9 MB/s | 127 kB 00:00 2023-11-16T18:13:09.567 INFO:teuthology.orchestra.run.smithi040.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 885 kB/s | 103 kB 00:00 2023-11-16T18:13:09.589 INFO:teuthology.orchestra.run.smithi040.stdout:(93/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 656 kB/s | 126 kB 00:00 2023-11-16T18:13:09.740 INFO:teuthology.orchestra.run.smithi040.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 1.3 MB/s | 198 kB 00:00 2023-11-16T18:13:09.781 INFO:teuthology.orchestra.run.smithi040.stdout:(95/131): libtool-2.4.6-25.el8.x86_64.rpm 3.0 MB/s | 709 kB 00:00 2023-11-16T18:13:09.905 INFO:teuthology.orchestra.run.smithi002.stdout:(29/131): ceph-mgr-dashboard-18.0.0-3905.g5e4a7 5.0 MB/s | 1.7 MB 00:00 2023-11-16T18:13:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 14 MB/s | 4.9 MB 00:00 2023-11-16T18:13:09.957 INFO:teuthology.orchestra.run.smithi040.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.1 MB/s | 202 kB 00:00 2023-11-16T18:13:09.997 INFO:teuthology.orchestra.run.smithi002.stdout:(30/131): rbd-mirror-18.0.0-3905.g5e4a743a.el8. 4.2 MB/s | 3.4 MB 00:00 2023-11-16T18:13:10.084 INFO:teuthology.orchestra.run.smithi040.stdout:(98/131): openblas-0.3.15-3.el8.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2023-11-16T18:13:10.103 INFO:teuthology.orchestra.run.smithi002.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3905.g5e 2.4 MB/s | 240 kB 00:00 2023-11-16T18:13:10.116 INFO:teuthology.orchestra.run.smithi040.stdout:(99/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.7 MB/s | 354 kB 00:00 2023-11-16T18:13:10.233 INFO:teuthology.orchestra.run.smithi040.stdout:(100/131): postgresql-server-10.21-2.module+el8 18 MB/s | 5.1 MB 00:00 2023-11-16T18:13:10.266 INFO:teuthology.orchestra.run.smithi040.stdout:(101/131): postgresql-contrib-10.21-2.module+el 4.3 MB/s | 808 kB 00:00 2023-11-16T18:13:10.338 INFO:teuthology.orchestra.run.smithi040.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 6.9 MB/s | 1.5 MB 00:00 2023-11-16T18:13:10.384 INFO:teuthology.orchestra.run.smithi040.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.7 MB/s | 200 kB 00:00 2023-11-16T18:13:10.417 INFO:teuthology.orchestra.run.smithi040.stdout:(104/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 254 kB/s | 47 kB 00:00 2023-11-16T18:13:10.492 INFO:teuthology.orchestra.run.smithi040.stdout:(105/131): python3-jwt-1.6.1-2.el8.noarch.rpm 268 kB/s | 43 kB 00:00 2023-11-16T18:13:10.518 INFO:teuthology.orchestra.run.smithi040.stdout:(106/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2023-11-16T18:13:10.525 INFO:teuthology.orchestra.run.smithi002.stdout:(32/131): ceph-test-18.0.0-3905.g5e4a743a.el8.x 9.4 MB/s | 41 MB 00:04 2023-11-16T18:13:10.551 INFO:teuthology.orchestra.run.smithi040.stdout:(107/131): keyutils-libs-devel-1.5.10-9.el8.x86 359 kB/s | 48 kB 00:00 2023-11-16T18:13:10.559 INFO:teuthology.orchestra.run.smithi002.stdout:(33/131): ceph-mgr-rook-18.0.0-3905.g5e4a743a.e 103 kB/s | 47 kB 00:00 2023-11-16T18:13:10.582 INFO:teuthology.orchestra.run.smithi002.stdout:(34/131): ceph-prometheus-alerts-18.0.0-3905.g5 251 kB/s | 15 kB 00:00 2023-11-16T18:13:10.632 INFO:teuthology.orchestra.run.smithi002.stdout:(35/131): ceph-volume-18.0.0-3905.g5e4a743a.el8 3.4 MB/s | 265 kB 00:00 2023-11-16T18:13:10.637 INFO:teuthology.orchestra.run.smithi040.stdout:(108/131): libsepol-devel-2.9-3.el8.x86_64.rpm 612 kB/s | 87 kB 00:00 2023-11-16T18:13:10.658 INFO:teuthology.orchestra.run.smithi002.stdout:(36/131): cephadm-18.0.0-3905.g5e4a743a.el8.noa 1.1 MB/s | 83 kB 00:00 2023-11-16T18:13:10.683 INFO:teuthology.orchestra.run.smithi002.stdout:(37/131): cephfs-top-18.0.0-3905.g5e4a743a.el8. 358 kB/s | 18 kB 00:00 2023-11-16T18:13:10.685 INFO:teuthology.orchestra.run.smithi040.stdout:(109/131): python3-requests-oauthlib-1.0.0-1.el 255 kB/s | 43 kB 00:00 2023-11-16T18:13:10.710 INFO:teuthology.orchestra.run.smithi040.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 687 kB/s | 109 kB 00:00 2023-11-16T18:13:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.6 MB/s | 229 kB 00:00 2023-11-16T18:13:10.791 INFO:teuthology.orchestra.run.smithi002.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 589 kB/s | 59 kB 00:00 2023-11-16T18:13:10.825 INFO:teuthology.orchestra.run.smithi002.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2023-11-16T18:13:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:(112/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-11-16T18:13:10.850 INFO:teuthology.orchestra.run.smithi002.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-11-16T18:13:10.853 INFO:teuthology.orchestra.run.smithi040.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 131 kB/s | 19 kB 00:00 2023-11-16T18:13:10.884 INFO:teuthology.orchestra.run.smithi002.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 7.9 MB/s | 476 kB 00:00 2023-11-16T18:13:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:(114/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 2.7 MB/s | 560 kB 00:00 2023-11-16T18:13:10.910 INFO:teuthology.orchestra.run.smithi002.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 745 kB/s | 44 kB 00:00 2023-11-16T18:13:10.937 INFO:teuthology.orchestra.run.smithi002.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 3.6 MB/s | 185 kB 00:00 2023-11-16T18:13:10.964 INFO:teuthology.orchestra.run.smithi002.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 658 kB/s | 33 kB 00:00 2023-11-16T18:13:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 4.1 MB/s | 559 kB 00:00 2023-11-16T18:13:10.985 INFO:teuthology.orchestra.run.smithi002.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 325 kB/s | 16 kB 00:00 2023-11-16T18:13:10.988 INFO:teuthology.orchestra.run.smithi040.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 133 kB/s | 18 kB 00:00 2023-11-16T18:13:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2023-11-16T18:13:11.022 INFO:teuthology.orchestra.run.smithi002.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2023-11-16T18:13:11.044 INFO:teuthology.orchestra.run.smithi002.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 6.4 MB/s | 384 kB 00:00 2023-11-16T18:13:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:(118/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.7 MB/s | 605 kB 00:00 2023-11-16T18:13:11.069 INFO:teuthology.orchestra.run.smithi002.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2023-11-16T18:13:11.094 INFO:teuthology.orchestra.run.smithi002.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-11-16T18:13:11.123 INFO:teuthology.orchestra.run.smithi040.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 17 MB/s | 2.3 MB 00:00 2023-11-16T18:13:11.123 INFO:teuthology.orchestra.run.smithi002.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 351 kB/s | 18 kB 00:00 2023-11-16T18:13:11.147 INFO:teuthology.orchestra.run.smithi040.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 825 kB/s | 83 kB 00:00 2023-11-16T18:13:11.162 INFO:teuthology.orchestra.run.smithi002.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 1.1 MB/s | 49 kB 00:00 2023-11-16T18:13:11.172 INFO:teuthology.orchestra.run.smithi040.stdout:(121/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 480 kB/s | 72 kB 00:00 2023-11-16T18:13:11.220 INFO:teuthology.orchestra.run.smithi002.stdout:(52/131): python3-kubernetes-11.0.0-6.el8.noarc 9.3 MB/s | 1.2 MB 00:00 2023-11-16T18:13:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:(122/131): elfutils-libelf-devel-0.186-1.el8.x8 478 kB/s | 60 kB 00:00 2023-11-16T18:13:11.249 INFO:teuthology.orchestra.run.smithi002.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 705 kB/s | 59 kB 00:00 2023-11-16T18:13:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 782 kB/s | 98 kB 00:00 2023-11-16T18:13:11.287 INFO:teuthology.orchestra.run.smithi002.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.4 MB/s | 60 kB 00:00 2023-11-16T18:13:11.298 INFO:teuthology.orchestra.run.smithi040.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 459 kB/s | 58 kB 00:00 2023-11-16T18:13:11.313 INFO:teuthology.orchestra.run.smithi002.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 4.1 MB/s | 283 kB 00:00 2023-11-16T18:13:11.399 INFO:teuthology.orchestra.run.smithi040.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 256 kB/s | 39 kB 00:00 2023-11-16T18:13:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.1 MB/s | 171 kB 00:00 2023-11-16T18:13:11.461 INFO:teuthology.orchestra.run.smithi002.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 90 kB/s | 16 kB 00:00 2023-11-16T18:13:11.482 INFO:teuthology.orchestra.run.smithi040.stdout:(127/131): NetworkManager-initscripts-updown-1. 1.6 MB/s | 138 kB 00:00 2023-11-16T18:13:11.505 INFO:teuthology.orchestra.run.smithi002.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 175 kB/s | 34 kB 00:00 2023-11-16T18:13:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:(128/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 2.6 MB/s | 644 kB 00:00 2023-11-16T18:13:11.531 INFO:teuthology.orchestra.run.smithi002.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 2.8 MB/s | 196 kB 00:00 2023-11-16T18:13:11.556 INFO:teuthology.orchestra.run.smithi002.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2023-11-16T18:13:11.573 INFO:teuthology.orchestra.run.smithi002.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 447 kB/s | 19 kB 00:00 2023-11-16T18:13:11.590 INFO:teuthology.orchestra.run.smithi002.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 708 kB/s | 24 kB 00:00 2023-11-16T18:13:11.611 INFO:teuthology.orchestra.run.smithi002.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 866 kB/s | 29 kB 00:00 2023-11-16T18:13:11.625 INFO:teuthology.orchestra.run.smithi002.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 820 kB/s | 27 kB 00:00 2023-11-16T18:13:11.649 INFO:teuthology.orchestra.run.smithi002.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 6.0 MB/s | 255 kB 00:00 2023-11-16T18:13:11.666 INFO:teuthology.orchestra.run.smithi040.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 5.2 MB/s | 798 kB 00:00 2023-11-16T18:13:11.685 INFO:teuthology.orchestra.run.smithi002.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 4.2 MB/s | 251 kB 00:00 2023-11-16T18:13:11.708 INFO:teuthology.orchestra.run.smithi002.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 1.0 MB/s | 61 kB 00:00 2023-11-16T18:13:11.734 INFO:teuthology.orchestra.run.smithi002.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2023-11-16T18:13:11.825 INFO:teuthology.orchestra.run.smithi002.stdout:(68/131): ceph-mgr-diskprediction-local-18.0.0- 3.9 MB/s | 7.4 MB 00:01 2023-11-16T18:13:11.851 INFO:teuthology.orchestra.run.smithi002.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 161 kB/s | 23 kB 00:00 2023-11-16T18:13:11.876 INFO:teuthology.orchestra.run.smithi002.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-11-16T18:13:11.951 INFO:teuthology.orchestra.run.smithi002.stdout:(71/131): thrift-0.13.0-2.el8.x86_64.rpm 8.0 MB/s | 1.7 MB 00:00 2023-11-16T18:13:11.994 INFO:teuthology.orchestra.run.smithi002.stdout:(72/131): python3-pyasn1-modules-0.3.7-6.el8.no 943 kB/s | 110 kB 00:00 2023-11-16T18:13:12.083 INFO:teuthology.orchestra.run.smithi040.stdout:(130/131): librbd1-18.0.0-3905.g5e4a743a.el8.x8 5.9 MB/s | 3.6 MB 00:00 2023-11-16T18:13:12.121 INFO:teuthology.orchestra.run.smithi002.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 325 kB/s | 54 kB 00:00 2023-11-16T18:13:12.152 INFO:teuthology.orchestra.run.smithi002.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.7 MB/s | 259 kB 00:00 2023-11-16T18:13:12.203 INFO:teuthology.orchestra.run.smithi002.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2023-11-16T18:13:12.286 INFO:teuthology.orchestra.run.smithi002.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 290 kB/s | 39 kB 00:00 2023-11-16T18:13:12.336 INFO:teuthology.orchestra.run.smithi002.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 5.0 MB/s | 688 kB 00:00 2023-11-16T18:13:12.383 INFO:teuthology.orchestra.run.smithi040.stdout:(131/131): librados2-18.0.0-3905.g5e4a743a.el8. 3.9 MB/s | 3.7 MB 00:00 2023-11-16T18:13:12.386 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-11-16T18:13:12.386 INFO:teuthology.orchestra.run.smithi040.stdout:Total 13 MB/s | 197 MB 00:15 2023-11-16T18:13:12.537 INFO:teuthology.orchestra.run.smithi002.stdout:(78/131): python3-scipy-1.0.0-21.module+el8.5.0 20 MB/s | 14 MB 00:00 2023-11-16T18:13:12.562 INFO:teuthology.orchestra.run.smithi002.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 696 kB/s | 157 kB 00:00 2023-11-16T18:13:12.595 INFO:teuthology.orchestra.run.smithi002.stdout:(80/131): python3-jinja2-2.10.1-3.el8.noarch.rp 1.7 MB/s | 538 kB 00:00 2023-11-16T18:13:12.695 INFO:teuthology.orchestra.run.smithi002.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-11-16T18:13:12.737 INFO:teuthology.orchestra.run.smithi002.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 4.0 MB/s | 713 kB 00:00 2023-11-16T18:13:12.871 INFO:teuthology.orchestra.run.smithi002.stdout:(83/131): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2023-11-16T18:13:12.896 INFO:teuthology.orchestra.run.smithi002.stdout:(84/131): python3-jsonpatch-1.21-2.el8.noarch.r 132 kB/s | 27 kB 00:00 2023-11-16T18:13:12.926 INFO:teuthology.orchestra.run.smithi002.stdout:(85/131): flex-2.6.1-9.el8.x86_64.rpm 1.7 MB/s | 320 kB 00:00 2023-11-16T18:13:12.991 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-16T18:13:12.991 INFO:teuthology.orchestra.run.smithi002.stdout:(86/131): python3-prettytable-0.7.2-14.el8.noar 376 kB/s | 44 kB 00:00 2023-11-16T18:13:13.097 INFO:teuthology.orchestra.run.smithi002.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 19 MB/s | 3.7 MB 00:00 2023-11-16T18:13:13.130 INFO:teuthology.orchestra.run.smithi002.stdout:(88/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.1 MB/s | 457 kB 00:00 2023-11-16T18:13:13.157 INFO:teuthology.orchestra.run.smithi002.stdout:(89/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.3 MB/s | 225 kB 00:00 2023-11-16T18:13:13.158 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-16T18:13:13.224 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-11-16T18:13:13.240 INFO:teuthology.orchestra.run.smithi002.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 450 kB/s | 64 kB 00:00 2023-11-16T18:13:13.307 INFO:teuthology.orchestra.run.smithi002.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 133 kB/s | 20 kB 00:00 2023-11-16T18:13:13.326 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-16T18:13:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:(92/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 629 kB/s | 126 kB 00:00 2023-11-16T18:13:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 882 kB/s | 103 kB 00:00 2023-11-16T18:13:13.358 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-11-16T18:13:13.359 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-11-16T18:13:13.468 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-16T18:13:13.482 INFO:teuthology.orchestra.run.smithi002.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 3.9 MB/s | 709 kB 00:00 2023-11-16T18:13:13.516 INFO:teuthology.orchestra.run.smithi002.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.2 MB/s | 198 kB 00:00 2023-11-16T18:13:13.573 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-16T18:13:13.626 INFO:teuthology.orchestra.run.smithi002.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 17 MB/s | 4.9 MB 00:00 2023-11-16T18:13:13.650 INFO:teuthology.orchestra.run.smithi002.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.5 MB/s | 202 kB 00:00 2023-11-16T18:13:13.758 INFO:teuthology.orchestra.run.smithi002.stdout:(98/131): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-11-16T18:13:13.792 INFO:teuthology.orchestra.run.smithi002.stdout:(99/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.1 MB/s | 354 kB 00:00 2023-11-16T18:13:13.809 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-16T18:13:13.909 INFO:teuthology.orchestra.run.smithi002.stdout:(100/131): postgresql-server-10.21-2.module+el8 20 MB/s | 5.1 MB 00:00 2023-11-16T18:13:13.946 INFO:teuthology.orchestra.run.smithi002.stdout:(101/131): postgresql-contrib-10.21-2.module+el 4.3 MB/s | 808 kB 00:00 2023-11-16T18:13:13.993 INFO:teuthology.orchestra.run.smithi002.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 7.5 MB/s | 1.5 MB 00:00 2023-11-16T18:13:14.051 INFO:teuthology.orchestra.run.smithi002.stdout:(103/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 329 kB/s | 47 kB 00:00 2023-11-16T18:13:14.085 INFO:teuthology.orchestra.run.smithi002.stdout:(104/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.4 MB/s | 200 kB 00:00 2023-11-16T18:13:14.146 INFO:teuthology.orchestra.run.smithi002.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.3 MB/s | 220 kB 00:00 2023-11-16T18:13:14.208 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-16T18:13:14.214 INFO:teuthology.orchestra.run.smithi002.stdout:(106/131): python3-jwt-1.6.1-2.el8.noarch.rpm 196 kB/s | 43 kB 00:00 2023-11-16T18:13:14.228 INFO:teuthology.orchestra.run.smithi002.stdout:(107/131): keyutils-libs-devel-1.5.10-9.el8.x86 338 kB/s | 48 kB 00:00 2023-11-16T18:13:14.278 INFO:teuthology.orchestra.run.smithi002.stdout:(108/131): libsepol-devel-2.9-3.el8.x86_64.rpm 650 kB/s | 87 kB 00:00 2023-11-16T18:13:14.333 INFO:teuthology.orchestra.run.smithi002.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 1.1 MB/s | 109 kB 00:00 2023-11-16T18:13:14.345 INFO:teuthology.orchestra.run.smithi002.stdout:(110/131): python3-requests-oauthlib-1.0.0-1.el 318 kB/s | 43 kB 00:00 2023-11-16T18:13:14.412 INFO:teuthology.orchestra.run.smithi002.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.7 MB/s | 229 kB 00:00 2023-11-16T18:13:14.419 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 5/134 2023-11-16T18:13:14.454 INFO:teuthology.orchestra.run.smithi002.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.4 MB/s | 560 kB 00:00 2023-11-16T18:13:14.496 INFO:teuthology.orchestra.run.smithi002.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 124 kB/s | 19 kB 00:00 2023-11-16T18:13:14.503 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-11-16T18:13:14.508 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-11-16T18:13:14.558 INFO:teuthology.orchestra.run.smithi002.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.1 MB/s | 155 kB 00:00 2023-11-16T18:13:14.627 INFO:teuthology.orchestra.run.smithi002.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 3.3 MB/s | 559 kB 00:00 2023-11-16T18:13:14.647 INFO:teuthology.orchestra.run.smithi002.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 119 kB/s | 18 kB 00:00 2023-11-16T18:13:14.725 INFO:teuthology.orchestra.run.smithi002.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.5 MB/s | 605 kB 00:00 2023-11-16T18:13:14.803 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 5/134 2023-11-16T18:13:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:(118/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 15 MB/s | 2.3 MB 00:00 2023-11-16T18:13:14.835 INFO:teuthology.orchestra.run.smithi002.stdout:(119/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.1 MB/s | 238 kB 00:00 2023-11-16T18:13:14.856 INFO:teuthology.orchestra.run.smithi002.stdout:(120/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 540 kB/s | 72 kB 00:00 2023-11-16T18:13:14.906 INFO:teuthology.orchestra.run.smithi002.stdout:(121/131): fuse-2.9.7-15.el8.x86_64.rpm 824 kB/s | 83 kB 00:00 2023-11-16T18:13:14.940 INFO:teuthology.orchestra.run.smithi002.stdout:(122/131): elfutils-libelf-devel-0.186-1.el8.x8 548 kB/s | 60 kB 00:00 2023-11-16T18:13:14.966 INFO:teuthology.orchestra.run.smithi002.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 900 kB/s | 98 kB 00:00 2023-11-16T18:13:14.967 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T18:13:15.049 INFO:teuthology.orchestra.run.smithi002.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 355 kB/s | 39 kB 00:00 2023-11-16T18:13:15.075 INFO:teuthology.orchestra.run.smithi002.stdout:(125/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 344 kB/s | 58 kB 00:00 2023-11-16T18:13:15.078 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T18:13:15.145 INFO:teuthology.orchestra.run.smithi002.stdout:(126/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 3.6 MB/s | 644 kB 00:00 2023-11-16T18:13:15.191 INFO:teuthology.orchestra.run.smithi002.stdout:(127/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.2 MB/s | 171 kB 00:00 2023-11-16T18:13:15.225 INFO:teuthology.orchestra.run.smithi002.stdout:(128/131): NetworkManager-initscripts-updown-1. 915 kB/s | 138 kB 00:00 2023-11-16T18:13:15.259 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T18:13:15.345 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T18:13:15.412 INFO:teuthology.orchestra.run.smithi002.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.3 MB/s | 798 kB 00:00 2023-11-16T18:13:15.506 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T18:13:15.568 INFO:teuthology.orchestra.run.smithi002.stdout:(130/131): librados2-18.0.0-3905.g5e4a743a.el8. 8.6 MB/s | 3.7 MB 00:00 2023-11-16T18:13:15.694 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-11-16T18:13:15.921 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-16T18:13:16.000 INFO:teuthology.orchestra.run.smithi002.stdout:(131/131): librbd1-18.0.0-3905.g5e4a743a.el8.x8 4.4 MB/s | 3.6 MB 00:00 2023-11-16T18:13:16.007 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-11-16T18:13:16.007 INFO:teuthology.orchestra.run.smithi002.stdout:Total 12 MB/s | 197 MB 00:16 2023-11-16T18:13:16.137 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-16T18:13:16.530 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-16T18:13:16.771 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-16T18:13:16.845 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-11-16T18:13:16.909 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T18:13:16.977 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-11-16T18:13:16.977 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-11-16T18:13:17.002 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T18:13:17.170 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T18:13:17.318 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T18:13:17.521 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-11-16T18:13:17.710 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-11-16T18:13:17.885 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-11-16T18:13:18.025 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-16T18:13:18.122 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-11-16T18:13:18.134 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-11-16T18:13:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-16T18:13:18.359 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-16T18:13:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-16T18:13:19.125 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-16T18:13:19.318 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-11-16T18:13:19.503 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-16T18:13:19.653 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-16T18:13:19.721 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-16T18:13:19.855 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-11-16T18:13:19.908 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-11-16T18:13:20.029 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T18:13:20.145 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T18:13:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T18:13:20.598 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-11-16T18:13:20.843 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-11-16T18:13:20.939 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T18:13:21.022 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-11-16T18:13:21.121 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T18:13:21.275 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-16T18:13:21.447 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 26/134 2023-11-16T18:13:21.523 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T18:13:21.572 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 27/134 2023-11-16T18:13:21.729 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T18:13:21.778 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 28/134 2023-11-16T18:13:21.854 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T18:13:21.977 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-16T18:13:22.006 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T18:13:22.107 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T18:13:22.266 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T18:13:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-16T18:13:22.699 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-16T18:13:22.933 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-11-16T18:13:23.085 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-16T18:13:23.088 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-16T18:13:23.268 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-16T18:13:23.335 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-16T18:13:23.445 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-11-16T18:13:23.462 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T18:13:23.563 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T18:13:23.598 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T18:13:23.691 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T18:13:23.738 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-16T18:13:23.854 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T18:13:23.920 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T18:13:23.957 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T18:13:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T18:13:24.255 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T18:13:24.379 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T18:13:24.405 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T18:13:24.542 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T18:13:24.834 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-11-16T18:13:24.964 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T18:13:25.059 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T18:13:25.114 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T18:13:25.154 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-11-16T18:13:25.237 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T18:13:25.247 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T18:13:25.418 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T18:13:25.458 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-11-16T18:13:25.516 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T18:13:25.675 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T18:13:25.699 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-11-16T18:13:25.871 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T18:13:25.884 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-16T18:13:25.913 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-16T18:13:26.077 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-16T18:13:26.083 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-16T18:13:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T18:13:26.271 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 36/134 2023-11-16T18:13:26.438 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 37/134 2023-11-16T18:13:26.457 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-16T18:13:26.592 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-11-16T18:13:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-16T18:13:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-11-16T18:13:26.798 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T18:13:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T18:13:26.905 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T18:13:26.923 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-11-16T18:13:26.999 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-11-16T18:13:27.031 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T18:13:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-11-16T18:13:27.112 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T18:13:27.222 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T18:13:27.229 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-11-16T18:13:27.286 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-11-16T18:13:27.363 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T18:13:27.460 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-11-16T18:13:27.514 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T18:13:27.535 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-11-16T18:13:27.606 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-16T18:13:27.670 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-16T18:13:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-11-16T18:13:27.869 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-11-16T18:13:27.941 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-11-16T18:13:28.026 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-11-16T18:13:28.174 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T18:13:28.177 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-16T18:13:28.193 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-11-16T18:13:28.339 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T18:13:28.349 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-11-16T18:13:28.360 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 26/134 2023-11-16T18:13:28.550 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-11-16T18:13:28.616 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 27/134 2023-11-16T18:13:28.710 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-11-16T18:13:28.912 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-11-16T18:13:28.981 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T18:13:29.111 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-11-16T18:13:29.164 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T18:13:29.465 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-11-16T18:13:29.565 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 28/134 2023-11-16T18:13:29.749 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-11-16T18:13:29.866 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-11-16T18:13:29.893 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T18:13:29.950 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-11-16T18:13:30.113 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-11-16T18:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-11-16T18:13:30.201 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-11-16T18:13:30.331 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-11-16T18:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-11-16T18:13:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-11-16T18:13:30.535 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-11-16T18:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T18:13:30.712 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-11-16T18:13:30.757 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-11-16T18:13:30.916 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-11-16T18:13:30.963 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-11-16T18:13:31.131 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-11-16T18:13:31.162 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-16T18:13:31.331 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 26/134 2023-11-16T18:13:31.359 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-11-16T18:13:31.467 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 27/134 2023-11-16T18:13:31.534 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-11-16T18:13:31.725 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-11-16T18:13:31.993 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-11-16T18:13:32.243 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-11-16T18:13:32.395 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 28/134 2023-11-16T18:13:32.451 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T18:13:32.497 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-11-16T18:13:32.709 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-11-16T18:13:32.737 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-11-16T18:13:32.750 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-11-16T18:13:32.856 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-16T18:13:32.959 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-16T18:13:32.961 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-11-16T18:13:33.015 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-16T18:13:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-16T18:13:33.212 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-11-16T18:13:33.263 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-16T18:13:33.384 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 36/134 2023-11-16T18:13:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T18:13:33.462 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-16T18:13:33.548 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 37/134 2023-11-16T18:13:33.586 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-16T18:13:33.686 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-11-16T18:13:33.724 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-16T18:13:33.844 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-11-16T18:13:33.929 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-16T18:13:34.030 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-11-16T18:13:34.141 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-16T18:13:34.209 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-11-16T18:13:34.323 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-16T18:13:34.499 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-16T18:13:34.646 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-16T18:13:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-11-16T18:13:34.874 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-16T18:13:34.900 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-11-16T18:13:35.097 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-16T18:13:35.261 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-16T18:13:35.282 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T18:13:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-11-16T18:13:35.493 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-16T18:13:35.550 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-11-16T18:13:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-11-16T18:13:35.687 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-16T18:13:35.722 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-16T18:13:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-11-16T18:13:35.847 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-16T18:13:35.914 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-16T18:13:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-11-16T18:13:36.064 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-16T18:13:36.103 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 36/134 2023-11-16T18:13:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-11-16T18:13:36.267 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 37/134 2023-11-16T18:13:36.404 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-11-16T18:13:36.487 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-11-16T18:13:36.591 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-11-16T18:13:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-11-16T18:13:36.759 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-11-16T18:13:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-11-16T18:13:36.895 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-11-16T18:13:37.076 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-11-16T18:13:37.082 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-11-16T18:13:37.182 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-16T18:13:37.258 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-11-16T18:13:37.406 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-11-16T18:13:37.438 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-16T18:13:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-11-16T18:13:37.653 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-16T18:13:37.736 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-11-16T18:13:37.769 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-11-16T18:13:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-16T18:13:37.988 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-11-16T18:13:38.171 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-11-16T18:13:38.219 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-11-16T18:13:38.228 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-16T18:13:38.349 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-11-16T18:13:38.413 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-16T18:13:38.417 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-11-16T18:13:38.558 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-11-16T18:13:38.584 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-11-16T18:13:38.737 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-11-16T18:13:38.787 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-11-16T18:13:38.910 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T18:13:38.984 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T18:13:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-11-16T18:13:39.183 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-16T18:13:39.251 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-11-16T18:13:39.312 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-11-16T18:13:39.370 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T18:13:39.443 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-11-16T18:13:39.444 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T18:13:39.473 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-11-16T18:13:39.572 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T18:13:39.595 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 93/134 2023-11-16T18:13:39.638 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T18:13:39.663 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-11-16T18:13:39.812 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-11-16T18:13:39.860 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-11-16T18:13:39.960 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-11-16T18:13:40.063 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-11-16T18:13:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-16T18:13:40.272 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-11-16T18:13:40.387 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-16T18:13:40.518 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-11-16T18:13:40.733 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-16T18:13:40.759 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-11-16T18:13:40.936 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-16T18:13:40.945 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-11-16T18:13:41.065 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-16T18:13:41.106 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-11-16T18:13:41.197 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-16T18:13:41.248 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-11-16T18:13:41.392 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-16T18:13:41.404 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T18:13:41.475 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T18:13:41.633 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-16T18:13:41.714 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-11-16T18:13:41.815 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-16T18:13:41.884 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-11-16T18:13:41.980 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-16T18:13:42.029 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T18:13:42.114 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T18:13:42.127 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-16T18:13:42.331 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-11-16T18:13:42.350 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-16T18:13:42.472 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-11-16T18:13:42.605 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-16T18:13:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-16T18:13:42.763 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-16T18:13:42.766 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-16T18:13:42.990 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-16T18:13:43.003 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-16T18:13:43.168 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-16T18:13:43.204 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T18:13:43.214 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-16T18:13:43.333 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-16T18:13:43.363 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-16T18:13:43.422 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T18:13:43.458 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-16T18:13:43.558 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-16T18:13:43.670 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-16T18:13:43.852 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-16T18:13:44.014 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T18:13:44.017 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-16T18:13:44.174 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-16T18:13:44.310 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-16T18:13:44.403 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T18:13:44.668 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-16T18:13:44.671 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T18:13:44.719 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-16T18:13:44.836 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T18:13:44.858 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-16T18:13:45.026 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-16T18:13:45.035 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-16T18:13:45.257 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-16T18:13:45.287 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-16T18:13:45.436 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-16T18:13:45.579 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-16T18:13:45.580 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-16T18:13:45.791 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-16T18:13:45.873 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-16T18:13:46.057 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-16T18:13:46.858 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-16T18:13:46.938 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-16T18:13:47.013 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T18:13:47.083 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T18:13:47.176 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-16T18:13:47.253 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 93/134 2023-11-16T18:13:47.387 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-16T18:13:47.695 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-16T18:13:47.969 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-16T18:13:48.151 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-16T18:13:48.975 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-16T18:13:49.131 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T18:13:49.207 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T18:13:49.330 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 93/134 2023-11-16T18:13:51.139 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T18:13:51.317 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T18:13:51.952 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T18:13:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T18:13:52.280 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T18:13:52.426 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T18:13:53.143 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T18:13:53.586 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T18:13:54.217 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T18:13:54.727 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T18:13:54.810 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T18:13:54.936 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T18:14:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T18:14:03.621 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2023-11-16T18:14:03.621 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2023-11-16T18:14:03.621 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2023-11-16T18:14:03.622 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /run 2023-11-16T18:14:03.622 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2023-11-16T18:14:03.622 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2023-11-16T18:14:03.622 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2023-11-16T18:14:03.622 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2023-11-16T18:14:03.622 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2023-11-16T18:14:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:14:03.990 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T18:14:05.550 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T18:14:05.714 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T18:14:05.933 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T18:14:06.098 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T18:14:06.548 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T18:14:06.714 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T18:14:06.733 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T18:14:08.381 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T18:14:08.600 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T18:14:08.738 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 102/134 2023-11-16T18:14:08.954 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 103/134 2023-11-16T18:14:09.148 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T18:14:09.362 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T18:14:09.647 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T18:14:10.610 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T18:14:10.788 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T18:14:10.879 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T18:14:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 107/134 2023-11-16T18:14:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T18:14:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T18:14:11.688 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T18:14:11.700 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T18:14:11.929 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T18:14:11.929 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2023-11-16T18:14:11.930 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2023-11-16T18:14:11.930 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /dev 2023-11-16T18:14:11.930 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /run 2023-11-16T18:14:11.930 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2023-11-16T18:14:11.930 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2023-11-16T18:14:11.930 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2023-11-16T18:14:11.930 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2023-11-16T18:14:11.930 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /dev 2023-11-16T18:14:11.930 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-11-16T18:14:12.272 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T18:14:12.970 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 110/134 2023-11-16T18:14:13.147 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T18:14:13.180 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T18:14:13.379 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T18:14:13.432 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T18:14:13.588 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T18:14:13.932 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T18:14:13.933 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T18:14:14.099 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T18:14:14.105 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T18:14:14.332 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T18:14:14.480 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T18:14:14.759 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T18:14:14.760 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /sys 2023-11-16T18:14:14.760 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /proc 2023-11-16T18:14:14.760 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /dev 2023-11-16T18:14:14.760 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /run 2023-11-16T18:14:14.760 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /mnt 2023-11-16T18:14:14.760 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /var/tmp 2023-11-16T18:14:14.760 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /home 2023-11-16T18:14:14.760 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /tmp 2023-11-16T18:14:14.760 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /dev 2023-11-16T18:14:14.760 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:14:14.958 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T18:14:15.107 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T18:14:15.118 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T18:14:15.132 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T18:14:16.773 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T18:14:16.915 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T18:14:17.085 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T18:14:17.145 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T18:14:17.276 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T18:14:17.299 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T18:14:17.395 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 102/134 2023-11-16T18:14:17.628 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 103/134 2023-11-16T18:14:17.778 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T18:14:17.805 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T18:14:17.907 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T18:14:17.925 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T18:14:18.017 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T18:14:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T18:14:18.259 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T18:14:18.922 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 115/134 2023-11-16T18:14:19.225 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-11-16T18:14:19.282 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T18:14:19.423 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T18:14:19.432 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 117/134 2023-11-16T18:14:19.519 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T18:14:19.719 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-11-16T18:14:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T18:14:19.855 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 107/134 2023-11-16T18:14:19.871 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-11-16T18:14:19.928 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T18:14:20.030 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 102/134 2023-11-16T18:14:20.074 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-11-16T18:14:20.104 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T18:14:20.142 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T18:14:20.225 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 103/134 2023-11-16T18:14:20.271 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T18:14:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T18:14:20.307 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-11-16T18:14:20.381 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T18:14:20.481 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 121/134 2023-11-16T18:14:20.607 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T18:14:20.630 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T18:14:20.888 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T18:14:21.626 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 110/134 2023-11-16T18:14:21.838 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T18:14:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T18:14:21.917 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T18:14:22.016 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T18:14:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T18:14:22.064 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T18:14:22.160 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T18:14:22.221 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T18:14:22.522 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 107/134 2023-11-16T18:14:22.586 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T18:14:22.744 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T18:14:22.746 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T18:14:22.768 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T18:14:22.780 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T18:14:22.964 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T18:14:22.978 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T18:14:23.052 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T18:14:23.070 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T18:14:23.228 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 124/134 2023-11-16T18:14:23.510 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 125/134 2023-11-16T18:14:23.804 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 126/134 2023-11-16T18:14:24.126 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-16T18:14:24.306 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 110/134 2023-11-16T18:14:24.310 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-11-16T18:14:24.399 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-11-16T18:14:24.484 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T18:14:24.519 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T18:14:24.626 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-11-16T18:14:24.639 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-11-16T18:14:24.665 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T18:14:24.711 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T18:14:24.805 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-16T18:14:24.813 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-16T18:14:24.839 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T18:14:24.929 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-11-16T18:14:24.930 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T18:14:25.104 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T18:14:25.105 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T18:14:25.206 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T18:14:25.340 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T18:14:25.439 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T18:14:25.533 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T18:14:26.116 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T18:14:26.117 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 1/134 2023-11-16T18:14:26.117 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2/134 2023-11-16T18:14:26.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 3/134 2023-11-16T18:14:26.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 4/134 2023-11-16T18:14:26.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 5/134 2023-11-16T18:14:26.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T18:14:26.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T18:14:26.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T18:14:26.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T18:14:26.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 10/134 2023-11-16T18:14:26.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 11/134 2023-11-16T18:14:26.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 12/134 2023-11-16T18:14:26.151 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 13/134 2023-11-16T18:14:26.157 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 14/134 2023-11-16T18:14:26.157 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 15/134 2023-11-16T18:14:26.162 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 16/134 2023-11-16T18:14:26.162 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 17/134 2023-11-16T18:14:26.162 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 18/134 2023-11-16T18:14:26.163 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 19/134 2023-11-16T18:14:26.163 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 20/134 2023-11-16T18:14:26.163 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 21/134 2023-11-16T18:14:26.163 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 22/134 2023-11-16T18:14:26.163 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 23/134 2023-11-16T18:14:26.163 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 24/134 2023-11-16T18:14:26.163 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 25/134 2023-11-16T18:14:26.163 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 26/134 2023-11-16T18:14:26.163 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 27/134 2023-11-16T18:14:26.163 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 28/134 2023-11-16T18:14:26.163 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 29/134 2023-11-16T18:14:26.163 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 30/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 31/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 32/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 33/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 34/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 35/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 36/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 37/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 38/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-16T18:14:26.164 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-16T18:14:26.165 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-16T18:14:26.165 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-16T18:14:26.165 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-16T18:14:26.165 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-16T18:14:26.165 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-16T18:14:26.165 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-16T18:14:26.165 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-16T18:14:26.182 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-16T18:14:26.182 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-16T18:14:26.182 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-16T18:14:26.182 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-16T18:14:26.182 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-16T18:14:26.182 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-16T18:14:26.182 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-16T18:14:26.182 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-16T18:14:26.183 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-16T18:14:26.183 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-16T18:14:26.183 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-16T18:14:26.183 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-16T18:14:26.183 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-16T18:14:26.183 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-16T18:14:26.183 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-16T18:14:26.183 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-16T18:14:26.187 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-16T18:14:26.188 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-16T18:14:26.189 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-16T18:14:26.190 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-16T18:14:26.190 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-11-16T18:14:26.191 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-11-16T18:14:26.193 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-11-16T18:14:26.193 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-11-16T18:14:26.193 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-11-16T18:14:26.193 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-11-16T18:14:26.193 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-11-16T18:14:26.193 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-11-16T18:14:26.194 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-11-16T18:14:26.194 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-11-16T18:14:26.194 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-11-16T18:14:26.194 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-11-16T18:14:26.194 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-11-16T18:14:26.194 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-11-16T18:14:26.194 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-11-16T18:14:26.194 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-11-16T18:14:26.194 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-11-16T18:14:26.194 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-11-16T18:14:26.194 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-11-16T18:14:26.195 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-11-16T18:14:26.195 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-11-16T18:14:26.195 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-11-16T18:14:26.195 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-11-16T18:14:26.195 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-11-16T18:14:26.195 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-11-16T18:14:26.195 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-11-16T18:14:26.195 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-11-16T18:14:26.195 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-11-16T18:14:26.195 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-11-16T18:14:26.195 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-11-16T18:14:26.195 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-11-16T18:14:26.196 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-11-16T18:14:26.196 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-11-16T18:14:26.196 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-11-16T18:14:26.196 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-11-16T18:14:26.196 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-11-16T18:14:26.196 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-11-16T18:14:26.196 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-11-16T18:14:26.196 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-11-16T18:14:26.196 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-11-16T18:14:26.196 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-11-16T18:14:26.196 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-11-16T18:14:26.202 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-11-16T18:14:26.203 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-11-16T18:14:26.203 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-11-16T18:14:26.203 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-11-16T18:14:26.203 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-11-16T18:14:26.203 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-11-16T18:14:26.203 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-11-16T18:14:26.203 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-11-16T18:14:26.203 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-11-16T18:14:26.203 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-11-16T18:14:26.203 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-11-16T18:14:26.204 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-11-16T18:14:26.204 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-11-16T18:14:26.204 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-11-16T18:14:26.204 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-11-16T18:14:26.204 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 129/134 2023-11-16T18:14:26.204 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-16T18:14:26.204 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 131/134 2023-11-16T18:14:26.204 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T18:14:26.204 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-16T18:14:26.906 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T18:14:27.153 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 115/134 2023-11-16T18:14:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T18:14:27.525 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T18:14:28.445 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T18:14:28.446 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:46 ago on Thu 16 Nov 2023 06:12:42 PM UTC. 2023-11-16T18:14:28.484 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-11-16T18:14:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-11-16T18:14:28.958 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 119/134 2023-11-16T18:14:29.116 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T18:14:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 121/134 2023-11-16T18:14:29.439 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T18:14:29.615 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T18:14:29.899 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 115/134 2023-11-16T18:14:30.047 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T18:14:30.253 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T18:14:31.148 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-11-16T18:14:31.450 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-11-16T18:14:31.625 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 119/134 2023-11-16T18:14:31.767 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T18:14:31.914 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 121/134 2023-11-16T18:14:32.039 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T18:14:32.109 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T18:14:32.344 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T18:14:32.363 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T18:14:32.538 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 124/134 2023-11-16T18:14:32.795 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 125/134 2023-11-16T18:14:33.079 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 126/134 2023-11-16T18:14:33.421 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-16T18:14:33.554 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T18:14:33.562 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T18:14:33.737 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-11-16T18:14:33.914 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T18:14:34.002 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T18:14:34.163 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T18:14:34.180 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T18:14:34.180 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T18:14:34.388 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T18:14:34.389 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T18:14:34.390 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T18:14:34.602 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T18:14:34.604 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T18:14:34.622 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T18:14:34.747 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 124/134 2023-11-16T18:14:34.802 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T18:14:34.925 INFO:teuthology.orchestra.run.smithi188.stdout:Installed products updated. 2023-11-16T18:14:35.063 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 125/134 2023-11-16T18:14:35.137 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:14:35.138 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2023-11-16T18:14:35.138 INFO:teuthology.orchestra.run.smithi188.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-16T18:14:35.138 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.139 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.139 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-16T18:14:35.139 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-16T18:14:35.139 INFO:teuthology.orchestra.run.smithi188.stdout: automake-1.16.1-7.el8.noarch 2023-11-16T18:14:35.140 INFO:teuthology.orchestra.run.smithi188.stdout: bison-3.0.4-10.el8.x86_64 2023-11-16T18:14:35.140 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.140 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.140 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.141 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.141 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:35.141 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.141 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.142 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.142 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:35.142 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:35.142 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:35.143 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:35.143 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:35.143 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.144 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.144 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:35.144 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.144 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.145 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.145 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:35.145 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:35.145 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.146 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:35.147 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-16T18:14:35.147 INFO:teuthology.orchestra.run.smithi188.stdout: flex-2.6.1-9.el8.x86_64 2023-11-16T18:14:35.147 INFO:teuthology.orchestra.run.smithi188.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-16T18:14:35.148 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T18:14:35.148 INFO:teuthology.orchestra.run.smithi188.stdout: jq-1.6-3.el8.x86_64 2023-11-16T18:14:35.149 INFO:teuthology.orchestra.run.smithi188.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-16T18:14:35.149 INFO:teuthology.orchestra.run.smithi188.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-16T18:14:35.149 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-16T18:14:35.149 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.150 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.150 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.150 INFO:teuthology.orchestra.run.smithi188.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-16T18:14:35.150 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-16T18:14:35.151 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T18:14:35.151 INFO:teuthology.orchestra.run.smithi188.stdout: libpq-13.5-1.el8.x86_64 2023-11-16T18:14:35.151 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-16T18:14:35.151 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-16T18:14:35.152 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.152 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.152 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T18:14:35.153 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.153 INFO:teuthology.orchestra.run.smithi188.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-16T18:14:35.153 INFO:teuthology.orchestra.run.smithi188.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-16T18:14:35.154 INFO:teuthology.orchestra.run.smithi188.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-16T18:14:35.154 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T18:14:35.155 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-16T18:14:35.155 INFO:teuthology.orchestra.run.smithi188.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-16T18:14:35.155 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T18:14:35.155 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T18:14:35.156 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-16T18:14:35.156 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-16T18:14:35.156 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-16T18:14:35.156 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-16T18:14:35.157 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-16T18:14:35.158 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-16T18:14:35.158 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T18:14:35.190 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T18:14:35.190 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T18:14:35.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-16T18:14:35.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T18:14:35.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T18:14:35.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T18:14:35.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T18:14:35.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T18:14:35.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-16T18:14:35.192 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T18:14:35.192 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T18:14:35.192 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-16T18:14:35.192 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T18:14:35.192 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T18:14:35.192 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T18:14:35.192 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T18:14:35.192 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T18:14:35.192 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T18:14:35.192 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T18:14:35.193 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T18:14:35.193 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T18:14:35.193 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-16T18:14:35.193 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T18:14:35.193 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T18:14:35.193 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-16T18:14:35.193 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T18:14:35.193 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T18:14:35.193 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T18:14:35.193 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T18:14:35.193 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T18:14:35.193 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T18:14:35.193 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T18:14:35.194 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T18:14:35.194 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T18:14:35.194 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T18:14:35.194 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-16T18:14:35.198 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.198 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.199 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T18:14:35.199 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T18:14:35.199 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.199 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T18:14:35.199 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T18:14:35.199 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-16T18:14:35.199 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T18:14:35.199 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T18:14:35.199 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T18:14:35.199 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T18:14:35.200 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T18:14:35.200 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T18:14:35.200 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T18:14:35.200 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T18:14:35.200 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T18:14:35.200 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T18:14:35.200 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.200 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.200 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:35.200 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T18:14:35.200 INFO:teuthology.orchestra.run.smithi188.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-16T18:14:35.200 INFO:teuthology.orchestra.run.smithi188.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-16T18:14:35.201 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-16T18:14:35.201 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T18:14:35.330 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-16T18:14:35.338 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:14:35.350 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-16T18:14:35.355 DEBUG:teuthology.parallel:result is None 2023-11-16T18:14:35.404 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 126/134 2023-11-16T18:14:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T18:14:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 1/134 2023-11-16T18:14:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2/134 2023-11-16T18:14:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 3/134 2023-11-16T18:14:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 4/134 2023-11-16T18:14:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 5/134 2023-11-16T18:14:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T18:14:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T18:14:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T18:14:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T18:14:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 10/134 2023-11-16T18:14:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 11/134 2023-11-16T18:14:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 12/134 2023-11-16T18:14:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 13/134 2023-11-16T18:14:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 14/134 2023-11-16T18:14:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 15/134 2023-11-16T18:14:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 16/134 2023-11-16T18:14:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 17/134 2023-11-16T18:14:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 18/134 2023-11-16T18:14:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 19/134 2023-11-16T18:14:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 20/134 2023-11-16T18:14:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 21/134 2023-11-16T18:14:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 22/134 2023-11-16T18:14:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 23/134 2023-11-16T18:14:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 24/134 2023-11-16T18:14:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 25/134 2023-11-16T18:14:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 26/134 2023-11-16T18:14:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 27/134 2023-11-16T18:14:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 28/134 2023-11-16T18:14:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 29/134 2023-11-16T18:14:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 30/134 2023-11-16T18:14:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 31/134 2023-11-16T18:14:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 32/134 2023-11-16T18:14:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 33/134 2023-11-16T18:14:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 34/134 2023-11-16T18:14:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 35/134 2023-11-16T18:14:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 36/134 2023-11-16T18:14:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 37/134 2023-11-16T18:14:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 38/134 2023-11-16T18:14:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-16T18:14:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-16T18:14:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-16T18:14:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-16T18:14:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-16T18:14:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-16T18:14:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-16T18:14:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-16T18:14:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-16T18:14:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-16T18:14:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-16T18:14:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-16T18:14:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-16T18:14:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-16T18:14:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-16T18:14:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-16T18:14:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-16T18:14:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-16T18:14:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-16T18:14:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-16T18:14:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-16T18:14:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-16T18:14:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-16T18:14:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-16T18:14:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-16T18:14:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-16T18:14:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-16T18:14:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-16T18:14:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-16T18:14:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-16T18:14:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-16T18:14:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-16T18:14:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-16T18:14:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-11-16T18:14:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-11-16T18:14:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-11-16T18:14:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-11-16T18:14:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-11-16T18:14:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-11-16T18:14:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-11-16T18:14:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-11-16T18:14:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-11-16T18:14:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-11-16T18:14:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-11-16T18:14:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-11-16T18:14:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-11-16T18:14:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-11-16T18:14:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-11-16T18:14:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-11-16T18:14:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-11-16T18:14:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-11-16T18:14:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-11-16T18:14:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-11-16T18:14:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-11-16T18:14:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-11-16T18:14:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-11-16T18:14:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-11-16T18:14:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-11-16T18:14:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-11-16T18:14:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-11-16T18:14:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-11-16T18:14:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-11-16T18:14:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-11-16T18:14:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-11-16T18:14:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-11-16T18:14:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-11-16T18:14:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-11-16T18:14:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-11-16T18:14:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-11-16T18:14:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-11-16T18:14:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-11-16T18:14:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-11-16T18:14:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-11-16T18:14:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-11-16T18:14:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-11-16T18:14:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-11-16T18:14:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-11-16T18:14:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-11-16T18:14:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-11-16T18:14:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-11-16T18:14:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-11-16T18:14:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T18:14:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-11-16T18:14:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T18:14:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-11-16T18:14:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-11-16T18:14:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-11-16T18:14:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-11-16T18:14:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-11-16T18:14:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T18:14:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 129/134 2023-11-16T18:14:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-16T18:14:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 131/134 2023-11-16T18:14:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T18:14:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-16T18:14:35.698 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-16T18:14:35.822 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T18:14:35.834 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T18:14:36.018 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-11-16T18:14:36.195 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T18:14:36.288 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T18:14:36.470 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T18:14:36.484 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T18:14:36.485 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T18:14:36.652 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T18:14:36.653 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T18:14:36.873 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T18:14:37.070 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T18:14:37.637 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T18:14:37.637 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 1/134 2023-11-16T18:14:37.637 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2/134 2023-11-16T18:14:37.637 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 3/134 2023-11-16T18:14:37.637 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 4/134 2023-11-16T18:14:37.637 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 5/134 2023-11-16T18:14:37.637 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T18:14:37.637 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T18:14:37.637 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T18:14:37.638 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T18:14:37.638 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 10/134 2023-11-16T18:14:37.638 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 11/134 2023-11-16T18:14:37.638 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 12/134 2023-11-16T18:14:37.638 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 13/134 2023-11-16T18:14:37.648 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 14/134 2023-11-16T18:14:37.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 15/134 2023-11-16T18:14:37.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 16/134 2023-11-16T18:14:37.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 17/134 2023-11-16T18:14:37.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 18/134 2023-11-16T18:14:37.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 19/134 2023-11-16T18:14:37.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 20/134 2023-11-16T18:14:37.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 21/134 2023-11-16T18:14:37.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 22/134 2023-11-16T18:14:37.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 23/134 2023-11-16T18:14:37.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 24/134 2023-11-16T18:14:37.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 25/134 2023-11-16T18:14:37.650 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 26/134 2023-11-16T18:14:37.650 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 27/134 2023-11-16T18:14:37.659 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 28/134 2023-11-16T18:14:37.659 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 29/134 2023-11-16T18:14:37.659 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 30/134 2023-11-16T18:14:37.659 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 31/134 2023-11-16T18:14:37.659 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 32/134 2023-11-16T18:14:37.659 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 33/134 2023-11-16T18:14:37.659 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 34/134 2023-11-16T18:14:37.659 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 35/134 2023-11-16T18:14:37.659 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 36/134 2023-11-16T18:14:37.659 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 37/134 2023-11-16T18:14:37.659 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 38/134 2023-11-16T18:14:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-16T18:14:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-16T18:14:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-16T18:14:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-16T18:14:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-16T18:14:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-16T18:14:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-16T18:14:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-16T18:14:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-16T18:14:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-16T18:14:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-16T18:14:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-16T18:14:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-16T18:14:37.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-16T18:14:37.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-16T18:14:37.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-16T18:14:37.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-16T18:14:37.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-16T18:14:37.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-16T18:14:37.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-16T18:14:37.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-16T18:14:37.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-16T18:14:37.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-16T18:14:37.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-16T18:14:37.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-11-16T18:14:37.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-11-16T18:14:37.663 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-11-16T18:14:37.663 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-11-16T18:14:37.663 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-11-16T18:14:37.663 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-11-16T18:14:37.663 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-11-16T18:14:37.663 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-11-16T18:14:37.663 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-11-16T18:14:37.663 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-11-16T18:14:37.663 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-11-16T18:14:37.663 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-11-16T18:14:37.663 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-11-16T18:14:37.663 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-11-16T18:14:37.664 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-11-16T18:14:37.664 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-11-16T18:14:37.664 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-11-16T18:14:37.664 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-11-16T18:14:37.664 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-11-16T18:14:37.664 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-11-16T18:14:37.664 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-11-16T18:14:37.664 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-11-16T18:14:37.664 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-11-16T18:14:37.664 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-11-16T18:14:37.664 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-11-16T18:14:37.664 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-11-16T18:14:37.664 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-11-16T18:14:37.665 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-11-16T18:14:37.665 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-11-16T18:14:37.665 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-11-16T18:14:37.665 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-11-16T18:14:37.665 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-11-16T18:14:37.665 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-11-16T18:14:37.665 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-11-16T18:14:37.665 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-11-16T18:14:37.665 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-11-16T18:14:37.665 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-11-16T18:14:37.665 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-11-16T18:14:37.665 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-11-16T18:14:37.666 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-11-16T18:14:37.666 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-11-16T18:14:37.666 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-11-16T18:14:37.666 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-11-16T18:14:37.666 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-11-16T18:14:37.683 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T18:14:37.683 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-11-16T18:14:37.683 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T18:14:37.683 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-11-16T18:14:37.684 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-11-16T18:14:37.684 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-11-16T18:14:37.684 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-11-16T18:14:37.684 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-11-16T18:14:37.684 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T18:14:37.685 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 129/134 2023-11-16T18:14:37.685 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-16T18:14:37.685 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 131/134 2023-11-16T18:14:37.685 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T18:14:37.685 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-16T18:14:37.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T18:14:37.855 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:53 ago on Thu 16 Nov 2023 06:12:44 PM UTC. 2023-11-16T18:14:40.052 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T18:14:40.053 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:54 ago on Thu 16 Nov 2023 06:12:46 PM UTC. 2023-11-16T18:14:46.515 INFO:teuthology.orchestra.run.smithi040.stdout:Installed products updated. 2023-11-16T18:14:46.751 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-11-16T18:14:46.752 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2023-11-16T18:14:46.752 INFO:teuthology.orchestra.run.smithi040.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-16T18:14:46.752 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.752 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.752 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-11-16T18:14:46.752 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-16T18:14:46.752 INFO:teuthology.orchestra.run.smithi040.stdout: automake-1.16.1-7.el8.noarch 2023-11-16T18:14:46.752 INFO:teuthology.orchestra.run.smithi040.stdout: bison-3.0.4-10.el8.x86_64 2023-11-16T18:14:46.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.753 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:46.753 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.753 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.753 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.753 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:46.753 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:46.753 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:46.753 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:46.753 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:46.753 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.753 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.753 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:46.753 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.754 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.754 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.754 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:46.754 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:46.754 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.754 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:46.754 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-16T18:14:46.754 INFO:teuthology.orchestra.run.smithi040.stdout: flex-2.6.1-9.el8.x86_64 2023-11-16T18:14:46.754 INFO:teuthology.orchestra.run.smithi040.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-16T18:14:46.754 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T18:14:46.754 INFO:teuthology.orchestra.run.smithi040.stdout: jq-1.6-3.el8.x86_64 2023-11-16T18:14:46.754 INFO:teuthology.orchestra.run.smithi040.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-16T18:14:46.755 INFO:teuthology.orchestra.run.smithi040.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-16T18:14:46.755 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-16T18:14:46.755 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.755 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.757 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.757 INFO:teuthology.orchestra.run.smithi040.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-16T18:14:46.757 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-16T18:14:46.757 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T18:14:46.757 INFO:teuthology.orchestra.run.smithi040.stdout: libpq-13.5-1.el8.x86_64 2023-11-16T18:14:46.757 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-16T18:14:46.757 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-16T18:14:46.757 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.757 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.757 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T18:14:46.758 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.758 INFO:teuthology.orchestra.run.smithi040.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-16T18:14:46.758 INFO:teuthology.orchestra.run.smithi040.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-16T18:14:46.758 INFO:teuthology.orchestra.run.smithi040.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-16T18:14:46.758 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T18:14:46.758 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-16T18:14:46.759 INFO:teuthology.orchestra.run.smithi040.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-16T18:14:46.759 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T18:14:46.759 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T18:14:46.759 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-16T18:14:46.759 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-16T18:14:46.759 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-16T18:14:46.759 INFO:teuthology.orchestra.run.smithi040.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-16T18:14:46.759 INFO:teuthology.orchestra.run.smithi040.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-16T18:14:46.759 INFO:teuthology.orchestra.run.smithi040.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-16T18:14:46.759 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T18:14:46.759 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T18:14:46.759 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T18:14:46.759 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-16T18:14:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T18:14:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T18:14:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T18:14:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T18:14:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T18:14:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-16T18:14:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T18:14:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T18:14:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-16T18:14:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T18:14:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T18:14:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T18:14:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T18:14:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T18:14:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T18:14:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T18:14:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T18:14:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T18:14:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-16T18:14:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T18:14:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T18:14:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-16T18:14:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T18:14:46.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T18:14:46.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T18:14:46.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T18:14:46.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T18:14:46.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T18:14:46.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T18:14:46.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T18:14:46.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T18:14:46.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T18:14:46.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-16T18:14:46.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T18:14:46.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T18:14:46.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T18:14:46.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T18:14:46.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-16T18:14:46.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T18:14:46.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T18:14:46.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T18:14:46.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T18:14:46.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T18:14:46.764 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T18:14:46.764 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T18:14:46.764 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T18:14:46.764 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T18:14:46.764 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T18:14:46.764 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.764 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.764 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:46.764 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T18:14:46.764 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-16T18:14:46.764 INFO:teuthology.orchestra.run.smithi040.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-16T18:14:46.764 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-16T18:14:46.764 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T18:14:46.765 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-16T18:14:46.765 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-11-16T18:14:46.765 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-11-16T18:14:47.106 INFO:teuthology.orchestra.run.smithi002.stdout:Installed products updated. 2023-11-16T18:14:47.343 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:14:47.343 INFO:teuthology.orchestra.run.smithi002.stdout:Upgraded: 2023-11-16T18:14:47.343 INFO:teuthology.orchestra.run.smithi002.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-16T18:14:47.343 INFO:teuthology.orchestra.run.smithi002.stdout: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.343 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.343 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-11-16T18:14:47.343 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-16T18:14:47.343 INFO:teuthology.orchestra.run.smithi002.stdout: automake-1.16.1-7.el8.noarch 2023-11-16T18:14:47.343 INFO:teuthology.orchestra.run.smithi002.stdout: bison-3.0.4-10.el8.x86_64 2023-11-16T18:14:47.343 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.343 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.344 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.344 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.344 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:47.344 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.344 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.344 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.344 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:47.344 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:47.344 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:47.344 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:47.344 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:47.344 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.344 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T18:14:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-16T18:14:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: flex-2.6.1-9.el8.x86_64 2023-11-16T18:14:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-16T18:14:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T18:14:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: jq-1.6-3.el8.x86_64 2023-11-16T18:14:47.346 INFO:teuthology.orchestra.run.smithi002.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-16T18:14:47.346 INFO:teuthology.orchestra.run.smithi002.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-16T18:14:47.346 INFO:teuthology.orchestra.run.smithi002.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-16T18:14:47.346 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.346 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.351 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.351 INFO:teuthology.orchestra.run.smithi002.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-16T18:14:47.351 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-16T18:14:47.351 INFO:teuthology.orchestra.run.smithi002.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T18:14:47.351 INFO:teuthology.orchestra.run.smithi002.stdout: libpq-13.5-1.el8.x86_64 2023-11-16T18:14:47.351 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-16T18:14:47.352 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-16T18:14:47.352 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.352 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.352 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T18:14:47.352 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.352 INFO:teuthology.orchestra.run.smithi002.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-16T18:14:47.352 INFO:teuthology.orchestra.run.smithi002.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-16T18:14:47.352 INFO:teuthology.orchestra.run.smithi002.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-16T18:14:47.352 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T18:14:47.352 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-16T18:14:47.352 INFO:teuthology.orchestra.run.smithi002.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-16T18:14:47.353 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T18:14:47.353 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T18:14:47.355 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-16T18:14:47.355 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-16T18:14:47.355 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-16T18:14:47.355 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-16T18:14:47.355 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-16T18:14:47.355 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-16T18:14:47.355 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T18:14:47.355 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T18:14:47.355 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T18:14:47.355 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-16T18:14:47.355 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T18:14:47.355 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T18:14:47.355 INFO:teuthology.orchestra.run.smithi002.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T18:14:47.356 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T18:14:47.356 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.356 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.356 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.356 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T18:14:47.356 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-16T18:14:47.356 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T18:14:47.356 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T18:14:47.356 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-16T18:14:47.357 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T18:14:47.358 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T18:14:47.358 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T18:14:47.358 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T18:14:47.358 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T18:14:47.358 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T18:14:47.358 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T18:14:47.359 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T18:14:47.359 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T18:14:47.359 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-16T18:14:47.360 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T18:14:47.360 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T18:14:47.360 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-16T18:14:47.360 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T18:14:47.360 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T18:14:47.360 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T18:14:47.361 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T18:14:47.361 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T18:14:47.361 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T18:14:47.361 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T18:14:47.361 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T18:14:47.361 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T18:14:47.361 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T18:14:47.361 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-16T18:14:47.362 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.362 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.362 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T18:14:47.362 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T18:14:47.362 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.362 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T18:14:47.362 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T18:14:47.363 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-16T18:14:47.363 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T18:14:47.363 INFO:teuthology.orchestra.run.smithi002.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T18:14:47.363 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T18:14:47.363 INFO:teuthology.orchestra.run.smithi002.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T18:14:47.363 INFO:teuthology.orchestra.run.smithi002.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T18:14:47.363 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T18:14:47.363 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T18:14:47.363 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T18:14:47.363 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T18:14:47.363 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T18:14:47.363 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.363 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.364 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T18:14:47.364 INFO:teuthology.orchestra.run.smithi002.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T18:14:47.364 INFO:teuthology.orchestra.run.smithi002.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-16T18:14:47.364 INFO:teuthology.orchestra.run.smithi002.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-16T18:14:47.364 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-16T18:14:47.364 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T18:14:47.364 INFO:teuthology.orchestra.run.smithi002.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-16T18:14:47.364 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:14:47.364 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-11-16T18:14:47.540 DEBUG:teuthology.parallel:result is None 2023-11-16T18:14:47.855 DEBUG:teuthology.parallel:result is None 2023-11-16T18:14:47.856 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T18:14:48.105 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T18:14:48.132 INFO:teuthology.orchestra.run.smithi002.stdout:18.0.0-3905.g5e4a743a.el8 2023-11-16T18:14:48.132 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3905.g5e4a743a.el8 2023-11-16T18:14:48.132 INFO:teuthology.task.install:The correct ceph version 18.0.0-3905.g5e4a743a is installed. 2023-11-16T18:14:48.133 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T18:14:48.283 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T18:14:48.313 INFO:teuthology.orchestra.run.smithi040.stdout:18.0.0-3905.g5e4a743a.el8 2023-11-16T18:14:48.313 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3905.g5e4a743a.el8 2023-11-16T18:14:48.313 INFO:teuthology.task.install:The correct ceph version 18.0.0-3905.g5e4a743a is installed. 2023-11-16T18:14:48.313 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T18:14:48.571 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T18:14:48.600 INFO:teuthology.orchestra.run.smithi188.stdout:18.0.0-3905.g5e4a743a.el8 2023-11-16T18:14:48.601 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3905.g5e4a743a.el8 2023-11-16T18:14:48.601 INFO:teuthology.task.install:The correct ceph version 18.0.0-3905.g5e4a743a is installed. 2023-11-16T18:14:48.602 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-16T18:14:48.603 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:14:48.603 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T18:14:48.656 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-11-16T18:14:48.656 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T18:14:48.711 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-16T18:14:48.712 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T18:14:48.759 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-16T18:14:48.760 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:14:48.760 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T18:14:48.796 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T18:14:48.867 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-11-16T18:14:48.868 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T18:14:48.902 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T18:14:48.974 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-16T18:14:48.974 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T18:14:49.028 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T18:14:49.099 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-16T18:14:49.100 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:14:49.100 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T18:14:49.134 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T18:14:49.212 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-11-16T18:14:49.217 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T18:14:49.251 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T18:14:49.326 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-16T18:14:49.327 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T18:14:49.357 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T18:14:49.431 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-16T18:14:49.431 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:14:49.431 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T18:14:49.465 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T18:14:49.537 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-11-16T18:14:49.537 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T18:14:49.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T18:14:49.647 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-16T18:14:49.647 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T18:14:49.678 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T18:14:49.751 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-16T18:14:49.996 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-11-16T18:14:49.996 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T18:14:49.997 INFO:tasks.cephadm:Cluster fsid is 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:14:49.997 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-16T18:14:49.997 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.2', 'mon.b': '172.21.15.40', 'mon.c': '172.21.15.188'} 2023-11-16T18:14:49.997 INFO:tasks.cephadm:First mon is mon.a on smithi002 2023-11-16T18:14:49.997 INFO:tasks.cephadm:First mgr is x 2023-11-16T18:14:49.997 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-16T18:14:49.998 DEBUG:teuthology.orchestra.run.smithi002:> sudo hostname $(hostname -s) 2023-11-16T18:14:50.033 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname $(hostname -s) 2023-11-16T18:14:50.068 DEBUG:teuthology.orchestra.run.smithi188:> sudo hostname $(hostname -s) 2023-11-16T18:14:50.101 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-11-16T18:14:50.101 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T18:14:50.330 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/74186/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-3905-g5e4a743a', 'node_name': '172.21.5.32+adami02', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-3905.g5e4a743a'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-11-16 00:50:05.467524', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-choffman-62793', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-11-16T18:14:50.602 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-choffman-62793, sha1 5e4a743ad7fd1422ebd6c4dc950a21a303279207 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T18:14:50.603 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/x86_64/flavors/default/cephadm 2023-11-16T18:14:50.610 DEBUG:teuthology.orchestra.run.smithi002:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T18:14:51.023 INFO:teuthology.orchestra.run.smithi002.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Nov 16 18:14 /home/ubuntu/cephtest/cephadm 2023-11-16T18:14:51.025 DEBUG:teuthology.orchestra.run.smithi040:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T18:14:51.537 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Nov 16 18:14 /home/ubuntu/cephtest/cephadm 2023-11-16T18:14:51.538 DEBUG:teuthology.orchestra.run.smithi188:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T18:14:51.982 INFO:teuthology.orchestra.run.smithi188.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Nov 16 18:14 /home/ubuntu/cephtest/cephadm 2023-11-16T18:14:51.984 DEBUG:teuthology.orchestra.run.smithi002:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T18:14:52.007 DEBUG:teuthology.orchestra.run.smithi040:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T18:14:52.033 DEBUG:teuthology.orchestra.run.smithi188:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T18:14:52.096 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 on all hosts... 2023-11-16T18:14:52.097 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 pull 2023-11-16T18:14:52.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 pull 2023-11-16T18:14:52.132 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 pull 2023-11-16T18:14:54.403 INFO:teuthology.orchestra.run.smithi188.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T18:14:56.649 INFO:teuthology.orchestra.run.smithi002.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T18:14:57.046 INFO:teuthology.orchestra.run.smithi040.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T18:15:28.651 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:15:28.651 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev)", 2023-11-16T18:15:28.651 INFO:teuthology.orchestra.run.smithi002.stdout: "image_id": "d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99", 2023-11-16T18:15:28.651 INFO:teuthology.orchestra.run.smithi002.stdout: "repo_digests": [ 2023-11-16T18:15:28.652 INFO:teuthology.orchestra.run.smithi002.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:209515a30c262c9be4300b2075ceb5fd8912e5a4eccd3c9021b5585705456b42" 2023-11-16T18:15:28.652 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T18:15:28.652 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:15:30.913 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2023-11-16T18:15:30.914 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev)", 2023-11-16T18:15:30.914 INFO:teuthology.orchestra.run.smithi188.stdout: "image_id": "d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99", 2023-11-16T18:15:30.914 INFO:teuthology.orchestra.run.smithi188.stdout: "repo_digests": [ 2023-11-16T18:15:30.914 INFO:teuthology.orchestra.run.smithi188.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:209515a30c262c9be4300b2075ceb5fd8912e5a4eccd3c9021b5585705456b42" 2023-11-16T18:15:30.914 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2023-11-16T18:15:30.914 INFO:teuthology.orchestra.run.smithi188.stdout:} 2023-11-16T18:15:33.248 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-11-16T18:15:33.248 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev)", 2023-11-16T18:15:33.248 INFO:teuthology.orchestra.run.smithi040.stdout: "image_id": "d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99", 2023-11-16T18:15:33.248 INFO:teuthology.orchestra.run.smithi040.stdout: "repo_digests": [ 2023-11-16T18:15:33.248 INFO:teuthology.orchestra.run.smithi040.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:209515a30c262c9be4300b2075ceb5fd8912e5a4eccd3c9021b5585705456b42" 2023-11-16T18:15:33.248 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-11-16T18:15:33.248 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-11-16T18:15:33.268 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph 2023-11-16T18:15:33.306 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph 2023-11-16T18:15:33.338 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph 2023-11-16T18:15:33.375 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /etc/ceph 2023-11-16T18:15:33.409 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /etc/ceph 2023-11-16T18:15:33.437 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /etc/ceph 2023-11-16T18:15:33.473 INFO:tasks.cephadm:Writing seed config... 2023-11-16T18:15:33.475 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-16T18:15:33.477 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-16T18:15:33.477 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-16T18:15:33.477 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-16T18:15:33.478 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-16T18:15:33.478 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-11-16T18:15:33.479 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-16T18:15:33.479 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-11-16T18:15:33.479 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-16T18:15:33.480 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-11-16T18:15:33.480 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-11-16T18:15:33.480 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-11-16T18:15:33.480 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-11-16T18:15:33.480 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-16T18:15:33.481 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-16T18:15:33.481 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-16T18:15:33.481 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-16T18:15:33.482 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2023-11-16T18:15:33.482 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-16T18:15:33.483 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-16T18:15:33.483 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-16T18:15:33.484 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-11-16T18:15:33.484 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-11-16T18:15:33.484 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-16T18:15:33.484 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-16T18:15:33.485 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-16T18:15:33.485 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-16T18:15:33.485 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-16T18:15:33.485 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-16T18:15:33.486 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-16T18:15:33.487 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2023-11-16T18:15:33.487 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-16T18:15:33.488 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-11-16T18:15:33.488 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-11-16T18:15:33.488 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-16T18:15:33.488 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-16T18:15:33.490 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-11-16T18:15:33.490 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-16T18:15:33.490 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-16T18:15:33.515 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-16T18:15:33.516 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-16T18:15:33.516 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-16T18:15:33.517 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-16T18:15:33.517 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-11-16T18:15:33.517 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-11-16T18:15:33.519 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:15:33.520 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-16T18:15:33.559 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 08160fac-84ac-11ee-8dbe-212e2dc638e7 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-16T18:15:33.576 DEBUG:teuthology.orchestra.run.smithi002:mon.a> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.a.service 2023-11-16T18:15:33.603 DEBUG:teuthology.orchestra.run.smithi002:mgr.x> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.x.service 2023-11-16T18:15:33.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:-- Logs begin at Thu 2023-11-16 17:59:05 UTC. -- 2023-11-16T18:15:33.655 INFO:tasks.cephadm:Bootstrapping... 2023-11-16T18:15:33.688 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 -v bootstrap --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.2 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-16T18:15:33.702 INFO:journalctl@ceph.mgr.x.smithi002.stdout:-- Logs begin at Thu 2023-11-16 17:59:05 UTC. -- 2023-11-16T18:15:33.980 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-11-16T18:15:33.980 INFO:teuthology.orchestra.run.smithi002.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207', '-v', 'bootstrap', '--fsid', '08160fac-84ac-11ee-8dbe-212e2dc638e7', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.2', '--skip-admin-label'] 2023-11-16T18:15:34.130 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.1.1 2023-11-16T18:15:34.131 INFO:teuthology.orchestra.run.smithi002.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-11-16T18:15:34.131 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying podman|docker is present... 2023-11-16T18:15:34.282 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.1.1 2023-11-16T18:15:34.282 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying lvm2 is present... 2023-11-16T18:15:34.282 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying time synchronization is in place... 2023-11-16T18:15:34.287 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-11-16T18:15:34.288 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-11-16T18:15:34.298 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-11-16T18:15:34.299 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2023-11-16T18:15:34.300 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout enabled 2023-11-16T18:15:34.306 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout active 2023-11-16T18:15:34.306 INFO:teuthology.orchestra.run.smithi002.stdout:Unit chronyd.service is enabled and running 2023-11-16T18:15:34.306 INFO:teuthology.orchestra.run.smithi002.stdout:Repeating the final host check... 2023-11-16T18:15:34.448 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.1.1 2023-11-16T18:15:34.448 INFO:teuthology.orchestra.run.smithi002.stdout:podman (/bin/podman) version 4.1.1 is present 2023-11-16T18:15:34.448 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl is present 2023-11-16T18:15:34.448 INFO:teuthology.orchestra.run.smithi002.stdout:lvcreate is present 2023-11-16T18:15:34.454 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-11-16T18:15:34.454 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-11-16T18:15:34.459 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-11-16T18:15:34.460 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2023-11-16T18:15:34.465 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout enabled 2023-11-16T18:15:34.472 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout active 2023-11-16T18:15:34.472 INFO:teuthology.orchestra.run.smithi002.stdout:Unit chronyd.service is enabled and running 2023-11-16T18:15:34.472 INFO:teuthology.orchestra.run.smithi002.stdout:Host looks OK 2023-11-16T18:15:34.473 INFO:teuthology.orchestra.run.smithi002.stdout:Cluster fsid: 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:15:34.473 INFO:teuthology.orchestra.run.smithi002.stdout:Acquiring lock 140522327886312 on /run/cephadm/08160fac-84ac-11ee-8dbe-212e2dc638e7.lock 2023-11-16T18:15:34.473 INFO:teuthology.orchestra.run.smithi002.stdout:Lock 140522327886312 acquired on /run/cephadm/08160fac-84ac-11ee-8dbe-212e2dc638e7.lock 2023-11-16T18:15:34.473 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying IP 172.21.15.2 port 3300 ... 2023-11-16T18:15:34.474 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying IP 172.21.15.2 port 6789 ... 2023-11-16T18:15:34.474 INFO:teuthology.orchestra.run.smithi002.stdout:Base mon IP(s) is [172.21.15.2:3300, 172.21.15.2:6789], mon addrv is [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2023-11-16T18:15:34.476 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-11-16T18:15:34.476 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.2 metric 100 2023-11-16T18:15:34.480 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-11-16T18:15:34.480 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-11-16T18:15:34.480 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1776sec hoplimit 64 pref medium 2023-11-16T18:15:34.484 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-16T18:15:34.484 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-11-16T18:15:34.484 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-11-16T18:15:34.484 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 3: ens1f0: mtu 1500 state UP qlen 1000 2023-11-16T18:15:34.485 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1920/64 scope link 2023-11-16T18:15:34.485 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-11-16T18:15:34.485 INFO:teuthology.orchestra.run.smithi002.stdout:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2023-11-16T18:15:34.485 INFO:teuthology.orchestra.run.smithi002.stdout:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2023-11-16T18:15:34.486 INFO:teuthology.orchestra.run.smithi002.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-11-16T18:15:34.487 INFO:teuthology.orchestra.run.smithi002.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-16T18:15:34.490 INFO:teuthology.orchestra.run.smithi002.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T18:15:35.436 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99 2023-11-16T18:15:35.438 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T18:15:35.438 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Getting image source signatures 2023-11-16T18:15:35.438 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying blob sha256:3a8f6e475435f9e3b5d92795b454ceb30085f6221d261249c26042723227dea7 2023-11-16T18:15:35.438 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying blob sha256:837e66e3ff89804ed1bb42786f80e6c23534ff65fe48526d5007449f4e1f79b0 2023-11-16T18:15:35.439 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying config sha256:d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99 2023-11-16T18:15:35.439 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Writing manifest to image destination 2023-11-16T18:15:35.439 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Storing signatures 2023-11-16T18:15:36.190 INFO:teuthology.orchestra.run.smithi002.stdout:ceph: stdout ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev) 2023-11-16T18:15:36.191 INFO:teuthology.orchestra.run.smithi002.stdout:Ceph version: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev) 2023-11-16T18:15:36.191 INFO:teuthology.orchestra.run.smithi002.stdout:Extracting ceph user uid/gid from container image... 2023-11-16T18:15:36.939 INFO:teuthology.orchestra.run.smithi002.stdout:stat: stdout 167 167 2023-11-16T18:15:36.939 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial keys... 2023-11-16T18:15:37.590 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQBJXFZlZg1iEhAA5Azzvcr1KyOfiUCBxnNgHQ== 2023-11-16T18:15:38.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQBJXFZleT4SOhAA7lYs0hwwWSBMsQoI0hwJGw== 2023-11-16T18:15:38.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQBKXFZl6LKuKBAASeDqlTqUjfhbm7jkSKktpA== 2023-11-16T18:15:38.991 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial monmap... 2023-11-16T18:15:39.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-16T18:15:39.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-11-16T18:15:39.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:15:39.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-16T18:15:39.652 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool for a [v2:172.21.15.2:3300,v1:172.21.15.2:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-16T18:15:39.652 INFO:teuthology.orchestra.run.smithi002.stdout:setting min_mon_release = pacific 2023-11-16T18:15:39.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: set fsid to 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:15:39.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-16T18:15:39.652 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:15:39.653 INFO:teuthology.orchestra.run.smithi002.stdout:Creating mon... 2023-11-16T18:15:40.402 INFO:teuthology.orchestra.run.smithi002.stdout:create mon.a on 2023-11-16T18:15:40.506 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-16T18:15:40.595 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-16T18:15:40.693 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7.target. 2023-11-16T18:15:40.693 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7.target. 2023-11-16T18:15:40.822 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.a 2023-11-16T18:15:40.822 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to reset failed state of unit ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.a.service: Unit ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.a.service not loaded. 2023-11-16T18:15:40.914 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7.target.wants/ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.a.service → /etc/systemd/system/ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@.service. 2023-11-16T18:15:41.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:40 smithi002 systemd[1]: Starting Ceph mon.a for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:15:41.420 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T18:15:41.420 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2023-11-16T18:15:41.428 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T18:15:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2023-11-16T18:15:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2023-11-16T18:15:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to enable service . firewalld.service is not available 2023-11-16T18:15:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mon to start... 2023-11-16T18:15:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mon... 2023-11-16T18:15:41.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:41 smithi002 podman[98274]: 2023-11-16T18:15:41.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:41 smithi002 bash[98274]: ed0bd85715ad1d05f5537d13e5bba3540f02c409d27f76f49111a70ab8be3212 2023-11-16T18:15:41.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:41 smithi002 systemd[1]: Started Ceph mon.a for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:15:41.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:41 smithi002 ceph-mon[98313]: mkfs 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:15:41.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:41 smithi002 ceph-mon[98313]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-16T18:15:42.282 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout cluster: 2023-11-16T18:15:42.282 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout id: 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:15:42.282 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-11-16T18:15:42.282 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T18:15:42.282 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout services: 2023-11-16T18:15:42.282 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.585002s) 2023-11-16T18:15:42.282 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-11-16T18:15:42.282 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-11-16T18:15:42.283 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T18:15:42.283 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout data: 2023-11-16T18:15:42.283 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-11-16T18:15:42.283 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-11-16T18:15:42.283 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-11-16T18:15:42.283 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout pgs: 2023-11-16T18:15:42.283 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T18:15:42.283 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.876+0000 7fa5251dd700 1 Processor -- start 2023-11-16T18:15:42.283 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.876+0000 7fa5251dd700 1 -- start start 2023-11-16T18:15:42.283 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.876+0000 7fa5251dd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 0x7fa5201088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:42.283 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.876+0000 7fa5251dd700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa520079e20 con 0x7fa5201064d0 2023-11-16T18:15:42.284 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.876+0000 7fa51ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 0x7fa5201088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:42.284 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.876+0000 7fa51ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 0x7fa5201088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46472/0 (socket says 172.21.15.2:46472) 2023-11-16T18:15:42.284 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.876+0000 7fa51ed9d700 1 -- 172.21.15.2:0/3971887657 learned_addr learned my addr 172.21.15.2:0/3971887657 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:42.284 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.877+0000 7fa51ed9d700 1 -- 172.21.15.2:0/3971887657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa520108df0 con 0x7fa5201064d0 2023-11-16T18:15:42.284 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.877+0000 7fa51ed9d700 1 --2- 172.21.15.2:0/3971887657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 0x7fa5201088b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa5100098a0 tx=0x7fa510009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1fda8a1e0a0baa8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:42.284 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.877+0000 7fa51dd9b700 1 -- 172.21.15.2:0/3971887657 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa510004160 con 0x7fa5201064d0 2023-11-16T18:15:42.284 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.877+0000 7fa51dd9b700 1 -- 172.21.15.2:0/3971887657 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5100042c0 con 0x7fa5201064d0 2023-11-16T18:15:42.284 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.877+0000 7fa51dd9b700 1 -- 172.21.15.2:0/3971887657 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5100045a0 con 0x7fa5201064d0 2023-11-16T18:15:42.284 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.878+0000 7fa5251dd700 1 -- 172.21.15.2:0/3971887657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 msgr2=0x7fa5201088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:42.284 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.878+0000 7fa5251dd700 1 --2- 172.21.15.2:0/3971887657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 0x7fa5201088b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa5100098a0 tx=0x7fa510009bb0 comp rx=0 tx=0).stop 2023-11-16T18:15:42.285 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.878+0000 7fa5251dd700 1 -- 172.21.15.2:0/3971887657 shutdown_connections 2023-11-16T18:15:42.285 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.878+0000 7fa5251dd700 1 --2- 172.21.15.2:0/3971887657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 0x7fa5201088b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:42.285 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.878+0000 7fa5251dd700 1 -- 172.21.15.2:0/3971887657 >> 172.21.15.2:0/3971887657 conn(0x7fa5200ffed0 msgr2=0x7fa520102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:42.285 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.878+0000 7fa5251dd700 1 -- 172.21.15.2:0/3971887657 shutdown_connections 2023-11-16T18:15:42.285 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.878+0000 7fa5251dd700 1 -- 172.21.15.2:0/3971887657 wait complete. 2023-11-16T18:15:42.285 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.878+0000 7fa5251dd700 1 Processor -- start 2023-11-16T18:15:42.285 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.879+0000 7fa5251dd700 1 -- start start 2023-11-16T18:15:42.285 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.879+0000 7fa5251dd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 0x7fa520105e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:42.285 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.879+0000 7fa5251dd700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa520106370 con 0x7fa5201064d0 2023-11-16T18:15:42.285 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.879+0000 7fa51ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 0x7fa520105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:42.285 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.879+0000 7fa51ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 0x7fa520105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46474/0 (socket says 172.21.15.2:46474) 2023-11-16T18:15:42.285 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.879+0000 7fa51ed9d700 1 -- 172.21.15.2:0/3795761828 learned_addr learned my addr 172.21.15.2:0/3795761828 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:42.285 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.879+0000 7fa51ed9d700 1 -- 172.21.15.2:0/3795761828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa510009550 con 0x7fa5201064d0 2023-11-16T18:15:42.286 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.879+0000 7fa51ed9d700 1 --2- 172.21.15.2:0/3795761828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 0x7fa520105e30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa510009fb0 tx=0x7fa510004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:42.286 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.880+0000 7fa507fff700 1 -- 172.21.15.2:0/3795761828 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa510003dd0 con 0x7fa5201064d0 2023-11-16T18:15:42.286 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.880+0000 7fa507fff700 1 -- 172.21.15.2:0/3795761828 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa510011630 con 0x7fa5201064d0 2023-11-16T18:15:42.286 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.880+0000 7fa5251dd700 1 -- 172.21.15.2:0/3795761828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa520109230 con 0x7fa5201064d0 2023-11-16T18:15:42.286 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.880+0000 7fa507fff700 1 -- 172.21.15.2:0/3795761828 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa510011910 con 0x7fa5201064d0 2023-11-16T18:15:42.286 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.880+0000 7fa5251dd700 1 -- 172.21.15.2:0/3795761828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5201090a0 con 0x7fa5201064d0 2023-11-16T18:15:42.286 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.880+0000 7fa507fff700 1 -- 172.21.15.2:0/3795761828 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa51001c460 con 0x7fa5201064d0 2023-11-16T18:15:42.286 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.880+0000 7fa507fff700 1 -- 172.21.15.2:0/3795761828 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa51001a880 con 0x7fa5201064d0 2023-11-16T18:15:42.286 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.881+0000 7fa505ffb700 1 -- 172.21.15.2:0/3795761828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5000052f0 con 0x7fa5201064d0 2023-11-16T18:15:42.286 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:41.883+0000 7fa507fff700 1 -- 172.21.15.2:0/3795761828 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fa51001aae0 con 0x7fa5201064d0 2023-11-16T18:15:42.286 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.015+0000 7fa505ffb700 1 -- 172.21.15.2:0/3795761828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fa500005c90 con 0x7fa5201064d0 2023-11-16T18:15:42.286 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.015+0000 7fa507fff700 1 -- 172.21.15.2:0/3795761828 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fa51001ac80 con 0x7fa5201064d0 2023-11-16T18:15:42.287 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.016+0000 7fa505ffb700 1 -- 172.21.15.2:0/3795761828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 msgr2=0x7fa520105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:42.287 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.016+0000 7fa505ffb700 1 --2- 172.21.15.2:0/3795761828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 0x7fa520105e30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa510009fb0 tx=0x7fa510004750 comp rx=0 tx=0).stop 2023-11-16T18:15:42.287 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.016+0000 7fa505ffb700 1 -- 172.21.15.2:0/3795761828 shutdown_connections 2023-11-16T18:15:42.292 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.016+0000 7fa505ffb700 1 --2- 172.21.15.2:0/3795761828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5201064d0 0x7fa520105e30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:42.292 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.016+0000 7fa505ffb700 1 -- 172.21.15.2:0/3795761828 >> 172.21.15.2:0/3795761828 conn(0x7fa5200ffed0 msgr2=0x7fa520101830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:42.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.016+0000 7fa505ffb700 1 -- 172.21.15.2:0/3795761828 shutdown_connections 2023-11-16T18:15:42.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.016+0000 7fa505ffb700 1 -- 172.21.15.2:0/3795761828 wait complete. 2023-11-16T18:15:42.293 INFO:teuthology.orchestra.run.smithi002.stdout:mon is available 2023-11-16T18:15:42.293 INFO:teuthology.orchestra.run.smithi002.stdout:Assimilating anything we can from ceph.conf... 2023-11-16T18:15:42.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:42 smithi002 ceph-mon[98313]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-16T18:15:42.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:42 smithi002 ceph-mon[98313]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} removed_ranks: {} 2023-11-16T18:15:42.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:42 smithi002 ceph-mon[98313]: fsmap 2023-11-16T18:15:42.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:42 smithi002 ceph-mon[98313]: osdmap e1: 0 total, 0 up, 0 in 2023-11-16T18:15:42.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:42 smithi002 ceph-mon[98313]: mgrmap e1: no daemons active 2023-11-16T18:15:42.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:42 smithi002 ceph-mon[98313]: from='client.? 172.21.15.2:0/3795761828' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-16T18:15:43.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T18:15:43.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [global] 2023-11-16T18:15:43.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout fsid = 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:15:43.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2023-11-16T18:15:43.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-11-16T18:15:43.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-11-16T18:15:43.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-11-16T18:15:43.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-11-16T18:15:43.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T18:15:43.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [mgr] 2023-11-16T18:15:43.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-11-16T18:15:43.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T18:15:43.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [osd] 2023-11-16T18:15:43.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-11-16T18:15:43.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-11-16T18:15:43.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-11-16T18:15:43.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-11-16T18:15:43.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-11-16T18:15:43.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.754+0000 7f6698d2e700 1 Processor -- start 2023-11-16T18:15:43.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.754+0000 7f6698d2e700 1 -- start start 2023-11-16T18:15:43.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.755+0000 7f6698d2e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 0x7f6694104d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:43.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.755+0000 7f6698d2e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6694105300 con 0x7f6694104920 2023-11-16T18:15:43.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.755+0000 7f669259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 0x7f6694104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:43.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.755+0000 7f669259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 0x7f6694104d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46476/0 (socket says 172.21.15.2:46476) 2023-11-16T18:15:43.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.755+0000 7f669259c700 1 -- 172.21.15.2:0/3257326640 learned_addr learned my addr 172.21.15.2:0/3257326640 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:43.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.755+0000 7f669259c700 1 -- 172.21.15.2:0/3257326640 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6694105b20 con 0x7f6694104920 2023-11-16T18:15:43.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.756+0000 7f669259c700 1 --2- 172.21.15.2:0/3257326640 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 0x7f6694104d30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f668400bb40 tx=0x7f668400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495bd466323ce954 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:43.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.756+0000 7f669159a700 1 -- 172.21.15.2:0/3257326640 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f668400dbe0 con 0x7f6694104920 2023-11-16T18:15:43.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.756+0000 7f669159a700 1 -- 172.21.15.2:0/3257326640 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668400dd40 con 0x7f6694104920 2023-11-16T18:15:43.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.756+0000 7f669159a700 1 -- 172.21.15.2:0/3257326640 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6684004030 con 0x7f6694104920 2023-11-16T18:15:43.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.756+0000 7f6698d2e700 1 -- 172.21.15.2:0/3257326640 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 msgr2=0x7f6694104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:43.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.756+0000 7f6698d2e700 1 --2- 172.21.15.2:0/3257326640 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 0x7f6694104d30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f668400bb40 tx=0x7f668400be50 comp rx=0 tx=0).stop 2023-11-16T18:15:43.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.756+0000 7f6698d2e700 1 -- 172.21.15.2:0/3257326640 shutdown_connections 2023-11-16T18:15:43.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.756+0000 7f6698d2e700 1 --2- 172.21.15.2:0/3257326640 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 0x7f6694104d30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:43.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.756+0000 7f6698d2e700 1 -- 172.21.15.2:0/3257326640 >> 172.21.15.2:0/3257326640 conn(0x7f66940ffed0 msgr2=0x7f6694102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:43.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.756+0000 7f6698d2e700 1 -- 172.21.15.2:0/3257326640 shutdown_connections 2023-11-16T18:15:43.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.756+0000 7f6698d2e700 1 -- 172.21.15.2:0/3257326640 wait complete. 2023-11-16T18:15:43.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.757+0000 7f6698d2e700 1 Processor -- start 2023-11-16T18:15:43.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.757+0000 7f6698d2e700 1 -- start start 2023-11-16T18:15:43.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.757+0000 7f6698d2e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 0x7f669419b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:43.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.757+0000 7f6698d2e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669419b6a0 con 0x7f6694104920 2023-11-16T18:15:43.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.757+0000 7f669259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 0x7f669419b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:43.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.758+0000 7f669259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 0x7f669419b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46478/0 (socket says 172.21.15.2:46478) 2023-11-16T18:15:43.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.758+0000 7f669259c700 1 -- 172.21.15.2:0/3818296956 learned_addr learned my addr 172.21.15.2:0/3818296956 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:43.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.758+0000 7f669259c700 1 -- 172.21.15.2:0/3818296956 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668400b7f0 con 0x7f6694104920 2023-11-16T18:15:43.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.758+0000 7f669259c700 1 --2- 172.21.15.2:0/3818296956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 0x7f669419b160 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f668400b340 tx=0x7f6684004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:43.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.758+0000 7f66837fe700 1 -- 172.21.15.2:0/3818296956 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6684003d80 con 0x7f6694104920 2023-11-16T18:15:43.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.758+0000 7f66837fe700 1 -- 172.21.15.2:0/3818296956 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6684010470 con 0x7f6694104920 2023-11-16T18:15:43.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.758+0000 7f66837fe700 1 -- 172.21.15.2:0/3818296956 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6684010750 con 0x7f6694104920 2023-11-16T18:15:43.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.758+0000 7f6698d2e700 1 -- 172.21.15.2:0/3818296956 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6694105f60 con 0x7f6694104920 2023-11-16T18:15:43.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.758+0000 7f6698d2e700 1 -- 172.21.15.2:0/3818296956 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6694105dd0 con 0x7f6694104920 2023-11-16T18:15:43.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.758+0000 7f66837fe700 1 -- 172.21.15.2:0/3818296956 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6684010c00 con 0x7f6694104920 2023-11-16T18:15:43.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.758+0000 7f66837fe700 1 -- 172.21.15.2:0/3818296956 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f668402c3f0 con 0x7f6694104920 2023-11-16T18:15:43.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.759+0000 7f6698d2e700 1 -- 172.21.15.2:0/3818296956 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6694190620 con 0x7f6694104920 2023-11-16T18:15:43.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.762+0000 7f66837fe700 1 -- 172.21.15.2:0/3818296956 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f6684023440 con 0x7f6694104920 2023-11-16T18:15:43.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.888+0000 7f6698d2e700 1 -- 172.21.15.2:0/3818296956 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f669419be10 con 0x7f6694104920 2023-11-16T18:15:43.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.892+0000 7f66837fe700 1 -- 172.21.15.2:0/3818296956 <== mon.0 v2:172.21.15.2:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66840235e0 con 0x7f6694104920 2023-11-16T18:15:43.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.892+0000 7f66837fe700 1 -- 172.21.15.2:0/3818296956 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+427 (secure 0 0 0) 0x7f668402c600 con 0x7f6694104920 2023-11-16T18:15:43.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.893+0000 7f6698d2e700 1 -- 172.21.15.2:0/3818296956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 msgr2=0x7f669419b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:43.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.893+0000 7f6698d2e700 1 --2- 172.21.15.2:0/3818296956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 0x7f669419b160 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f668400b340 tx=0x7f6684004750 comp rx=0 tx=0).stop 2023-11-16T18:15:43.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.893+0000 7f6698d2e700 1 -- 172.21.15.2:0/3818296956 shutdown_connections 2023-11-16T18:15:43.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.893+0000 7f6698d2e700 1 --2- 172.21.15.2:0/3818296956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6694104920 0x7f669419b160 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:43.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.893+0000 7f6698d2e700 1 -- 172.21.15.2:0/3818296956 >> 172.21.15.2:0/3818296956 conn(0x7f66940ffed0 msgr2=0x7f66941021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:43.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.893+0000 7f6698d2e700 1 -- 172.21.15.2:0/3818296956 shutdown_connections 2023-11-16T18:15:43.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:42.893+0000 7f6698d2e700 1 -- 172.21.15.2:0/3818296956 wait complete. 2023-11-16T18:15:43.252 INFO:teuthology.orchestra.run.smithi002.stdout:Generating new minimal ceph.conf... 2023-11-16T18:15:43.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:43 smithi002 ceph-mon[98313]: from='client.? 172.21.15.2:0/3818296956' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-16T18:15:43.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:43 smithi002 ceph-mon[98313]: from='client.? 172.21.15.2:0/3818296956' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-16T18:15:44.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.656+0000 7f5b87626700 1 Processor -- start 2023-11-16T18:15:44.184 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.657+0000 7f5b87626700 1 -- start start 2023-11-16T18:15:44.184 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.657+0000 7f5b87626700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 0x7f5b80104690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:44.185 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.657+0000 7f5b87626700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b80079e60 con 0x7f5b80104280 2023-11-16T18:15:44.185 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.657+0000 7f5b853c2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 0x7f5b80104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:44.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.657+0000 7f5b853c2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 0x7f5b80104690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46480/0 (socket says 172.21.15.2:46480) 2023-11-16T18:15:44.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.657+0000 7f5b853c2700 1 -- 172.21.15.2:0/663315661 learned_addr learned my addr 172.21.15.2:0/663315661 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:44.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.657+0000 7f5b853c2700 1 -- 172.21.15.2:0/663315661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b80104bd0 con 0x7f5b80104280 2023-11-16T18:15:44.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.657+0000 7f5b853c2700 1 --2- 172.21.15.2:0/663315661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 0x7f5b80104690 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c0098a0 tx=0x7f5b7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f777fc126eb024d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:44.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.658+0000 7f5b77fff700 1 -- 172.21.15.2:0/663315661 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b7c004160 con 0x7f5b80104280 2023-11-16T18:15:44.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.658+0000 7f5b77fff700 1 -- 172.21.15.2:0/663315661 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b7c0042c0 con 0x7f5b80104280 2023-11-16T18:15:44.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.658+0000 7f5b77fff700 1 -- 172.21.15.2:0/663315661 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b7c011a00 con 0x7f5b80104280 2023-11-16T18:15:44.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.658+0000 7f5b87626700 1 -- 172.21.15.2:0/663315661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 msgr2=0x7f5b80104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:44.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.658+0000 7f5b87626700 1 --2- 172.21.15.2:0/663315661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 0x7f5b80104690 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c0098a0 tx=0x7f5b7c009bb0 comp rx=0 tx=0).stop 2023-11-16T18:15:44.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.658+0000 7f5b87626700 1 -- 172.21.15.2:0/663315661 shutdown_connections 2023-11-16T18:15:44.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.658+0000 7f5b87626700 1 --2- 172.21.15.2:0/663315661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 0x7f5b80104690 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:44.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.658+0000 7f5b87626700 1 -- 172.21.15.2:0/663315661 >> 172.21.15.2:0/663315661 conn(0x7f5b800ffed0 msgr2=0x7f5b80102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:44.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.658+0000 7f5b87626700 1 -- 172.21.15.2:0/663315661 shutdown_connections 2023-11-16T18:15:44.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.658+0000 7f5b87626700 1 -- 172.21.15.2:0/663315661 wait complete. 2023-11-16T18:15:44.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.659+0000 7f5b87626700 1 Processor -- start 2023-11-16T18:15:44.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.659+0000 7f5b87626700 1 -- start start 2023-11-16T18:15:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.659+0000 7f5b87626700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 0x7f5b8019b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.659+0000 7f5b87626700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8019be70 con 0x7f5b80104280 2023-11-16T18:15:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.659+0000 7f5b853c2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 0x7f5b8019b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.659+0000 7f5b853c2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 0x7f5b8019b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46482/0 (socket says 172.21.15.2:46482) 2023-11-16T18:15:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.659+0000 7f5b853c2700 1 -- 172.21.15.2:0/506174336 learned_addr learned my addr 172.21.15.2:0/506174336 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.660+0000 7f5b853c2700 1 -- 172.21.15.2:0/506174336 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c009550 con 0x7f5b80104280 2023-11-16T18:15:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.660+0000 7f5b853c2700 1 --2- 172.21.15.2:0/506174336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 0x7f5b8019b930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c006af0 tx=0x7f5b7c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.660+0000 7f5b767fc700 1 -- 172.21.15.2:0/506174336 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b7c003a40 con 0x7f5b80104280 2023-11-16T18:15:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.660+0000 7f5b767fc700 1 -- 172.21.15.2:0/506174336 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b7c003ba0 con 0x7f5b80104280 2023-11-16T18:15:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.660+0000 7f5b87626700 1 -- 172.21.15.2:0/506174336 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b80105bd0 con 0x7f5b80104280 2023-11-16T18:15:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.660+0000 7f5b767fc700 1 -- 172.21.15.2:0/506174336 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b7c01a6e0 con 0x7f5b80104280 2023-11-16T18:15:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.660+0000 7f5b87626700 1 -- 172.21.15.2:0/506174336 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b80105a40 con 0x7f5b80104280 2023-11-16T18:15:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.660+0000 7f5b767fc700 1 -- 172.21.15.2:0/506174336 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5b7c01ac50 con 0x7f5b80104280 2023-11-16T18:15:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.660+0000 7f5b767fc700 1 -- 172.21.15.2:0/506174336 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5b7c019700 con 0x7f5b80104280 2023-11-16T18:15:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.661+0000 7f5b87626700 1 -- 172.21.15.2:0/506174336 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b80105d60 con 0x7f5b80104280 2023-11-16T18:15:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.664+0000 7f5b767fc700 1 -- 172.21.15.2:0/506174336 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f5b7c024a60 con 0x7f5b80104280 2023-11-16T18:15:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.792+0000 7f5b87626700 1 -- 172.21.15.2:0/506174336 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f5b80195210 con 0x7f5b80104280 2023-11-16T18:15:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.792+0000 7f5b767fc700 1 -- 172.21.15.2:0/506174336 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7f5b7c024c00 con 0x7f5b80104280 2023-11-16T18:15:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.793+0000 7f5b87626700 1 -- 172.21.15.2:0/506174336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 msgr2=0x7f5b8019b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.793+0000 7f5b87626700 1 --2- 172.21.15.2:0/506174336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 0x7f5b8019b930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c006af0 tx=0x7f5b7c011ad0 comp rx=0 tx=0).stop 2023-11-16T18:15:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.793+0000 7f5b87626700 1 -- 172.21.15.2:0/506174336 shutdown_connections 2023-11-16T18:15:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.793+0000 7f5b87626700 1 --2- 172.21.15.2:0/506174336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b80104280 0x7f5b8019b930 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.793+0000 7f5b87626700 1 -- 172.21.15.2:0/506174336 >> 172.21.15.2:0/506174336 conn(0x7f5b800ffed0 msgr2=0x7f5b8018fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.793+0000 7f5b87626700 1 -- 172.21.15.2:0/506174336 shutdown_connections 2023-11-16T18:15:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:43.793+0000 7f5b87626700 1 -- 172.21.15.2:0/506174336 wait complete. 2023-11-16T18:15:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:Restarting the monitor... 2023-11-16T18:15:44.451 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:44 smithi002 systemd[1]: Stopping Ceph mon.a for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:15:44.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:44 smithi002 ceph-mon[98313]: from='client.? 172.21.15.2:0/506174336' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:15:44.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:44 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-a[98289]: 2023-11-16T18:15:44.500+0000 7fb635db1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:15:44.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:44 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-a[98289]: 2023-11-16T18:15:44.500+0000 7fb635db1700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-16T18:15:45.151 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:44 smithi002 bash[98644]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-a 2023-11-16T18:15:45.439 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 systemd[1]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.a.service: Succeeded. 2023-11-16T18:15:45.439 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 systemd[1]: Stopped Ceph mon.a for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:15:45.440 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 systemd[1]: Starting Ceph mon.a for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:15:45.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 podman[98767]: 2023-11-16T18:15:45.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 bash[98767]: a0ce6cc9456c825872fe8dd8e8a0911de5449c987a6dabbd2bcf85e1730234b2 2023-11-16T18:15:45.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: set uid:gid to 167:167 (ceph:ceph) 2023-11-16T18:15:45.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev), process ceph-mon, pid 7 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: pidfile_write: ignore empty --pid-file 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: load: jerasure load: lrc 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: RocksDB version: 7.9.2 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Git sha 0 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Compile date 2023-11-15 23:53:54 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: DB SUMMARY 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: DB Session ID: 8N246I84VUAR9BVBMS6F 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: CURRENT file: CURRENT 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: IDENTITY file: IDENTITY 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98685 ; 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.error_if_exists: 0 2023-11-16T18:15:45.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.create_if_missing: 0 2023-11-16T18:15:45.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.paranoid_checks: 1 2023-11-16T18:15:45.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.flush_verify_memtable_count: 1 2023-11-16T18:15:45.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-16T18:15:45.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-11-16T18:15:45.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.env: 0x55bbee547720 2023-11-16T18:15:45.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.fs: PosixFileSystem 2023-11-16T18:15:45.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.info_log: 0x55bbefea73c0 2023-11-16T18:15:45.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_file_opening_threads: 16 2023-11-16T18:15:45.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.statistics: (nil) 2023-11-16T18:15:45.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.use_fsync: 0 2023-11-16T18:15:45.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_log_file_size: 0 2023-11-16T18:15:45.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T18:15:45.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T18:15:45.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.keep_log_file_num: 1000 2023-11-16T18:15:45.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.recycle_log_file_num: 0 2023-11-16T18:15:45.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.allow_fallocate: 1 2023-11-16T18:15:45.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.allow_mmap_reads: 0 2023-11-16T18:15:45.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.allow_mmap_writes: 0 2023-11-16T18:15:45.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.use_direct_reads: 0 2023-11-16T18:15:45.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T18:15:45.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.create_missing_column_families: 0 2023-11-16T18:15:45.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.db_log_dir: 2023-11-16T18:15:45.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.wal_dir: 2023-11-16T18:15:45.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T18:15:45.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T18:15:45.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T18:15:45.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T18:15:45.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T18:15:45.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T18:15:45.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.advise_random_on_open: 1 2023-11-16T18:15:45.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.db_write_buffer_size: 0 2023-11-16T18:15:45.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.write_buffer_manager: 0x55bbef1183c0 2023-11-16T18:15:45.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T18:15:45.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T18:15:45.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T18:15:45.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.rate_limiter: (nil) 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.wal_recovery_mode: 2 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.enable_thread_tracking: 0 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.enable_pipelined_write: 0 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.unordered_write: 0 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.row_cache: None 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.wal_filter: None 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.allow_ingest_behind: 0 2023-11-16T18:15:45.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.two_write_queues: 0 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.manual_wal_flush: 0 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.wal_compression: 0 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.atomic_flush: 0 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.log_readahead_size: 0 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.best_efforts_recovery: 0 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.allow_data_in_errors: 0 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.db_host_id: __hostname__ 2023-11-16T18:15:45.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.enforce_single_del_contracts: true 2023-11-16T18:15:45.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_background_jobs: 2 2023-11-16T18:15:45.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_background_compactions: -1 2023-11-16T18:15:45.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_subcompactions: 1 2023-11-16T18:15:45.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T18:15:45.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T18:15:45.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T18:15:45.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_total_wal_size: 0 2023-11-16T18:15:45.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T18:15:45.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T18:15:45.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T18:15:45.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T18:15:45.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_open_files: -1 2023-11-16T18:15:45.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.bytes_per_sync: 0 2023-11-16T18:15:45.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T18:15:45.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T18:15:45.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compaction_readahead_size: 0 2023-11-16T18:15:45.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_background_flushes: -1 2023-11-16T18:15:45.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Compression algorithms supported: 2023-11-16T18:15:45.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T18:15:45.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: kZSTD supported: 0 2023-11-16T18:15:45.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: kXpressCompression supported: 0 2023-11-16T18:15:45.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: kLZ4HCCompression supported: 1 2023-11-16T18:15:45.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: kZlibCompression supported: 1 2023-11-16T18:15:45.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: kSnappyCompression supported: 1 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: kLZ4Compression supported: 1 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: kBZip2Compression supported: 0 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: DMutex implementation: pthread_mutex_t 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.merge_operator: 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compaction_filter: None 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compaction_filter_factory: None 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.sst_partitioner_factory: None 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T18:15:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bbefea74e0) 2023-11-16T18:15:45.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks: 1 2023-11-16T18:15:45.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T18:15:45.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T18:15:45.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_top_level_index_and_filter: 1 2023-11-16T18:15:45.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_type: 0 2023-11-16T18:15:45.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_index_type: 0 2023-11-16T18:15:45.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_shortening: 1 2023-11-16T18:15:45.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2023-11-16T18:15:45.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: checksum: 4 2023-11-16T18:15:45.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: no_block_cache: 0 2023-11-16T18:15:45.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache: 0x55bbef1b91f0 2023-11-16T18:15:45.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_name: BinnedLRUCache 2023-11-16T18:15:45.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_options: 2023-11-16T18:15:45.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: capacity : 536870912 2023-11-16T18:15:45.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_shard_bits : 4 2023-11-16T18:15:45.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: strict_capacity_limit : 0 2023-11-16T18:15:45.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: high_pri_pool_ratio: 0.000 2023-11-16T18:15:45.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_compressed: (nil) 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: persistent_cache: (nil) 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size: 4096 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size_deviation: 10 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_restart_interval: 16 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_block_restart_interval: 1 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: metadata_block_size: 4096 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: partition_filters: 0 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: use_delta_encoding: 1 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: filter_policy: bloomfilter 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: whole_key_filtering: 1 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: verify_compression: 0 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: read_amp_bytes_per_bit: 0 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: format_version: 5 2023-11-16T18:15:45.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_index_compression: 1 2023-11-16T18:15:45.757 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_align: 0 2023-11-16T18:15:45.757 INFO:journalctl@ceph.mon.a.smithi002.stdout: max_auto_readahead_size: 262144 2023-11-16T18:15:45.757 INFO:journalctl@ceph.mon.a.smithi002.stdout: prepopulate_block_cache: 0 2023-11-16T18:15:45.757 INFO:journalctl@ceph.mon.a.smithi002.stdout: initial_auto_readahead_size: 8192 2023-11-16T18:15:45.757 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_file_reads_for_auto_readahead: 2 2023-11-16T18:15:45.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.write_buffer_size: 33554432 2023-11-16T18:15:45.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_write_buffer_number: 2 2023-11-16T18:15:45.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compression: NoCompression 2023-11-16T18:15:45.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.bottommost_compression: Disabled 2023-11-16T18:15:45.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.prefix_extractor: nullptr 2023-11-16T18:15:45.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T18:15:45.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.num_levels: 7 2023-11-16T18:15:45.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T18:15:45.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T18:15:45.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T18:15:45.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T18:15:45.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T18:15:45.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T18:15:45.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T18:15:45.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T18:15:45.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-16T18:15:45.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T18:15:45.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-16T18:15:45.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-16T18:15:45.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T18:15:45.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compression_opts.level: 32767 2023-11-16T18:15:45.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compression_opts.strategy: 0 2023-11-16T18:15:45.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T18:15:45.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T18:15:45.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-16T18:15:45.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-16T18:15:45.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compression_opts.enabled: false 2023-11-16T18:15:45.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-16T18:15:45.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T18:15:45.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T18:15:45.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T18:15:45.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.target_file_size_base: 67108864 2023-11-16T18:15:45.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T18:15:45.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T18:15:45.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T18:15:45.785 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T18:15:45.785 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T18:15:45.786 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T18:15:45.786 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T18:15:45.786 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T18:15:45.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T18:15:45.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T18:15:45.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T18:15:45.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T18:15:45.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T18:15:45.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-16T18:15:45.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.arena_block_size: 1048576 2023-11-16T18:15:45.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T18:15:45.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T18:15:45.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.disable_auto_compactions: 0 2023-11-16T18:15:45.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T18:15:45.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T18:15:45.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T18:15:45.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T18:15:45.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T18:15:45.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.table_properties_collectors: 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.inplace_update_support: 0 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.bloom_locality: 0 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.max_successive_merges: 0 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T18:15:45.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.paranoid_file_checks: 0 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.force_consistency_checks: 1 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.report_bg_io_stats: 0 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.ttl: 2592000 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.enable_blob_files: false 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.min_blob_size: 0 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.blob_file_size: 268435456 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-16T18:15:45.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-16T18:15:45.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.blob_file_starting_level: 0 2023-11-16T18:15:45.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-16T18:15:45.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-11-16T18:15:45.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-11-16T18:15:45.805 INFO:teuthology.orchestra.run.smithi002.stdout:Setting mon public_network to 172.21.0.0/20 2023-11-16T18:15:45.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dc6bda62-4033-4939-a0d6-f0f23f7a91d4 2023-11-16T18:15:45.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700158545671980, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-11-16T18:15:45.841 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-11-16T18:15:45.841 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700158545672970, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93938, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91937, "index_size": 283, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13658, "raw_average_key_size": 50, "raw_value_size": 84478, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1700158545, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dc6bda62-4033-4939-a0d6-f0f23f7a91d4", "db_session_id": "8N246I84VUAR9BVBMS6F", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-11-16T18:15:45.841 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700158545673042, "job": 1, "event": "recovery_finished"} 2023-11-16T18:15:45.841 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-11-16T18:15:45.841 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-16T18:15:45.842 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bbef256000 2023-11-16T18:15:45.842 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: DB pointer 0x55bbef242000 2023-11-16T18:15:45.842 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-16T18:15:45.842 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-16T18:15:45.842 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** DB Stats ** 2023-11-16T18:15:45.842 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T18:15:45.842 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T18:15:45.842 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T18:15:45.842 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T18:15:45.842 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T18:15:45.843 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T18:15:45.843 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T18:15:45.843 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2023-11-16T18:15:45.843 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2023-11-16T18:15:45.843 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T18:15:45.843 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-16T18:15:45.843 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 2/0 93.57 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T18:15:45.843 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 93.57 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T18:15:45.843 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T18:15:45.843 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2023-11-16T18:15:45.843 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2023-11-16T18:15:45.843 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T18:15:45.843 INFO:journalctl@ceph.mon.a.smithi002.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T18:15:45.844 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 155.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T18:15:45.844 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2023-11-16T18:15:45.844 INFO:journalctl@ceph.mon.a.smithi002.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-16T18:15:45.844 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2023-11-16T18:15:45.844 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T18:15:45.844 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T18:15:45.844 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T18:15:45.844 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T18:15:45.844 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T18:15:45.844 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-16T18:15:45.844 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 27.97 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T18:15:45.844 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 27.97 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T18:15:45.844 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout: Block cache BinnedLRUCache@0x55bbef1b91f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: starting mon.a rank 0 at public addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] at bind addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: mon.a@-1(???) e1 preinit fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: mon.a@-1(???).mds e1 new map 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: mon.a@-1(???).mds e1 print_map 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout: e1 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout: legacy client fscid: -1 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2023-11-16T18:15:45.845 INFO:journalctl@ceph.mon.a.smithi002.stdout: No filesystems configured 2023-11-16T18:15:45.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T18:15:45.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T18:15:45.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T18:15:45.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T18:15:45.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-16T18:15:45.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 systemd[1]: Started Ceph mon.a for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:15:45.987 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-16T18:15:45.988 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} removed_ranks: {} 2023-11-16T18:15:45.988 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: fsmap 2023-11-16T18:15:45.988 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: osdmap e1: 0 total, 0 up, 0 in 2023-11-16T18:15:45.988 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:45 smithi002 ceph-mon[98807]: mgrmap e1: no daemons active 2023-11-16T18:15:46.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.241+0000 7fb6b36fc700 1 Processor -- start 2023-11-16T18:15:46.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.241+0000 7fb6b36fc700 1 -- start start 2023-11-16T18:15:46.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.242+0000 7fb6b36fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 0x7fb6ac104120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:46.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.242+0000 7fb6b36fc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac1046f0 con 0x7fb6ac103d10 2023-11-16T18:15:46.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.242+0000 7fb6b1498700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 0x7fb6ac104120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:46.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.242+0000 7fb6b1498700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 0x7fb6ac104120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46484/0 (socket says 172.21.15.2:46484) 2023-11-16T18:15:46.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.242+0000 7fb6b1498700 1 -- 172.21.15.2:0/1896134851 learned_addr learned my addr 172.21.15.2:0/1896134851 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:46.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.243+0000 7fb6b1498700 1 -- 172.21.15.2:0/1896134851 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac106ca0 con 0x7fb6ac103d10 2023-11-16T18:15:46.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.243+0000 7fb6b1498700 1 --2- 172.21.15.2:0/1896134851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 0x7fb6ac104120 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb69c0098a0 tx=0x7fb69c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11808627d409d69 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:46.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.243+0000 7fb6a3fff700 1 -- 172.21.15.2:0/1896134851 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb69c004160 con 0x7fb6ac103d10 2023-11-16T18:15:46.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.243+0000 7fb6a3fff700 1 -- 172.21.15.2:0/1896134851 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb69c0042c0 con 0x7fb6ac103d10 2023-11-16T18:15:46.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.243+0000 7fb6a3fff700 1 -- 172.21.15.2:0/1896134851 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb69c011a00 con 0x7fb6ac103d10 2023-11-16T18:15:46.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.244+0000 7fb6b36fc700 1 -- 172.21.15.2:0/1896134851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 msgr2=0x7fb6ac104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:46.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.244+0000 7fb6b36fc700 1 --2- 172.21.15.2:0/1896134851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 0x7fb6ac104120 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb69c0098a0 tx=0x7fb69c009bb0 comp rx=0 tx=0).stop 2023-11-16T18:15:46.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.244+0000 7fb6b36fc700 1 -- 172.21.15.2:0/1896134851 shutdown_connections 2023-11-16T18:15:46.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.244+0000 7fb6b36fc700 1 --2- 172.21.15.2:0/1896134851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 0x7fb6ac104120 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:46.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.244+0000 7fb6b36fc700 1 -- 172.21.15.2:0/1896134851 >> 172.21.15.2:0/1896134851 conn(0x7fb6ac0ff650 msgr2=0x7fb6ac101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:46.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.244+0000 7fb6b36fc700 1 -- 172.21.15.2:0/1896134851 shutdown_connections 2023-11-16T18:15:46.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.244+0000 7fb6b36fc700 1 -- 172.21.15.2:0/1896134851 wait complete. 2023-11-16T18:15:46.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.245+0000 7fb6b36fc700 1 Processor -- start 2023-11-16T18:15:46.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.245+0000 7fb6b36fc700 1 -- start start 2023-11-16T18:15:46.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.245+0000 7fb6b36fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 0x7fb6ac1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:46.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.245+0000 7fb6b36fc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac199e20 con 0x7fb6ac103d10 2023-11-16T18:15:46.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.245+0000 7fb6b1498700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 0x7fb6ac1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:46.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.245+0000 7fb6b1498700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 0x7fb6ac1998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46486/0 (socket says 172.21.15.2:46486) 2023-11-16T18:15:46.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.245+0000 7fb6b1498700 1 -- 172.21.15.2:0/937616733 learned_addr learned my addr 172.21.15.2:0/937616733 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:46.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.245+0000 7fb6b1498700 1 -- 172.21.15.2:0/937616733 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c009550 con 0x7fb6ac103d10 2023-11-16T18:15:46.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.245+0000 7fb6b1498700 1 --2- 172.21.15.2:0/937616733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 0x7fb6ac1998e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb69c006af0 tx=0x7fb69c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:46.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.246+0000 7fb6a27fc700 1 -- 172.21.15.2:0/937616733 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb69c003a40 con 0x7fb6ac103d10 2023-11-16T18:15:46.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.246+0000 7fb6b36fc700 1 -- 172.21.15.2:0/937616733 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ac109ed0 con 0x7fb6ac103d10 2023-11-16T18:15:46.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.246+0000 7fb6a27fc700 1 -- 172.21.15.2:0/937616733 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb69c003ba0 con 0x7fb6ac103d10 2023-11-16T18:15:46.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.246+0000 7fb6a27fc700 1 -- 172.21.15.2:0/937616733 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb69c01a700 con 0x7fb6ac103d10 2023-11-16T18:15:46.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.246+0000 7fb6b36fc700 1 -- 172.21.15.2:0/937616733 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ac10a060 con 0x7fb6ac103d10 2023-11-16T18:15:46.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.246+0000 7fb6a27fc700 1 -- 172.21.15.2:0/937616733 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb69c01a930 con 0x7fb6ac103d10 2023-11-16T18:15:46.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.246+0000 7fb6a27fc700 1 -- 172.21.15.2:0/937616733 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb69c02dd50 con 0x7fb6ac103d10 2023-11-16T18:15:46.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.247+0000 7fb6b36fc700 1 -- 172.21.15.2:0/937616733 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ac107b10 con 0x7fb6ac103d10 2023-11-16T18:15:46.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.249+0000 7fb6a27fc700 1 -- 172.21.15.2:0/937616733 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fb6ac107b10 con 0x7fb6ac103d10 2023-11-16T18:15:46.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.374+0000 7fb6b36fc700 1 -- 172.21.15.2:0/937616733 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fb6ac19a530 con 0x7fb6ac103d10 2023-11-16T18:15:46.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.377+0000 7fb6a27fc700 1 -- 172.21.15.2:0/937616733 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb69c0248d0 con 0x7fb6ac103d10 2023-11-16T18:15:46.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.377+0000 7fb6b36fc700 1 -- 172.21.15.2:0/937616733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 msgr2=0x7fb6ac1998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:46.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.377+0000 7fb6b36fc700 1 --2- 172.21.15.2:0/937616733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 0x7fb6ac1998e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb69c006af0 tx=0x7fb69c011ad0 comp rx=0 tx=0).stop 2023-11-16T18:15:46.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.377+0000 7fb6b36fc700 1 -- 172.21.15.2:0/937616733 shutdown_connections 2023-11-16T18:15:46.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.377+0000 7fb6b36fc700 1 --2- 172.21.15.2:0/937616733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6ac103d10 0x7fb6ac1998e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:46.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.377+0000 7fb6b36fc700 1 -- 172.21.15.2:0/937616733 >> 172.21.15.2:0/937616733 conn(0x7fb6ac0ff650 msgr2=0x7fb6ac100410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:46.697 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.377+0000 7fb6b36fc700 1 -- 172.21.15.2:0/937616733 shutdown_connections 2023-11-16T18:15:46.697 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:46.378+0000 7fb6b36fc700 1 -- 172.21.15.2:0/937616733 wait complete. 2023-11-16T18:15:46.697 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote config to /etc/ceph/ceph.conf 2023-11-16T18:15:46.697 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-16T18:15:46.697 INFO:teuthology.orchestra.run.smithi002.stdout:Creating mgr... 2023-11-16T18:15:46.697 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying port 9283 ... 2023-11-16T18:15:46.697 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying port 8765 ... 2023-11-16T18:15:46.827 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.x 2023-11-16T18:15:46.827 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to reset failed state of unit ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.x.service: Unit ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.x.service not loaded. 2023-11-16T18:15:46.923 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7.target.wants/ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.x.service → /etc/systemd/system/ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@.service. 2023-11-16T18:15:46.928 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:46 smithi002 systemd[1]: Starting Ceph mgr.x for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:15:47.433 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T18:15:47.433 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2023-11-16T18:15:47.441 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T18:15:47.442 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2023-11-16T18:15:47.442 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2023-11-16T18:15:47.442 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to enable service . firewalld.service is not available 2023-11-16T18:15:47.453 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T18:15:47.453 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2023-11-16T18:15:47.466 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T18:15:47.467 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2023-11-16T18:15:47.467 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2023-11-16T18:15:47.467 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-11-16T18:15:47.467 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr to start... 2023-11-16T18:15:47.468 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr... 2023-11-16T18:15:47.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:47 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/937616733' entity='client.admin' 2023-11-16T18:15:47.575 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:47 smithi002 podman[99036]: 2023-11-16T18:15:47.576 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:47 smithi002 bash[99036]: a85ff49ab4d1e3053e657f25ac4a974f069f19940bea276be7ee57f975d4e023 2023-11-16T18:15:47.576 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:47 smithi002 systemd[1]: Started Ceph mgr.x for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:15:47.858 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:47.571+0000 7f410ea99200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T18:15:47.858 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:47.671+0000 7f410ea99200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T18:15:47.867 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:47.784+0000 7f410ea99200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T18:15:48.259 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:47.898+0000 7f410ea99200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T18:15:48.259 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:48 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:48.108+0000 7f410ea99200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T18:15:48.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T18:15:48.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T18:15:48.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "08160fac-84ac-11ee-8dbe-212e2dc638e7", 2023-11-16T18:15:48.430 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2023-11-16T18:15:48.430 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-16T18:15:48.430 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-16T18:15:48.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-16T18:15:48.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:48.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-16T18:15:48.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-16T18:15:48.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2023-11-16T18:15:48.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T18:15:48.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-16T18:15:48.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2023-11-16T18:15:48.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T18:15:48.438 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-11-16T18:15:48.444 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-16T18:15:48.444 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:15:48.470 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-11-16T18:15:48.471 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-16T18:15:48.484 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:48.484 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-16T18:15:48.485 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:15:48.485 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-16T18:15:48.485 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-16T18:15:48.486 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-16T18:15:48.486 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-16T18:15:48.486 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-16T18:15:48.487 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-16T18:15:48.487 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:48.487 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-16T18:15:48.488 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-16T18:15:48.489 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-16T18:15:48.489 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-16T18:15:48.490 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-16T18:15:48.491 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-16T18:15:48.494 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-16T18:15:48.494 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-16T18:15:48.503 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-16T18:15:48.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:48.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-16T18:15:48.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:15:48.537 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-16T18:15:48.538 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-16T18:15:48.538 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:48.538 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-16T18:15:48.538 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2023-11-16T18:15:48.539 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-16T18:15:48.563 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-16T18:15:48.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-16T18:15:48.594 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-16T18:15:48.596 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2023-11-16T18:15:48.596 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T18:15:48.597 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T18:15:48.598 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:48.598 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-16T18:15:48.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:15:48.602 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2023-11-16T18:15:41.444940+0000", 2023-11-16T18:15:48.602 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T18:15:48.603 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:48.603 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-16T18:15:48.604 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T18:15:48.604 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.976+0000 7fc284de7700 1 Processor -- start 2023-11-16T18:15:48.605 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.976+0000 7fc284de7700 1 -- start start 2023-11-16T18:15:48.605 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.977+0000 7fc284de7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 0x7fc2801088f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:48.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.977+0000 7fc284de7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc280079de0 con 0x7fc280106510 2023-11-16T18:15:48.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.977+0000 7fc27e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 0x7fc2801088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:48.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.977+0000 7fc27e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 0x7fc2801088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46492/0 (socket says 172.21.15.2:46492) 2023-11-16T18:15:48.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.977+0000 7fc27e59c700 1 -- 172.21.15.2:0/1870841538 learned_addr learned my addr 172.21.15.2:0/1870841538 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:48.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.977+0000 7fc27e59c700 1 -- 172.21.15.2:0/1870841538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc280108e30 con 0x7fc280106510 2023-11-16T18:15:48.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.977+0000 7fc27e59c700 1 --2- 172.21.15.2:0/1870841538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 0x7fc2801088f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc2700098a0 tx=0x7fc270009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117c2881169e0be0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:48.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.978+0000 7fc27d59a700 1 -- 172.21.15.2:0/1870841538 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc270004160 con 0x7fc280106510 2023-11-16T18:15:48.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.978+0000 7fc27d59a700 1 -- 172.21.15.2:0/1870841538 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2700042c0 con 0x7fc280106510 2023-11-16T18:15:48.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.978+0000 7fc27d59a700 1 -- 172.21.15.2:0/1870841538 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc270011a00 con 0x7fc280106510 2023-11-16T18:15:48.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.978+0000 7fc284de7700 1 -- 172.21.15.2:0/1870841538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 msgr2=0x7fc2801088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:48.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.978+0000 7fc284de7700 1 --2- 172.21.15.2:0/1870841538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 0x7fc2801088f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc2700098a0 tx=0x7fc270009bb0 comp rx=0 tx=0).stop 2023-11-16T18:15:48.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.978+0000 7fc284de7700 1 -- 172.21.15.2:0/1870841538 shutdown_connections 2023-11-16T18:15:48.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.978+0000 7fc284de7700 1 --2- 172.21.15.2:0/1870841538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 0x7fc2801088f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:48.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.978+0000 7fc284de7700 1 -- 172.21.15.2:0/1870841538 >> 172.21.15.2:0/1870841538 conn(0x7fc2800ffed0 msgr2=0x7fc280102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:48.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.979+0000 7fc284de7700 1 -- 172.21.15.2:0/1870841538 shutdown_connections 2023-11-16T18:15:48.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.979+0000 7fc284de7700 1 -- 172.21.15.2:0/1870841538 wait complete. 2023-11-16T18:15:48.630 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.980+0000 7fc284de7700 1 Processor -- start 2023-11-16T18:15:48.630 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.980+0000 7fc284de7700 1 -- start start 2023-11-16T18:15:48.630 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.980+0000 7fc284de7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 0x7fc280199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:48.630 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.980+0000 7fc284de7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc280199cb0 con 0x7fc280106510 2023-11-16T18:15:48.630 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.980+0000 7fc27e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 0x7fc280199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:48.630 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.980+0000 7fc27e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 0x7fc280199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46494/0 (socket says 172.21.15.2:46494) 2023-11-16T18:15:48.630 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.980+0000 7fc27e59c700 1 -- 172.21.15.2:0/3259594547 learned_addr learned my addr 172.21.15.2:0/3259594547 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:48.630 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.981+0000 7fc27e59c700 1 -- 172.21.15.2:0/3259594547 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc270009550 con 0x7fc280106510 2023-11-16T18:15:48.630 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.981+0000 7fc27e59c700 1 --2- 172.21.15.2:0/3259594547 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 0x7fc280199770 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc270009fb0 tx=0x7fc270011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:48.630 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.981+0000 7fc26f7fe700 1 -- 172.21.15.2:0/3259594547 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc270003930 con 0x7fc280106510 2023-11-16T18:15:48.640 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.981+0000 7fc26f7fe700 1 -- 172.21.15.2:0/3259594547 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc270003a90 con 0x7fc280106510 2023-11-16T18:15:48.640 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.981+0000 7fc284de7700 1 -- 172.21.15.2:0/3259594547 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc280109d10 con 0x7fc280106510 2023-11-16T18:15:48.640 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.981+0000 7fc26f7fe700 1 -- 172.21.15.2:0/3259594547 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc270019610 con 0x7fc280106510 2023-11-16T18:15:48.640 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.981+0000 7fc284de7700 1 -- 172.21.15.2:0/3259594547 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc280109b80 con 0x7fc280106510 2023-11-16T18:15:48.640 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.981+0000 7fc26f7fe700 1 -- 172.21.15.2:0/3259594547 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc270019bf0 con 0x7fc280106510 2023-11-16T18:15:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.981+0000 7fc26f7fe700 1 -- 172.21.15.2:0/3259594547 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc2700243f0 con 0x7fc280106510 2023-11-16T18:15:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.982+0000 7fc284de7700 1 -- 172.21.15.2:0/3259594547 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc280062120 con 0x7fc280106510 2023-11-16T18:15:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:47.984+0000 7fc26f7fe700 1 -- 172.21.15.2:0/3259594547 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fc270003c00 con 0x7fc280106510 2023-11-16T18:15:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:48.117+0000 7fc284de7700 1 -- 172.21.15.2:0/3259594547 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc28019a1f0 con 0x7fc280106510 2023-11-16T18:15:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:48.117+0000 7fc26f7fe700 1 -- 172.21.15.2:0/3259594547 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fc27001a9d0 con 0x7fc280106510 2023-11-16T18:15:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:48.118+0000 7fc284de7700 1 -- 172.21.15.2:0/3259594547 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 msgr2=0x7fc280199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:48.118+0000 7fc284de7700 1 --2- 172.21.15.2:0/3259594547 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 0x7fc280199770 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc270009fb0 tx=0x7fc270011ad0 comp rx=0 tx=0).stop 2023-11-16T18:15:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:48.118+0000 7fc284de7700 1 -- 172.21.15.2:0/3259594547 shutdown_connections 2023-11-16T18:15:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:48.118+0000 7fc284de7700 1 --2- 172.21.15.2:0/3259594547 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc280106510 0x7fc280199770 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:48.118+0000 7fc284de7700 1 -- 172.21.15.2:0/3259594547 >> 172.21.15.2:0/3259594547 conn(0x7fc2800ffed0 msgr2=0x7fc280100cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:48.119+0000 7fc284de7700 1 -- 172.21.15.2:0/3259594547 shutdown_connections 2023-11-16T18:15:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:48.119+0000 7fc284de7700 1 -- 172.21.15.2:0/3259594547 wait complete. 2023-11-16T18:15:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (1/15)... 2023-11-16T18:15:48.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:48 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3259594547' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T18:15:48.778 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:48 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:48.518+0000 7f410ea99200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T18:15:48.778 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:48 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:48.626+0000 7f410ea99200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T18:15:49.106 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:48 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:48.776+0000 7f410ea99200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T18:15:49.106 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:48 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:48.891+0000 7f410ea99200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T18:15:49.855 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:49 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:49.400+0000 7f410ea99200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T18:15:50.355 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:50 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:50.094+0000 7f410ea99200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T18:15:50.356 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:50 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:50.284+0000 7f410ea99200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T18:15:51.593 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:51.258+0000 7f410ea99200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:15:51.594 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:51.369+0000 7f410ea99200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T18:15:51.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T18:15:51.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T18:15:51.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "08160fac-84ac-11ee-8dbe-212e2dc638e7", 2023-11-16T18:15:51.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2023-11-16T18:15:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-16T18:15:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-16T18:15:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-16T18:15:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-16T18:15:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-16T18:15:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2023-11-16T18:15:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T18:15:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-16T18:15:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2023-11-16T18:15:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T18:15:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-11-16T18:15:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-16T18:15:51.665 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:15:51.665 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-11-16T18:15:51.665 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-16T18:15:51.665 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:51.665 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-16T18:15:51.665 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:15:51.665 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-16T18:15:51.665 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-16T18:15:51.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-16T18:15:51.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-16T18:15:51.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-16T18:15:51.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-16T18:15:51.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:51.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-16T18:15:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-16T18:15:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-16T18:15:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-16T18:15:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-16T18:15:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-16T18:15:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-16T18:15:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-16T18:15:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-16T18:15:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-16T18:15:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:15:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-16T18:15:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-16T18:15:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-16T18:15:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2023-11-16T18:15:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-16T18:15:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-16T18:15:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-16T18:15:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-16T18:15:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2023-11-16T18:15:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T18:15:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T18:15:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-16T18:15:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:15:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2023-11-16T18:15:41.444940+0000", 2023-11-16T18:15:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T18:15:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-16T18:15:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T18:15:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.237+0000 7f4f27ab5700 1 Processor -- start 2023-11-16T18:15:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.237+0000 7f4f27ab5700 1 -- start start 2023-11-16T18:15:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.238+0000 7f4f27ab5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 0x7f4f20104d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.238+0000 7f4f27ab5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f20105300 con 0x7f4f20104920 2023-11-16T18:15:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.238+0000 7f4f25851700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 0x7f4f20104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.238+0000 7f4f25851700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 0x7f4f20104d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46496/0 (socket says 172.21.15.2:46496) 2023-11-16T18:15:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.238+0000 7f4f25851700 1 -- 172.21.15.2:0/3489011472 learned_addr learned my addr 172.21.15.2:0/3489011472 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.238+0000 7f4f25851700 1 -- 172.21.15.2:0/3489011472 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f20105b20 con 0x7f4f20104920 2023-11-16T18:15:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.238+0000 7f4f25851700 1 --2- 172.21.15.2:0/3489011472 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 0x7f4f20104d30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00bb40 tx=0x7f4f1c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f374e0ef2ee71df2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.239+0000 7f4f2484f700 1 -- 172.21.15.2:0/3489011472 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f1c00dbe0 con 0x7f4f20104920 2023-11-16T18:15:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.239+0000 7f4f2484f700 1 -- 172.21.15.2:0/3489011472 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f1c0044d0 con 0x7f4f20104920 2023-11-16T18:15:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.239+0000 7f4f2484f700 1 -- 172.21.15.2:0/3489011472 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f1c010470 con 0x7f4f20104920 2023-11-16T18:15:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.239+0000 7f4f27ab5700 1 -- 172.21.15.2:0/3489011472 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 msgr2=0x7f4f20104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.239+0000 7f4f27ab5700 1 --2- 172.21.15.2:0/3489011472 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 0x7f4f20104d30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00bb40 tx=0x7f4f1c00be50 comp rx=0 tx=0).stop 2023-11-16T18:15:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.239+0000 7f4f27ab5700 1 -- 172.21.15.2:0/3489011472 shutdown_connections 2023-11-16T18:15:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.239+0000 7f4f27ab5700 1 --2- 172.21.15.2:0/3489011472 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 0x7f4f20104d30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.239+0000 7f4f27ab5700 1 -- 172.21.15.2:0/3489011472 >> 172.21.15.2:0/3489011472 conn(0x7f4f200ffed0 msgr2=0x7f4f20102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.239+0000 7f4f27ab5700 1 -- 172.21.15.2:0/3489011472 shutdown_connections 2023-11-16T18:15:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.239+0000 7f4f27ab5700 1 -- 172.21.15.2:0/3489011472 wait complete. 2023-11-16T18:15:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.240+0000 7f4f27ab5700 1 Processor -- start 2023-11-16T18:15:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.240+0000 7f4f27ab5700 1 -- start start 2023-11-16T18:15:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.240+0000 7f4f27ab5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 0x7f4f2019b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.240+0000 7f4f27ab5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2019bee0 con 0x7f4f20104920 2023-11-16T18:15:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.240+0000 7f4f25851700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 0x7f4f2019b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.240+0000 7f4f25851700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 0x7f4f2019b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46498/0 (socket says 172.21.15.2:46498) 2023-11-16T18:15:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.240+0000 7f4f25851700 1 -- 172.21.15.2:0/2525250054 learned_addr learned my addr 172.21.15.2:0/2525250054 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.241+0000 7f4f25851700 1 -- 172.21.15.2:0/2525250054 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c00b7f0 con 0x7f4f20104920 2023-11-16T18:15:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.241+0000 7f4f25851700 1 --2- 172.21.15.2:0/2525250054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 0x7f4f2019b9a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c003a80 tx=0x7f4f1c003df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.241+0000 7f4f16ffd700 1 -- 172.21.15.2:0/2525250054 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f1c0106e0 con 0x7f4f20104920 2023-11-16T18:15:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.241+0000 7f4f16ffd700 1 -- 172.21.15.2:0/2525250054 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f1c010ce0 con 0x7f4f20104920 2023-11-16T18:15:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.241+0000 7f4f27ab5700 1 -- 172.21.15.2:0/2525250054 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f20078f80 con 0x7f4f20104920 2023-11-16T18:15:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.241+0000 7f4f16ffd700 1 -- 172.21.15.2:0/2525250054 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f1c019c40 con 0x7f4f20104920 2023-11-16T18:15:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.241+0000 7f4f27ab5700 1 -- 172.21.15.2:0/2525250054 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f201068d0 con 0x7f4f20104920 2023-11-16T18:15:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.242+0000 7f4f16ffd700 1 -- 172.21.15.2:0/2525250054 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4f1c02a450 con 0x7f4f20104920 2023-11-16T18:15:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.242+0000 7f4f27ab5700 1 -- 172.21.15.2:0/2525250054 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f201911b0 con 0x7f4f20104920 2023-11-16T18:15:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.242+0000 7f4f16ffd700 1 -- 172.21.15.2:0/2525250054 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4f1c021750 con 0x7f4f20104920 2023-11-16T18:15:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.244+0000 7f4f16ffd700 1 -- 172.21.15.2:0/2525250054 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f4f1c0193f0 con 0x7f4f20104920 2023-11-16T18:15:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.377+0000 7f4f27ab5700 1 -- 172.21.15.2:0/2525250054 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4f20062120 con 0x7f4f20104920 2023-11-16T18:15:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.377+0000 7f4f16ffd700 1 -- 172.21.15.2:0/2525250054 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f4f1c019590 con 0x7f4f20104920 2023-11-16T18:15:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.378+0000 7f4f27ab5700 1 -- 172.21.15.2:0/2525250054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 msgr2=0x7f4f2019b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.378+0000 7f4f27ab5700 1 --2- 172.21.15.2:0/2525250054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 0x7f4f2019b9a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c003a80 tx=0x7f4f1c003df0 comp rx=0 tx=0).stop 2023-11-16T18:15:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.378+0000 7f4f27ab5700 1 -- 172.21.15.2:0/2525250054 shutdown_connections 2023-11-16T18:15:51.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.378+0000 7f4f27ab5700 1 --2- 172.21.15.2:0/2525250054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20104920 0x7f4f2019b9a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:51.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.378+0000 7f4f27ab5700 1 -- 172.21.15.2:0/2525250054 >> 172.21.15.2:0/2525250054 conn(0x7f4f200ffed0 msgr2=0x7f4f2018e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:51.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.378+0000 7f4f27ab5700 1 -- 172.21.15.2:0/2525250054 shutdown_connections 2023-11-16T18:15:51.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:51.378+0000 7f4f27ab5700 1 -- 172.21.15.2:0/2525250054 wait complete. 2023-11-16T18:15:51.694 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (2/15)... 2023-11-16T18:15:51.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:51 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2525250054' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T18:15:51.861 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:51.592+0000 7f410ea99200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:15:52.193 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:51.937+0000 7f410ea99200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T18:15:52.605 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:52.188+0000 7f410ea99200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T18:15:52.606 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:52.368+0000 7f410ea99200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T18:15:53.106 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:52.693+0000 7f410ea99200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T18:15:53.839 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:53 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:53.512+0000 7f410ea99200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T18:15:53.840 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:53 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:53.668+0000 7f410ea99200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T18:15:54.105 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:53 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:53.917+0000 7f410ea99200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T18:15:54.105 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:54 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:54.054+0000 7f410ea99200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T18:15:54.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:54 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3460016866' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T18:15:54.604 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:54 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:54.341+0000 7f410ea99200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T18:15:54.634 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T18:15:54.634 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T18:15:54.635 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "08160fac-84ac-11ee-8dbe-212e2dc638e7", 2023-11-16T18:15:54.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2023-11-16T18:15:54.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-16T18:15:54.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-16T18:15:54.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-16T18:15:54.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:54.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-16T18:15:54.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-16T18:15:54.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2023-11-16T18:15:54.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T18:15:54.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-16T18:15:54.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2023-11-16T18:15:54.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T18:15:54.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-11-16T18:15:54.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-16T18:15:54.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:15:54.644 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-11-16T18:15:54.644 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-16T18:15:54.644 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:54.644 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-16T18:15:54.644 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:15:54.645 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-16T18:15:54.645 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-16T18:15:54.645 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-16T18:15:54.645 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-16T18:15:54.645 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-16T18:15:54.645 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-16T18:15:54.645 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:54.645 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-16T18:15:54.645 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-16T18:15:54.645 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-16T18:15:54.645 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-16T18:15:54.645 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-16T18:15:54.646 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-16T18:15:54.646 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-16T18:15:54.646 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-16T18:15:54.646 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-16T18:15:54.646 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:54.646 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-16T18:15:54.646 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:15:54.646 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-16T18:15:54.646 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-16T18:15:54.646 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:54.646 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-16T18:15:54.646 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2023-11-16T18:15:54.646 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-16T18:15:54.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-16T18:15:54.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-16T18:15:54.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-16T18:15:54.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2023-11-16T18:15:54.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T18:15:54.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T18:15:54.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:54.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-16T18:15:54.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:15:54.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2023-11-16T18:15:41.444940+0000", 2023-11-16T18:15:54.648 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T18:15:54.648 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:15:54.648 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-16T18:15:54.648 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T18:15:54.648 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.109+0000 7f77aae84700 1 Processor -- start 2023-11-16T18:15:54.649 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.109+0000 7f77aae84700 1 -- start start 2023-11-16T18:15:54.649 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.109+0000 7f77aae84700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 0x7f77a4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:54.649 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.109+0000 7f77aae84700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a4079d80 con 0x7f77a4106570 2023-11-16T18:15:54.649 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.109+0000 7f77a8c20700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 0x7f77a4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:54.649 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.110+0000 7f77a8c20700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 0x7f77a4108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46500/0 (socket says 172.21.15.2:46500) 2023-11-16T18:15:54.649 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.110+0000 7f77a8c20700 1 -- 172.21.15.2:0/2885873655 learned_addr learned my addr 172.21.15.2:0/2885873655 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:54.649 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.110+0000 7f77a8c20700 1 -- 172.21.15.2:0/2885873655 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a4079ec0 con 0x7f77a4106570 2023-11-16T18:15:54.649 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.110+0000 7f77a8c20700 1 --2- 172.21.15.2:0/2885873655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 0x7f77a4108950 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f77980098a0 tx=0x7f7798009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba8518fa064a1a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:54.649 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.110+0000 7f77a37fe700 1 -- 172.21.15.2:0/2885873655 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7798004160 con 0x7f77a4106570 2023-11-16T18:15:54.649 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.110+0000 7f77a37fe700 1 -- 172.21.15.2:0/2885873655 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77980042c0 con 0x7f77a4106570 2023-11-16T18:15:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.110+0000 7f77a37fe700 1 -- 172.21.15.2:0/2885873655 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7798011a00 con 0x7f77a4106570 2023-11-16T18:15:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.111+0000 7f77aae84700 1 -- 172.21.15.2:0/2885873655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 msgr2=0x7f77a4108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.111+0000 7f77aae84700 1 --2- 172.21.15.2:0/2885873655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 0x7f77a4108950 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f77980098a0 tx=0x7f7798009bb0 comp rx=0 tx=0).stop 2023-11-16T18:15:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.111+0000 7f77aae84700 1 -- 172.21.15.2:0/2885873655 shutdown_connections 2023-11-16T18:15:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.111+0000 7f77aae84700 1 --2- 172.21.15.2:0/2885873655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 0x7f77a4108950 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.111+0000 7f77aae84700 1 -- 172.21.15.2:0/2885873655 >> 172.21.15.2:0/2885873655 conn(0x7f77a40ffed0 msgr2=0x7f77a4102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.111+0000 7f77aae84700 1 -- 172.21.15.2:0/2885873655 shutdown_connections 2023-11-16T18:15:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.111+0000 7f77aae84700 1 -- 172.21.15.2:0/2885873655 wait complete. 2023-11-16T18:15:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.112+0000 7f77aae84700 1 Processor -- start 2023-11-16T18:15:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.112+0000 7f77aae84700 1 -- start start 2023-11-16T18:15:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.112+0000 7f77aae84700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 0x7f77a4199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:15:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.112+0000 7f77aae84700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a4199cb0 con 0x7f77a4106570 2023-11-16T18:15:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.112+0000 7f77a8c20700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 0x7f77a4199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:15:54.651 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.112+0000 7f77a8c20700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 0x7f77a4199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46502/0 (socket says 172.21.15.2:46502) 2023-11-16T18:15:54.651 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.112+0000 7f77a8c20700 1 -- 172.21.15.2:0/3460016866 learned_addr learned my addr 172.21.15.2:0/3460016866 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:15:54.651 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.112+0000 7f77a8c20700 1 -- 172.21.15.2:0/3460016866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798009550 con 0x7f77a4106570 2023-11-16T18:15:54.651 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.112+0000 7f77a8c20700 1 --2- 172.21.15.2:0/3460016866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 0x7f77a4199770 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7798009fb0 tx=0x7f7798011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:15:54.651 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.112+0000 7f77a1ffb700 1 -- 172.21.15.2:0/3460016866 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7798003930 con 0x7f77a4106570 2023-11-16T18:15:54.651 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.113+0000 7f77a1ffb700 1 -- 172.21.15.2:0/3460016866 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7798003a90 con 0x7f77a4106570 2023-11-16T18:15:54.651 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.113+0000 7f77a1ffb700 1 -- 172.21.15.2:0/3460016866 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7798019610 con 0x7f77a4106570 2023-11-16T18:15:54.651 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.113+0000 7f77aae84700 1 -- 172.21.15.2:0/3460016866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a4109d50 con 0x7f77a4106570 2023-11-16T18:15:54.651 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.113+0000 7f77aae84700 1 -- 172.21.15.2:0/3460016866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a4109bc0 con 0x7f77a4106570 2023-11-16T18:15:54.651 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.113+0000 7f77a1ffb700 1 -- 172.21.15.2:0/3460016866 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7798019b80 con 0x7f77a4106570 2023-11-16T18:15:54.651 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.113+0000 7f77a1ffb700 1 -- 172.21.15.2:0/3460016866 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f77980243f0 con 0x7f77a4106570 2023-11-16T18:15:54.651 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.114+0000 7f77aae84700 1 -- 172.21.15.2:0/3460016866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a4109ee0 con 0x7f77a4106570 2023-11-16T18:15:54.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.116+0000 7f77a1ffb700 1 -- 172.21.15.2:0/3460016866 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f779802de40 con 0x7f77a4106570 2023-11-16T18:15:54.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.255+0000 7f77aae84700 1 -- 172.21.15.2:0/3460016866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f77a419a2b0 con 0x7f77a4106570 2023-11-16T18:15:54.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.256+0000 7f77a1ffb700 1 -- 172.21.15.2:0/3460016866 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f7798003cb0 con 0x7f77a4106570 2023-11-16T18:15:54.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.257+0000 7f77aae84700 1 -- 172.21.15.2:0/3460016866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 msgr2=0x7f77a4199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:15:54.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.257+0000 7f77aae84700 1 --2- 172.21.15.2:0/3460016866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 0x7f77a4199770 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7798009fb0 tx=0x7f7798011ad0 comp rx=0 tx=0).stop 2023-11-16T18:15:54.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.257+0000 7f77aae84700 1 -- 172.21.15.2:0/3460016866 shutdown_connections 2023-11-16T18:15:54.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.257+0000 7f77aae84700 1 --2- 172.21.15.2:0/3460016866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77a4106570 0x7f77a4199770 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:15:54.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.257+0000 7f77aae84700 1 -- 172.21.15.2:0/3460016866 >> 172.21.15.2:0/3460016866 conn(0x7f77a40ffed0 msgr2=0x7f77a4101570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:15:54.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.257+0000 7f77aae84700 1 -- 172.21.15.2:0/3460016866 shutdown_connections 2023-11-16T18:15:54.652 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:54.257+0000 7f77aae84700 1 -- 172.21.15.2:0/3460016866 wait complete. 2023-11-16T18:15:54.652 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (3/15)... 2023-11-16T18:15:55.856 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:15:55.410+0000 7f410ea99200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T18:15:55.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-mon[98807]: Activating manager daemon x 2023-11-16T18:15:55.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-mon[98807]: mgrmap e2: x(active, starting, since 0.00316179s) 2023-11-16T18:15:55.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-mon[98807]: from='mgr.14100 172.21.15.2:0/4143795671' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T18:15:55.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-mon[98807]: from='mgr.14100 172.21.15.2:0/4143795671' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T18:15:55.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-mon[98807]: from='mgr.14100 172.21.15.2:0/4143795671' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T18:15:55.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-mon[98807]: from='mgr.14100 172.21.15.2:0/4143795671' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:15:55.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-mon[98807]: from='mgr.14100 172.21.15.2:0/4143795671' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T18:15:55.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-mon[98807]: Manager daemon x is now available 2023-11-16T18:15:55.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-mon[98807]: from='mgr.14100 172.21.15.2:0/4143795671' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T18:15:55.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-mon[98807]: from='mgr.14100 172.21.15.2:0/4143795671' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T18:15:55.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-mon[98807]: from='mgr.14100 172.21.15.2:0/4143795671' entity='mgr.x' 2023-11-16T18:15:55.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-mon[98807]: from='mgr.14100 172.21.15.2:0/4143795671' entity='mgr.x' 2023-11-16T18:15:55.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:55 smithi002 ceph-mon[98807]: from='mgr.14100 172.21.15.2:0/4143795671' entity='mgr.x' 2023-11-16T18:15:57.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:57 smithi002 ceph-mon[98807]: mgrmap e3: x(active, since 1.00754s) 2023-11-16T18:15:59.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:59 smithi002 ceph-mon[98807]: mgrmap e4: x(active, since 3s) 2023-11-16T18:15:59.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:15:59 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1761621715' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T18:16:01.193 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T18:16:01.194 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T18:16:01.194 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "08160fac-84ac-11ee-8dbe-212e2dc638e7", 2023-11-16T18:16:01.194 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2023-11-16T18:16:01.194 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-16T18:16:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-16T18:16:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-16T18:16:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:16:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-16T18:16:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-16T18:16:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2023-11-16T18:16:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T18:16:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-16T18:16:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2023-11-16T18:16:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T18:16:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2023-11-16T18:16:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-16T18:16:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:16:01.196 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-11-16T18:16:01.196 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-16T18:16:01.196 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:16:01.205 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-16T18:16:01.205 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:16:01.205 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-16T18:16:01.205 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-16T18:16:01.205 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-16T18:16:01.205 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-16T18:16:01.205 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-16T18:16:01.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-16T18:16:01.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:16:01.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-16T18:16:01.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-16T18:16:01.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-16T18:16:01.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-16T18:16:01.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-16T18:16:01.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-16T18:16:01.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-16T18:16:01.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-16T18:16:01.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-16T18:16:01.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:16:01.217 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-16T18:16:01.217 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:16:01.217 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-16T18:16:01.218 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-16T18:16:01.232 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:16:01.232 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-16T18:16:01.233 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-16T18:16:01.233 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-16T18:16:01.233 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-16T18:16:01.237 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-16T18:16:01.237 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-16T18:16:01.237 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2023-11-16T18:16:01.238 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T18:16:01.238 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T18:16:01.238 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:16:01.238 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-16T18:16:01.238 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T18:16:01.238 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2023-11-16T18:15:41.444940+0000", 2023-11-16T18:16:01.238 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T18:16:01.238 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T18:16:01.238 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-16T18:16:01.238 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T18:16:01.238 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.449+0000 7fc0f4770700 1 Processor -- start 2023-11-16T18:16:01.239 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.449+0000 7fc0f4770700 1 -- start start 2023-11-16T18:16:01.239 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.450+0000 7fc0f4770700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 0x7fc0ec108870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:01.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.450+0000 7fc0f4770700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec079e60 con 0x7fc0ec106490 2023-11-16T18:16:01.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.450+0000 7fc0f250c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 0x7fc0ec108870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:01.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.450+0000 7fc0f250c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 0x7fc0ec108870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46528/0 (socket says 172.21.15.2:46528) 2023-11-16T18:16:01.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.450+0000 7fc0f250c700 1 -- 172.21.15.2:0/2978977704 learned_addr learned my addr 172.21.15.2:0/2978977704 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:01.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.450+0000 7fc0f250c700 1 -- 172.21.15.2:0/2978977704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ec108db0 con 0x7fc0ec106490 2023-11-16T18:16:01.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.450+0000 7fc0f250c700 1 --2- 172.21.15.2:0/2978977704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 0x7fc0ec108870 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80098a0 tx=0x7fc0e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1030756efa1a1158 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:01.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.450+0000 7fc0f150a700 1 -- 172.21.15.2:0/2978977704 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0e8004160 con 0x7fc0ec106490 2023-11-16T18:16:01.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.451+0000 7fc0f150a700 1 -- 172.21.15.2:0/2978977704 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0e80042c0 con 0x7fc0ec106490 2023-11-16T18:16:01.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.451+0000 7fc0f150a700 1 -- 172.21.15.2:0/2978977704 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0e8011a00 con 0x7fc0ec106490 2023-11-16T18:16:01.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.451+0000 7fc0f4770700 1 -- 172.21.15.2:0/2978977704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 msgr2=0x7fc0ec108870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:01.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.451+0000 7fc0f4770700 1 --2- 172.21.15.2:0/2978977704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 0x7fc0ec108870 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80098a0 tx=0x7fc0e8009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:01.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.451+0000 7fc0f4770700 1 -- 172.21.15.2:0/2978977704 shutdown_connections 2023-11-16T18:16:01.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.451+0000 7fc0f4770700 1 --2- 172.21.15.2:0/2978977704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 0x7fc0ec108870 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:01.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.451+0000 7fc0f4770700 1 -- 172.21.15.2:0/2978977704 >> 172.21.15.2:0/2978977704 conn(0x7fc0ec0ffed0 msgr2=0x7fc0ec102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:01.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.451+0000 7fc0f4770700 1 -- 172.21.15.2:0/2978977704 shutdown_connections 2023-11-16T18:16:01.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.451+0000 7fc0f4770700 1 -- 172.21.15.2:0/2978977704 wait complete. 2023-11-16T18:16:01.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.452+0000 7fc0f4770700 1 Processor -- start 2023-11-16T18:16:01.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.452+0000 7fc0f4770700 1 -- start start 2023-11-16T18:16:01.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.452+0000 7fc0f4770700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 0x7fc0ec199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:01.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.452+0000 7fc0f4770700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec199ca0 con 0x7fc0ec106490 2023-11-16T18:16:01.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.452+0000 7fc0f250c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 0x7fc0ec199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:01.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.452+0000 7fc0f250c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 0x7fc0ec199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46530/0 (socket says 172.21.15.2:46530) 2023-11-16T18:16:01.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.452+0000 7fc0f250c700 1 -- 172.21.15.2:0/1761621715 learned_addr learned my addr 172.21.15.2:0/1761621715 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:01.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.452+0000 7fc0f250c700 1 -- 172.21.15.2:0/1761621715 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e8009550 con 0x7fc0ec106490 2023-11-16T18:16:01.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.453+0000 7fc0f250c700 1 --2- 172.21.15.2:0/1761621715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 0x7fc0ec199760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009fb0 tx=0x7fc0e8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:01.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.453+0000 7fc0e37fe700 1 -- 172.21.15.2:0/1761621715 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0e8003930 con 0x7fc0ec106490 2023-11-16T18:16:01.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.453+0000 7fc0e37fe700 1 -- 172.21.15.2:0/1761621715 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0e8003a90 con 0x7fc0ec106490 2023-11-16T18:16:01.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.453+0000 7fc0f4770700 1 -- 172.21.15.2:0/1761621715 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ec109d40 con 0x7fc0ec106490 2023-11-16T18:16:01.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.453+0000 7fc0e37fe700 1 -- 172.21.15.2:0/1761621715 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0e8019610 con 0x7fc0ec106490 2023-11-16T18:16:01.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.453+0000 7fc0f4770700 1 -- 172.21.15.2:0/1761621715 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ec109bb0 con 0x7fc0ec106490 2023-11-16T18:16:01.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.454+0000 7fc0e37fe700 1 -- 172.21.15.2:0/1761621715 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fc0e80198d0 con 0x7fc0ec106490 2023-11-16T18:16:01.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.454+0000 7fc0e37fe700 1 --2- 172.21.15.2:0/1761621715 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7fc0d8038670 0x7fc0d803ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:01.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.454+0000 7fc0e17fa700 1 -- 172.21.15.2:0/1761621715 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d40052f0 con 0x7fc0ec106490 2023-11-16T18:16:01.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.454+0000 7fc0e37fe700 1 -- 172.21.15.2:0/1761621715 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc0e804c710 con 0x7fc0ec106490 2023-11-16T18:16:01.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.454+0000 7fc0f1d0b700 1 --2- 172.21.15.2:0/1761621715 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7fc0d8038670 0x7fc0d803ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:01.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.455+0000 7fc0f1d0b700 1 --2- 172.21.15.2:0/1761621715 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7fc0d8038670 0x7fc0d803ab20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc006fd0 tx=0x7fc0dc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:01.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.459+0000 7fc0e37fe700 1 -- 172.21.15.2:0/1761621715 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc0e8019b80 con 0x7fc0ec106490 2023-11-16T18:16:01.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.974+0000 7fc0e17fa700 1 -- 172.21.15.2:0/1761621715 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc0d40059c0 con 0x7fc0ec106490 2023-11-16T18:16:01.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.974+0000 7fc0e37fe700 1 -- 172.21.15.2:0/1761621715 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fc0e8018350 con 0x7fc0ec106490 2023-11-16T18:16:01.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.976+0000 7fc0e17fa700 1 -- 172.21.15.2:0/1761621715 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7fc0d8038670 msgr2=0x7fc0d803ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:01.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.976+0000 7fc0e17fa700 1 --2- 172.21.15.2:0/1761621715 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7fc0d8038670 0x7fc0d803ab20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc006fd0 tx=0x7fc0dc006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:01.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.976+0000 7fc0e17fa700 1 -- 172.21.15.2:0/1761621715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 msgr2=0x7fc0ec199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:01.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.976+0000 7fc0e17fa700 1 --2- 172.21.15.2:0/1761621715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 0x7fc0ec199760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009fb0 tx=0x7fc0e8011ad0 comp rx=0 tx=0).stop 2023-11-16T18:16:01.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.976+0000 7fc0e17fa700 1 -- 172.21.15.2:0/1761621715 shutdown_connections 2023-11-16T18:16:01.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.976+0000 7fc0e17fa700 1 --2- 172.21.15.2:0/1761621715 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7fc0d8038670 0x7fc0d803ab20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:01.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.976+0000 7fc0e17fa700 1 --2- 172.21.15.2:0/1761621715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0ec106490 0x7fc0ec199760 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:01.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.976+0000 7fc0e17fa700 1 -- 172.21.15.2:0/1761621715 >> 172.21.15.2:0/1761621715 conn(0x7fc0ec0ffed0 msgr2=0x7fc0ec100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:01.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.976+0000 7fc0e17fa700 1 -- 172.21.15.2:0/1761621715 shutdown_connections 2023-11-16T18:16:01.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:15:58.976+0000 7fc0e17fa700 1 -- 172.21.15.2:0/1761621715 wait complete. 2023-11-16T18:16:01.258 INFO:teuthology.orchestra.run.smithi002.stdout:mgr is available 2023-11-16T18:16:04.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:03 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1290285283' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-16T18:16:04.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T18:16:04.250 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [global] 2023-11-16T18:16:04.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout fsid = 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:16:04.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-11-16T18:16:04.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-11-16T18:16:04.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-11-16T18:16:04.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-11-16T18:16:04.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T18:16:04.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [mgr] 2023-11-16T18:16:04.253 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-11-16T18:16:04.253 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T18:16:04.253 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [osd] 2023-11-16T18:16:04.253 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-11-16T18:16:04.253 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-11-16T18:16:04.253 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-11-16T18:16:04.253 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-11-16T18:16:04.253 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-11-16T18:16:04.253 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.504+0000 7fd644bd8700 1 Processor -- start 2023-11-16T18:16:04.253 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.504+0000 7fd644bd8700 1 -- start start 2023-11-16T18:16:04.253 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.504+0000 7fd644bd8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 0x7fd64007a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:04.254 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.504+0000 7fd644bd8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64007a7f0 con 0x7fd64007bdb0 2023-11-16T18:16:04.254 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.505+0000 7fd63e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 0x7fd64007a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:04.255 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.505+0000 7fd63e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 0x7fd64007a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46534/0 (socket says 172.21.15.2:46534) 2023-11-16T18:16:04.255 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.505+0000 7fd63e59c700 1 -- 172.21.15.2:0/2760578890 learned_addr learned my addr 172.21.15.2:0/2760578890 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:04.255 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.505+0000 7fd63e59c700 1 -- 172.21.15.2:0/2760578890 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64007a930 con 0x7fd64007bdb0 2023-11-16T18:16:04.255 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.505+0000 7fd63e59c700 1 --2- 172.21.15.2:0/2760578890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 0x7fd64007a2b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd6300098a0 tx=0x7fd630009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4255db471c08022a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:04.255 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.505+0000 7fd63d59a700 1 -- 172.21.15.2:0/2760578890 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd630004160 con 0x7fd64007bdb0 2023-11-16T18:16:04.255 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.505+0000 7fd63d59a700 1 -- 172.21.15.2:0/2760578890 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6300042c0 con 0x7fd64007bdb0 2023-11-16T18:16:04.255 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.505+0000 7fd63d59a700 1 -- 172.21.15.2:0/2760578890 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd630011a00 con 0x7fd64007bdb0 2023-11-16T18:16:04.255 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.506+0000 7fd644bd8700 1 -- 172.21.15.2:0/2760578890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 msgr2=0x7fd64007a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:04.255 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.506+0000 7fd644bd8700 1 --2- 172.21.15.2:0/2760578890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 0x7fd64007a2b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd6300098a0 tx=0x7fd630009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:04.255 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.506+0000 7fd644bd8700 1 -- 172.21.15.2:0/2760578890 shutdown_connections 2023-11-16T18:16:04.255 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.506+0000 7fd644bd8700 1 --2- 172.21.15.2:0/2760578890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 0x7fd64007a2b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:04.256 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.506+0000 7fd644bd8700 1 -- 172.21.15.2:0/2760578890 >> 172.21.15.2:0/2760578890 conn(0x7fd640101170 msgr2=0x7fd640103580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:04.256 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.506+0000 7fd644bd8700 1 -- 172.21.15.2:0/2760578890 shutdown_connections 2023-11-16T18:16:04.256 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.506+0000 7fd644bd8700 1 -- 172.21.15.2:0/2760578890 wait complete. 2023-11-16T18:16:04.256 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.507+0000 7fd644bd8700 1 Processor -- start 2023-11-16T18:16:04.256 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.507+0000 7fd644bd8700 1 -- start start 2023-11-16T18:16:04.256 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.507+0000 7fd644bd8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 0x7fd64019bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:04.256 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.507+0000 7fd644bd8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64019c020 con 0x7fd64007bdb0 2023-11-16T18:16:04.256 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.507+0000 7fd63e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 0x7fd64019bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:04.256 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.507+0000 7fd63e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 0x7fd64019bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46536/0 (socket says 172.21.15.2:46536) 2023-11-16T18:16:04.256 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.507+0000 7fd63e59c700 1 -- 172.21.15.2:0/1290285283 learned_addr learned my addr 172.21.15.2:0/1290285283 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:04.256 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.507+0000 7fd63e59c700 1 -- 172.21.15.2:0/1290285283 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd630009550 con 0x7fd64007bdb0 2023-11-16T18:16:04.256 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.508+0000 7fd63e59c700 1 --2- 172.21.15.2:0/1290285283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 0x7fd64019bae0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd630006b20 tx=0x7fd630011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:04.256 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.508+0000 7fd6377fe700 1 -- 172.21.15.2:0/1290285283 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd630003b70 con 0x7fd64007bdb0 2023-11-16T18:16:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.508+0000 7fd6377fe700 1 -- 172.21.15.2:0/1290285283 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd630003cd0 con 0x7fd64007bdb0 2023-11-16T18:16:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.508+0000 7fd644bd8700 1 -- 172.21.15.2:0/1290285283 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64010bf90 con 0x7fd64007bdb0 2023-11-16T18:16:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.508+0000 7fd6377fe700 1 -- 172.21.15.2:0/1290285283 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6300247c0 con 0x7fd64007bdb0 2023-11-16T18:16:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.508+0000 7fd644bd8700 1 -- 172.21.15.2:0/1290285283 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64010be00 con 0x7fd64007bdb0 2023-11-16T18:16:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.509+0000 7fd6377fe700 1 -- 172.21.15.2:0/1290285283 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fd63001ab40 con 0x7fd64007bdb0 2023-11-16T18:16:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.509+0000 7fd6377fe700 1 --2- 172.21.15.2:0/1290285283 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7fd62c038710 0x7fd62c03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.509+0000 7fd6377fe700 1 -- 172.21.15.2:0/1290285283 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd63004c4f0 con 0x7fd64007bdb0 2023-11-16T18:16:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.509+0000 7fd63dd9b700 1 --2- 172.21.15.2:0/1290285283 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7fd62c038710 0x7fd62c03abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.509+0000 7fd644bd8700 1 -- 172.21.15.2:0/1290285283 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64010c120 con 0x7fd64007bdb0 2023-11-16T18:16:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.510+0000 7fd63dd9b700 1 --2- 172.21.15.2:0/1290285283 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7fd62c038710 0x7fd62c03abc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd628006fd0 tx=0x7fd628006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.514+0000 7fd6377fe700 1 -- 172.21.15.2:0/1290285283 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd63001a430 con 0x7fd64007bdb0 2023-11-16T18:16:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.870+0000 7fd644bd8700 1 -- 172.21.15.2:0/1290285283 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fd640062120 con 0x7fd64007bdb0 2023-11-16T18:16:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.872+0000 7fd6377fe700 1 -- 172.21.15.2:0/1290285283 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fd63002cb00 con 0x7fd64007bdb0 2023-11-16T18:16:04.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.874+0000 7fd644bd8700 1 -- 172.21.15.2:0/1290285283 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7fd62c038710 msgr2=0x7fd62c03abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:04.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.874+0000 7fd644bd8700 1 --2- 172.21.15.2:0/1290285283 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7fd62c038710 0x7fd62c03abc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd628006fd0 tx=0x7fd628006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:04.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.874+0000 7fd644bd8700 1 -- 172.21.15.2:0/1290285283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 msgr2=0x7fd64019bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:04.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.874+0000 7fd644bd8700 1 --2- 172.21.15.2:0/1290285283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 0x7fd64019bae0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd630006b20 tx=0x7fd630011ad0 comp rx=0 tx=0).stop 2023-11-16T18:16:04.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.874+0000 7fd644bd8700 1 -- 172.21.15.2:0/1290285283 shutdown_connections 2023-11-16T18:16:04.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.874+0000 7fd644bd8700 1 --2- 172.21.15.2:0/1290285283 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7fd62c038710 0x7fd62c03abc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:04.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.874+0000 7fd644bd8700 1 --2- 172.21.15.2:0/1290285283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd64007bdb0 0x7fd64019bae0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:04.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.874+0000 7fd644bd8700 1 -- 172.21.15.2:0/1290285283 >> 172.21.15.2:0/1290285283 conn(0x7fd640101170 msgr2=0x7fd640102860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:04.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.874+0000 7fd644bd8700 1 -- 172.21.15.2:0/1290285283 shutdown_connections 2023-11-16T18:16:04.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:03.874+0000 7fd644bd8700 1 -- 172.21.15.2:0/1290285283 wait complete. 2023-11-16T18:16:04.258 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling cephadm module... 2023-11-16T18:16:05.530 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:05 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3684855805' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-16T18:16:05.530 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: ignoring --setuser ceph since I am not root 2023-11-16T18:16:05.531 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: ignoring --setgroup ceph since I am not root 2023-11-16T18:16:05.531 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:05.432+0000 7f285e361200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T18:16:05.599 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.709+0000 7ff72658b700 1 Processor -- start 2023-11-16T18:16:05.599 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.709+0000 7ff72658b700 1 -- start start 2023-11-16T18:16:05.599 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.709+0000 7ff72658b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 0x7ff7201044b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:05.599 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.710+0000 7ff72658b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff720104a80 con 0x7ff7201040a0 2023-11-16T18:16:05.599 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.710+0000 7ff71ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 0x7ff7201044b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:05.599 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.710+0000 7ff71ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 0x7ff7201044b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46540/0 (socket says 172.21.15.2:46540) 2023-11-16T18:16:05.600 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.710+0000 7ff71ffff700 1 -- 172.21.15.2:0/677263263 learned_addr learned my addr 172.21.15.2:0/677263263 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:05.600 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.710+0000 7ff71ffff700 1 -- 172.21.15.2:0/677263263 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7201052a0 con 0x7ff7201040a0 2023-11-16T18:16:05.600 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.710+0000 7ff71ffff700 1 --2- 172.21.15.2:0/677263263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 0x7ff7201044b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff71000bb40 tx=0x7ff71000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8efb0459c8097b8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:05.600 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.710+0000 7ff71effd700 1 -- 172.21.15.2:0/677263263 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff71000dad0 con 0x7ff7201040a0 2023-11-16T18:16:05.600 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.710+0000 7ff71effd700 1 -- 172.21.15.2:0/677263263 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7100044d0 con 0x7ff7201040a0 2023-11-16T18:16:05.600 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.710+0000 7ff71effd700 1 -- 172.21.15.2:0/677263263 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff710003e40 con 0x7ff7201040a0 2023-11-16T18:16:05.600 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.711+0000 7ff72658b700 1 -- 172.21.15.2:0/677263263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 msgr2=0x7ff7201044b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:05.600 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.711+0000 7ff72658b700 1 --2- 172.21.15.2:0/677263263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 0x7ff7201044b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff71000bb40 tx=0x7ff71000be50 comp rx=0 tx=0).stop 2023-11-16T18:16:05.600 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.711+0000 7ff72658b700 1 -- 172.21.15.2:0/677263263 shutdown_connections 2023-11-16T18:16:05.600 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.711+0000 7ff72658b700 1 --2- 172.21.15.2:0/677263263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 0x7ff7201044b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:05.600 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.711+0000 7ff72658b700 1 -- 172.21.15.2:0/677263263 >> 172.21.15.2:0/677263263 conn(0x7ff7200ff650 msgr2=0x7ff720101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:05.600 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.711+0000 7ff72658b700 1 -- 172.21.15.2:0/677263263 shutdown_connections 2023-11-16T18:16:05.600 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.711+0000 7ff72658b700 1 -- 172.21.15.2:0/677263263 wait complete. 2023-11-16T18:16:05.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.712+0000 7ff72658b700 1 Processor -- start 2023-11-16T18:16:05.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.712+0000 7ff72658b700 1 -- start start 2023-11-16T18:16:05.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.712+0000 7ff72658b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 0x7ff72019b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:05.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.712+0000 7ff72658b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72019bf20 con 0x7ff7201040a0 2023-11-16T18:16:05.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.712+0000 7ff71ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 0x7ff72019b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:05.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.712+0000 7ff71ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 0x7ff72019b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46542/0 (socket says 172.21.15.2:46542) 2023-11-16T18:16:05.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.712+0000 7ff71ffff700 1 -- 172.21.15.2:0/3684855805 learned_addr learned my addr 172.21.15.2:0/3684855805 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:05.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.712+0000 7ff71ffff700 1 -- 172.21.15.2:0/3684855805 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71000b7f0 con 0x7ff7201040a0 2023-11-16T18:16:05.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.713+0000 7ff71ffff700 1 --2- 172.21.15.2:0/3684855805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 0x7ff72019b9e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff7100039a0 tx=0x7ff710003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:05.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.713+0000 7ff71d7fa700 1 -- 172.21.15.2:0/3684855805 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff710023070 con 0x7ff7201040a0 2023-11-16T18:16:05.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.713+0000 7ff71d7fa700 1 -- 172.21.15.2:0/3684855805 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7100179a0 con 0x7ff7201040a0 2023-11-16T18:16:05.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.713+0000 7ff72658b700 1 -- 172.21.15.2:0/3684855805 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff720105990 con 0x7ff7201040a0 2023-11-16T18:16:05.601 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.713+0000 7ff71d7fa700 1 -- 172.21.15.2:0/3684855805 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff710009040 con 0x7ff7201040a0 2023-11-16T18:16:05.602 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.713+0000 7ff72658b700 1 -- 172.21.15.2:0/3684855805 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff720105800 con 0x7ff7201040a0 2023-11-16T18:16:05.602 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.714+0000 7ff71d7fa700 1 -- 172.21.15.2:0/3684855805 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7ff710015070 con 0x7ff7201040a0 2023-11-16T18:16:05.602 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.714+0000 7ff72658b700 1 -- 172.21.15.2:0/3684855805 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff720105b20 con 0x7ff7201040a0 2023-11-16T18:16:05.602 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.714+0000 7ff71d7fa700 1 --2- 172.21.15.2:0/3684855805 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7ff7080386f0 0x7ff70803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:05.602 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.714+0000 7ff71d7fa700 1 -- 172.21.15.2:0/3684855805 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff71004be70 con 0x7ff7201040a0 2023-11-16T18:16:05.602 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.714+0000 7ff71f7fe700 1 --2- 172.21.15.2:0/3684855805 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7ff7080386f0 0x7ff70803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:05.603 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.715+0000 7ff71f7fe700 1 --2- 172.21.15.2:0/3684855805 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7ff7080386f0 0x7ff70803aba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff714006fd0 tx=0x7ff714006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:05.603 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:04.719+0000 7ff71d7fa700 1 -- 172.21.15.2:0/3684855805 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff710026970 con 0x7ff7201040a0 2023-11-16T18:16:05.603 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:05.183+0000 7ff72658b700 1 -- 172.21.15.2:0/3684855805 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7ff72002cfe0 con 0x7ff7201040a0 2023-11-16T18:16:05.603 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:05.236+0000 7ff71d7fa700 1 -- 172.21.15.2:0/3684855805 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ff71001e380 con 0x7ff7201040a0 2023-11-16T18:16:05.603 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:05.237+0000 7ff71d7fa700 1 -- 172.21.15.2:0/3684855805 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7ff71001d2e0 con 0x7ff7201040a0 2023-11-16T18:16:05.603 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:05.240+0000 7ff72658b700 1 -- 172.21.15.2:0/3684855805 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7ff7080386f0 msgr2=0x7ff70803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:05.603 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:05.240+0000 7ff72658b700 1 --2- 172.21.15.2:0/3684855805 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7ff7080386f0 0x7ff70803aba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff714006fd0 tx=0x7ff714006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:05.603 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:05.240+0000 7ff72658b700 1 -- 172.21.15.2:0/3684855805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 msgr2=0x7ff72019b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:05.603 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:05.240+0000 7ff72658b700 1 --2- 172.21.15.2:0/3684855805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 0x7ff72019b9e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff7100039a0 tx=0x7ff710003f40 comp rx=0 tx=0).stop 2023-11-16T18:16:05.603 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:05.241+0000 7ff72658b700 1 -- 172.21.15.2:0/3684855805 shutdown_connections 2023-11-16T18:16:05.604 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:05.241+0000 7ff72658b700 1 --2- 172.21.15.2:0/3684855805 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7ff7080386f0 0x7ff70803aba0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:05.604 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:05.241+0000 7ff72658b700 1 --2- 172.21.15.2:0/3684855805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7201040a0 0x7ff72019b9e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:05.604 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:05.241+0000 7ff72658b700 1 -- 172.21.15.2:0/3684855805 >> 172.21.15.2:0/3684855805 conn(0x7ff7200ff650 msgr2=0x7ff72018ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:05.604 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:05.241+0000 7ff72658b700 1 -- 172.21.15.2:0/3684855805 shutdown_connections 2023-11-16T18:16:05.604 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:05.241+0000 7ff72658b700 1 -- 172.21.15.2:0/3684855805 wait complete. 2023-11-16T18:16:05.787 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:05.527+0000 7f285e361200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T18:16:05.788 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:05.639+0000 7f285e361200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T18:16:05.788 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:05.752+0000 7f285e361200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T18:16:06.107 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:05.965+0000 7f285e361200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T18:16:06.552 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:06 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3684855805' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-16T18:16:06.552 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:06 smithi002 ceph-mon[98807]: mgrmap e5: x(active, since 9s) 2023-11-16T18:16:06.552 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:06 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:06.363+0000 7f285e361200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T18:16:06.552 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:06 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:06.471+0000 7f285e361200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T18:16:06.847 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T18:16:06.847 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-11-16T18:16:06.848 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-16T18:16:06.848 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-11-16T18:16:06.848 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-11-16T18:16:06.848 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T18:16:06.848 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.049+0000 7f9859cc2700 1 Processor -- start 2023-11-16T18:16:06.848 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.049+0000 7f9859cc2700 1 -- start start 2023-11-16T18:16:06.848 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.049+0000 7f9859cc2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 0x7f9854104710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:06.848 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.050+0000 7f9859cc2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9854079de0 con 0x7f9854104300 2023-11-16T18:16:06.848 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.050+0000 7f98537fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 0x7f9854104710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:06.848 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.050+0000 7f98537fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 0x7f9854104710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46556/0 (socket says 172.21.15.2:46556) 2023-11-16T18:16:06.848 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.050+0000 7f98537fe700 1 -- 172.21.15.2:0/2506961892 learned_addr learned my addr 172.21.15.2:0/2506961892 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:06.848 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.050+0000 7f98537fe700 1 -- 172.21.15.2:0/2506961892 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9854104c50 con 0x7f9854104300 2023-11-16T18:16:06.848 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.050+0000 7f98537fe700 1 --2- 172.21.15.2:0/2506961892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 0x7f9854104710 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f98440098a0 tx=0x7f9844009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2375455f42665221 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:06.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.050+0000 7f98527fc700 1 -- 172.21.15.2:0/2506961892 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9844004160 con 0x7f9854104300 2023-11-16T18:16:06.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.050+0000 7f98527fc700 1 -- 172.21.15.2:0/2506961892 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98440042c0 con 0x7f9854104300 2023-11-16T18:16:06.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.050+0000 7f98527fc700 1 -- 172.21.15.2:0/2506961892 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9844011a00 con 0x7f9854104300 2023-11-16T18:16:06.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.051+0000 7f9859cc2700 1 -- 172.21.15.2:0/2506961892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 msgr2=0x7f9854104710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:06.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.051+0000 7f9859cc2700 1 --2- 172.21.15.2:0/2506961892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 0x7f9854104710 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f98440098a0 tx=0x7f9844009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:06.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.051+0000 7f9859cc2700 1 -- 172.21.15.2:0/2506961892 shutdown_connections 2023-11-16T18:16:06.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.051+0000 7f9859cc2700 1 --2- 172.21.15.2:0/2506961892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 0x7f9854104710 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:06.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.051+0000 7f9859cc2700 1 -- 172.21.15.2:0/2506961892 >> 172.21.15.2:0/2506961892 conn(0x7f98540ffed0 msgr2=0x7f9854102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:06.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.051+0000 7f9859cc2700 1 -- 172.21.15.2:0/2506961892 shutdown_connections 2023-11-16T18:16:06.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.051+0000 7f9859cc2700 1 -- 172.21.15.2:0/2506961892 wait complete. 2023-11-16T18:16:06.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.052+0000 7f9859cc2700 1 Processor -- start 2023-11-16T18:16:06.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.052+0000 7f9859cc2700 1 -- start start 2023-11-16T18:16:06.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.052+0000 7f9859cc2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 0x7f985419b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:06.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.052+0000 7f9859cc2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985419bee0 con 0x7f9854104300 2023-11-16T18:16:06.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.052+0000 7f98537fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 0x7f985419b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:06.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.052+0000 7f98537fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 0x7f985419b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46558/0 (socket says 172.21.15.2:46558) 2023-11-16T18:16:06.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.052+0000 7f98537fe700 1 -- 172.21.15.2:0/1243987617 learned_addr learned my addr 172.21.15.2:0/1243987617 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:06.856 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.053+0000 7f98537fe700 1 -- 172.21.15.2:0/1243987617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9844009550 con 0x7f9854104300 2023-11-16T18:16:06.857 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.053+0000 7f98537fe700 1 --2- 172.21.15.2:0/1243987617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 0x7f985419b9a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9844009fb0 tx=0x7f9844011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:06.857 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.053+0000 7f9850ff9700 1 -- 172.21.15.2:0/1243987617 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9844003930 con 0x7f9854104300 2023-11-16T18:16:06.857 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.053+0000 7f9850ff9700 1 -- 172.21.15.2:0/1243987617 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9844003a90 con 0x7f9854104300 2023-11-16T18:16:06.857 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.053+0000 7f9859cc2700 1 -- 172.21.15.2:0/1243987617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9854105c50 con 0x7f9854104300 2023-11-16T18:16:06.857 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.053+0000 7f9850ff9700 1 -- 172.21.15.2:0/1243987617 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9844019610 con 0x7f9854104300 2023-11-16T18:16:06.857 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.053+0000 7f9859cc2700 1 -- 172.21.15.2:0/1243987617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9854105ac0 con 0x7f9854104300 2023-11-16T18:16:06.857 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.054+0000 7f9850ff9700 1 -- 172.21.15.2:0/1243987617 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f98440198c0 con 0x7f9854104300 2023-11-16T18:16:06.857 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.054+0000 7f9859cc2700 1 -- 172.21.15.2:0/1243987617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9854195250 con 0x7f9854104300 2023-11-16T18:16:06.857 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.054+0000 7f9850ff9700 1 --2- 172.21.15.2:0/1243987617 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f983c038710 0x7f983c03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:06.857 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.054+0000 7f9852ffd700 1 -- 172.21.15.2:0/1243987617 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f983c038710 msgr2=0x7f983c03abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1609662822 2023-11-16T18:16:06.857 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.054+0000 7f9850ff9700 1 -- 172.21.15.2:0/1243987617 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f984404c4b0 con 0x7f9854104300 2023-11-16T18:16:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.054+0000 7f9852ffd700 1 --2- 172.21.15.2:0/1243987617 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f983c038710 0x7f983c03abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T18:16:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.059+0000 7f9850ff9700 1 -- 172.21.15.2:0/1243987617 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f984402d440 con 0x7f9854104300 2023-11-16T18:16:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.255+0000 7f9852ffd700 1 -- 172.21.15.2:0/1243987617 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f983c038710 msgr2=0x7f983c03abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1609662822 2023-11-16T18:16:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.255+0000 7f9852ffd700 1 --2- 172.21.15.2:0/1243987617 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f983c038710 0x7f983c03abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T18:16:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.525+0000 7f9859cc2700 1 -- 172.21.15.2:0/1243987617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f985402cfe0 con 0x7f9854104300 2023-11-16T18:16:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.526+0000 7f9850ff9700 1 -- 172.21.15.2:0/1243987617 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f9844018350 con 0x7f9854104300 2023-11-16T18:16:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.528+0000 7f9859cc2700 1 -- 172.21.15.2:0/1243987617 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f983c038710 msgr2=0x7f983c03abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T18:16:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.528+0000 7f9859cc2700 1 --2- 172.21.15.2:0/1243987617 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f983c038710 0x7f983c03abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.528+0000 7f9859cc2700 1 -- 172.21.15.2:0/1243987617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 msgr2=0x7f985419b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.528+0000 7f9859cc2700 1 --2- 172.21.15.2:0/1243987617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 0x7f985419b9a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9844009fb0 tx=0x7f9844011ad0 comp rx=0 tx=0).stop 2023-11-16T18:16:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.528+0000 7f9859cc2700 1 -- 172.21.15.2:0/1243987617 shutdown_connections 2023-11-16T18:16:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.528+0000 7f9859cc2700 1 --2- 172.21.15.2:0/1243987617 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f983c038710 0x7f983c03abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.528+0000 7f9859cc2700 1 --2- 172.21.15.2:0/1243987617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9854104300 0x7f985419b9a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.528+0000 7f9859cc2700 1 -- 172.21.15.2:0/1243987617 >> 172.21.15.2:0/1243987617 conn(0x7f98540ffed0 msgr2=0x7f985418fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.529+0000 7f9859cc2700 1 -- 172.21.15.2:0/1243987617 shutdown_connections 2023-11-16T18:16:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:06.529+0000 7f9859cc2700 1 -- 172.21.15.2:0/1243987617 wait complete. 2023-11-16T18:16:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for the mgr to restart... 2023-11-16T18:16:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr epoch 5... 2023-11-16T18:16:06.859 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:06 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:06.627+0000 7f285e361200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T18:16:06.859 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:06 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:06.744+0000 7f285e361200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T18:16:07.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:07 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1243987617' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-16T18:16:07.608 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:07 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:07.265+0000 7f285e361200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T18:16:08.355 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:07 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:07.947+0000 7f285e361200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T18:16:08.356 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:08 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:08.138+0000 7f285e361200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T18:16:09.355 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:09 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:09.096+0000 7f285e361200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:16:10.335 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:09 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:09.206+0000 7f285e361200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T18:16:10.336 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:09 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:09.424+0000 7f285e361200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:16:10.336 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:09 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:09.769+0000 7f285e361200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T18:16:10.355 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:10 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:10.020+0000 7f285e361200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T18:16:10.355 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:10 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:10.201+0000 7f285e361200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T18:16:10.855 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:10 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:10.527+0000 7f285e361200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T18:16:11.605 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:11 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:11.345+0000 7f285e361200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T18:16:11.606 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:11 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:11.502+0000 7f285e361200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T18:16:12.105 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:11 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:11.743+0000 7f285e361200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T18:16:12.106 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:11 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:11.879+0000 7f285e361200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T18:16:12.605 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:12 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:12.150+0000 7f285e361200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T18:16:13.606 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:13.203+0000 7f285e361200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T18:16:13.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: Active manager daemon x restarted 2023-11-16T18:16:13.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: Activating manager daemon x 2023-11-16T18:16:13.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: osdmap e2: 0 total, 0 up, 0 in 2023-11-16T18:16:13.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: mgrmap e6: x(active, starting, since 0.00357215s) 2023-11-16T18:16:13.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:16:13.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T18:16:13.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T18:16:13.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T18:16:13.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T18:16:13.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: Manager daemon x is now available 2023-11-16T18:16:13.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:13.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:13.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:16:13.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:13 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:16:14.506 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T18:16:14.506 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-11-16T18:16:14.507 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "initialized": true 2023-11-16T18:16:14.507 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T18:16:14.507 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.343+0000 7f11311a6700 1 Processor -- start 2023-11-16T18:16:14.507 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.343+0000 7f11311a6700 1 -- start start 2023-11-16T18:16:14.507 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.344+0000 7f11311a6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 0x7f112c108870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:14.507 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.344+0000 7f11311a6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c079e60 con 0x7f112c106490 2023-11-16T18:16:14.507 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.344+0000 7f112ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 0x7f112c108870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:14.507 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.344+0000 7f112ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 0x7f112c108870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46566/0 (socket says 172.21.15.2:46566) 2023-11-16T18:16:14.507 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.344+0000 7f112ad9d700 1 -- 172.21.15.2:0/3209449692 learned_addr learned my addr 172.21.15.2:0/3209449692 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:14.507 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.344+0000 7f112ad9d700 1 -- 172.21.15.2:0/3209449692 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c108db0 con 0x7f112c106490 2023-11-16T18:16:14.507 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.344+0000 7f112ad9d700 1 --2- 172.21.15.2:0/3209449692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 0x7f112c108870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f111c0098a0 tx=0x7f111c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500e5eb3fe9ed4bf server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:14.507 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.344+0000 7f1129d9b700 1 -- 172.21.15.2:0/3209449692 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f111c004160 con 0x7f112c106490 2023-11-16T18:16:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.344+0000 7f1129d9b700 1 -- 172.21.15.2:0/3209449692 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f111c0042c0 con 0x7f112c106490 2023-11-16T18:16:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.344+0000 7f1129d9b700 1 -- 172.21.15.2:0/3209449692 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f111c011a00 con 0x7f112c106490 2023-11-16T18:16:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.345+0000 7f11311a6700 1 -- 172.21.15.2:0/3209449692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 msgr2=0x7f112c108870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.345+0000 7f11311a6700 1 --2- 172.21.15.2:0/3209449692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 0x7f112c108870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f111c0098a0 tx=0x7f111c009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.345+0000 7f11311a6700 1 -- 172.21.15.2:0/3209449692 shutdown_connections 2023-11-16T18:16:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.345+0000 7f11311a6700 1 --2- 172.21.15.2:0/3209449692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 0x7f112c108870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.345+0000 7f11311a6700 1 -- 172.21.15.2:0/3209449692 >> 172.21.15.2:0/3209449692 conn(0x7f112c0ffed0 msgr2=0x7f112c102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.345+0000 7f11311a6700 1 -- 172.21.15.2:0/3209449692 shutdown_connections 2023-11-16T18:16:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.345+0000 7f11311a6700 1 -- 172.21.15.2:0/3209449692 wait complete. 2023-11-16T18:16:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.346+0000 7f11311a6700 1 Processor -- start 2023-11-16T18:16:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.346+0000 7f11311a6700 1 -- start start 2023-11-16T18:16:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.346+0000 7f11311a6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 0x7f112c19b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.346+0000 7f11311a6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c19beb0 con 0x7f112c106490 2023-11-16T18:16:14.509 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.347+0000 7f112ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 0x7f112c19b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:14.509 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.347+0000 7f112ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 0x7f112c19b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46568/0 (socket says 172.21.15.2:46568) 2023-11-16T18:16:14.509 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.347+0000 7f112ad9d700 1 -- 172.21.15.2:0/699208204 learned_addr learned my addr 172.21.15.2:0/699208204 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:14.509 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.347+0000 7f112ad9d700 1 -- 172.21.15.2:0/699208204 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c009550 con 0x7f112c106490 2023-11-16T18:16:14.509 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.347+0000 7f112ad9d700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 0x7f112c19b970 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f111c006af0 tx=0x7f111c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:14.509 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.347+0000 7f1113fff700 1 -- 172.21.15.2:0/699208204 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f111c003a40 con 0x7f112c106490 2023-11-16T18:16:14.509 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.347+0000 7f1113fff700 1 -- 172.21.15.2:0/699208204 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f111c003ba0 con 0x7f112c106490 2023-11-16T18:16:14.509 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.347+0000 7f11311a6700 1 -- 172.21.15.2:0/699208204 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f112c109d40 con 0x7f112c106490 2023-11-16T18:16:14.509 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.347+0000 7f1113fff700 1 -- 172.21.15.2:0/699208204 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f111c01a6e0 con 0x7f112c106490 2023-11-16T18:16:14.509 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.348+0000 7f11311a6700 1 -- 172.21.15.2:0/699208204 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f112c109bb0 con 0x7f112c106490 2023-11-16T18:16:14.509 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.349+0000 7f1113fff700 1 -- 172.21.15.2:0/699208204 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f111c01a840 con 0x7f112c106490 2023-11-16T18:16:14.509 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.349+0000 7f1113fff700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 0x7f111403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:14.510 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.349+0000 7f1113fff700 1 -- 172.21.15.2:0/699208204 --> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f111403b320 con 0x7f1114038760 2023-11-16T18:16:14.510 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.349+0000 7f1113fff700 1 -- 172.21.15.2:0/699208204 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f111c04c6b0 con 0x7f112c106490 2023-11-16T18:16:14.510 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.349+0000 7f112a59c700 1 -- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 msgr2=0x7f111403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1609662822 2023-11-16T18:16:14.510 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.349+0000 7f112a59c700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 0x7f111403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T18:16:14.510 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.550+0000 7f112a59c700 1 -- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 msgr2=0x7f111403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1609662822 2023-11-16T18:16:14.510 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.550+0000 7f112a59c700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 0x7f111403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T18:16:14.510 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.950+0000 7f112a59c700 1 -- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 msgr2=0x7f111403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1609662822 2023-11-16T18:16:14.510 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:07.950+0000 7f112a59c700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 0x7f111403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-16T18:16:14.511 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:08.751+0000 7f112a59c700 1 -- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 msgr2=0x7f111403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1609662822 2023-11-16T18:16:14.511 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:08.751+0000 7f112a59c700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 0x7f111403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-16T18:16:14.511 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:10.352+0000 7f112a59c700 1 -- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 msgr2=0x7f111403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1609662822 2023-11-16T18:16:14.511 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:10.352+0000 7f112a59c700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 0x7f111403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-16T18:16:14.511 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:13.218+0000 7f1113fff700 1 -- 172.21.15.2:0/699208204 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7f111c02d910 con 0x7f112c106490 2023-11-16T18:16:14.511 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:13.218+0000 7f1113fff700 1 -- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 msgr2=0x7f111403ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T18:16:14.511 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:13.218+0000 7f1113fff700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 0x7f111403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:14.511 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.219+0000 7f1113fff700 1 -- 172.21.15.2:0/699208204 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f111c04e180 con 0x7f112c106490 2023-11-16T18:16:14.511 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.219+0000 7f1113fff700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f111403bfe0 0x7f111403e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:14.511 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.219+0000 7f1113fff700 1 -- 172.21.15.2:0/699208204 --> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f111403b320 con 0x7f111403bfe0 2023-11-16T18:16:14.511 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.220+0000 7f112a59c700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f111403bfe0 0x7f111403e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:14.511 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.220+0000 7f112a59c700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f111403bfe0 0x7f111403e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1120003a10 tx=0x7f11200092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:14.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.221+0000 7f1113fff700 1 -- 172.21.15.2:0/699208204 <== mgr.14120 v2:172.21.15.2:6800/1605535303 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f111403b320 con 0x7f111403bfe0 2023-11-16T18:16:14.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.232+0000 7f11311a6700 1 -- 172.21.15.2:0/699208204 --> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f112c19ccb0 con 0x7f111403bfe0 2023-11-16T18:16:14.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.232+0000 7f1113fff700 1 -- 172.21.15.2:0/699208204 <== mgr.14120 v2:172.21.15.2:6800/1605535303 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f112c19ccb0 con 0x7f111403bfe0 2023-11-16T18:16:14.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.233+0000 7f11311a6700 1 -- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f111403bfe0 msgr2=0x7f111403e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:14.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.233+0000 7f11311a6700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f111403bfe0 0x7f111403e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1120003a10 tx=0x7f11200092b0 comp rx=0 tx=0).stop 2023-11-16T18:16:14.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.233+0000 7f11311a6700 1 -- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 msgr2=0x7f112c19b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:14.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.233+0000 7f11311a6700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 0x7f112c19b970 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f111c006af0 tx=0x7f111c011ad0 comp rx=0 tx=0).stop 2023-11-16T18:16:14.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.233+0000 7f11311a6700 1 -- 172.21.15.2:0/699208204 shutdown_connections 2023-11-16T18:16:14.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.233+0000 7f11311a6700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f111403bfe0 0x7f111403e3c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:14.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.233+0000 7f11311a6700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f112c106490 0x7f112c19b970 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:14.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.233+0000 7f11311a6700 1 --2- 172.21.15.2:0/699208204 >> [v2:172.21.15.2:6800/1609662822,v1:172.21.15.2:6801/1609662822] conn(0x7f1114038760 0x7f111403ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:14.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.233+0000 7f11311a6700 1 -- 172.21.15.2:0/699208204 >> 172.21.15.2:0/699208204 conn(0x7f112c0ffed0 msgr2=0x7f112c100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:14.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.233+0000 7f11311a6700 1 -- 172.21.15.2:0/699208204 shutdown_connections 2023-11-16T18:16:14.513 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:14.233+0000 7f11311a6700 1 -- 172.21.15.2:0/699208204 wait complete. 2023-11-16T18:16:14.513 INFO:teuthology.orchestra.run.smithi002.stdout:mgr epoch 5 is available 2023-11-16T18:16:14.513 INFO:teuthology.orchestra.run.smithi002.stdout:Setting orchestrator backend to cephadm... 2023-11-16T18:16:14.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:14 smithi002 ceph-mon[98807]: Found migration_current of "None". Setting to last migration. 2023-11-16T18:16:14.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:14 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:16:14.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:14 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T18:16:14.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:14 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T18:16:14.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:14 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:14.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:14 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:14.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:14 smithi002 ceph-mon[98807]: mgrmap e7: x(active, since 1.00551s) 2023-11-16T18:16:15.545 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:15 smithi002 ceph-mon[98807]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-16T18:16:15.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:15 smithi002 ceph-mon[98807]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-16T18:16:15.745 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.008+0000 7fb83c94f700 1 Processor -- start 2023-11-16T18:16:15.752 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.009+0000 7fb83c94f700 1 -- start start 2023-11-16T18:16:15.752 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.009+0000 7fb83c94f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 0x7fb838108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:15.752 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.009+0000 7fb83c94f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb838079da0 con 0x7fb838106550 2023-11-16T18:16:15.752 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.009+0000 7fb83659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 0x7fb838108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:15.752 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.009+0000 7fb83659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 0x7fb838108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46612/0 (socket says 172.21.15.2:46612) 2023-11-16T18:16:15.752 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.009+0000 7fb83659c700 1 -- 172.21.15.2:0/3463198254 learned_addr learned my addr 172.21.15.2:0/3463198254 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:15.752 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.010+0000 7fb83659c700 1 -- 172.21.15.2:0/3463198254 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb838079ee0 con 0x7fb838106550 2023-11-16T18:16:15.753 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.010+0000 7fb83659c700 1 --2- 172.21.15.2:0/3463198254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 0x7fb838108930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb8280098a0 tx=0x7fb828009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b73f8c4a8060d7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:15.753 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.010+0000 7fb83559a700 1 -- 172.21.15.2:0/3463198254 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb828004160 con 0x7fb838106550 2023-11-16T18:16:15.753 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.010+0000 7fb83559a700 1 -- 172.21.15.2:0/3463198254 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8280042c0 con 0x7fb838106550 2023-11-16T18:16:15.753 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.010+0000 7fb83559a700 1 -- 172.21.15.2:0/3463198254 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb828011a00 con 0x7fb838106550 2023-11-16T18:16:15.753 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.011+0000 7fb83c94f700 1 -- 172.21.15.2:0/3463198254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 msgr2=0x7fb838108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:15.753 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.011+0000 7fb83c94f700 1 --2- 172.21.15.2:0/3463198254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 0x7fb838108930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb8280098a0 tx=0x7fb828009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:15.753 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.011+0000 7fb83c94f700 1 -- 172.21.15.2:0/3463198254 shutdown_connections 2023-11-16T18:16:15.753 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.011+0000 7fb83c94f700 1 --2- 172.21.15.2:0/3463198254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 0x7fb838108930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:15.753 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.011+0000 7fb83c94f700 1 -- 172.21.15.2:0/3463198254 >> 172.21.15.2:0/3463198254 conn(0x7fb8380ffed0 msgr2=0x7fb838102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:15.753 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.011+0000 7fb83c94f700 1 -- 172.21.15.2:0/3463198254 shutdown_connections 2023-11-16T18:16:15.753 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.011+0000 7fb83c94f700 1 -- 172.21.15.2:0/3463198254 wait complete. 2023-11-16T18:16:15.754 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.012+0000 7fb83c94f700 1 Processor -- start 2023-11-16T18:16:15.754 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.012+0000 7fb83c94f700 1 -- start start 2023-11-16T18:16:15.754 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.012+0000 7fb83c94f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 0x7fb838199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:15.754 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.012+0000 7fb83c94f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb838199cb0 con 0x7fb838106550 2023-11-16T18:16:15.754 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.012+0000 7fb83659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 0x7fb838199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:15.754 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.012+0000 7fb83659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 0x7fb838199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46614/0 (socket says 172.21.15.2:46614) 2023-11-16T18:16:15.754 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.012+0000 7fb83659c700 1 -- 172.21.15.2:0/705863819 learned_addr learned my addr 172.21.15.2:0/705863819 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:15.754 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.012+0000 7fb83659c700 1 -- 172.21.15.2:0/705863819 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb828009550 con 0x7fb838106550 2023-11-16T18:16:15.754 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.013+0000 7fb83659c700 1 --2- 172.21.15.2:0/705863819 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 0x7fb838199770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb828006af0 tx=0x7fb828011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:15.754 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.013+0000 7fb8277fe700 1 -- 172.21.15.2:0/705863819 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb828003a40 con 0x7fb838106550 2023-11-16T18:16:15.754 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.013+0000 7fb8277fe700 1 -- 172.21.15.2:0/705863819 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb828003ba0 con 0x7fb838106550 2023-11-16T18:16:15.755 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.013+0000 7fb8277fe700 1 -- 172.21.15.2:0/705863819 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb82801a6e0 con 0x7fb838106550 2023-11-16T18:16:15.755 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.013+0000 7fb83c94f700 1 -- 172.21.15.2:0/705863819 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb838109d30 con 0x7fb838106550 2023-11-16T18:16:15.755 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.013+0000 7fb83c94f700 1 -- 172.21.15.2:0/705863819 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb838109ba0 con 0x7fb838106550 2023-11-16T18:16:15.755 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.014+0000 7fb8277fe700 1 -- 172.21.15.2:0/705863819 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fb828024a60 con 0x7fb838106550 2023-11-16T18:16:15.755 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.014+0000 7fb83c94f700 1 -- 172.21.15.2:0/705863819 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb838062120 con 0x7fb838106550 2023-11-16T18:16:15.755 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.014+0000 7fb8277fe700 1 --2- 172.21.15.2:0/705863819 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fb820038630 0x7fb82003aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:15.755 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.014+0000 7fb8277fe700 1 -- 172.21.15.2:0/705863819 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb82804c540 con 0x7fb838106550 2023-11-16T18:16:15.756 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.014+0000 7fb835d9b700 1 --2- 172.21.15.2:0/705863819 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fb820038630 0x7fb82003aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:15.756 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.015+0000 7fb835d9b700 1 --2- 172.21.15.2:0/705863819 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fb820038630 0x7fb82003aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb82c006fd0 tx=0x7fb82c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:15.756 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.019+0000 7fb8277fe700 1 -- 172.21.15.2:0/705863819 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb828024430 con 0x7fb838106550 2023-11-16T18:16:15.756 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.270+0000 7fb8277fe700 1 -- 172.21.15.2:0/705863819 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fb8280246c0 con 0x7fb838106550 2023-11-16T18:16:15.756 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.406+0000 7fb83c94f700 1 -- 172.21.15.2:0/705863819 --> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fb838102190 con 0x7fb820038630 2023-11-16T18:16:15.756 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.411+0000 7fb8277fe700 1 -- 172.21.15.2:0/705863819 <== mgr.14120 v2:172.21.15.2:6800/1605535303 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb838102190 con 0x7fb820038630 2023-11-16T18:16:15.756 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.414+0000 7fb83c94f700 1 -- 172.21.15.2:0/705863819 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fb820038630 msgr2=0x7fb82003aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:15.756 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.414+0000 7fb83c94f700 1 --2- 172.21.15.2:0/705863819 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fb820038630 0x7fb82003aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb82c006fd0 tx=0x7fb82c006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:15.756 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.414+0000 7fb83c94f700 1 -- 172.21.15.2:0/705863819 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 msgr2=0x7fb838199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:15.756 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.414+0000 7fb83c94f700 1 --2- 172.21.15.2:0/705863819 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 0x7fb838199770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb828006af0 tx=0x7fb828011ad0 comp rx=0 tx=0).stop 2023-11-16T18:16:15.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.414+0000 7fb83c94f700 1 -- 172.21.15.2:0/705863819 shutdown_connections 2023-11-16T18:16:15.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.414+0000 7fb83c94f700 1 --2- 172.21.15.2:0/705863819 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fb820038630 0x7fb82003aae0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:15.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.414+0000 7fb83c94f700 1 --2- 172.21.15.2:0/705863819 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb838106550 0x7fb838199770 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:15.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.414+0000 7fb83c94f700 1 -- 172.21.15.2:0/705863819 >> 172.21.15.2:0/705863819 conn(0x7fb8380ffed0 msgr2=0x7fb838101570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:15.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.414+0000 7fb83c94f700 1 -- 172.21.15.2:0/705863819 shutdown_connections 2023-11-16T18:16:15.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:15.414+0000 7fb83c94f700 1 -- 172.21.15.2:0/705863819 wait complete. 2023-11-16T18:16:16.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:16 smithi002 ceph-mon[98807]: mgrmap e8: x(active, since 2s) 2023-11-16T18:16:16.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:16 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:16.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:16 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:16:16.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:16 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:16:16.897 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout value unchanged 2023-11-16T18:16:16.897 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.194+0000 7f0cce788700 1 Processor -- start 2023-11-16T18:16:16.898 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.195+0000 7f0cce788700 1 -- start start 2023-11-16T18:16:16.898 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.195+0000 7f0cce788700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 0x7f0cc8105ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:16.898 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.195+0000 7f0cce788700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc8106410 con 0x7f0cc8105ac0 2023-11-16T18:16:16.898 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.195+0000 7f0cc7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 0x7f0cc8105ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:16.899 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.195+0000 7f0cc7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 0x7f0cc8105ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46626/0 (socket says 172.21.15.2:46626) 2023-11-16T18:16:16.899 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.195+0000 7f0cc7fff700 1 -- 172.21.15.2:0/2973892376 learned_addr learned my addr 172.21.15.2:0/2973892376 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:16.899 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.195+0000 7f0cc7fff700 1 -- 172.21.15.2:0/2973892376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc8106550 con 0x7f0cc8105ac0 2023-11-16T18:16:16.899 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.195+0000 7f0cc7fff700 1 --2- 172.21.15.2:0/2973892376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 0x7f0cc8105ed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0cb80098a0 tx=0x7f0cb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87ac58c8696ba1f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:16.900 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.196+0000 7f0cc6ffd700 1 -- 172.21.15.2:0/2973892376 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0cb8004160 con 0x7f0cc8105ac0 2023-11-16T18:16:16.900 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.196+0000 7f0cc6ffd700 1 -- 172.21.15.2:0/2973892376 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cb80042c0 con 0x7f0cc8105ac0 2023-11-16T18:16:16.900 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.196+0000 7f0cc6ffd700 1 -- 172.21.15.2:0/2973892376 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0cb8011a00 con 0x7f0cc8105ac0 2023-11-16T18:16:16.900 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.196+0000 7f0cce788700 1 -- 172.21.15.2:0/2973892376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 msgr2=0x7f0cc8105ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:16.900 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.196+0000 7f0cce788700 1 --2- 172.21.15.2:0/2973892376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 0x7f0cc8105ed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0cb80098a0 tx=0x7f0cb8009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:16.901 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.196+0000 7f0cce788700 1 -- 172.21.15.2:0/2973892376 shutdown_connections 2023-11-16T18:16:16.901 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.196+0000 7f0cce788700 1 --2- 172.21.15.2:0/2973892376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 0x7f0cc8105ed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:16.901 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.196+0000 7f0cce788700 1 -- 172.21.15.2:0/2973892376 >> 172.21.15.2:0/2973892376 conn(0x7f0cc8101170 msgr2=0x7f0cc8103580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:16.901 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.196+0000 7f0cce788700 1 -- 172.21.15.2:0/2973892376 shutdown_connections 2023-11-16T18:16:16.901 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.196+0000 7f0cce788700 1 -- 172.21.15.2:0/2973892376 wait complete. 2023-11-16T18:16:16.902 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.197+0000 7f0cce788700 1 Processor -- start 2023-11-16T18:16:16.902 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.197+0000 7f0cce788700 1 -- start start 2023-11-16T18:16:16.902 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.197+0000 7f0cce788700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 0x7f0cc8199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:16.902 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.197+0000 7f0cce788700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc8199d60 con 0x7f0cc8105ac0 2023-11-16T18:16:16.903 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.197+0000 7f0cc7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 0x7f0cc8199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:16.903 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.197+0000 7f0cc7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 0x7f0cc8199820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46628/0 (socket says 172.21.15.2:46628) 2023-11-16T18:16:16.903 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.197+0000 7f0cc7fff700 1 -- 172.21.15.2:0/4132467859 learned_addr learned my addr 172.21.15.2:0/4132467859 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:16.903 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.198+0000 7f0cc7fff700 1 -- 172.21.15.2:0/4132467859 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb8009550 con 0x7f0cc8105ac0 2023-11-16T18:16:16.903 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.198+0000 7f0cc7fff700 1 --2- 172.21.15.2:0/4132467859 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 0x7f0cc8199820 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009fb0 tx=0x7f0cb8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:16.904 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.198+0000 7f0cc57fa700 1 -- 172.21.15.2:0/4132467859 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0cb8003930 con 0x7f0cc8105ac0 2023-11-16T18:16:16.904 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.198+0000 7f0cc57fa700 1 -- 172.21.15.2:0/4132467859 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cb8003a90 con 0x7f0cc8105ac0 2023-11-16T18:16:16.904 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.198+0000 7f0cce788700 1 -- 172.21.15.2:0/4132467859 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc8107b10 con 0x7f0cc8105ac0 2023-11-16T18:16:16.904 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.198+0000 7f0cc57fa700 1 -- 172.21.15.2:0/4132467859 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0cb8019610 con 0x7f0cc8105ac0 2023-11-16T18:16:16.905 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.198+0000 7f0cce788700 1 -- 172.21.15.2:0/4132467859 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc8107980 con 0x7f0cc8105ac0 2023-11-16T18:16:16.905 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.199+0000 7f0cc57fa700 1 -- 172.21.15.2:0/4132467859 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f0cb80198d0 con 0x7f0cc8105ac0 2023-11-16T18:16:16.905 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.199+0000 7f0cce788700 1 -- 172.21.15.2:0/4132467859 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc8109f90 con 0x7f0cc8105ac0 2023-11-16T18:16:16.905 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.199+0000 7f0cc57fa700 1 --2- 172.21.15.2:0/4132467859 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f0cb00386c0 0x7f0cb003ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:16.906 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.200+0000 7f0cc57fa700 1 -- 172.21.15.2:0/4132467859 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0cb804cad0 con 0x7f0cc8105ac0 2023-11-16T18:16:16.906 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.200+0000 7f0cc77fe700 1 --2- 172.21.15.2:0/4132467859 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f0cb00386c0 0x7f0cb003ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:16.906 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.200+0000 7f0cc77fe700 1 --2- 172.21.15.2:0/4132467859 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f0cb00386c0 0x7f0cb003ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc006fd0 tx=0x7f0cbc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:16.906 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.205+0000 7f0cc57fa700 1 -- 172.21.15.2:0/4132467859 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0cb802b340 con 0x7f0cc8105ac0 2023-11-16T18:16:16.907 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.569+0000 7f0cce788700 1 -- 172.21.15.2:0/4132467859 --> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cc8102630 con 0x7f0cb00386c0 2023-11-16T18:16:16.907 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.569+0000 7f0cc57fa700 1 -- 172.21.15.2:0/4132467859 <== mgr.14120 v2:172.21.15.2:6800/1605535303 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f0cc8102630 con 0x7f0cb00386c0 2023-11-16T18:16:16.907 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.571+0000 7f0cce788700 1 -- 172.21.15.2:0/4132467859 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f0cb00386c0 msgr2=0x7f0cb003ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:16.908 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.571+0000 7f0cce788700 1 --2- 172.21.15.2:0/4132467859 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f0cb00386c0 0x7f0cb003ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc006fd0 tx=0x7f0cbc006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:16.908 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.571+0000 7f0cce788700 1 -- 172.21.15.2:0/4132467859 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 msgr2=0x7f0cc8199820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:16.909 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.571+0000 7f0cce788700 1 --2- 172.21.15.2:0/4132467859 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 0x7f0cc8199820 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009fb0 tx=0x7f0cb8011ad0 comp rx=0 tx=0).stop 2023-11-16T18:16:16.909 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.572+0000 7f0cce788700 1 -- 172.21.15.2:0/4132467859 shutdown_connections 2023-11-16T18:16:16.909 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.572+0000 7f0cce788700 1 --2- 172.21.15.2:0/4132467859 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f0cb00386c0 0x7f0cb003ab70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:16.910 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.572+0000 7f0cce788700 1 --2- 172.21.15.2:0/4132467859 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cc8105ac0 0x7f0cc8199820 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:16.914 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.572+0000 7f0cce788700 1 -- 172.21.15.2:0/4132467859 >> 172.21.15.2:0/4132467859 conn(0x7f0cc8101170 msgr2=0x7f0cc8101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:16.923 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.572+0000 7f0cce788700 1 -- 172.21.15.2:0/4132467859 shutdown_connections 2023-11-16T18:16:16.924 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:16.572+0000 7f0cce788700 1 -- 172.21.15.2:0/4132467859 wait complete. 2023-11-16T18:16:16.925 INFO:teuthology.orchestra.run.smithi002.stdout:Generating ssh key... 2023-11-16T18:16:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-mon[98807]: [16/Nov/2023:18:16:15] ENGINE Bus STARTING 2023-11-16T18:16:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-mon[98807]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:16:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-mon[98807]: [16/Nov/2023:18:16:15] ENGINE Serving on http://172.21.15.2:8765 2023-11-16T18:16:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-mon[98807]: [16/Nov/2023:18:16:15] ENGINE Serving on https://172.21.15.2:7150 2023-11-16T18:16:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-mon[98807]: [16/Nov/2023:18:16:15] ENGINE Bus STARTED 2023-11-16T18:16:18.204 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: Generating public/private rsa key pair. 2023-11-16T18:16:18.204 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: Your identification has been saved in /tmp/tmpi7yu9d85/key. 2023-11-16T18:16:18.204 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: Your public key has been saved in /tmp/tmpi7yu9d85/key.pub. 2023-11-16T18:16:18.204 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: The key fingerprint is: 2023-11-16T18:16:18.204 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: SHA256:XLhcCCUxs4uqgtO9PCnDOxris0604nc3DW+aKWtrPR0 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:16:18.204 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: The key's randomart image is: 2023-11-16T18:16:18.204 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: +---[RSA 3072]----+ 2023-11-16T18:16:18.204 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: | *o. | 2023-11-16T18:16:18.204 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: | * o | 2023-11-16T18:16:18.204 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: | . o o | 2023-11-16T18:16:18.205 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: | . + + | 2023-11-16T18:16:18.205 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: | . . . S | 2023-11-16T18:16:18.205 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: |. .. . E | 2023-11-16T18:16:18.205 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: |==.. .. = . | 2023-11-16T18:16:18.205 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: |X+*o+= =o= | 2023-11-16T18:16:18.205 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: |=B==*==+= | 2023-11-16T18:16:18.205 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: +----[SHA256]-----+ 2023-11-16T18:16:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.339+0000 7f71e361c700 1 Processor -- start 2023-11-16T18:16:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.339+0000 7f71e361c700 1 -- start start 2023-11-16T18:16:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.339+0000 7f71e361c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 0x7f71dc104710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.340+0000 7f71e361c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc079de0 con 0x7f71dc104300 2023-11-16T18:16:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.340+0000 7f71e13b8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 0x7f71dc104710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.340+0000 7f71e13b8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 0x7f71dc104710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46632/0 (socket says 172.21.15.2:46632) 2023-11-16T18:16:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.340+0000 7f71e13b8700 1 -- 172.21.15.2:0/2538471376 learned_addr learned my addr 172.21.15.2:0/2538471376 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.340+0000 7f71e13b8700 1 -- 172.21.15.2:0/2538471376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc104c50 con 0x7f71dc104300 2023-11-16T18:16:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.340+0000 7f71e13b8700 1 --2- 172.21.15.2:0/2538471376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 0x7f71dc104710 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f71d80098a0 tx=0x7f71d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7061ea551738f717 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.340+0000 7f71cffff700 1 -- 172.21.15.2:0/2538471376 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71d8004160 con 0x7f71dc104300 2023-11-16T18:16:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.340+0000 7f71cffff700 1 -- 172.21.15.2:0/2538471376 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71d80042c0 con 0x7f71dc104300 2023-11-16T18:16:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.340+0000 7f71cffff700 1 -- 172.21.15.2:0/2538471376 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71d8011a00 con 0x7f71dc104300 2023-11-16T18:16:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.341+0000 7f71e361c700 1 -- 172.21.15.2:0/2538471376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 msgr2=0x7f71dc104710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.341+0000 7f71e361c700 1 --2- 172.21.15.2:0/2538471376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 0x7f71dc104710 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f71d80098a0 tx=0x7f71d8009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.341+0000 7f71e361c700 1 -- 172.21.15.2:0/2538471376 shutdown_connections 2023-11-16T18:16:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.341+0000 7f71e361c700 1 --2- 172.21.15.2:0/2538471376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 0x7f71dc104710 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.341+0000 7f71e361c700 1 -- 172.21.15.2:0/2538471376 >> 172.21.15.2:0/2538471376 conn(0x7f71dc0ffed0 msgr2=0x7f71dc102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.341+0000 7f71e361c700 1 -- 172.21.15.2:0/2538471376 shutdown_connections 2023-11-16T18:16:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.341+0000 7f71e361c700 1 -- 172.21.15.2:0/2538471376 wait complete. 2023-11-16T18:16:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.342+0000 7f71e361c700 1 Processor -- start 2023-11-16T18:16:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.342+0000 7f71e361c700 1 -- start start 2023-11-16T18:16:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.342+0000 7f71e361c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 0x7f71dc19b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.342+0000 7f71e361c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc19beb0 con 0x7f71dc104300 2023-11-16T18:16:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.342+0000 7f71e13b8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 0x7f71dc19b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.342+0000 7f71e13b8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 0x7f71dc19b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46634/0 (socket says 172.21.15.2:46634) 2023-11-16T18:16:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.342+0000 7f71e13b8700 1 -- 172.21.15.2:0/3460174920 learned_addr learned my addr 172.21.15.2:0/3460174920 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.342+0000 7f71e13b8700 1 -- 172.21.15.2:0/3460174920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d8009550 con 0x7f71dc104300 2023-11-16T18:16:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.342+0000 7f71e13b8700 1 --2- 172.21.15.2:0/3460174920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 0x7f71dc19b970 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f71d8006af0 tx=0x7f71d8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.343+0000 7f71ce7fc700 1 -- 172.21.15.2:0/3460174920 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71d8003a40 con 0x7f71dc104300 2023-11-16T18:16:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.343+0000 7f71ce7fc700 1 -- 172.21.15.2:0/3460174920 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71d8003ba0 con 0x7f71dc104300 2023-11-16T18:16:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.343+0000 7f71e361c700 1 -- 172.21.15.2:0/3460174920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71dc105c50 con 0x7f71dc104300 2023-11-16T18:16:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.343+0000 7f71ce7fc700 1 -- 172.21.15.2:0/3460174920 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71d801a6e0 con 0x7f71dc104300 2023-11-16T18:16:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.343+0000 7f71e361c700 1 -- 172.21.15.2:0/3460174920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71dc105ac0 con 0x7f71dc104300 2023-11-16T18:16:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.344+0000 7f71ce7fc700 1 -- 172.21.15.2:0/3460174920 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f71d801a860 con 0x7f71dc104300 2023-11-16T18:16:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.344+0000 7f71ce7fc700 1 --2- 172.21.15.2:0/3460174920 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f71c8038760 0x7f71c803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.344+0000 7f71ce7fc700 1 -- 172.21.15.2:0/3460174920 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f71d804c960 con 0x7f71dc104300 2023-11-16T18:16:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.344+0000 7f71e0bb7700 1 --2- 172.21.15.2:0/3460174920 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f71c8038760 0x7f71c803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.344+0000 7f71e361c700 1 -- 172.21.15.2:0/3460174920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71dc195240 con 0x7f71dc104300 2023-11-16T18:16:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.344+0000 7f71e0bb7700 1 --2- 172.21.15.2:0/3460174920 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f71c8038760 0x7f71c803ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f71d0006fd0 tx=0x7f71d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.349+0000 7f71ce7fc700 1 -- 172.21.15.2:0/3460174920 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f71d801ab10 con 0x7f71dc104300 2023-11-16T18:16:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.711+0000 7f71e361c700 1 -- 172.21.15.2:0/3460174920 --> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f71dc19c690 con 0x7f71c8038760 2023-11-16T18:16:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.863+0000 7f71ce7fc700 1 -- 172.21.15.2:0/3460174920 <== mgr.14120 v2:172.21.15.2:6800/1605535303 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f71dc19c690 con 0x7f71c8038760 2023-11-16T18:16:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.865+0000 7f71e361c700 1 -- 172.21.15.2:0/3460174920 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f71c8038760 msgr2=0x7f71c803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.865+0000 7f71e361c700 1 --2- 172.21.15.2:0/3460174920 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f71c8038760 0x7f71c803ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f71d0006fd0 tx=0x7f71d0006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.865+0000 7f71e361c700 1 -- 172.21.15.2:0/3460174920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 msgr2=0x7f71dc19b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.865+0000 7f71e361c700 1 --2- 172.21.15.2:0/3460174920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 0x7f71dc19b970 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f71d8006af0 tx=0x7f71d8011ad0 comp rx=0 tx=0).stop 2023-11-16T18:16:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.866+0000 7f71e361c700 1 -- 172.21.15.2:0/3460174920 shutdown_connections 2023-11-16T18:16:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.866+0000 7f71e361c700 1 --2- 172.21.15.2:0/3460174920 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f71c8038760 0x7f71c803ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.866+0000 7f71e361c700 1 --2- 172.21.15.2:0/3460174920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71dc104300 0x7f71dc19b970 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.866+0000 7f71e361c700 1 -- 172.21.15.2:0/3460174920 >> 172.21.15.2:0/3460174920 conn(0x7f71dc0ffed0 msgr2=0x7f71dc18fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.866+0000 7f71e361c700 1 -- 172.21.15.2:0/3460174920 shutdown_connections 2023-11-16T18:16:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:17.866+0000 7f71e361c700 1 -- 172.21.15.2:0/3460174920 wait complete. 2023-11-16T18:16:18.469 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:18 smithi002 ceph-mon[98807]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:16:18.469 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:18 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:18.469 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:18 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:19.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:19 smithi002 ceph-mon[98807]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:16:19.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:19 smithi002 ceph-mon[98807]: Generating ssh key... 2023-11-16T18:16:19.388 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+0R8beOieZYos5G8AZwNJgU5L9F8r/fdqji0sI5xDZQpfN+FWquOcbr5u/Pn+kPPxTWcPcU1d2Q/dI8qcciTMFTF8FIqORgv1cH7eoNNB8c5Iy9o/l18kQPZ//38jyiLrutX0bLKWWr9Npg70flkUTgs9DSFgjZUOL+pIMmBK3IlkQwk4imWENBR83G9GJ1U3HGIW1l+qZO1j6JJUiQxctTDfdf9LaGqXSObZojkLuiLOmMitF8BLs/RQ9BOrzYRuiCqHzrxJaEMGdLB7kuqDizlu8FLHvvYYHyZ27pQ3kAx+cnbQonyYXJctAy+rCrD6txNEZWlXDBNnawmRPnEkWxV3/H7/RIVlXPF7STtZEZBq+AccyFi1lTNqILKGzwTT+mGN/Fvr8Nl05aDCYPaEe+LW3KHlDkHEoq4NZwzaA88oQmUMpVcSYgjISgvI3QixqjepCCVELU7RKv7GYiTM9xfwHcEdjBnzNH1gQ2+N/5++VtyoSzmkRoxvNIKzlo8= ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:16:19.388 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.696+0000 7f53687fc700 1 Processor -- start 2023-11-16T18:16:19.388 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.696+0000 7f53687fc700 1 -- start start 2023-11-16T18:16:19.388 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.696+0000 7f53687fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 0x7f53601044b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:19.388 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.696+0000 7f53687fc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5360104a80 con 0x7f53601040a0 2023-11-16T18:16:19.388 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.696+0000 7f5366598700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 0x7f53601044b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:19.388 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.696+0000 7f5366598700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 0x7f53601044b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46638/0 (socket says 172.21.15.2:46638) 2023-11-16T18:16:19.388 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.696+0000 7f5366598700 1 -- 172.21.15.2:0/4239316328 learned_addr learned my addr 172.21.15.2:0/4239316328 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:19.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.697+0000 7f5366598700 1 -- 172.21.15.2:0/4239316328 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53601052a0 con 0x7f53601040a0 2023-11-16T18:16:19.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.697+0000 7f5366598700 1 --2- 172.21.15.2:0/4239316328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 0x7f53601044b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f535c00aa40 tx=0x7f535c00ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c70988c76b381c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:19.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.697+0000 7f5365596700 1 -- 172.21.15.2:0/4239316328 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f535c004160 con 0x7f53601040a0 2023-11-16T18:16:19.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.697+0000 7f5365596700 1 -- 172.21.15.2:0/4239316328 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f535c00cd90 con 0x7f53601040a0 2023-11-16T18:16:19.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.697+0000 7f5365596700 1 -- 172.21.15.2:0/4239316328 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f535c003c80 con 0x7f53601040a0 2023-11-16T18:16:19.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.698+0000 7f53687fc700 1 -- 172.21.15.2:0/4239316328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 msgr2=0x7f53601044b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:19.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.698+0000 7f53687fc700 1 --2- 172.21.15.2:0/4239316328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 0x7f53601044b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f535c00aa40 tx=0x7f535c00ad50 comp rx=0 tx=0).stop 2023-11-16T18:16:19.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.698+0000 7f53687fc700 1 -- 172.21.15.2:0/4239316328 shutdown_connections 2023-11-16T18:16:19.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.698+0000 7f53687fc700 1 --2- 172.21.15.2:0/4239316328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 0x7f53601044b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:19.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.698+0000 7f53687fc700 1 -- 172.21.15.2:0/4239316328 >> 172.21.15.2:0/4239316328 conn(0x7f53600ff650 msgr2=0x7f5360101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:19.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.698+0000 7f53687fc700 1 -- 172.21.15.2:0/4239316328 shutdown_connections 2023-11-16T18:16:19.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.698+0000 7f53687fc700 1 -- 172.21.15.2:0/4239316328 wait complete. 2023-11-16T18:16:19.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.698+0000 7f53687fc700 1 Processor -- start 2023-11-16T18:16:19.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.698+0000 7f53687fc700 1 -- start start 2023-11-16T18:16:19.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.699+0000 7f53687fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 0x7f53601997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:19.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.699+0000 7f53687fc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5360199d20 con 0x7f53601040a0 2023-11-16T18:16:19.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.699+0000 7f5366598700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 0x7f53601997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:19.392 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.699+0000 7f5366598700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 0x7f53601997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46640/0 (socket says 172.21.15.2:46640) 2023-11-16T18:16:19.392 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.699+0000 7f5366598700 1 -- 172.21.15.2:0/3946734141 learned_addr learned my addr 172.21.15.2:0/3946734141 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:19.392 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.699+0000 7f5366598700 1 -- 172.21.15.2:0/3946734141 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535c00a6f0 con 0x7f53601040a0 2023-11-16T18:16:19.392 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.699+0000 7f5366598700 1 --2- 172.21.15.2:0/3946734141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 0x7f53601997e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f535c00afd0 tx=0x7f535c0041d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:19.392 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.699+0000 7f53577fe700 1 -- 172.21.15.2:0/3946734141 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f535c020400 con 0x7f53601040a0 2023-11-16T18:16:19.392 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.699+0000 7f53577fe700 1 -- 172.21.15.2:0/3946734141 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f535c020a00 con 0x7f53601040a0 2023-11-16T18:16:19.392 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.699+0000 7f53687fc700 1 -- 172.21.15.2:0/3946734141 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5360105990 con 0x7f53601040a0 2023-11-16T18:16:19.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.699+0000 7f53577fe700 1 -- 172.21.15.2:0/3946734141 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f535c0108e0 con 0x7f53601040a0 2023-11-16T18:16:19.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.700+0000 7f53687fc700 1 -- 172.21.15.2:0/3946734141 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5360105800 con 0x7f53601040a0 2023-11-16T18:16:19.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.700+0000 7f53577fe700 1 -- 172.21.15.2:0/3946734141 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f535c020b70 con 0x7f53601040a0 2023-11-16T18:16:19.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.701+0000 7f53687fc700 1 -- 172.21.15.2:0/3946734141 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5360193090 con 0x7f53601040a0 2023-11-16T18:16:19.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.701+0000 7f53577fe700 1 --2- 172.21.15.2:0/3946734141 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f534c038760 0x7f534c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:19.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.701+0000 7f53577fe700 1 -- 172.21.15.2:0/3946734141 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f535c04c530 con 0x7f53601040a0 2023-11-16T18:16:19.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.701+0000 7f5365d97700 1 --2- 172.21.15.2:0/3946734141 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f534c038760 0x7f534c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:19.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.701+0000 7f5365d97700 1 --2- 172.21.15.2:0/3946734141 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f534c038760 0x7f534c03ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5350006fd0 tx=0x7f5350006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:19.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:18.706+0000 7f53577fe700 1 -- 172.21.15.2:0/3946734141 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f535c025020 con 0x7f53601040a0 2023-11-16T18:16:19.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.067+0000 7f53687fc700 1 -- 172.21.15.2:0/3946734141 --> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f5360060f30 con 0x7f534c038760 2023-11-16T18:16:19.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.067+0000 7f53577fe700 1 -- 172.21.15.2:0/3946734141 <== mgr.14120 v2:172.21.15.2:6800/1605535303 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f5360060f30 con 0x7f534c038760 2023-11-16T18:16:19.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.070+0000 7f53687fc700 1 -- 172.21.15.2:0/3946734141 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f534c038760 msgr2=0x7f534c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:19.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.070+0000 7f53687fc700 1 --2- 172.21.15.2:0/3946734141 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f534c038760 0x7f534c03ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5350006fd0 tx=0x7f5350006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:19.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.070+0000 7f53687fc700 1 -- 172.21.15.2:0/3946734141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 msgr2=0x7f53601997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:19.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.070+0000 7f53687fc700 1 --2- 172.21.15.2:0/3946734141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 0x7f53601997e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f535c00afd0 tx=0x7f535c0041d0 comp rx=0 tx=0).stop 2023-11-16T18:16:19.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.070+0000 7f53687fc700 1 -- 172.21.15.2:0/3946734141 shutdown_connections 2023-11-16T18:16:19.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.070+0000 7f53687fc700 1 --2- 172.21.15.2:0/3946734141 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f534c038760 0x7f534c03ac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:19.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.070+0000 7f53687fc700 1 --2- 172.21.15.2:0/3946734141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53601040a0 0x7f53601997e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:19.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.070+0000 7f53687fc700 1 -- 172.21.15.2:0/3946734141 >> 172.21.15.2:0/3946734141 conn(0x7f53600ff650 msgr2=0x7f536018ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:19.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.070+0000 7f53687fc700 1 -- 172.21.15.2:0/3946734141 shutdown_connections 2023-11-16T18:16:19.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.070+0000 7f53687fc700 1 -- 172.21.15.2:0/3946734141 wait complete. 2023-11-16T18:16:19.394 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-16T18:16:19.394 INFO:teuthology.orchestra.run.smithi002.stdout:Adding key to root@localhost authorized_keys... 2023-11-16T18:16:19.395 INFO:teuthology.orchestra.run.smithi002.stdout:Adding host smithi002... 2023-11-16T18:16:20.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:20 smithi002 ceph-mon[98807]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:16:21.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:21 smithi002 ceph-mon[98807]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:16:22.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:22 smithi002 ceph-mon[98807]: Deploying cephadm binary to smithi002 2023-11-16T18:16:23.050 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Added host 'smithi002' with addr '172.21.15.2' 2023-11-16T18:16:23.057 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.833+0000 7f29ba813700 1 Processor -- start 2023-11-16T18:16:23.057 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.833+0000 7f29ba813700 1 -- start start 2023-11-16T18:16:23.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.833+0000 7f29ba813700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 0x7f29b4104d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:23.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.833+0000 7f29ba813700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b4105300 con 0x7f29b4104920 2023-11-16T18:16:23.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.833+0000 7f29b3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 0x7f29b4104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:23.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.833+0000 7f29b3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 0x7f29b4104d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46644/0 (socket says 172.21.15.2:46644) 2023-11-16T18:16:23.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.833+0000 7f29b3fff700 1 -- 172.21.15.2:0/2662157562 learned_addr learned my addr 172.21.15.2:0/2662157562 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:23.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.834+0000 7f29b3fff700 1 -- 172.21.15.2:0/2662157562 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b4105b20 con 0x7f29b4104920 2023-11-16T18:16:23.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.834+0000 7f29b3fff700 1 --2- 172.21.15.2:0/2662157562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 0x7f29b4104d30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f29a40098a0 tx=0x7f29a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b716ca379613a389 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:23.060 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.834+0000 7f29b2ffd700 1 -- 172.21.15.2:0/2662157562 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29a4004160 con 0x7f29b4104920 2023-11-16T18:16:23.060 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.834+0000 7f29b2ffd700 1 -- 172.21.15.2:0/2662157562 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29a40042c0 con 0x7f29b4104920 2023-11-16T18:16:23.060 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.834+0000 7f29b2ffd700 1 -- 172.21.15.2:0/2662157562 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29a4011a00 con 0x7f29b4104920 2023-11-16T18:16:23.060 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.835+0000 7f29ba813700 1 -- 172.21.15.2:0/2662157562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 msgr2=0x7f29b4104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:23.062 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.835+0000 7f29ba813700 1 --2- 172.21.15.2:0/2662157562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 0x7f29b4104d30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f29a40098a0 tx=0x7f29a4009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:23.062 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.835+0000 7f29ba813700 1 -- 172.21.15.2:0/2662157562 shutdown_connections 2023-11-16T18:16:23.062 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.835+0000 7f29ba813700 1 --2- 172.21.15.2:0/2662157562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 0x7f29b4104d30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:23.062 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.835+0000 7f29ba813700 1 -- 172.21.15.2:0/2662157562 >> 172.21.15.2:0/2662157562 conn(0x7f29b40ffed0 msgr2=0x7f29b4102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:23.062 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.835+0000 7f29ba813700 1 -- 172.21.15.2:0/2662157562 shutdown_connections 2023-11-16T18:16:23.062 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.835+0000 7f29ba813700 1 -- 172.21.15.2:0/2662157562 wait complete. 2023-11-16T18:16:23.062 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.835+0000 7f29ba813700 1 Processor -- start 2023-11-16T18:16:23.062 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.835+0000 7f29ba813700 1 -- start start 2023-11-16T18:16:23.062 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.836+0000 7f29ba813700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 0x7f29b4079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:23.062 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.836+0000 7f29ba813700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b419c680 con 0x7f29b4104920 2023-11-16T18:16:23.063 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.836+0000 7f29b3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 0x7f29b4079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:23.063 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.836+0000 7f29b3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 0x7f29b4079ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46646/0 (socket says 172.21.15.2:46646) 2023-11-16T18:16:23.063 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.836+0000 7f29b3fff700 1 -- 172.21.15.2:0/219340076 learned_addr learned my addr 172.21.15.2:0/219340076 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:23.063 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.836+0000 7f29b3fff700 1 -- 172.21.15.2:0/219340076 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a4009550 con 0x7f29b4104920 2023-11-16T18:16:23.063 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.836+0000 7f29b3fff700 1 --2- 172.21.15.2:0/219340076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 0x7f29b4079ae0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009fb0 tx=0x7f29a4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:23.063 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.836+0000 7f29b17fa700 1 -- 172.21.15.2:0/219340076 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29a4003930 con 0x7f29b4104920 2023-11-16T18:16:23.063 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.836+0000 7f29b17fa700 1 -- 172.21.15.2:0/219340076 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29a4003a90 con 0x7f29b4104920 2023-11-16T18:16:23.063 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.836+0000 7f29ba813700 1 -- 172.21.15.2:0/219340076 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b4108d50 con 0x7f29b4104920 2023-11-16T18:16:23.063 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.836+0000 7f29b17fa700 1 -- 172.21.15.2:0/219340076 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29a40195e0 con 0x7f29b4104920 2023-11-16T18:16:23.063 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.836+0000 7f29ba813700 1 -- 172.21.15.2:0/219340076 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b4108ee0 con 0x7f29b4104920 2023-11-16T18:16:23.064 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.837+0000 7f29b17fa700 1 -- 172.21.15.2:0/219340076 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f29a4019850 con 0x7f29b4104920 2023-11-16T18:16:23.064 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.838+0000 7f29b17fa700 1 --2- 172.21.15.2:0/219340076 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f299c038760 0x7f299c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:23.064 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.838+0000 7f29b17fa700 1 -- 172.21.15.2:0/219340076 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f29a404c880 con 0x7f29b4104920 2023-11-16T18:16:23.064 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.838+0000 7f29ba813700 1 -- 172.21.15.2:0/219340076 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b4106990 con 0x7f29b4104920 2023-11-16T18:16:23.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.838+0000 7f29b37fe700 1 --2- 172.21.15.2:0/219340076 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f299c038760 0x7f299c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:23.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.838+0000 7f29b37fe700 1 --2- 172.21.15.2:0/219340076 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f299c038760 0x7f299c03ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f29a8006fd0 tx=0x7f29a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:23.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:19.843+0000 7f29b17fa700 1 -- 172.21.15.2:0/219340076 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f29b4106990 con 0x7f29b4104920 2023-11-16T18:16:23.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:20.207+0000 7f29ba813700 1 -- 172.21.15.2:0/219340076 --> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}) v1 -- 0x7f29b4106990 con 0x7f299c038760 2023-11-16T18:16:23.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:22.625+0000 7f29b17fa700 1 -- 172.21.15.2:0/219340076 <== mgr.14120 v2:172.21.15.2:6800/1605535303 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f29b4106990 con 0x7f299c038760 2023-11-16T18:16:23.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:22.628+0000 7f29ba813700 1 -- 172.21.15.2:0/219340076 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f299c038760 msgr2=0x7f299c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:23.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:22.628+0000 7f29ba813700 1 --2- 172.21.15.2:0/219340076 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f299c038760 0x7f299c03ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f29a8006fd0 tx=0x7f29a8006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:23.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:22.628+0000 7f29ba813700 1 -- 172.21.15.2:0/219340076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 msgr2=0x7f29b4079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:23.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:22.628+0000 7f29ba813700 1 --2- 172.21.15.2:0/219340076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 0x7f29b4079ae0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009fb0 tx=0x7f29a4011ad0 comp rx=0 tx=0).stop 2023-11-16T18:16:23.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:22.629+0000 7f29ba813700 1 -- 172.21.15.2:0/219340076 shutdown_connections 2023-11-16T18:16:23.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:22.629+0000 7f29ba813700 1 --2- 172.21.15.2:0/219340076 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f299c038760 0x7f299c03ac10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:23.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:22.629+0000 7f29ba813700 1 --2- 172.21.15.2:0/219340076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b4104920 0x7f29b4079ae0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:23.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:22.629+0000 7f29ba813700 1 -- 172.21.15.2:0/219340076 >> 172.21.15.2:0/219340076 conn(0x7f29b40ffed0 msgr2=0x7f29b4101540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:23.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:22.629+0000 7f29ba813700 1 -- 172.21.15.2:0/219340076 shutdown_connections 2023-11-16T18:16:23.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:22.629+0000 7f29ba813700 1 -- 172.21.15.2:0/219340076 wait complete. 2023-11-16T18:16:23.069 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying unmanaged mon service... 2023-11-16T18:16:23.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:23 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:23.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:23 smithi002 ceph-mon[98807]: Added host smithi002 2023-11-16T18:16:23.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:23 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:16:24.387 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-11-16T18:16:24.387 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.496+0000 7f985ea8b700 1 Processor -- start 2023-11-16T18:16:24.387 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.496+0000 7f985ea8b700 1 -- start start 2023-11-16T18:16:24.387 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.497+0000 7f985ea8b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 0x7f9858105e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:24.387 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.497+0000 7f985ea8b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9858106400 con 0x7f9858105a20 2023-11-16T18:16:24.388 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.497+0000 7f985c827700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 0x7f9858105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:24.388 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.497+0000 7f985c827700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 0x7f9858105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46680/0 (socket says 172.21.15.2:46680) 2023-11-16T18:16:24.388 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.497+0000 7f985c827700 1 -- 172.21.15.2:0/3006980443 learned_addr learned my addr 172.21.15.2:0/3006980443 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:24.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.497+0000 7f985c827700 1 -- 172.21.15.2:0/3006980443 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9858106c20 con 0x7f9858105a20 2023-11-16T18:16:24.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.497+0000 7f985c827700 1 --2- 172.21.15.2:0/3006980443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 0x7f9858105e30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f984c0098a0 tx=0x7f984c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd432a36b40acfc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:24.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.497+0000 7f98577fe700 1 -- 172.21.15.2:0/3006980443 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f984c004160 con 0x7f9858105a20 2023-11-16T18:16:24.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.498+0000 7f98577fe700 1 -- 172.21.15.2:0/3006980443 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f984c0042c0 con 0x7f9858105a20 2023-11-16T18:16:24.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.498+0000 7f98577fe700 1 -- 172.21.15.2:0/3006980443 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f984c011a00 con 0x7f9858105a20 2023-11-16T18:16:24.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.498+0000 7f985ea8b700 1 -- 172.21.15.2:0/3006980443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 msgr2=0x7f9858105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:24.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.498+0000 7f985ea8b700 1 --2- 172.21.15.2:0/3006980443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 0x7f9858105e30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f984c0098a0 tx=0x7f984c009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:24.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.498+0000 7f985ea8b700 1 -- 172.21.15.2:0/3006980443 shutdown_connections 2023-11-16T18:16:24.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.498+0000 7f985ea8b700 1 --2- 172.21.15.2:0/3006980443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 0x7f9858105e30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:24.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.498+0000 7f985ea8b700 1 -- 172.21.15.2:0/3006980443 >> 172.21.15.2:0/3006980443 conn(0x7f9858100fd0 msgr2=0x7f9858103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:24.392 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.498+0000 7f985ea8b700 1 -- 172.21.15.2:0/3006980443 shutdown_connections 2023-11-16T18:16:24.392 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.498+0000 7f985ea8b700 1 -- 172.21.15.2:0/3006980443 wait complete. 2023-11-16T18:16:24.392 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.499+0000 7f985ea8b700 1 Processor -- start 2023-11-16T18:16:24.392 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.499+0000 7f985ea8b700 1 -- start start 2023-11-16T18:16:24.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.499+0000 7f985ea8b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 0x7f985819bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:24.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.499+0000 7f985ea8b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985819c090 con 0x7f9858105a20 2023-11-16T18:16:24.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.499+0000 7f985c827700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 0x7f985819bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:24.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.499+0000 7f985c827700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 0x7f985819bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46682/0 (socket says 172.21.15.2:46682) 2023-11-16T18:16:24.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.499+0000 7f985c827700 1 -- 172.21.15.2:0/2529610098 learned_addr learned my addr 172.21.15.2:0/2529610098 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:24.395 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.499+0000 7f985c827700 1 -- 172.21.15.2:0/2529610098 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c009550 con 0x7f9858105a20 2023-11-16T18:16:24.395 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.500+0000 7f985c827700 1 --2- 172.21.15.2:0/2529610098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 0x7f985819bb50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f984c009fb0 tx=0x7f984c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:24.395 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.500+0000 7f9855ffb700 1 -- 172.21.15.2:0/2529610098 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f984c003930 con 0x7f9858105a20 2023-11-16T18:16:24.395 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.500+0000 7f9855ffb700 1 -- 172.21.15.2:0/2529610098 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f984c003a90 con 0x7f9858105a20 2023-11-16T18:16:24.396 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.500+0000 7f985ea8b700 1 -- 172.21.15.2:0/2529610098 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9858109e50 con 0x7f9858105a20 2023-11-16T18:16:24.396 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.500+0000 7f9855ffb700 1 -- 172.21.15.2:0/2529610098 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f984c0195e0 con 0x7f9858105a20 2023-11-16T18:16:24.396 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.500+0000 7f985ea8b700 1 -- 172.21.15.2:0/2529610098 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9858109fe0 con 0x7f9858105a20 2023-11-16T18:16:24.396 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.501+0000 7f9855ffb700 1 -- 172.21.15.2:0/2529610098 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f984c019850 con 0x7f9858105a20 2023-11-16T18:16:24.397 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.501+0000 7f9855ffb700 1 --2- 172.21.15.2:0/2529610098 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f9848038710 0x7f984803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:24.397 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.501+0000 7f9855ffb700 1 -- 172.21.15.2:0/2529610098 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f984c04ca60 con 0x7f9858105a20 2023-11-16T18:16:24.398 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.501+0000 7f985ea8b700 1 -- 172.21.15.2:0/2529610098 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985810a170 con 0x7f9858105a20 2023-11-16T18:16:24.398 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.501+0000 7f9857fff700 1 --2- 172.21.15.2:0/2529610098 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f9848038710 0x7f984803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:24.398 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.502+0000 7f9857fff700 1 --2- 172.21.15.2:0/2529610098 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f9848038710 0x7f984803abc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9844006fd0 tx=0x7f9844006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:24.404 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.506+0000 7f9855ffb700 1 -- 172.21.15.2:0/2529610098 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f984c019b50 con 0x7f9858105a20 2023-11-16T18:16:24.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.904+0000 7f985ea8b700 1 -- 172.21.15.2:0/2529610098 --> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f98580621c0 con 0x7f9848038710 2023-11-16T18:16:24.413 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.909+0000 7f9855ffb700 1 -- 172.21.15.2:0/2529610098 <== mgr.14120 v2:172.21.15.2:6800/1605535303 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f98580621c0 con 0x7f9848038710 2023-11-16T18:16:24.418 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.911+0000 7f985ea8b700 1 -- 172.21.15.2:0/2529610098 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f9848038710 msgr2=0x7f984803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:24.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.911+0000 7f985ea8b700 1 --2- 172.21.15.2:0/2529610098 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f9848038710 0x7f984803abc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9844006fd0 tx=0x7f9844006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:24.428 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.911+0000 7f985ea8b700 1 -- 172.21.15.2:0/2529610098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 msgr2=0x7f985819bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:24.428 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.911+0000 7f985ea8b700 1 --2- 172.21.15.2:0/2529610098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 0x7f985819bb50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f984c009fb0 tx=0x7f984c011ad0 comp rx=0 tx=0).stop 2023-11-16T18:16:24.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.911+0000 7f985ea8b700 1 -- 172.21.15.2:0/2529610098 shutdown_connections 2023-11-16T18:16:24.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.911+0000 7f985ea8b700 1 --2- 172.21.15.2:0/2529610098 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f9848038710 0x7f984803abc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:24.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.911+0000 7f985ea8b700 1 --2- 172.21.15.2:0/2529610098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9858105a20 0x7f985819bb50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:24.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.911+0000 7f985ea8b700 1 -- 172.21.15.2:0/2529610098 >> 172.21.15.2:0/2529610098 conn(0x7f9858100fd0 msgr2=0x7f9858102640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:24.439 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.911+0000 7f985ea8b700 1 -- 172.21.15.2:0/2529610098 shutdown_connections 2023-11-16T18:16:24.439 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:23.911+0000 7f985ea8b700 1 -- 172.21.15.2:0/2529610098 wait complete. 2023-11-16T18:16:24.442 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying unmanaged mgr service... 2023-11-16T18:16:25.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:24 smithi002 ceph-mon[98807]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:16:25.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:24 smithi002 ceph-mon[98807]: Saving service mon spec with placement count:5 2023-11-16T18:16:25.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:24 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:25.635 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-11-16T18:16:25.635 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.877+0000 7fce3cdcc700 1 Processor -- start 2023-11-16T18:16:25.635 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.878+0000 7fce3cdcc700 1 -- start start 2023-11-16T18:16:25.635 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.878+0000 7fce3cdcc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 0x7fce38105ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:25.635 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.878+0000 7fce3cdcc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce38106410 con 0x7fce38105ac0 2023-11-16T18:16:25.635 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.878+0000 7fce3659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 0x7fce38105ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:25.635 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.878+0000 7fce3659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 0x7fce38105ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46686/0 (socket says 172.21.15.2:46686) 2023-11-16T18:16:25.635 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.878+0000 7fce3659c700 1 -- 172.21.15.2:0/3117109328 learned_addr learned my addr 172.21.15.2:0/3117109328 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:25.636 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.878+0000 7fce3659c700 1 -- 172.21.15.2:0/3117109328 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce38106550 con 0x7fce38105ac0 2023-11-16T18:16:25.636 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.879+0000 7fce3659c700 1 --2- 172.21.15.2:0/3117109328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 0x7fce38105ed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fce280098a0 tx=0x7fce28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72118e43f509687 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:25.636 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.879+0000 7fce3559a700 1 -- 172.21.15.2:0/3117109328 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce28004160 con 0x7fce38105ac0 2023-11-16T18:16:25.636 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.879+0000 7fce3559a700 1 -- 172.21.15.2:0/3117109328 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce280042c0 con 0x7fce38105ac0 2023-11-16T18:16:25.636 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.879+0000 7fce3559a700 1 -- 172.21.15.2:0/3117109328 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce28011a00 con 0x7fce38105ac0 2023-11-16T18:16:25.636 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.879+0000 7fce3cdcc700 1 -- 172.21.15.2:0/3117109328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 msgr2=0x7fce38105ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:25.636 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.879+0000 7fce3cdcc700 1 --2- 172.21.15.2:0/3117109328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 0x7fce38105ed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fce280098a0 tx=0x7fce28009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:25.636 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.879+0000 7fce3cdcc700 1 -- 172.21.15.2:0/3117109328 shutdown_connections 2023-11-16T18:16:25.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.879+0000 7fce3cdcc700 1 --2- 172.21.15.2:0/3117109328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 0x7fce38105ed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:25.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.879+0000 7fce3cdcc700 1 -- 172.21.15.2:0/3117109328 >> 172.21.15.2:0/3117109328 conn(0x7fce38101170 msgr2=0x7fce38103580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:25.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.880+0000 7fce3cdcc700 1 -- 172.21.15.2:0/3117109328 shutdown_connections 2023-11-16T18:16:25.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.880+0000 7fce3cdcc700 1 -- 172.21.15.2:0/3117109328 wait complete. 2023-11-16T18:16:25.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.881+0000 7fce3cdcc700 1 Processor -- start 2023-11-16T18:16:25.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.881+0000 7fce3cdcc700 1 -- start start 2023-11-16T18:16:25.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.881+0000 7fce3cdcc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 0x7fce38199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:25.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.881+0000 7fce3cdcc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce38199d60 con 0x7fce38105ac0 2023-11-16T18:16:25.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.881+0000 7fce3659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 0x7fce38199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:25.637 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.881+0000 7fce3659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 0x7fce38199820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46688/0 (socket says 172.21.15.2:46688) 2023-11-16T18:16:25.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.881+0000 7fce3659c700 1 -- 172.21.15.2:0/1742352814 learned_addr learned my addr 172.21.15.2:0/1742352814 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:25.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.881+0000 7fce3659c700 1 -- 172.21.15.2:0/1742352814 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce28009550 con 0x7fce38105ac0 2023-11-16T18:16:25.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.881+0000 7fce3659c700 1 --2- 172.21.15.2:0/1742352814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 0x7fce38199820 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fce28006af0 tx=0x7fce28011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:25.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.882+0000 7fce277fe700 1 -- 172.21.15.2:0/1742352814 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce28003a40 con 0x7fce38105ac0 2023-11-16T18:16:25.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.882+0000 7fce277fe700 1 -- 172.21.15.2:0/1742352814 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce28003ba0 con 0x7fce38105ac0 2023-11-16T18:16:25.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.882+0000 7fce3cdcc700 1 -- 172.21.15.2:0/1742352814 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce38107b10 con 0x7fce38105ac0 2023-11-16T18:16:25.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.882+0000 7fce277fe700 1 -- 172.21.15.2:0/1742352814 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce2801a6e0 con 0x7fce38105ac0 2023-11-16T18:16:25.638 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.882+0000 7fce3cdcc700 1 -- 172.21.15.2:0/1742352814 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce38107980 con 0x7fce38105ac0 2023-11-16T18:16:25.639 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.883+0000 7fce277fe700 1 -- 172.21.15.2:0/1742352814 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fce28024430 con 0x7fce38105ac0 2023-11-16T18:16:25.639 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.883+0000 7fce3cdcc700 1 -- 172.21.15.2:0/1742352814 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce38109f90 con 0x7fce38105ac0 2023-11-16T18:16:25.639 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.883+0000 7fce277fe700 1 --2- 172.21.15.2:0/1742352814 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fce20038710 0x7fce2003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:25.639 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.883+0000 7fce277fe700 1 -- 172.21.15.2:0/1742352814 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fce2804c730 con 0x7fce38105ac0 2023-11-16T18:16:25.639 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.883+0000 7fce35d9b700 1 --2- 172.21.15.2:0/1742352814 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fce20038710 0x7fce2003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:25.639 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.884+0000 7fce35d9b700 1 --2- 172.21.15.2:0/1742352814 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fce20038710 0x7fce2003abc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fce2c006fd0 tx=0x7fce2c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:25.639 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:24.888+0000 7fce277fe700 1 -- 172.21.15.2:0/1742352814 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fce280246e0 con 0x7fce38105ac0 2023-11-16T18:16:25.639 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:25.095+0000 7fce277fe700 1 -- 172.21.15.2:0/1742352814 <== mon.0 v2:172.21.15.2:3300/0 7 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce2804c140 con 0x7fce38105ac0 2023-11-16T18:16:25.639 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:25.278+0000 7fce3cdcc700 1 -- 172.21.15.2:0/1742352814 --> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fce38102630 con 0x7fce20038710 2023-11-16T18:16:25.640 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:25.283+0000 7fce277fe700 1 -- 172.21.15.2:0/1742352814 <== mgr.14120 v2:172.21.15.2:6800/1605535303 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fce38102630 con 0x7fce20038710 2023-11-16T18:16:25.640 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:25.285+0000 7fce3cdcc700 1 -- 172.21.15.2:0/1742352814 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fce20038710 msgr2=0x7fce2003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:25.640 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:25.285+0000 7fce3cdcc700 1 --2- 172.21.15.2:0/1742352814 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fce20038710 0x7fce2003abc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fce2c006fd0 tx=0x7fce2c006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:25.641 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:25.285+0000 7fce3cdcc700 1 -- 172.21.15.2:0/1742352814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 msgr2=0x7fce38199820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:25.642 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:25.285+0000 7fce3cdcc700 1 --2- 172.21.15.2:0/1742352814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 0x7fce38199820 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fce28006af0 tx=0x7fce28011ad0 comp rx=0 tx=0).stop 2023-11-16T18:16:25.642 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:25.285+0000 7fce3cdcc700 1 -- 172.21.15.2:0/1742352814 shutdown_connections 2023-11-16T18:16:25.643 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:25.285+0000 7fce3cdcc700 1 --2- 172.21.15.2:0/1742352814 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fce20038710 0x7fce2003abc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:25.644 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:25.285+0000 7fce3cdcc700 1 --2- 172.21.15.2:0/1742352814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce38105ac0 0x7fce38199820 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:25.644 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:25.285+0000 7fce3cdcc700 1 -- 172.21.15.2:0/1742352814 >> 172.21.15.2:0/1742352814 conn(0x7fce38101170 msgr2=0x7fce38101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:25.644 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:25.286+0000 7fce3cdcc700 1 -- 172.21.15.2:0/1742352814 shutdown_connections 2023-11-16T18:16:25.657 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:25.286+0000 7fce3cdcc700 1 -- 172.21.15.2:0/1742352814 wait complete. 2023-11-16T18:16:26.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:26 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:26 smithi002 ceph-mon[98807]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:16:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:26 smithi002 ceph-mon[98807]: Saving service mgr spec with placement count:2 2023-11-16T18:16:26.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:26 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:26.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:26 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:26.888 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.104+0000 7fdd99968700 1 Processor -- start 2023-11-16T18:16:26.896 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.104+0000 7fdd99968700 1 -- start start 2023-11-16T18:16:26.896 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.104+0000 7fdd99968700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 0x7fdd941088d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:26.896 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.104+0000 7fdd99968700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd94079e00 con 0x7fdd941064f0 2023-11-16T18:16:26.897 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.104+0000 7fdd92ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 0x7fdd941088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:26.897 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.105+0000 7fdd92ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 0x7fdd941088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46692/0 (socket says 172.21.15.2:46692) 2023-11-16T18:16:26.897 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.105+0000 7fdd92ffd700 1 -- 172.21.15.2:0/2166303475 learned_addr learned my addr 172.21.15.2:0/2166303475 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:26.898 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.105+0000 7fdd92ffd700 1 -- 172.21.15.2:0/2166303475 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd94108e10 con 0x7fdd941064f0 2023-11-16T18:16:26.898 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.105+0000 7fdd92ffd700 1 --2- 172.21.15.2:0/2166303475 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 0x7fdd941088d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdd880098a0 tx=0x7fdd88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba61ea92fb59d3dd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:26.898 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.105+0000 7fdd91ffb700 1 -- 172.21.15.2:0/2166303475 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd88004160 con 0x7fdd941064f0 2023-11-16T18:16:26.900 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.105+0000 7fdd91ffb700 1 -- 172.21.15.2:0/2166303475 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd880042c0 con 0x7fdd941064f0 2023-11-16T18:16:26.900 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.105+0000 7fdd91ffb700 1 -- 172.21.15.2:0/2166303475 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd880119b0 con 0x7fdd941064f0 2023-11-16T18:16:26.901 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.106+0000 7fdd99968700 1 -- 172.21.15.2:0/2166303475 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 msgr2=0x7fdd941088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:26.901 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.106+0000 7fdd99968700 1 --2- 172.21.15.2:0/2166303475 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 0x7fdd941088d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdd880098a0 tx=0x7fdd88009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:26.902 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.106+0000 7fdd99968700 1 -- 172.21.15.2:0/2166303475 shutdown_connections 2023-11-16T18:16:26.902 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.106+0000 7fdd99968700 1 --2- 172.21.15.2:0/2166303475 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 0x7fdd941088d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:26.919 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.106+0000 7fdd99968700 1 -- 172.21.15.2:0/2166303475 >> 172.21.15.2:0/2166303475 conn(0x7fdd940ffed0 msgr2=0x7fdd94102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:26.920 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.106+0000 7fdd99968700 1 -- 172.21.15.2:0/2166303475 shutdown_connections 2023-11-16T18:16:26.923 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.106+0000 7fdd99968700 1 -- 172.21.15.2:0/2166303475 wait complete. 2023-11-16T18:16:26.924 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.107+0000 7fdd99968700 1 Processor -- start 2023-11-16T18:16:26.925 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.107+0000 7fdd99968700 1 -- start start 2023-11-16T18:16:26.925 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.107+0000 7fdd99968700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 0x7fdd9419b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:26.928 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.107+0000 7fdd99968700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9419bed0 con 0x7fdd941064f0 2023-11-16T18:16:26.928 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.107+0000 7fdd92ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 0x7fdd9419b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:26.929 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.107+0000 7fdd92ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 0x7fdd9419b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46694/0 (socket says 172.21.15.2:46694) 2023-11-16T18:16:26.963 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.107+0000 7fdd92ffd700 1 -- 172.21.15.2:0/4054312908 learned_addr learned my addr 172.21.15.2:0/4054312908 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:26.981 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.107+0000 7fdd92ffd700 1 -- 172.21.15.2:0/4054312908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd88009550 con 0x7fdd941064f0 2023-11-16T18:16:26.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.108+0000 7fdd92ffd700 1 --2- 172.21.15.2:0/4054312908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 0x7fdd9419b990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdd88011980 tx=0x7fdd88011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:26.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.108+0000 7fdd98966700 1 -- 172.21.15.2:0/4054312908 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd88003a60 con 0x7fdd941064f0 2023-11-16T18:16:26.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.108+0000 7fdd98966700 1 -- 172.21.15.2:0/4054312908 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd88003bc0 con 0x7fdd941064f0 2023-11-16T18:16:26.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.108+0000 7fdd98966700 1 -- 172.21.15.2:0/4054312908 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd8801a670 con 0x7fdd941064f0 2023-11-16T18:16:26.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.108+0000 7fdd99968700 1 -- 172.21.15.2:0/4054312908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd94109d80 con 0x7fdd941064f0 2023-11-16T18:16:26.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.108+0000 7fdd99968700 1 -- 172.21.15.2:0/4054312908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd94109bf0 con 0x7fdd941064f0 2023-11-16T18:16:26.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.109+0000 7fdd98966700 1 -- 172.21.15.2:0/4054312908 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fdd8801a8e0 con 0x7fdd941064f0 2023-11-16T18:16:26.997 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.109+0000 7fdd98966700 1 --2- 172.21.15.2:0/4054312908 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fdd7c038760 0x7fdd7c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:26.997 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.109+0000 7fdd98966700 1 -- 172.21.15.2:0/4054312908 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdd8804c950 con 0x7fdd941064f0 2023-11-16T18:16:26.997 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.109+0000 7fdd99968700 1 -- 172.21.15.2:0/4054312908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd94109f10 con 0x7fdd941064f0 2023-11-16T18:16:26.998 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.109+0000 7fdd927fc700 1 --2- 172.21.15.2:0/4054312908 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fdd7c038760 0x7fdd7c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:27.011 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.110+0000 7fdd927fc700 1 --2- 172.21.15.2:0/4054312908 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fdd7c038760 0x7fdd7c03ac10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdd84006fd0 tx=0x7fdd84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:27.016 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.114+0000 7fdd98966700 1 -- 172.21.15.2:0/4054312908 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdd8801ab90 con 0x7fdd941064f0 2023-11-16T18:16:27.018 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.491+0000 7fdd99968700 1 -- 172.21.15.2:0/4054312908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fdd94062120 con 0x7fdd941064f0 2023-11-16T18:16:27.019 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.494+0000 7fdd98966700 1 -- 172.21.15.2:0/4054312908 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fdd88018350 con 0x7fdd941064f0 2023-11-16T18:16:27.019 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.496+0000 7fdd99968700 1 -- 172.21.15.2:0/4054312908 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fdd7c038760 msgr2=0x7fdd7c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:27.021 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.496+0000 7fdd99968700 1 --2- 172.21.15.2:0/4054312908 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fdd7c038760 0x7fdd7c03ac10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdd84006fd0 tx=0x7fdd84006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:27.022 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.496+0000 7fdd99968700 1 -- 172.21.15.2:0/4054312908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 msgr2=0x7fdd9419b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:27.023 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.496+0000 7fdd99968700 1 --2- 172.21.15.2:0/4054312908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 0x7fdd9419b990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdd88011980 tx=0x7fdd88011a60 comp rx=0 tx=0).stop 2023-11-16T18:16:27.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.496+0000 7fdd99968700 1 -- 172.21.15.2:0/4054312908 shutdown_connections 2023-11-16T18:16:27.028 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.496+0000 7fdd99968700 1 --2- 172.21.15.2:0/4054312908 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7fdd7c038760 0x7fdd7c03ac10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:27.034 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.496+0000 7fdd99968700 1 --2- 172.21.15.2:0/4054312908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd941064f0 0x7fdd9419b990 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:27.034 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.496+0000 7fdd99968700 1 -- 172.21.15.2:0/4054312908 >> 172.21.15.2:0/4054312908 conn(0x7fdd940ffed0 msgr2=0x7fdd94100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:27.035 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.496+0000 7fdd99968700 1 -- 172.21.15.2:0/4054312908 shutdown_connections 2023-11-16T18:16:27.037 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:26.497+0000 7fdd99968700 1 -- 172.21.15.2:0/4054312908 wait complete. 2023-11-16T18:16:27.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:27 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/4054312908' entity='client.admin' 2023-11-16T18:16:28.325 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.659+0000 7f6a48825700 1 Processor -- start 2023-11-16T18:16:28.326 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.659+0000 7f6a48825700 1 -- start start 2023-11-16T18:16:28.326 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.659+0000 7f6a48825700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 0x7f6a44104d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:28.327 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.659+0000 7f6a48825700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a44105300 con 0x7f6a44104920 2023-11-16T18:16:28.327 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.660+0000 7f6a4259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 0x7f6a44104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:28.327 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.660+0000 7f6a4259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 0x7f6a44104d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46698/0 (socket says 172.21.15.2:46698) 2023-11-16T18:16:28.327 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.660+0000 7f6a4259c700 1 -- 172.21.15.2:0/2798986453 learned_addr learned my addr 172.21.15.2:0/2798986453 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:28.327 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.660+0000 7f6a4259c700 1 -- 172.21.15.2:0/2798986453 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a44105b20 con 0x7f6a44104920 2023-11-16T18:16:28.328 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.660+0000 7f6a4259c700 1 --2- 172.21.15.2:0/2798986453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 0x7f6a44104d30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6a340098a0 tx=0x7f6a34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eebd9394056af7f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:28.328 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.660+0000 7f6a4159a700 1 -- 172.21.15.2:0/2798986453 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a34004160 con 0x7f6a44104920 2023-11-16T18:16:28.328 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.660+0000 7f6a4159a700 1 -- 172.21.15.2:0/2798986453 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a340042c0 con 0x7f6a44104920 2023-11-16T18:16:28.328 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.660+0000 7f6a4159a700 1 -- 172.21.15.2:0/2798986453 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a340119b0 con 0x7f6a44104920 2023-11-16T18:16:28.329 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.661+0000 7f6a48825700 1 -- 172.21.15.2:0/2798986453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 msgr2=0x7f6a44104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:28.329 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.661+0000 7f6a48825700 1 --2- 172.21.15.2:0/2798986453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 0x7f6a44104d30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6a340098a0 tx=0x7f6a34009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:28.329 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.661+0000 7f6a48825700 1 -- 172.21.15.2:0/2798986453 shutdown_connections 2023-11-16T18:16:28.329 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.661+0000 7f6a48825700 1 --2- 172.21.15.2:0/2798986453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 0x7f6a44104d30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:28.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.661+0000 7f6a48825700 1 -- 172.21.15.2:0/2798986453 >> 172.21.15.2:0/2798986453 conn(0x7f6a440ffed0 msgr2=0x7f6a44102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:28.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.661+0000 7f6a48825700 1 -- 172.21.15.2:0/2798986453 shutdown_connections 2023-11-16T18:16:28.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.661+0000 7f6a48825700 1 -- 172.21.15.2:0/2798986453 wait complete. 2023-11-16T18:16:28.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.662+0000 7f6a48825700 1 Processor -- start 2023-11-16T18:16:28.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.662+0000 7f6a48825700 1 -- start start 2023-11-16T18:16:28.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.662+0000 7f6a48825700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 0x7f6a441997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:28.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.662+0000 7f6a48825700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a44199d20 con 0x7f6a44104920 2023-11-16T18:16:28.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.663+0000 7f6a4259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 0x7f6a441997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:28.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.663+0000 7f6a4259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 0x7f6a441997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46700/0 (socket says 172.21.15.2:46700) 2023-11-16T18:16:28.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.663+0000 7f6a4259c700 1 -- 172.21.15.2:0/2421046399 learned_addr learned my addr 172.21.15.2:0/2421046399 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:28.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.663+0000 7f6a4259c700 1 -- 172.21.15.2:0/2421046399 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a34009550 con 0x7f6a44104920 2023-11-16T18:16:28.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.663+0000 7f6a4259c700 1 --2- 172.21.15.2:0/2421046399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 0x7f6a441997e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6a34000c00 tx=0x7f6a34011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:28.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.663+0000 7f6a337fe700 1 -- 172.21.15.2:0/2421046399 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a340039f0 con 0x7f6a44104920 2023-11-16T18:16:28.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.663+0000 7f6a48825700 1 -- 172.21.15.2:0/2421046399 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a44108ee0 con 0x7f6a44104920 2023-11-16T18:16:28.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.663+0000 7f6a337fe700 1 -- 172.21.15.2:0/2421046399 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a34003b50 con 0x7f6a44104920 2023-11-16T18:16:28.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.663+0000 7f6a337fe700 1 -- 172.21.15.2:0/2421046399 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a3401a780 con 0x7f6a44104920 2023-11-16T18:16:28.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.663+0000 7f6a48825700 1 -- 172.21.15.2:0/2421046399 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a44108d50 con 0x7f6a44104920 2023-11-16T18:16:28.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.664+0000 7f6a337fe700 1 -- 172.21.15.2:0/2421046399 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f6a3401aa40 con 0x7f6a44104920 2023-11-16T18:16:28.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.664+0000 7f6a337fe700 1 --2- 172.21.15.2:0/2421046399 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f6a2c038710 0x7f6a2c03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:28.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.664+0000 7f6a337fe700 1 -- 172.21.15.2:0/2421046399 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6a3404d660 con 0x7f6a44104920 2023-11-16T18:16:28.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.664+0000 7f6a41d9b700 1 --2- 172.21.15.2:0/2421046399 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f6a2c038710 0x7f6a2c03abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:28.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.665+0000 7f6a48825700 1 -- 172.21.15.2:0/2421046399 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a44062120 con 0x7f6a44104920 2023-11-16T18:16:28.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.665+0000 7f6a41d9b700 1 --2- 172.21.15.2:0/2421046399 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f6a2c038710 0x7f6a2c03abc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6a38006fd0 tx=0x7f6a38006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:28.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:27.674+0000 7f6a337fe700 1 -- 172.21.15.2:0/2421046399 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6a3401acf0 con 0x7f6a44104920 2023-11-16T18:16:28.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.037+0000 7f6a48825700 1 -- 172.21.15.2:0/2421046399 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f6a44109070 con 0x7f6a44104920 2023-11-16T18:16:28.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.040+0000 7f6a337fe700 1 -- 172.21.15.2:0/2421046399 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f6a34018350 con 0x7f6a44104920 2023-11-16T18:16:28.341 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.042+0000 7f6a48825700 1 -- 172.21.15.2:0/2421046399 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f6a2c038710 msgr2=0x7f6a2c03abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:28.341 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.042+0000 7f6a48825700 1 --2- 172.21.15.2:0/2421046399 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f6a2c038710 0x7f6a2c03abc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6a38006fd0 tx=0x7f6a38006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:28.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.042+0000 7f6a48825700 1 -- 172.21.15.2:0/2421046399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 msgr2=0x7f6a441997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:28.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.042+0000 7f6a48825700 1 --2- 172.21.15.2:0/2421046399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 0x7f6a441997e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6a34000c00 tx=0x7f6a34011a30 comp rx=0 tx=0).stop 2023-11-16T18:16:28.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.042+0000 7f6a48825700 1 -- 172.21.15.2:0/2421046399 shutdown_connections 2023-11-16T18:16:28.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.042+0000 7f6a48825700 1 --2- 172.21.15.2:0/2421046399 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f6a2c038710 0x7f6a2c03abc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:28.380 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.042+0000 7f6a48825700 1 --2- 172.21.15.2:0/2421046399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a44104920 0x7f6a441997e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:28.381 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.042+0000 7f6a48825700 1 -- 172.21.15.2:0/2421046399 >> 172.21.15.2:0/2421046399 conn(0x7f6a440ffed0 msgr2=0x7f6a44101590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:28.381 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.042+0000 7f6a48825700 1 -- 172.21.15.2:0/2421046399 shutdown_connections 2023-11-16T18:16:28.381 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.043+0000 7f6a48825700 1 -- 172.21.15.2:0/2421046399 wait complete. 2023-11-16T18:16:28.381 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling the dashboard module... 2023-11-16T18:16:28.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:28 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:28.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:28 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2421046399' entity='client.admin' 2023-11-16T18:16:29.899 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:29 smithi002 ceph-mon[98807]: from='mgr.14120 172.21.15.2:0/2528627981' entity='mgr.x' 2023-11-16T18:16:29.899 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:29 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1744644836' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-16T18:16:29.900 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:29 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: ignoring --setuser ceph since I am not root 2023-11-16T18:16:29.900 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:29 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: ignoring --setgroup ceph since I am not root 2023-11-16T18:16:29.900 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:29 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:29.895+0000 7f4573972200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T18:16:30.080 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.794+0000 7ff170def700 1 Processor -- start 2023-11-16T18:16:30.087 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.794+0000 7ff170def700 1 -- start start 2023-11-16T18:16:30.088 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.795+0000 7ff170def700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 0x7ff16c1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:30.088 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.795+0000 7ff170def700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16c079e00 con 0x7ff16c1064f0 2023-11-16T18:16:30.089 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.795+0000 7ff16a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 0x7ff16c1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:30.089 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.795+0000 7ff16a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 0x7ff16c1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46704/0 (socket says 172.21.15.2:46704) 2023-11-16T18:16:30.089 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.795+0000 7ff16a59c700 1 -- 172.21.15.2:0/1340619879 learned_addr learned my addr 172.21.15.2:0/1340619879 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:30.090 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.795+0000 7ff16a59c700 1 -- 172.21.15.2:0/1340619879 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16c108e10 con 0x7ff16c1064f0 2023-11-16T18:16:30.090 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.796+0000 7ff16a59c700 1 --2- 172.21.15.2:0/1340619879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 0x7ff16c1088d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff15c0098a0 tx=0x7ff15c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41ab46f5a0d3af2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:30.090 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.796+0000 7ff16959a700 1 -- 172.21.15.2:0/1340619879 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff15c004160 con 0x7ff16c1064f0 2023-11-16T18:16:30.090 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.796+0000 7ff16959a700 1 -- 172.21.15.2:0/1340619879 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff15c0042c0 con 0x7ff16c1064f0 2023-11-16T18:16:30.090 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.796+0000 7ff16959a700 1 -- 172.21.15.2:0/1340619879 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff15c0119b0 con 0x7ff16c1064f0 2023-11-16T18:16:30.090 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.797+0000 7ff170def700 1 -- 172.21.15.2:0/1340619879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 msgr2=0x7ff16c1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:30.090 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.797+0000 7ff170def700 1 --2- 172.21.15.2:0/1340619879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 0x7ff16c1088d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff15c0098a0 tx=0x7ff15c009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:30.091 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.797+0000 7ff170def700 1 -- 172.21.15.2:0/1340619879 shutdown_connections 2023-11-16T18:16:30.091 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.797+0000 7ff170def700 1 --2- 172.21.15.2:0/1340619879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 0x7ff16c1088d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:30.091 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.797+0000 7ff170def700 1 -- 172.21.15.2:0/1340619879 >> 172.21.15.2:0/1340619879 conn(0x7ff16c0ffed0 msgr2=0x7ff16c102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:30.091 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.797+0000 7ff170def700 1 -- 172.21.15.2:0/1340619879 shutdown_connections 2023-11-16T18:16:30.091 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.797+0000 7ff170def700 1 -- 172.21.15.2:0/1340619879 wait complete. 2023-11-16T18:16:30.099 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.798+0000 7ff170def700 1 Processor -- start 2023-11-16T18:16:30.099 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.798+0000 7ff170def700 1 -- start start 2023-11-16T18:16:30.099 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.798+0000 7ff170def700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 0x7ff16c1997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:30.099 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.798+0000 7ff170def700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16c199d10 con 0x7ff16c1064f0 2023-11-16T18:16:30.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.798+0000 7ff16a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 0x7ff16c1997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:30.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.798+0000 7ff16a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 0x7ff16c1997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46706/0 (socket says 172.21.15.2:46706) 2023-11-16T18:16:30.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.798+0000 7ff16a59c700 1 -- 172.21.15.2:0/1744644836 learned_addr learned my addr 172.21.15.2:0/1744644836 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:30.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.798+0000 7ff16a59c700 1 -- 172.21.15.2:0/1744644836 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c009550 con 0x7ff16c1064f0 2023-11-16T18:16:30.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.799+0000 7ff16a59c700 1 --2- 172.21.15.2:0/1744644836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 0x7ff16c1997d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff15c011980 tx=0x7ff15c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:30.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.799+0000 7ff15b7fe700 1 -- 172.21.15.2:0/1744644836 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff15c003a60 con 0x7ff16c1064f0 2023-11-16T18:16:30.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.799+0000 7ff170def700 1 -- 172.21.15.2:0/1744644836 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff16c109d80 con 0x7ff16c1064f0 2023-11-16T18:16:30.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.799+0000 7ff15b7fe700 1 -- 172.21.15.2:0/1744644836 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff15c003bc0 con 0x7ff16c1064f0 2023-11-16T18:16:30.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.799+0000 7ff15b7fe700 1 -- 172.21.15.2:0/1744644836 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff15c01a670 con 0x7ff16c1064f0 2023-11-16T18:16:30.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.799+0000 7ff170def700 1 -- 172.21.15.2:0/1744644836 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff16c109bf0 con 0x7ff16c1064f0 2023-11-16T18:16:30.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.800+0000 7ff15b7fe700 1 -- 172.21.15.2:0/1744644836 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ff15c01a930 con 0x7ff16c1064f0 2023-11-16T18:16:30.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.800+0000 7ff15b7fe700 1 --2- 172.21.15.2:0/1744644836 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7ff154038710 0x7ff15403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:30.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.800+0000 7ff170def700 1 -- 172.21.15.2:0/1744644836 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff16c109f10 con 0x7ff16c1064f0 2023-11-16T18:16:30.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.800+0000 7ff15b7fe700 1 -- 172.21.15.2:0/1744644836 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff15c04c7e0 con 0x7ff16c1064f0 2023-11-16T18:16:30.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.800+0000 7ff169d9b700 1 --2- 172.21.15.2:0/1744644836 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7ff154038710 0x7ff15403abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:30.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.801+0000 7ff169d9b700 1 --2- 172.21.15.2:0/1744644836 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7ff154038710 0x7ff15403abc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff160006fd0 tx=0x7ff160006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:30.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:28.805+0000 7ff15b7fe700 1 -- 172.21.15.2:0/1744644836 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff15c01abe0 con 0x7ff16c1064f0 2023-11-16T18:16:30.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:29.283+0000 7ff170def700 1 -- 172.21.15.2:0/1744644836 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7ff16c0621a0 con 0x7ff16c1064f0 2023-11-16T18:16:30.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:29.681+0000 7ff15b7fe700 1 -- 172.21.15.2:0/1744644836 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7ff15c04b610 con 0x7ff16c1064f0 2023-11-16T18:16:30.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:29.683+0000 7ff15b7fe700 1 -- 172.21.15.2:0/1744644836 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7ff15c04b990 con 0x7ff16c1064f0 2023-11-16T18:16:30.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:29.685+0000 7ff170def700 1 -- 172.21.15.2:0/1744644836 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7ff154038710 msgr2=0x7ff15403abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:30.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:29.685+0000 7ff170def700 1 --2- 172.21.15.2:0/1744644836 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7ff154038710 0x7ff15403abc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff160006fd0 tx=0x7ff160006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:30.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:29.685+0000 7ff170def700 1 -- 172.21.15.2:0/1744644836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 msgr2=0x7ff16c1997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:30.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:29.685+0000 7ff170def700 1 --2- 172.21.15.2:0/1744644836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 0x7ff16c1997d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff15c011980 tx=0x7ff15c011a60 comp rx=0 tx=0).stop 2023-11-16T18:16:30.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:29.685+0000 7ff170def700 1 -- 172.21.15.2:0/1744644836 shutdown_connections 2023-11-16T18:16:30.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:29.685+0000 7ff170def700 1 --2- 172.21.15.2:0/1744644836 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7ff154038710 0x7ff15403abc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:30.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:29.685+0000 7ff170def700 1 --2- 172.21.15.2:0/1744644836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff16c1064f0 0x7ff16c1997d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:30.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:29.685+0000 7ff170def700 1 -- 172.21.15.2:0/1744644836 >> 172.21.15.2:0/1744644836 conn(0x7ff16c0ffed0 msgr2=0x7ff16c1015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:30.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:29.685+0000 7ff170def700 1 -- 172.21.15.2:0/1744644836 shutdown_connections 2023-11-16T18:16:30.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:29.685+0000 7ff170def700 1 -- 172.21.15.2:0/1744644836 wait complete. 2023-11-16T18:16:30.229 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:29 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:29.991+0000 7f4573972200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T18:16:30.230 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:30 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:30.108+0000 7f4573972200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T18:16:30.230 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:30 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:30.228+0000 7f4573972200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T18:16:30.528 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:30 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:30.447+0000 7f4573972200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T18:16:30.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:30 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1744644836' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-16T18:16:30.864 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:30 smithi002 ceph-mon[98807]: mgrmap e9: x(active, since 16s) 2023-11-16T18:16:31.120 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:30 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:30.857+0000 7f4573972200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T18:16:31.120 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:30 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:30.966+0000 7f4573972200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T18:16:31.121 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:31 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:31.118+0000 7f4573972200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T18:16:31.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T18:16:31.451 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-11-16T18:16:31.452 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-16T18:16:31.452 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-11-16T18:16:31.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-11-16T18:16:31.454 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T18:16:31.454 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.578+0000 7f7743884700 1 Processor -- start 2023-11-16T18:16:31.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.578+0000 7f7743884700 1 -- start start 2023-11-16T18:16:31.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.579+0000 7f7743884700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 0x7f773c104d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:31.456 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.579+0000 7f7743884700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c105300 con 0x7f773c104920 2023-11-16T18:16:31.456 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.579+0000 7f7741620700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 0x7f773c104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:31.456 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.579+0000 7f7741620700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 0x7f773c104d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46720/0 (socket says 172.21.15.2:46720) 2023-11-16T18:16:31.457 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.579+0000 7f7741620700 1 -- 172.21.15.2:0/594197179 learned_addr learned my addr 172.21.15.2:0/594197179 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:31.457 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.579+0000 7f7741620700 1 -- 172.21.15.2:0/594197179 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c105b20 con 0x7f773c104920 2023-11-16T18:16:31.458 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.579+0000 7f7741620700 1 --2- 172.21.15.2:0/594197179 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 0x7f773c104d30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f773800bb40 tx=0x7f773800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c88b25292c725b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:31.458 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.580+0000 7f7733fff700 1 -- 172.21.15.2:0/594197179 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f773800dbe0 con 0x7f773c104920 2023-11-16T18:16:31.459 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.580+0000 7f7733fff700 1 -- 172.21.15.2:0/594197179 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77380044f0 con 0x7f773c104920 2023-11-16T18:16:31.459 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.580+0000 7f7733fff700 1 -- 172.21.15.2:0/594197179 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7738003e60 con 0x7f773c104920 2023-11-16T18:16:31.459 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.580+0000 7f7743884700 1 -- 172.21.15.2:0/594197179 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 msgr2=0x7f773c104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:31.468 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.580+0000 7f7743884700 1 --2- 172.21.15.2:0/594197179 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 0x7f773c104d30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f773800bb40 tx=0x7f773800be50 comp rx=0 tx=0).stop 2023-11-16T18:16:31.468 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.580+0000 7f7743884700 1 -- 172.21.15.2:0/594197179 shutdown_connections 2023-11-16T18:16:31.469 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.580+0000 7f7743884700 1 --2- 172.21.15.2:0/594197179 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 0x7f773c104d30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:31.495 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.580+0000 7f7743884700 1 -- 172.21.15.2:0/594197179 >> 172.21.15.2:0/594197179 conn(0x7f773c0ffed0 msgr2=0x7f773c102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:31.495 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.580+0000 7f7743884700 1 -- 172.21.15.2:0/594197179 shutdown_connections 2023-11-16T18:16:31.495 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.581+0000 7f7743884700 1 -- 172.21.15.2:0/594197179 wait complete. 2023-11-16T18:16:31.495 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.581+0000 7f7743884700 1 Processor -- start 2023-11-16T18:16:31.495 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.581+0000 7f7743884700 1 -- start start 2023-11-16T18:16:31.495 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.581+0000 7f7743884700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 0x7f773c19ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:31.495 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.581+0000 7f7743884700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c19bf70 con 0x7f773c104920 2023-11-16T18:16:31.495 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.581+0000 7f7741620700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 0x7f773c19ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:31.496 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.581+0000 7f7741620700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 0x7f773c19ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46722/0 (socket says 172.21.15.2:46722) 2023-11-16T18:16:31.496 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.581+0000 7f7741620700 1 -- 172.21.15.2:0/279322146 learned_addr learned my addr 172.21.15.2:0/279322146 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:31.496 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.582+0000 7f7741620700 1 -- 172.21.15.2:0/279322146 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773800b7f0 con 0x7f773c104920 2023-11-16T18:16:31.496 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.582+0000 7f7741620700 1 --2- 172.21.15.2:0/279322146 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 0x7f773c19ba30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7738006fa0 tx=0x7f7738004020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:31.496 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.582+0000 7f77327fc700 1 -- 172.21.15.2:0/279322146 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7738010630 con 0x7f773c104920 2023-11-16T18:16:31.496 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.582+0000 7f7743884700 1 -- 172.21.15.2:0/279322146 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c078f80 con 0x7f773c104920 2023-11-16T18:16:31.496 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.582+0000 7f7743884700 1 -- 172.21.15.2:0/279322146 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c106910 con 0x7f773c104920 2023-11-16T18:16:31.496 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.582+0000 7f77327fc700 1 -- 172.21.15.2:0/279322146 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7738010c50 con 0x7f773c104920 2023-11-16T18:16:31.496 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.582+0000 7f77327fc700 1 -- 172.21.15.2:0/279322146 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7738019b10 con 0x7f773c104920 2023-11-16T18:16:31.496 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.583+0000 7f77327fc700 1 -- 172.21.15.2:0/279322146 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f7738019c70 con 0x7f773c104920 2023-11-16T18:16:31.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.583+0000 7f77327fc700 1 --2- 172.21.15.2:0/279322146 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f7728038760 0x7f772803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:31.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.583+0000 7f77327fc700 1 -- 172.21.15.2:0/279322146 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f773804c990 con 0x7f773c104920 2023-11-16T18:16:31.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.583+0000 7f7740e1f700 1 -- 172.21.15.2:0/279322146 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f7728038760 msgr2=0x7f772803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1605535303 2023-11-16T18:16:31.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.583+0000 7f7740e1f700 1 --2- 172.21.15.2:0/279322146 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f7728038760 0x7f772803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T18:16:31.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.584+0000 7f7743884700 1 -- 172.21.15.2:0/279322146 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7720005320 con 0x7f773c104920 2023-11-16T18:16:31.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.588+0000 7f77327fc700 1 -- 172.21.15.2:0/279322146 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7738028940 con 0x7f773c104920 2023-11-16T18:16:31.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.784+0000 7f7740e1f700 1 -- 172.21.15.2:0/279322146 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f7728038760 msgr2=0x7f772803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1605535303 2023-11-16T18:16:31.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:30.784+0000 7f7740e1f700 1 --2- 172.21.15.2:0/279322146 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f7728038760 0x7f772803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T18:16:31.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.069+0000 7f7743884700 1 -- 172.21.15.2:0/279322146 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7720006200 con 0x7f773c104920 2023-11-16T18:16:31.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.070+0000 7f77327fc700 1 -- 172.21.15.2:0/279322146 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f773804a030 con 0x7f773c104920 2023-11-16T18:16:31.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.072+0000 7f7743884700 1 -- 172.21.15.2:0/279322146 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f7728038760 msgr2=0x7f772803ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T18:16:31.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.072+0000 7f7743884700 1 --2- 172.21.15.2:0/279322146 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f7728038760 0x7f772803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:31.510 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.072+0000 7f7743884700 1 -- 172.21.15.2:0/279322146 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 msgr2=0x7f773c19ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:31.510 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.072+0000 7f7743884700 1 --2- 172.21.15.2:0/279322146 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 0x7f773c19ba30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7738006fa0 tx=0x7f7738004020 comp rx=0 tx=0).stop 2023-11-16T18:16:31.511 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.072+0000 7f7743884700 1 -- 172.21.15.2:0/279322146 shutdown_connections 2023-11-16T18:16:31.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.072+0000 7f7743884700 1 --2- 172.21.15.2:0/279322146 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f7728038760 0x7f772803ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:31.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.072+0000 7f7743884700 1 --2- 172.21.15.2:0/279322146 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c104920 0x7f773c19ba30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:31.512 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.072+0000 7f7743884700 1 -- 172.21.15.2:0/279322146 >> 172.21.15.2:0/279322146 conn(0x7f773c0ffed0 msgr2=0x7f773c18e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:31.513 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.072+0000 7f7743884700 1 -- 172.21.15.2:0/279322146 shutdown_connections 2023-11-16T18:16:31.513 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.072+0000 7f7743884700 1 -- 172.21.15.2:0/279322146 wait complete. 2023-11-16T18:16:31.516 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for the mgr to restart... 2023-11-16T18:16:31.516 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr epoch 9... 2023-11-16T18:16:31.606 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:31 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:31.244+0000 7f4573972200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T18:16:32.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:31 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/279322146' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-16T18:16:32.106 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:31 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:31.771+0000 7f4573972200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T18:16:32.856 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:32 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:32.458+0000 7f4573972200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T18:16:32.858 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:32 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:32.649+0000 7f4573972200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T18:16:33.939 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:33 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:33.608+0000 7f4573972200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:16:33.940 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:33 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:33.719+0000 7f4573972200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T18:16:34.538 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:33 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:33.937+0000 7f4573972200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:16:34.538 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:34 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:34.285+0000 7f4573972200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T18:16:34.539 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:34 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:34.537+0000 7f4573972200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T18:16:34.855 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:34 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:34.718+0000 7f4573972200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T18:16:35.355 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:35 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:35.047+0000 7f4573972200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T18:16:36.273 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:35 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:35.870+0000 7f4573972200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T18:16:36.278 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:36 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:36.027+0000 7f4573972200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T18:16:36.279 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:36 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:36.270+0000 7f4573972200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T18:16:36.607 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:36 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:36.406+0000 7f4573972200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T18:16:37.106 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:36 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:36.676+0000 7f4573972200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T18:16:38.106 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:16:37 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:16:37.710+0000 7f4573972200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T18:16:38.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:37 smithi002 ceph-mon[98807]: Active manager daemon x restarted 2023-11-16T18:16:38.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:37 smithi002 ceph-mon[98807]: Activating manager daemon x 2023-11-16T18:16:38.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:37 smithi002 ceph-mon[98807]: osdmap e3: 0 total, 0 up, 0 in 2023-11-16T18:16:38.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:37 smithi002 ceph-mon[98807]: mgrmap e10: x(active, starting, since 0.00304884s) 2023-11-16T18:16:38.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:16:38.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T18:16:38.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T18:16:38.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T18:16:38.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T18:16:38.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:37 smithi002 ceph-mon[98807]: Manager daemon x is now available 2023-11-16T18:16:38.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:16:38.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:16:39.045 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T18:16:39.045 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T18:16:39.045 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:38 smithi002 ceph-mon[98807]: mgrmap e11: x(active, since 1.00571s) 2023-11-16T18:16:39.045 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:38 smithi002 ceph-mon[98807]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-16T18:16:39.045 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:38 smithi002 ceph-mon[98807]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-16T18:16:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T18:16:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-11-16T18:16:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "initialized": true 2023-11-16T18:16:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T18:16:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.940+0000 7f57eeaee700 1 Processor -- start 2023-11-16T18:16:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.940+0000 7f57eeaee700 1 -- start start 2023-11-16T18:16:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.940+0000 7f57eeaee700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 0x7f57e8108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.940+0000 7f57eeaee700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e8079d80 con 0x7f57e8106570 2023-11-16T18:16:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.941+0000 7f57ec88a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 0x7f57e8108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.941+0000 7f57ec88a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 0x7f57e8108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46730/0 (socket says 172.21.15.2:46730) 2023-11-16T18:16:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.941+0000 7f57ec88a700 1 -- 172.21.15.2:0/3112905861 learned_addr learned my addr 172.21.15.2:0/3112905861 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.941+0000 7f57ec88a700 1 -- 172.21.15.2:0/3112905861 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e8079ec0 con 0x7f57e8106570 2023-11-16T18:16:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.941+0000 7f57ec88a700 1 --2- 172.21.15.2:0/3112905861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 0x7f57e8108950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0098a0 tx=0x7f57dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c144bb7009ef38a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.941+0000 7f57e77fe700 1 -- 172.21.15.2:0/3112905861 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57dc004160 con 0x7f57e8106570 2023-11-16T18:16:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.941+0000 7f57e77fe700 1 -- 172.21.15.2:0/3112905861 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57dc0042c0 con 0x7f57e8106570 2023-11-16T18:16:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.941+0000 7f57e77fe700 1 -- 172.21.15.2:0/3112905861 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57dc0119b0 con 0x7f57e8106570 2023-11-16T18:16:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.942+0000 7f57eeaee700 1 -- 172.21.15.2:0/3112905861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 msgr2=0x7f57e8108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:39.114 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.942+0000 7f57eeaee700 1 --2- 172.21.15.2:0/3112905861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 0x7f57e8108950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0098a0 tx=0x7f57dc009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:39.114 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.942+0000 7f57eeaee700 1 -- 172.21.15.2:0/3112905861 shutdown_connections 2023-11-16T18:16:39.114 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.942+0000 7f57eeaee700 1 --2- 172.21.15.2:0/3112905861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 0x7f57e8108950 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:39.114 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.942+0000 7f57eeaee700 1 -- 172.21.15.2:0/3112905861 >> 172.21.15.2:0/3112905861 conn(0x7f57e80ffed0 msgr2=0x7f57e8102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:39.114 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.942+0000 7f57eeaee700 1 -- 172.21.15.2:0/3112905861 shutdown_connections 2023-11-16T18:16:39.115 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.942+0000 7f57eeaee700 1 -- 172.21.15.2:0/3112905861 wait complete. 2023-11-16T18:16:39.115 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.943+0000 7f57eeaee700 1 Processor -- start 2023-11-16T18:16:39.116 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.943+0000 7f57eeaee700 1 -- start start 2023-11-16T18:16:39.116 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.943+0000 7f57eeaee700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 0x7f57e8105ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:39.116 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.943+0000 7f57eeaee700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e8106410 con 0x7f57e8106570 2023-11-16T18:16:39.116 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.943+0000 7f57ec88a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 0x7f57e8105ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:39.116 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.943+0000 7f57ec88a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 0x7f57e8105ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46732/0 (socket says 172.21.15.2:46732) 2023-11-16T18:16:39.116 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.943+0000 7f57ec88a700 1 -- 172.21.15.2:0/2139856765 learned_addr learned my addr 172.21.15.2:0/2139856765 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:39.116 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.943+0000 7f57ec88a700 1 -- 172.21.15.2:0/2139856765 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57dc009550 con 0x7f57e8106570 2023-11-16T18:16:39.116 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.944+0000 7f57ec88a700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 0x7f57e8105ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f57dc003e60 tx=0x7f57dc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:39.116 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.944+0000 7f57e5ffb700 1 -- 172.21.15.2:0/2139856765 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57dc003970 con 0x7f57e8106570 2023-11-16T18:16:39.116 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.944+0000 7f57e5ffb700 1 -- 172.21.15.2:0/2139856765 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57dc003ad0 con 0x7f57e8106570 2023-11-16T18:16:39.116 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.944+0000 7f57eeaee700 1 -- 172.21.15.2:0/2139856765 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e8109d90 con 0x7f57e8106570 2023-11-16T18:16:39.117 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.944+0000 7f57e5ffb700 1 -- 172.21.15.2:0/2139856765 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57dc0195e0 con 0x7f57e8106570 2023-11-16T18:16:39.117 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.944+0000 7f57eeaee700 1 -- 172.21.15.2:0/2139856765 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e8109c00 con 0x7f57e8106570 2023-11-16T18:16:39.117 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.945+0000 7f57e5ffb700 1 -- 172.21.15.2:0/2139856765 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f57dc019870 con 0x7f57e8106570 2023-11-16T18:16:39.117 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.945+0000 7f57e5ffb700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 0x7f57d803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:39.117 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.945+0000 7f57eeaee700 1 -- 172.21.15.2:0/2139856765 --> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57e8062120 con 0x7f57d8038760 2023-11-16T18:16:39.117 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.945+0000 7f57e5ffb700 1 -- 172.21.15.2:0/2139856765 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f57dc04c890 con 0x7f57e8106570 2023-11-16T18:16:39.117 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.945+0000 7f57e7fff700 1 -- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 msgr2=0x7f57d803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1605535303 2023-11-16T18:16:39.117 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:31.945+0000 7f57e7fff700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 0x7f57d803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T18:16:39.117 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:32.146+0000 7f57e7fff700 1 -- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 msgr2=0x7f57d803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1605535303 2023-11-16T18:16:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:32.146+0000 7f57e7fff700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 0x7f57d803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T18:16:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:32.547+0000 7f57e7fff700 1 -- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 msgr2=0x7f57d803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1605535303 2023-11-16T18:16:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:32.547+0000 7f57e7fff700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 0x7f57d803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-16T18:16:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:33.348+0000 7f57e7fff700 1 -- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 msgr2=0x7f57d803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1605535303 2023-11-16T18:16:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:33.348+0000 7f57e7fff700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 0x7f57d803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-16T18:16:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:34.949+0000 7f57e7fff700 1 -- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 msgr2=0x7f57d803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1605535303 2023-11-16T18:16:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:34.949+0000 7f57e7fff700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 0x7f57d803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-16T18:16:39.118 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:37.725+0000 7f57e5ffb700 1 -- 172.21.15.2:0/2139856765 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f57dc02d9b0 con 0x7f57e8106570 2023-11-16T18:16:39.134 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:37.725+0000 7f57e5ffb700 1 -- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 msgr2=0x7f57d803ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T18:16:39.134 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:37.725+0000 7f57e5ffb700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 0x7f57d803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:39.135 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.726+0000 7f57e5ffb700 1 -- 172.21.15.2:0/2139856765 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f57dc04e520 con 0x7f57e8106570 2023-11-16T18:16:39.135 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.727+0000 7f57e5ffb700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f57d803bfc0 0x7f57d803e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:39.135 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.727+0000 7f57e5ffb700 1 -- 172.21.15.2:0/2139856765 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57e8062120 con 0x7f57d803bfc0 2023-11-16T18:16:39.135 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.727+0000 7f57e7fff700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f57d803bfc0 0x7f57d803e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:39.135 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.729+0000 7f57e7fff700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f57d803bfc0 0x7f57d803e3a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f57d4003a10 tx=0x7f57d4009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:39.135 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.729+0000 7f57e5ffb700 1 -- 172.21.15.2:0/2139856765 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f57e8062120 con 0x7f57d803bfc0 2023-11-16T18:16:39.135 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.740+0000 7f57eeaee700 1 -- 172.21.15.2:0/2139856765 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f57e8105a70 con 0x7f57d803bfc0 2023-11-16T18:16:39.135 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.741+0000 7f57e5ffb700 1 -- 172.21.15.2:0/2139856765 <== mgr.14152 v2:172.21.15.2:6800/3369431713 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f57e8105a70 con 0x7f57d803bfc0 2023-11-16T18:16:39.136 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.741+0000 7f57eeaee700 1 -- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f57d803bfc0 msgr2=0x7f57d803e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:39.136 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.741+0000 7f57eeaee700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f57d803bfc0 0x7f57d803e3a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f57d4003a10 tx=0x7f57d4009270 comp rx=0 tx=0).stop 2023-11-16T18:16:39.136 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.741+0000 7f57eeaee700 1 -- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 msgr2=0x7f57e8105ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:39.136 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.741+0000 7f57eeaee700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 0x7f57e8105ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f57dc003e60 tx=0x7f57dc011aa0 comp rx=0 tx=0).stop 2023-11-16T18:16:39.136 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.741+0000 7f57eeaee700 1 -- 172.21.15.2:0/2139856765 shutdown_connections 2023-11-16T18:16:39.136 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.741+0000 7f57eeaee700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f57d803bfc0 0x7f57d803e3a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:39.136 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.741+0000 7f57eeaee700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57e8106570 0x7f57e8105ed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:39.136 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.741+0000 7f57eeaee700 1 --2- 172.21.15.2:0/2139856765 >> [v2:172.21.15.2:6800/1605535303,v1:172.21.15.2:6801/1605535303] conn(0x7f57d8038760 0x7f57d803ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:39.136 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.741+0000 7f57eeaee700 1 -- 172.21.15.2:0/2139856765 >> 172.21.15.2:0/2139856765 conn(0x7f57e80ffed0 msgr2=0x7f57e8101570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:39.136 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.741+0000 7f57eeaee700 1 -- 172.21.15.2:0/2139856765 shutdown_connections 2023-11-16T18:16:39.137 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:38.741+0000 7f57eeaee700 1 -- 172.21.15.2:0/2139856765 wait complete. 2023-11-16T18:16:39.137 INFO:teuthology.orchestra.run.smithi002.stdout:mgr epoch 9 is available 2023-11-16T18:16:39.137 INFO:teuthology.orchestra.run.smithi002.stdout:Generating a dashboard self-signed certificate... 2023-11-16T18:16:40.566 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-11-16T18:16:40.567 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.706+0000 7f18c8e56700 1 Processor -- start 2023-11-16T18:16:40.567 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.706+0000 7f18c8e56700 1 -- start start 2023-11-16T18:16:40.567 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.706+0000 7f18c8e56700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 0x7f18c41088d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:40.567 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.706+0000 7f18c8e56700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c4079e00 con 0x7f18c41064f0 2023-11-16T18:16:40.575 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.706+0000 7f18c259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 0x7f18c41088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:40.576 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.707+0000 7f18c259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 0x7f18c41088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46786/0 (socket says 172.21.15.2:46786) 2023-11-16T18:16:40.576 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.707+0000 7f18c259c700 1 -- 172.21.15.2:0/2228212838 learned_addr learned my addr 172.21.15.2:0/2228212838 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:40.576 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.707+0000 7f18c259c700 1 -- 172.21.15.2:0/2228212838 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c4108e10 con 0x7f18c41064f0 2023-11-16T18:16:40.576 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.707+0000 7f18c259c700 1 --2- 172.21.15.2:0/2228212838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 0x7f18c41088d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f18b40098a0 tx=0x7f18b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e15935dfca67627 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:40.576 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.707+0000 7f18c159a700 1 -- 172.21.15.2:0/2228212838 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18b4004160 con 0x7f18c41064f0 2023-11-16T18:16:40.576 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.707+0000 7f18c159a700 1 -- 172.21.15.2:0/2228212838 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18b40042c0 con 0x7f18c41064f0 2023-11-16T18:16:40.577 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.707+0000 7f18c159a700 1 -- 172.21.15.2:0/2228212838 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18b40119b0 con 0x7f18c41064f0 2023-11-16T18:16:40.577 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.708+0000 7f18c8e56700 1 -- 172.21.15.2:0/2228212838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 msgr2=0x7f18c41088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:40.577 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.708+0000 7f18c8e56700 1 --2- 172.21.15.2:0/2228212838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 0x7f18c41088d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f18b40098a0 tx=0x7f18b4009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:40.577 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.708+0000 7f18c8e56700 1 -- 172.21.15.2:0/2228212838 shutdown_connections 2023-11-16T18:16:40.577 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.708+0000 7f18c8e56700 1 --2- 172.21.15.2:0/2228212838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 0x7f18c41088d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:40.577 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.708+0000 7f18c8e56700 1 -- 172.21.15.2:0/2228212838 >> 172.21.15.2:0/2228212838 conn(0x7f18c40ffed0 msgr2=0x7f18c4102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:40.577 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.708+0000 7f18c8e56700 1 -- 172.21.15.2:0/2228212838 shutdown_connections 2023-11-16T18:16:40.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.708+0000 7f18c8e56700 1 -- 172.21.15.2:0/2228212838 wait complete. 2023-11-16T18:16:40.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.709+0000 7f18c8e56700 1 Processor -- start 2023-11-16T18:16:40.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.709+0000 7f18c8e56700 1 -- start start 2023-11-16T18:16:40.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.709+0000 7f18c8e56700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 0x7f18c4199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:40.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.709+0000 7f18c8e56700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c4199cd0 con 0x7f18c41064f0 2023-11-16T18:16:40.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.709+0000 7f18c259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 0x7f18c4199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:40.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.709+0000 7f18c259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 0x7f18c4199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46788/0 (socket says 172.21.15.2:46788) 2023-11-16T18:16:40.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.709+0000 7f18c259c700 1 -- 172.21.15.2:0/3956668721 learned_addr learned my addr 172.21.15.2:0/3956668721 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:40.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.710+0000 7f18c259c700 1 -- 172.21.15.2:0/3956668721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b4009550 con 0x7f18c41064f0 2023-11-16T18:16:40.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.710+0000 7f18c259c700 1 --2- 172.21.15.2:0/3956668721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 0x7f18c4199790 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f18b4011980 tx=0x7f18b4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:40.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.710+0000 7f18b37fe700 1 -- 172.21.15.2:0/3956668721 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18b4003a60 con 0x7f18c41064f0 2023-11-16T18:16:40.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.710+0000 7f18b37fe700 1 -- 172.21.15.2:0/3956668721 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18b4003bc0 con 0x7f18c41064f0 2023-11-16T18:16:40.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.710+0000 7f18c8e56700 1 -- 172.21.15.2:0/3956668721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c4109d80 con 0x7f18c41064f0 2023-11-16T18:16:40.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.710+0000 7f18b37fe700 1 -- 172.21.15.2:0/3956668721 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18b401a670 con 0x7f18c41064f0 2023-11-16T18:16:40.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.710+0000 7f18c8e56700 1 -- 172.21.15.2:0/3956668721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c4109bf0 con 0x7f18c41064f0 2023-11-16T18:16:40.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.712+0000 7f18c8e56700 1 -- 172.21.15.2:0/3956668721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c4109f10 con 0x7f18c41064f0 2023-11-16T18:16:40.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.712+0000 7f18b37fe700 1 -- 172.21.15.2:0/3956668721 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f18b4024a00 con 0x7f18c41064f0 2023-11-16T18:16:40.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.712+0000 7f18b37fe700 1 --2- 172.21.15.2:0/3956668721 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f18ac038630 0x7f18ac03aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:40.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.712+0000 7f18b37fe700 1 -- 172.21.15.2:0/3956668721 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f18b404cba0 con 0x7f18c41064f0 2023-11-16T18:16:40.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.713+0000 7f18c1d9b700 1 --2- 172.21.15.2:0/3956668721 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f18ac038630 0x7f18ac03aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:40.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.713+0000 7f18c1d9b700 1 --2- 172.21.15.2:0/3956668721 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f18ac038630 0x7f18ac03aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f18b8006fd0 tx=0x7f18b8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:40.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:39.717+0000 7f18b37fe700 1 -- 172.21.15.2:0/3956668721 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f18b4018760 con 0x7f18c41064f0 2023-11-16T18:16:40.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:40.087+0000 7f18c8e56700 1 -- 172.21.15.2:0/3956668721 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f18c4102190 con 0x7f18ac038630 2023-11-16T18:16:40.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:40.177+0000 7f18b37fe700 1 -- 172.21.15.2:0/3956668721 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f18c4102190 con 0x7f18ac038630 2023-11-16T18:16:40.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:40.179+0000 7f18c8e56700 1 -- 172.21.15.2:0/3956668721 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f18ac038630 msgr2=0x7f18ac03aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:40.588 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:40.179+0000 7f18c8e56700 1 --2- 172.21.15.2:0/3956668721 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f18ac038630 0x7f18ac03aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f18b8006fd0 tx=0x7f18b8006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:40.588 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:40.179+0000 7f18c8e56700 1 -- 172.21.15.2:0/3956668721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 msgr2=0x7f18c4199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:40.588 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:40.179+0000 7f18c8e56700 1 --2- 172.21.15.2:0/3956668721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 0x7f18c4199790 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f18b4011980 tx=0x7f18b4011a60 comp rx=0 tx=0).stop 2023-11-16T18:16:40.597 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:40.179+0000 7f18c8e56700 1 -- 172.21.15.2:0/3956668721 shutdown_connections 2023-11-16T18:16:40.597 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:40.179+0000 7f18c8e56700 1 --2- 172.21.15.2:0/3956668721 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f18ac038630 0x7f18ac03aae0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:40.597 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:40.179+0000 7f18c8e56700 1 --2- 172.21.15.2:0/3956668721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18c41064f0 0x7f18c4199790 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:40.598 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:40.179+0000 7f18c8e56700 1 -- 172.21.15.2:0/3956668721 >> 172.21.15.2:0/3956668721 conn(0x7f18c40ffed0 msgr2=0x7f18c4101530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:40.599 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:40.179+0000 7f18c8e56700 1 -- 172.21.15.2:0/3956668721 shutdown_connections 2023-11-16T18:16:40.599 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:40.179+0000 7f18c8e56700 1 -- 172.21.15.2:0/3956668721 wait complete. 2023-11-16T18:16:40.600 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial admin user... 2023-11-16T18:16:40.781 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:40 smithi002 ceph-mon[98807]: [16/Nov/2023:18:16:39] ENGINE Bus STARTING 2023-11-16T18:16:40.782 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:40 smithi002 ceph-mon[98807]: [16/Nov/2023:18:16:39] ENGINE Serving on http://172.21.15.2:8765 2023-11-16T18:16:40.782 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:40 smithi002 ceph-mon[98807]: [16/Nov/2023:18:16:39] ENGINE Serving on https://172.21.15.2:7150 2023-11-16T18:16:40.782 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:40 smithi002 ceph-mon[98807]: [16/Nov/2023:18:16:39] ENGINE Bus STARTED 2023-11-16T18:16:40.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:40.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:40.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:41.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:41 smithi002 ceph-mon[98807]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:16:41.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:41 smithi002 ceph-mon[98807]: mgrmap e12: x(active, since 2s) 2023-11-16T18:16:41.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:41 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:42.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$iIKAUiRR3.phOSYGPhJVUuT.gd0kv8v8PEsXLFvc2bzF38tYjsw9u", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700158601, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-16T18:16:42.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.074+0000 7fb029f4b700 1 Processor -- start 2023-11-16T18:16:42.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.074+0000 7fb029f4b700 1 -- start start 2023-11-16T18:16:42.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.074+0000 7fb029f4b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 0x7fb0241088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:42.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.074+0000 7fb029f4b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb024079e20 con 0x7fb0241064d0 2023-11-16T18:16:42.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.075+0000 7fb0237fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 0x7fb0241088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:42.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.075+0000 7fb0237fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 0x7fb0241088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46792/0 (socket says 172.21.15.2:46792) 2023-11-16T18:16:42.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.075+0000 7fb0237fe700 1 -- 172.21.15.2:0/1498682984 learned_addr learned my addr 172.21.15.2:0/1498682984 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:42.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.075+0000 7fb0237fe700 1 -- 172.21.15.2:0/1498682984 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb024108df0 con 0x7fb0241064d0 2023-11-16T18:16:42.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.075+0000 7fb0237fe700 1 --2- 172.21.15.2:0/1498682984 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 0x7fb0241088b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0140098a0 tx=0x7fb014009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6996b31ea33dae0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:42.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.075+0000 7fb022ffd700 1 -- 172.21.15.2:0/1498682984 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb014004160 con 0x7fb0241064d0 2023-11-16T18:16:42.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.075+0000 7fb022ffd700 1 -- 172.21.15.2:0/1498682984 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0140042c0 con 0x7fb0241064d0 2023-11-16T18:16:42.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.075+0000 7fb022ffd700 1 -- 172.21.15.2:0/1498682984 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0140119b0 con 0x7fb0241064d0 2023-11-16T18:16:42.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.077+0000 7fb029f4b700 1 -- 172.21.15.2:0/1498682984 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 msgr2=0x7fb0241088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:42.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.077+0000 7fb029f4b700 1 --2- 172.21.15.2:0/1498682984 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 0x7fb0241088b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0140098a0 tx=0x7fb014009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:42.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.077+0000 7fb029f4b700 1 -- 172.21.15.2:0/1498682984 shutdown_connections 2023-11-16T18:16:42.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.077+0000 7fb029f4b700 1 --2- 172.21.15.2:0/1498682984 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 0x7fb0241088b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:42.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.077+0000 7fb029f4b700 1 -- 172.21.15.2:0/1498682984 >> 172.21.15.2:0/1498682984 conn(0x7fb0240ffed0 msgr2=0x7fb024102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:42.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.077+0000 7fb029f4b700 1 -- 172.21.15.2:0/1498682984 shutdown_connections 2023-11-16T18:16:42.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.077+0000 7fb029f4b700 1 -- 172.21.15.2:0/1498682984 wait complete. 2023-11-16T18:16:42.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.077+0000 7fb029f4b700 1 Processor -- start 2023-11-16T18:16:42.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.078+0000 7fb029f4b700 1 -- start start 2023-11-16T18:16:42.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.078+0000 7fb029f4b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 0x7fb0241997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:42.143 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.078+0000 7fb029f4b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb024199ce0 con 0x7fb0241064d0 2023-11-16T18:16:42.143 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.078+0000 7fb0237fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 0x7fb0241997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:42.143 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.078+0000 7fb0237fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 0x7fb0241997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46794/0 (socket says 172.21.15.2:46794) 2023-11-16T18:16:42.143 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.078+0000 7fb0237fe700 1 -- 172.21.15.2:0/3135115458 learned_addr learned my addr 172.21.15.2:0/3135115458 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:42.144 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.078+0000 7fb0237fe700 1 -- 172.21.15.2:0/3135115458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb014009550 con 0x7fb0241064d0 2023-11-16T18:16:42.144 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.078+0000 7fb0237fe700 1 --2- 172.21.15.2:0/3135115458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 0x7fb0241997a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb014003e60 tx=0x7fb014011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:42.144 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.079+0000 7fb0217fa700 1 -- 172.21.15.2:0/3135115458 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb014003970 con 0x7fb0241064d0 2023-11-16T18:16:42.144 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.079+0000 7fb029f4b700 1 -- 172.21.15.2:0/3135115458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb024109d60 con 0x7fb0241064d0 2023-11-16T18:16:42.145 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.079+0000 7fb0217fa700 1 -- 172.21.15.2:0/3135115458 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb014003ad0 con 0x7fb0241064d0 2023-11-16T18:16:42.145 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.079+0000 7fb0217fa700 1 -- 172.21.15.2:0/3135115458 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0140195e0 con 0x7fb0241064d0 2023-11-16T18:16:42.145 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.079+0000 7fb029f4b700 1 -- 172.21.15.2:0/3135115458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb024109bd0 con 0x7fb0241064d0 2023-11-16T18:16:42.145 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.080+0000 7fb0217fa700 1 -- 172.21.15.2:0/3135115458 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fb0140198a0 con 0x7fb0241064d0 2023-11-16T18:16:42.146 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.080+0000 7fb0217fa700 1 --2- 172.21.15.2:0/3135115458 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb010038710 0x7fb01003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:42.146 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.080+0000 7fb0217fa700 1 -- 172.21.15.2:0/3135115458 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb01404c9c0 con 0x7fb0241064d0 2023-11-16T18:16:42.146 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.080+0000 7fb029f4b700 1 -- 172.21.15.2:0/3135115458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb024109ef0 con 0x7fb0241064d0 2023-11-16T18:16:42.147 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.080+0000 7fb01bfff700 1 --2- 172.21.15.2:0/3135115458 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb010038710 0x7fb01003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:42.147 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.081+0000 7fb01bfff700 1 --2- 172.21.15.2:0/3135115458 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb010038710 0x7fb01003abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb00c006fd0 tx=0x7fb00c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:42.147 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.089+0000 7fb0217fa700 1 -- 172.21.15.2:0/3135115458 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb014019bf0 con 0x7fb0241064d0 2023-11-16T18:16:42.148 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.493+0000 7fb029f4b700 1 -- 172.21.15.2:0/3135115458 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb024061f40 con 0x7fb010038710 2023-11-16T18:16:42.148 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.747+0000 7fb0217fa700 1 -- 172.21.15.2:0/3135115458 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fb024061f40 con 0x7fb010038710 2023-11-16T18:16:42.148 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.749+0000 7fb029f4b700 1 -- 172.21.15.2:0/3135115458 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb010038710 msgr2=0x7fb01003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:42.148 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.749+0000 7fb029f4b700 1 --2- 172.21.15.2:0/3135115458 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb010038710 0x7fb01003abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb00c006fd0 tx=0x7fb00c006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:42.149 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.749+0000 7fb029f4b700 1 -- 172.21.15.2:0/3135115458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 msgr2=0x7fb0241997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:42.149 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.749+0000 7fb029f4b700 1 --2- 172.21.15.2:0/3135115458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 0x7fb0241997a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb014003e60 tx=0x7fb014011aa0 comp rx=0 tx=0).stop 2023-11-16T18:16:42.149 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.750+0000 7fb029f4b700 1 -- 172.21.15.2:0/3135115458 shutdown_connections 2023-11-16T18:16:42.150 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.750+0000 7fb029f4b700 1 --2- 172.21.15.2:0/3135115458 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb010038710 0x7fb01003abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:42.150 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.750+0000 7fb029f4b700 1 --2- 172.21.15.2:0/3135115458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0241064d0 0x7fb0241997a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:42.150 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.750+0000 7fb029f4b700 1 -- 172.21.15.2:0/3135115458 >> 172.21.15.2:0/3135115458 conn(0x7fb0240ffed0 msgr2=0x7fb0241015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:42.151 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.750+0000 7fb029f4b700 1 -- 172.21.15.2:0/3135115458 shutdown_connections 2023-11-16T18:16:42.151 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:41.750+0000 7fb029f4b700 1 -- 172.21.15.2:0/3135115458 wait complete. 2023-11-16T18:16:42.152 INFO:teuthology.orchestra.run.smithi002.stdout:Fetching dashboard port number... 2023-11-16T18:16:43.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:42 smithi002 ceph-mon[98807]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:16:43.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:43.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 8443 2023-11-16T18:16:43.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.734+0000 7fb5fd72e700 1 Processor -- start 2023-11-16T18:16:43.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.734+0000 7fb5fd72e700 1 -- start start 2023-11-16T18:16:43.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.735+0000 7fb5fd72e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 0x7fb5f81088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:43.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.735+0000 7fb5fd72e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f8079e20 con 0x7fb5f81064d0 2023-11-16T18:16:43.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.735+0000 7fb5f6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 0x7fb5f81088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:43.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.735+0000 7fb5f6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 0x7fb5f81088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46798/0 (socket says 172.21.15.2:46798) 2023-11-16T18:16:43.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.735+0000 7fb5f6ffd700 1 -- 172.21.15.2:0/1462571462 learned_addr learned my addr 172.21.15.2:0/1462571462 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:43.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.735+0000 7fb5f6ffd700 1 -- 172.21.15.2:0/1462571462 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f8108df0 con 0x7fb5f81064d0 2023-11-16T18:16:43.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.735+0000 7fb5f6ffd700 1 --2- 172.21.15.2:0/1462571462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 0x7fb5f81088b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb5f00098a0 tx=0x7fb5f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda6b4a7c0a71df server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:43.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.735+0000 7fb5f5ffb700 1 -- 172.21.15.2:0/1462571462 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5f0004160 con 0x7fb5f81064d0 2023-11-16T18:16:43.427 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.736+0000 7fb5f5ffb700 1 -- 172.21.15.2:0/1462571462 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5f00042c0 con 0x7fb5f81064d0 2023-11-16T18:16:43.427 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.736+0000 7fb5f5ffb700 1 -- 172.21.15.2:0/1462571462 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5f00119b0 con 0x7fb5f81064d0 2023-11-16T18:16:43.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.736+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1462571462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 msgr2=0x7fb5f81088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:43.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.736+0000 7fb5fd72e700 1 --2- 172.21.15.2:0/1462571462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 0x7fb5f81088b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb5f00098a0 tx=0x7fb5f0009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:43.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.736+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1462571462 shutdown_connections 2023-11-16T18:16:43.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.736+0000 7fb5fd72e700 1 --2- 172.21.15.2:0/1462571462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 0x7fb5f81088b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:43.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.736+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1462571462 >> 172.21.15.2:0/1462571462 conn(0x7fb5f80ffed0 msgr2=0x7fb5f8102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:43.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.736+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1462571462 shutdown_connections 2023-11-16T18:16:43.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.736+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1462571462 wait complete. 2023-11-16T18:16:43.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.737+0000 7fb5fd72e700 1 Processor -- start 2023-11-16T18:16:43.438 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.737+0000 7fb5fd72e700 1 -- start start 2023-11-16T18:16:43.438 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.737+0000 7fb5fd72e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 0x7fb5f8105e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:43.438 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.737+0000 7fb5fd72e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f8106370 con 0x7fb5f81064d0 2023-11-16T18:16:43.438 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.737+0000 7fb5f6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 0x7fb5f8105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:43.439 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.737+0000 7fb5f6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 0x7fb5f8105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46800/0 (socket says 172.21.15.2:46800) 2023-11-16T18:16:43.439 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.737+0000 7fb5f6ffd700 1 -- 172.21.15.2:0/1049117804 learned_addr learned my addr 172.21.15.2:0/1049117804 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:43.439 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.738+0000 7fb5f6ffd700 1 -- 172.21.15.2:0/1049117804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f0009550 con 0x7fb5f81064d0 2023-11-16T18:16:43.440 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.738+0000 7fb5f6ffd700 1 --2- 172.21.15.2:0/1049117804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 0x7fb5f8105e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0003e60 tx=0x7fb5f0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:43.440 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.738+0000 7fb5effff700 1 -- 172.21.15.2:0/1049117804 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5f0003970 con 0x7fb5f81064d0 2023-11-16T18:16:43.440 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.738+0000 7fb5effff700 1 -- 172.21.15.2:0/1049117804 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5f0003ad0 con 0x7fb5f81064d0 2023-11-16T18:16:43.441 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.738+0000 7fb5effff700 1 -- 172.21.15.2:0/1049117804 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5f00195e0 con 0x7fb5f81064d0 2023-11-16T18:16:43.443 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.738+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1049117804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f8109d60 con 0x7fb5f81064d0 2023-11-16T18:16:43.445 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.738+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1049117804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f8109bd0 con 0x7fb5f81064d0 2023-11-16T18:16:43.445 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.739+0000 7fb5effff700 1 -- 172.21.15.2:0/1049117804 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fb5f001a430 con 0x7fb5f81064d0 2023-11-16T18:16:43.445 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.739+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1049117804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f8062120 con 0x7fb5f81064d0 2023-11-16T18:16:43.446 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.739+0000 7fb5effff700 1 --2- 172.21.15.2:0/1049117804 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb5e4038760 0x7fb5e403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:43.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.739+0000 7fb5effff700 1 -- 172.21.15.2:0/1049117804 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb5f004c940 con 0x7fb5f81064d0 2023-11-16T18:16:43.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.740+0000 7fb5f67fc700 1 --2- 172.21.15.2:0/1049117804 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb5e4038760 0x7fb5e403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:43.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.740+0000 7fb5f67fc700 1 --2- 172.21.15.2:0/1049117804 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb5e4038760 0x7fb5e403ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0006fd0 tx=0x7fb5e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:43.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:42.746+0000 7fb5effff700 1 -- 172.21.15.2:0/1049117804 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb5f0003c40 con 0x7fb5f81064d0 2023-11-16T18:16:43.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:43.110+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1049117804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb5f8104be0 con 0x7fb5f81064d0 2023-11-16T18:16:43.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:43.110+0000 7fb5effff700 1 -- 172.21.15.2:0/1049117804 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb5f0023b80 con 0x7fb5f81064d0 2023-11-16T18:16:43.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:43.112+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1049117804 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb5e4038760 msgr2=0x7fb5e403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:43.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:43.112+0000 7fb5fd72e700 1 --2- 172.21.15.2:0/1049117804 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb5e4038760 0x7fb5e403ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0006fd0 tx=0x7fb5e0006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:43.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:43.112+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1049117804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 msgr2=0x7fb5f8105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:43.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:43.112+0000 7fb5fd72e700 1 --2- 172.21.15.2:0/1049117804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 0x7fb5f8105e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0003e60 tx=0x7fb5f0011aa0 comp rx=0 tx=0).stop 2023-11-16T18:16:43.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:43.113+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1049117804 shutdown_connections 2023-11-16T18:16:43.451 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:43.113+0000 7fb5fd72e700 1 --2- 172.21.15.2:0/1049117804 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb5e4038760 0x7fb5e403ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:43.451 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:43.113+0000 7fb5fd72e700 1 --2- 172.21.15.2:0/1049117804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5f81064d0 0x7fb5f8105e30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:43.451 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:43.113+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1049117804 >> 172.21.15.2:0/1049117804 conn(0x7fb5f80ffed0 msgr2=0x7fb5f8101570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:43.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:43.113+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1049117804 shutdown_connections 2023-11-16T18:16:43.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:43.113+0000 7fb5fd72e700 1 -- 172.21.15.2:0/1049117804 wait complete. 2023-11-16T18:16:43.505 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T18:16:43.505 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2023-11-16T18:16:43.505 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T18:16:43.505 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2023-11-16T18:16:43.505 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2023-11-16T18:16:43.505 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-16T18:16:43.505 INFO:teuthology.orchestra.run.smithi002.stdout:Ceph Dashboard is now available at: 2023-11-16T18:16:43.505 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:16:43.505 INFO:teuthology.orchestra.run.smithi002.stdout: URL: https://smithi002:8443/ 2023-11-16T18:16:43.506 INFO:teuthology.orchestra.run.smithi002.stdout: User: admin 2023-11-16T18:16:43.506 INFO:teuthology.orchestra.run.smithi002.stdout: Password: l0yas50k7y 2023-11-16T18:16:43.506 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:16:43.506 INFO:teuthology.orchestra.run.smithi002.stdout:Saving cluster configuration to /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config directory 2023-11-16T18:16:43.506 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling autotune for osd_memory_target 2023-11-16T18:16:44.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:43 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1049117804' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-16T18:16:44.693 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.011+0000 7fec8f7d7700 1 Processor -- start 2023-11-16T18:16:44.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.012+0000 7fec8f7d7700 1 -- start start 2023-11-16T18:16:44.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.012+0000 7fec8f7d7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 0x7fec881088f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:44.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.012+0000 7fec8f7d7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec88079de0 con 0x7fec88106510 2023-11-16T18:16:44.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.012+0000 7fec8d573700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 0x7fec881088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:44.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.012+0000 7fec8d573700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 0x7fec881088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46808/0 (socket says 172.21.15.2:46808) 2023-11-16T18:16:44.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.012+0000 7fec8d573700 1 -- 172.21.15.2:0/1837145942 learned_addr learned my addr 172.21.15.2:0/1837145942 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:44.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.012+0000 7fec8d573700 1 -- 172.21.15.2:0/1837145942 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec88108e30 con 0x7fec88106510 2023-11-16T18:16:44.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.012+0000 7fec8d573700 1 --2- 172.21.15.2:0/1837145942 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 0x7fec881088f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0098a0 tx=0x7fec7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ba0ae8455d8044 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:44.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.013+0000 7fec7bfff700 1 -- 172.21.15.2:0/1837145942 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec7c004160 con 0x7fec88106510 2023-11-16T18:16:44.694 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.013+0000 7fec7bfff700 1 -- 172.21.15.2:0/1837145942 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec7c0042c0 con 0x7fec88106510 2023-11-16T18:16:44.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.013+0000 7fec7bfff700 1 -- 172.21.15.2:0/1837145942 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec7c0119b0 con 0x7fec88106510 2023-11-16T18:16:44.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.013+0000 7fec8f7d7700 1 -- 172.21.15.2:0/1837145942 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 msgr2=0x7fec881088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:44.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.013+0000 7fec8f7d7700 1 --2- 172.21.15.2:0/1837145942 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 0x7fec881088f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0098a0 tx=0x7fec7c009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:44.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.013+0000 7fec8f7d7700 1 -- 172.21.15.2:0/1837145942 shutdown_connections 2023-11-16T18:16:44.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.013+0000 7fec8f7d7700 1 --2- 172.21.15.2:0/1837145942 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 0x7fec881088f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:44.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.013+0000 7fec8f7d7700 1 -- 172.21.15.2:0/1837145942 >> 172.21.15.2:0/1837145942 conn(0x7fec880ffed0 msgr2=0x7fec88102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:44.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.013+0000 7fec8f7d7700 1 -- 172.21.15.2:0/1837145942 shutdown_connections 2023-11-16T18:16:44.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.013+0000 7fec8f7d7700 1 -- 172.21.15.2:0/1837145942 wait complete. 2023-11-16T18:16:44.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.014+0000 7fec8f7d7700 1 Processor -- start 2023-11-16T18:16:44.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.014+0000 7fec8f7d7700 1 -- start start 2023-11-16T18:16:44.695 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.014+0000 7fec8f7d7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 0x7fec88199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:44.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.014+0000 7fec8f7d7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec88199cd0 con 0x7fec88106510 2023-11-16T18:16:44.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.014+0000 7fec8d573700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 0x7fec88199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:44.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.014+0000 7fec8d573700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 0x7fec88199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46810/0 (socket says 172.21.15.2:46810) 2023-11-16T18:16:44.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.014+0000 7fec8d573700 1 -- 172.21.15.2:0/193548396 learned_addr learned my addr 172.21.15.2:0/193548396 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:44.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.015+0000 7fec8d573700 1 -- 172.21.15.2:0/193548396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c009550 con 0x7fec88106510 2023-11-16T18:16:44.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.015+0000 7fec8d573700 1 --2- 172.21.15.2:0/193548396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 0x7fec88199790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fec7c003e60 tx=0x7fec7c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:44.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.015+0000 7fec7a7fc700 1 -- 172.21.15.2:0/193548396 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec7c003970 con 0x7fec88106510 2023-11-16T18:16:44.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.015+0000 7fec7a7fc700 1 -- 172.21.15.2:0/193548396 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec7c003ad0 con 0x7fec88106510 2023-11-16T18:16:44.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.015+0000 7fec8f7d7700 1 -- 172.21.15.2:0/193548396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec88109d50 con 0x7fec88106510 2023-11-16T18:16:44.696 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.015+0000 7fec7a7fc700 1 -- 172.21.15.2:0/193548396 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec7c0195e0 con 0x7fec88106510 2023-11-16T18:16:44.713 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.015+0000 7fec8f7d7700 1 -- 172.21.15.2:0/193548396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec88109bc0 con 0x7fec88106510 2023-11-16T18:16:44.713 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.016+0000 7fec7a7fc700 1 -- 172.21.15.2:0/193548396 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fec7c01a430 con 0x7fec88106510 2023-11-16T18:16:44.714 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.016+0000 7fec8f7d7700 1 -- 172.21.15.2:0/193548396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec88109ee0 con 0x7fec88106510 2023-11-16T18:16:44.714 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.016+0000 7fec7a7fc700 1 --2- 172.21.15.2:0/193548396 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fec74038760 0x7fec7403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:44.714 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.017+0000 7fec7a7fc700 1 -- 172.21.15.2:0/193548396 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fec7c04c940 con 0x7fec88106510 2023-11-16T18:16:44.715 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.017+0000 7fec8cd72700 1 --2- 172.21.15.2:0/193548396 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fec74038760 0x7fec7403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:44.715 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.017+0000 7fec8cd72700 1 --2- 172.21.15.2:0/193548396 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fec74038760 0x7fec7403ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fec84006fd0 tx=0x7fec84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:44.716 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.021+0000 7fec7a7fc700 1 -- 172.21.15.2:0/193548396 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fec7c023b40 con 0x7fec88106510 2023-11-16T18:16:44.716 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.376+0000 7fec8f7d7700 1 -- 172.21.15.2:0/193548396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fec8819d410 con 0x7fec88106510 2023-11-16T18:16:44.716 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.376+0000 7fec7a7fc700 1 -- 172.21.15.2:0/193548396 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fec7c023b40 con 0x7fec88106510 2023-11-16T18:16:44.716 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.378+0000 7fec8f7d7700 1 -- 172.21.15.2:0/193548396 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fec74038760 msgr2=0x7fec7403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:44.716 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.378+0000 7fec8f7d7700 1 --2- 172.21.15.2:0/193548396 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fec74038760 0x7fec7403ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fec84006fd0 tx=0x7fec84006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:44.716 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.378+0000 7fec8f7d7700 1 -- 172.21.15.2:0/193548396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 msgr2=0x7fec88199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:44.716 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.378+0000 7fec8f7d7700 1 --2- 172.21.15.2:0/193548396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 0x7fec88199790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fec7c003e60 tx=0x7fec7c011aa0 comp rx=0 tx=0).stop 2023-11-16T18:16:44.716 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.378+0000 7fec8f7d7700 1 -- 172.21.15.2:0/193548396 shutdown_connections 2023-11-16T18:16:44.716 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.378+0000 7fec8f7d7700 1 --2- 172.21.15.2:0/193548396 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fec74038760 0x7fec7403ac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:44.716 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.378+0000 7fec8f7d7700 1 --2- 172.21.15.2:0/193548396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec88106510 0x7fec88199790 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:44.716 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.378+0000 7fec8f7d7700 1 -- 172.21.15.2:0/193548396 >> 172.21.15.2:0/193548396 conn(0x7fec880ffed0 msgr2=0x7fec88101530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:44.717 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.378+0000 7fec8f7d7700 1 -- 172.21.15.2:0/193548396 shutdown_connections 2023-11-16T18:16:44.717 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:44.378+0000 7fec8f7d7700 1 -- 172.21.15.2:0/193548396 wait complete. 2023-11-16T18:16:46.031 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.142+0000 7f128042c700 1 Processor -- start 2023-11-16T18:16:46.031 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.142+0000 7f128042c700 1 -- start start 2023-11-16T18:16:46.032 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.143+0000 7f128042c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 0x7f1278105230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:46.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.143+0000 7f128042c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1278105800 con 0x7f1278104e20 2023-11-16T18:16:46.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.143+0000 7f127e1c8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 0x7f1278105230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:46.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.143+0000 7f127e1c8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 0x7f1278105230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46814/0 (socket says 172.21.15.2:46814) 2023-11-16T18:16:46.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.143+0000 7f127e1c8700 1 -- 172.21.15.2:0/930864304 learned_addr learned my addr 172.21.15.2:0/930864304 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:46.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.143+0000 7f127e1c8700 1 -- 172.21.15.2:0/930864304 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1278105b40 con 0x7f1278104e20 2023-11-16T18:16:46.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.143+0000 7f127e1c8700 1 --2- 172.21.15.2:0/930864304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 0x7f1278105230 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f12680098a0 tx=0x7f1268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b76cbe8bb0887 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:46.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.143+0000 7f127d1c6700 1 -- 172.21.15.2:0/930864304 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1268004160 con 0x7f1278104e20 2023-11-16T18:16:46.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.143+0000 7f127d1c6700 1 -- 172.21.15.2:0/930864304 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12680042c0 con 0x7f1278104e20 2023-11-16T18:16:46.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.143+0000 7f127d1c6700 1 -- 172.21.15.2:0/930864304 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12680119b0 con 0x7f1278104e20 2023-11-16T18:16:46.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.144+0000 7f128042c700 1 -- 172.21.15.2:0/930864304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 msgr2=0x7f1278105230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:46.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.144+0000 7f128042c700 1 --2- 172.21.15.2:0/930864304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 0x7f1278105230 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f12680098a0 tx=0x7f1268009bb0 comp rx=0 tx=0).stop 2023-11-16T18:16:46.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.144+0000 7f128042c700 1 -- 172.21.15.2:0/930864304 shutdown_connections 2023-11-16T18:16:46.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.144+0000 7f128042c700 1 --2- 172.21.15.2:0/930864304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 0x7f1278105230 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:46.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.144+0000 7f128042c700 1 -- 172.21.15.2:0/930864304 >> 172.21.15.2:0/930864304 conn(0x7f12780ff650 msgr2=0x7f1278101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:46.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.144+0000 7f128042c700 1 -- 172.21.15.2:0/930864304 shutdown_connections 2023-11-16T18:16:46.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.144+0000 7f128042c700 1 -- 172.21.15.2:0/930864304 wait complete. 2023-11-16T18:16:46.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.145+0000 7f128042c700 1 Processor -- start 2023-11-16T18:16:46.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.145+0000 7f128042c700 1 -- start start 2023-11-16T18:16:46.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.145+0000 7f128042c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 0x7f1278198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:46.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.145+0000 7f128042c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1278198bd0 con 0x7f1278104e20 2023-11-16T18:16:46.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.145+0000 7f127e1c8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 0x7f1278198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:46.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.145+0000 7f127e1c8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 0x7f1278198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46816/0 (socket says 172.21.15.2:46816) 2023-11-16T18:16:46.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.145+0000 7f127e1c8700 1 -- 172.21.15.2:0/1166786837 learned_addr learned my addr 172.21.15.2:0/1166786837 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:46.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.145+0000 7f127e1c8700 1 -- 172.21.15.2:0/1166786837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1268009550 con 0x7f1278104e20 2023-11-16T18:16:46.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.146+0000 7f127e1c8700 1 --2- 172.21.15.2:0/1166786837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 0x7f1278198690 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1268003e60 tx=0x7f1268011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:46.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.146+0000 7f126f7fe700 1 -- 172.21.15.2:0/1166786837 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1268003970 con 0x7f1278104e20 2023-11-16T18:16:46.047 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.146+0000 7f126f7fe700 1 -- 172.21.15.2:0/1166786837 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1268003ad0 con 0x7f1278104e20 2023-11-16T18:16:46.047 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.146+0000 7f128042c700 1 -- 172.21.15.2:0/1166786837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1278108f00 con 0x7f1278104e20 2023-11-16T18:16:46.047 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.146+0000 7f126f7fe700 1 -- 172.21.15.2:0/1166786837 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12680195e0 con 0x7f1278104e20 2023-11-16T18:16:46.047 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.146+0000 7f128042c700 1 -- 172.21.15.2:0/1166786837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1278108d70 con 0x7f1278104e20 2023-11-16T18:16:46.047 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.147+0000 7f126f7fe700 1 -- 172.21.15.2:0/1166786837 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f12680198e0 con 0x7f1278104e20 2023-11-16T18:16:46.047 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.147+0000 7f128042c700 1 -- 172.21.15.2:0/1166786837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1278109090 con 0x7f1278104e20 2023-11-16T18:16:46.047 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.147+0000 7f126f7fe700 1 --2- 172.21.15.2:0/1166786837 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f12640387b0 0x7f126403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:46.047 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.147+0000 7f126f7fe700 1 -- 172.21.15.2:0/1166786837 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f126804ca90 con 0x7f1278104e20 2023-11-16T18:16:46.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.147+0000 7f127d9c7700 1 --2- 172.21.15.2:0/1166786837 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f12640387b0 0x7f126403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:46.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.147+0000 7f127d9c7700 1 --2- 172.21.15.2:0/1166786837 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f12640387b0 0x7f126403ac60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1274006fd0 tx=0x7f1274006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:46.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.152+0000 7f126f7fe700 1 -- 172.21.15.2:0/1166786837 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1268023b80 con 0x7f1278104e20 2023-11-16T18:16:46.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.724+0000 7f128042c700 1 -- 172.21.15.2:0/1166786837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f12780621a0 con 0x7f1278104e20 2023-11-16T18:16:46.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.726+0000 7f126f7fe700 1 -- 172.21.15.2:0/1166786837 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f1268023b80 con 0x7f1278104e20 2023-11-16T18:16:46.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.728+0000 7f128042c700 1 -- 172.21.15.2:0/1166786837 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f12640387b0 msgr2=0x7f126403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:46.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.728+0000 7f128042c700 1 --2- 172.21.15.2:0/1166786837 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f12640387b0 0x7f126403ac60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1274006fd0 tx=0x7f1274006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:46.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.728+0000 7f128042c700 1 -- 172.21.15.2:0/1166786837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 msgr2=0x7f1278198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:46.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.728+0000 7f128042c700 1 --2- 172.21.15.2:0/1166786837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 0x7f1278198690 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1268003e60 tx=0x7f1268011aa0 comp rx=0 tx=0).stop 2023-11-16T18:16:46.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.728+0000 7f128042c700 1 -- 172.21.15.2:0/1166786837 shutdown_connections 2023-11-16T18:16:46.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.728+0000 7f128042c700 1 --2- 172.21.15.2:0/1166786837 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f12640387b0 0x7f126403ac60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:46.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.728+0000 7f128042c700 1 --2- 172.21.15.2:0/1166786837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1278104e20 0x7f1278198690 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:46.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.728+0000 7f128042c700 1 -- 172.21.15.2:0/1166786837 >> 172.21.15.2:0/1166786837 conn(0x7f12780ff650 msgr2=0x7f1278100d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:46.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.728+0000 7f128042c700 1 -- 172.21.15.2:0/1166786837 shutdown_connections 2023-11-16T18:16:46.060 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T18:16:45.728+0000 7f128042c700 1 -- 172.21.15.2:0/1166786837 wait complete. 2023-11-16T18:16:46.060 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-11-16T18:16:46.060 INFO:teuthology.orchestra.run.smithi002.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-11-16T18:16:46.060 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:16:46.060 INFO:teuthology.orchestra.run.smithi002.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-16T18:16:46.060 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:16:46.060 INFO:teuthology.orchestra.run.smithi002.stdout:Or, if you are only running a single cluster on this host: 2023-11-16T18:16:46.060 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:16:46.060 INFO:teuthology.orchestra.run.smithi002.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-11-16T18:16:46.060 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:16:46.060 INFO:teuthology.orchestra.run.smithi002.stdout:Please consider enabling telemetry to help improve Ceph: 2023-11-16T18:16:46.061 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:16:46.061 INFO:teuthology.orchestra.run.smithi002.stdout: ceph telemetry on 2023-11-16T18:16:46.061 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:16:46.061 INFO:teuthology.orchestra.run.smithi002.stdout:For more information see: 2023-11-16T18:16:46.061 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:16:46.062 INFO:teuthology.orchestra.run.smithi002.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-11-16T18:16:46.062 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:16:46.062 INFO:teuthology.orchestra.run.smithi002.stdout:Bootstrap complete. 2023-11-16T18:16:46.062 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:45 smithi002 ceph-mon[98807]: mgrmap e13: x(active, since 7s) 2023-11-16T18:16:46.062 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:45 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1166786837' entity='client.admin' 2023-11-16T18:16:46.103 INFO:tasks.cephadm:Fetching config... 2023-11-16T18:16:46.107 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:16:46.122 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-16T18:16:46.177 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-16T18:16:46.178 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:16:46.178 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-16T18:16:46.243 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-16T18:16:46.244 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:16:46.245 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/keyring of=/dev/stdout 2023-11-16T18:16:46.354 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-16T18:16:46.355 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:16:46.356 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-16T18:16:46.416 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-16T18:16:46.417 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+0R8beOieZYos5G8AZwNJgU5L9F8r/fdqji0sI5xDZQpfN+FWquOcbr5u/Pn+kPPxTWcPcU1d2Q/dI8qcciTMFTF8FIqORgv1cH7eoNNB8c5Iy9o/l18kQPZ//38jyiLrutX0bLKWWr9Npg70flkUTgs9DSFgjZUOL+pIMmBK3IlkQwk4imWENBR83G9GJ1U3HGIW1l+qZO1j6JJUiQxctTDfdf9LaGqXSObZojkLuiLOmMitF8BLs/RQ9BOrzYRuiCqHzrxJaEMGdLB7kuqDizlu8FLHvvYYHyZ27pQ3kAx+cnbQonyYXJctAy+rCrD6txNEZWlXDBNnawmRPnEkWxV3/H7/RIVlXPF7STtZEZBq+AccyFi1lTNqILKGzwTT+mGN/Fvr8Nl05aDCYPaEe+LW3KHlDkHEoq4NZwzaA88oQmUMpVcSYgjISgvI3QixqjepCCVELU7RKv7GYiTM9xfwHcEdjBnzNH1gQ2+N/5++VtyoSzmkRoxvNIKzlo8= ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T18:16:46.506 INFO:teuthology.orchestra.run.smithi002.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+0R8beOieZYos5G8AZwNJgU5L9F8r/fdqji0sI5xDZQpfN+FWquOcbr5u/Pn+kPPxTWcPcU1d2Q/dI8qcciTMFTF8FIqORgv1cH7eoNNB8c5Iy9o/l18kQPZ//38jyiLrutX0bLKWWr9Npg70flkUTgs9DSFgjZUOL+pIMmBK3IlkQwk4imWENBR83G9GJ1U3HGIW1l+qZO1j6JJUiQxctTDfdf9LaGqXSObZojkLuiLOmMitF8BLs/RQ9BOrzYRuiCqHzrxJaEMGdLB7kuqDizlu8FLHvvYYHyZ27pQ3kAx+cnbQonyYXJctAy+rCrD6txNEZWlXDBNnawmRPnEkWxV3/H7/RIVlXPF7STtZEZBq+AccyFi1lTNqILKGzwTT+mGN/Fvr8Nl05aDCYPaEe+LW3KHlDkHEoq4NZwzaA88oQmUMpVcSYgjISgvI3QixqjepCCVELU7RKv7GYiTM9xfwHcEdjBnzNH1gQ2+N/5++VtyoSzmkRoxvNIKzlo8= ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:16:46.519 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+0R8beOieZYos5G8AZwNJgU5L9F8r/fdqji0sI5xDZQpfN+FWquOcbr5u/Pn+kPPxTWcPcU1d2Q/dI8qcciTMFTF8FIqORgv1cH7eoNNB8c5Iy9o/l18kQPZ//38jyiLrutX0bLKWWr9Npg70flkUTgs9DSFgjZUOL+pIMmBK3IlkQwk4imWENBR83G9GJ1U3HGIW1l+qZO1j6JJUiQxctTDfdf9LaGqXSObZojkLuiLOmMitF8BLs/RQ9BOrzYRuiCqHzrxJaEMGdLB7kuqDizlu8FLHvvYYHyZ27pQ3kAx+cnbQonyYXJctAy+rCrD6txNEZWlXDBNnawmRPnEkWxV3/H7/RIVlXPF7STtZEZBq+AccyFi1lTNqILKGzwTT+mGN/Fvr8Nl05aDCYPaEe+LW3KHlDkHEoq4NZwzaA88oQmUMpVcSYgjISgvI3QixqjepCCVELU7RKv7GYiTM9xfwHcEdjBnzNH1gQ2+N/5++VtyoSzmkRoxvNIKzlo8= ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T18:16:46.569 INFO:teuthology.orchestra.run.smithi040.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+0R8beOieZYos5G8AZwNJgU5L9F8r/fdqji0sI5xDZQpfN+FWquOcbr5u/Pn+kPPxTWcPcU1d2Q/dI8qcciTMFTF8FIqORgv1cH7eoNNB8c5Iy9o/l18kQPZ//38jyiLrutX0bLKWWr9Npg70flkUTgs9DSFgjZUOL+pIMmBK3IlkQwk4imWENBR83G9GJ1U3HGIW1l+qZO1j6JJUiQxctTDfdf9LaGqXSObZojkLuiLOmMitF8BLs/RQ9BOrzYRuiCqHzrxJaEMGdLB7kuqDizlu8FLHvvYYHyZ27pQ3kAx+cnbQonyYXJctAy+rCrD6txNEZWlXDBNnawmRPnEkWxV3/H7/RIVlXPF7STtZEZBq+AccyFi1lTNqILKGzwTT+mGN/Fvr8Nl05aDCYPaEe+LW3KHlDkHEoq4NZwzaA88oQmUMpVcSYgjISgvI3QixqjepCCVELU7RKv7GYiTM9xfwHcEdjBnzNH1gQ2+N/5++VtyoSzmkRoxvNIKzlo8= ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:16:46.583 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+0R8beOieZYos5G8AZwNJgU5L9F8r/fdqji0sI5xDZQpfN+FWquOcbr5u/Pn+kPPxTWcPcU1d2Q/dI8qcciTMFTF8FIqORgv1cH7eoNNB8c5Iy9o/l18kQPZ//38jyiLrutX0bLKWWr9Npg70flkUTgs9DSFgjZUOL+pIMmBK3IlkQwk4imWENBR83G9GJ1U3HGIW1l+qZO1j6JJUiQxctTDfdf9LaGqXSObZojkLuiLOmMitF8BLs/RQ9BOrzYRuiCqHzrxJaEMGdLB7kuqDizlu8FLHvvYYHyZ27pQ3kAx+cnbQonyYXJctAy+rCrD6txNEZWlXDBNnawmRPnEkWxV3/H7/RIVlXPF7STtZEZBq+AccyFi1lTNqILKGzwTT+mGN/Fvr8Nl05aDCYPaEe+LW3KHlDkHEoq4NZwzaA88oQmUMpVcSYgjISgvI3QixqjepCCVELU7RKv7GYiTM9xfwHcEdjBnzNH1gQ2+N/5++VtyoSzmkRoxvNIKzlo8= ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T18:16:46.636 INFO:teuthology.orchestra.run.smithi188.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+0R8beOieZYos5G8AZwNJgU5L9F8r/fdqji0sI5xDZQpfN+FWquOcbr5u/Pn+kPPxTWcPcU1d2Q/dI8qcciTMFTF8FIqORgv1cH7eoNNB8c5Iy9o/l18kQPZ//38jyiLrutX0bLKWWr9Npg70flkUTgs9DSFgjZUOL+pIMmBK3IlkQwk4imWENBR83G9GJ1U3HGIW1l+qZO1j6JJUiQxctTDfdf9LaGqXSObZojkLuiLOmMitF8BLs/RQ9BOrzYRuiCqHzrxJaEMGdLB7kuqDizlu8FLHvvYYHyZ27pQ3kAx+cnbQonyYXJctAy+rCrD6txNEZWlXDBNnawmRPnEkWxV3/H7/RIVlXPF7STtZEZBq+AccyFi1lTNqILKGzwTT+mGN/Fvr8Nl05aDCYPaEe+LW3KHlDkHEoq4NZwzaA88oQmUMpVcSYgjISgvI3QixqjepCCVELU7RKv7GYiTM9xfwHcEdjBnzNH1gQ2+N/5++VtyoSzmkRoxvNIKzlo8= ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:16:46.649 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-16T18:16:47.456 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:16:48.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.858+0000 7f83c602a700 1 -- 172.21.15.2:0/2331441291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83c00febf0 msgr2=0x7f83c00ff000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:48.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.858+0000 7f83c602a700 1 --2- 172.21.15.2:0/2331441291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83c00febf0 0x7f83c00ff000 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009910 tx=0x7f83b0009c20 comp rx=0 tx=0).stop 2023-11-16T18:16:48.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.858+0000 7f83c602a700 1 -- 172.21.15.2:0/2331441291 shutdown_connections 2023-11-16T18:16:48.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.858+0000 7f83c602a700 1 --2- 172.21.15.2:0/2331441291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83c00febf0 0x7f83c00ff000 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:48.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.858+0000 7f83c602a700 1 -- 172.21.15.2:0/2331441291 >> 172.21.15.2:0/2331441291 conn(0x7f83c00fa240 msgr2=0x7f83c00fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:48.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.858+0000 7f83c602a700 1 -- 172.21.15.2:0/2331441291 shutdown_connections 2023-11-16T18:16:48.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.858+0000 7f83c602a700 1 -- 172.21.15.2:0/2331441291 wait complete. 2023-11-16T18:16:48.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.859+0000 7f83c602a700 1 Processor -- start 2023-11-16T18:16:48.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.859+0000 7f83c602a700 1 -- start start 2023-11-16T18:16:48.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.859+0000 7f83c602a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83c00febf0 0x7f83c0068870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:48.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.859+0000 7f83c602a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c01958e0 con 0x7f83c00febf0 2023-11-16T18:16:48.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.859+0000 7f83bf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83c00febf0 0x7f83c0068870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.859+0000 7f83bf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83c00febf0 0x7f83c0068870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46822/0 (socket says 172.21.15.2:46822) 2023-11-16T18:16:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.859+0000 7f83bf7fe700 1 -- 172.21.15.2:0/711305198 learned_addr learned my addr 172.21.15.2:0/711305198 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.859+0000 7f83bf7fe700 1 -- 172.21.15.2:0/711305198 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00095f0 con 0x7f83c00febf0 2023-11-16T18:16:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.860+0000 7f83bf7fe700 1 --2- 172.21.15.2:0/711305198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83c00febf0 0x7f83c0068870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f83b0004f40 tx=0x7f83b00057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.860+0000 7f83bcff9700 1 -- 172.21.15.2:0/711305198 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f83b001d070 con 0x7f83c00febf0 2023-11-16T18:16:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.860+0000 7f83c602a700 1 -- 172.21.15.2:0/711305198 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c0104a40 con 0x7f83c00febf0 2023-11-16T18:16:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.860+0000 7f83bcff9700 1 -- 172.21.15.2:0/711305198 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83b00044e0 con 0x7f83c00febf0 2023-11-16T18:16:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.860+0000 7f83bcff9700 1 -- 172.21.15.2:0/711305198 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f83b0003a20 con 0x7f83c00febf0 2023-11-16T18:16:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.860+0000 7f83c602a700 1 -- 172.21.15.2:0/711305198 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c01048b0 con 0x7f83c00febf0 2023-11-16T18:16:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.861+0000 7f83bcff9700 1 -- 172.21.15.2:0/711305198 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f83b0003ca0 con 0x7f83c00febf0 2023-11-16T18:16:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.861+0000 7f83c602a700 1 -- 172.21.15.2:0/711305198 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c018d770 con 0x7f83c00febf0 2023-11-16T18:16:48.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.861+0000 7f83bcff9700 1 --2- 172.21.15.2:0/711305198 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f83ac038870 0x7f83ac03ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:48.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.861+0000 7f83bcff9700 1 -- 172.21.15.2:0/711305198 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f83b004c990 con 0x7f83c00febf0 2023-11-16T18:16:48.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.861+0000 7f83beffd700 1 --2- 172.21.15.2:0/711305198 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f83ac038870 0x7f83ac03ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:48.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.861+0000 7f83beffd700 1 --2- 172.21.15.2:0/711305198 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f83ac038870 0x7f83ac03ad20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f83a8006fd0 tx=0x7f83a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:48.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:48.866+0000 7f83bcff9700 1 -- 172.21.15.2:0/711305198 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f83b0020b20 con 0x7f83c00febf0 2023-11-16T18:16:49.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:49.063+0000 7f83c602a700 1 -- 172.21.15.2:0/711305198 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f83c0062120 con 0x7f83c00febf0 2023-11-16T18:16:49.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:49.066+0000 7f83bcff9700 1 -- 172.21.15.2:0/711305198 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f83b0020b20 con 0x7f83c00febf0 2023-11-16T18:16:49.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:49.068+0000 7f83c602a700 1 -- 172.21.15.2:0/711305198 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f83ac038870 msgr2=0x7f83ac03ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:49.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:49.068+0000 7f83c602a700 1 --2- 172.21.15.2:0/711305198 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f83ac038870 0x7f83ac03ad20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f83a8006fd0 tx=0x7f83a8006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:49.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:49.068+0000 7f83c602a700 1 -- 172.21.15.2:0/711305198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83c00febf0 msgr2=0x7f83c0068870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:49.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:49.068+0000 7f83c602a700 1 --2- 172.21.15.2:0/711305198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83c00febf0 0x7f83c0068870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f83b0004f40 tx=0x7f83b00057c0 comp rx=0 tx=0).stop 2023-11-16T18:16:49.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:49.068+0000 7f83c602a700 1 -- 172.21.15.2:0/711305198 shutdown_connections 2023-11-16T18:16:49.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:49.068+0000 7f83c602a700 1 --2- 172.21.15.2:0/711305198 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f83ac038870 0x7f83ac03ad20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:49.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:49.068+0000 7f83c602a700 1 --2- 172.21.15.2:0/711305198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83c00febf0 0x7f83c0068870 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:49.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:49.068+0000 7f83c602a700 1 -- 172.21.15.2:0/711305198 >> 172.21.15.2:0/711305198 conn(0x7f83c00fa240 msgr2=0x7f83c00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:49.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:49.068+0000 7f83c602a700 1 -- 172.21.15.2:0/711305198 shutdown_connections 2023-11-16T18:16:49.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:49.068+0000 7f83c602a700 1 -- 172.21.15.2:0/711305198 wait complete. 2023-11-16T18:16:49.515 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-11-16T18:16:49.515 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-11-16T18:16:49.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:49 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:49.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:49 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:49.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:49 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:16:49.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:49 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:49.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:49 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:16:49.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:49 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:49.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:49 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/711305198' entity='client.admin' 2023-11-16T18:16:49.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:49 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:16:49.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:49 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:50.172 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:16:51.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.366+0000 7f5a72d72700 1 -- 172.21.15.2:0/981271912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a6c0ffd20 msgr2=0x7f5a6c100130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:51.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.366+0000 7f5a72d72700 1 --2- 172.21.15.2:0/981271912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a6c0ffd20 0x7f5a6c100130 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009910 tx=0x7f5a60009c20 comp rx=0 tx=0).stop 2023-11-16T18:16:51.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.366+0000 7f5a72d72700 1 -- 172.21.15.2:0/981271912 shutdown_connections 2023-11-16T18:16:51.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.366+0000 7f5a72d72700 1 --2- 172.21.15.2:0/981271912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a6c0ffd20 0x7f5a6c100130 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:51.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.366+0000 7f5a72d72700 1 -- 172.21.15.2:0/981271912 >> 172.21.15.2:0/981271912 conn(0x7f5a6c0fb2b0 msgr2=0x7f5a6c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:51.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.366+0000 7f5a72d72700 1 -- 172.21.15.2:0/981271912 shutdown_connections 2023-11-16T18:16:51.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.366+0000 7f5a72d72700 1 -- 172.21.15.2:0/981271912 wait complete. 2023-11-16T18:16:51.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.366+0000 7f5a72d72700 1 Processor -- start 2023-11-16T18:16:51.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.366+0000 7f5a72d72700 1 -- start start 2023-11-16T18:16:51.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.367+0000 7f5a72d72700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a6c0ffd20 0x7f5a6c074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:51.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.367+0000 7f5a72d72700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c074dd0 con 0x7f5a6c0ffd20 2023-11-16T18:16:51.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.367+0000 7f5a70b0e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a6c0ffd20 0x7f5a6c074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:51.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.367+0000 7f5a70b0e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a6c0ffd20 0x7f5a6c074890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46828/0 (socket says 172.21.15.2:46828) 2023-11-16T18:16:51.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.367+0000 7f5a70b0e700 1 -- 172.21.15.2:0/1094773860 learned_addr learned my addr 172.21.15.2:0/1094773860 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:51.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.367+0000 7f5a70b0e700 1 -- 172.21.15.2:0/1094773860 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a600095f0 con 0x7f5a6c0ffd20 2023-11-16T18:16:51.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.367+0000 7f5a70b0e700 1 --2- 172.21.15.2:0/1094773860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a6c0ffd20 0x7f5a6c074890 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5a60004f40 tx=0x7f5a600057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:51.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.367+0000 7f5a69ffb700 1 -- 172.21.15.2:0/1094773860 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a6001d070 con 0x7f5a6c0ffd20 2023-11-16T18:16:51.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.367+0000 7f5a72d72700 1 -- 172.21.15.2:0/1094773860 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c105bf0 con 0x7f5a6c0ffd20 2023-11-16T18:16:51.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.367+0000 7f5a69ffb700 1 -- 172.21.15.2:0/1094773860 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a6000bc10 con 0x7f5a6c0ffd20 2023-11-16T18:16:51.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.367+0000 7f5a69ffb700 1 -- 172.21.15.2:0/1094773860 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a60003a70 con 0x7f5a6c0ffd20 2023-11-16T18:16:51.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.368+0000 7f5a72d72700 1 -- 172.21.15.2:0/1094773860 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c105a60 con 0x7f5a6c0ffd20 2023-11-16T18:16:51.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.369+0000 7f5a69ffb700 1 -- 172.21.15.2:0/1094773860 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f5a60003d30 con 0x7f5a6c0ffd20 2023-11-16T18:16:51.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.369+0000 7f5a72d72700 1 -- 172.21.15.2:0/1094773860 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c105d80 con 0x7f5a6c0ffd20 2023-11-16T18:16:51.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.369+0000 7f5a69ffb700 1 --2- 172.21.15.2:0/1094773860 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5a5c038870 0x7f5a5c03ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:51.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.369+0000 7f5a69ffb700 1 -- 172.21.15.2:0/1094773860 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5a6004c6f0 con 0x7f5a6c0ffd20 2023-11-16T18:16:51.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.369+0000 7f5a6bfff700 1 --2- 172.21.15.2:0/1094773860 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5a5c038870 0x7f5a5c03ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:51.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.369+0000 7f5a6bfff700 1 --2- 172.21.15.2:0/1094773860 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5a5c038870 0x7f5a5c03ad20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5a58006fd0 tx=0x7f5a58006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:51.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.374+0000 7f5a69ffb700 1 -- 172.21.15.2:0/1094773860 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5a60029770 con 0x7f5a6c0ffd20 2023-11-16T18:16:51.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.575+0000 7f5a72d72700 1 -- 172.21.15.2:0/1094773860 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a6c073360 con 0x7f5a5c038870 2023-11-16T18:16:51.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.578+0000 7f5a69ffb700 1 -- 172.21.15.2:0/1094773860 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5a6c073360 con 0x7f5a5c038870 2023-11-16T18:16:51.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.581+0000 7f5a72d72700 1 -- 172.21.15.2:0/1094773860 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5a5c038870 msgr2=0x7f5a5c03ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:51.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.581+0000 7f5a72d72700 1 --2- 172.21.15.2:0/1094773860 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5a5c038870 0x7f5a5c03ad20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5a58006fd0 tx=0x7f5a58006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:51.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.581+0000 7f5a72d72700 1 -- 172.21.15.2:0/1094773860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a6c0ffd20 msgr2=0x7f5a6c074890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:51.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.581+0000 7f5a72d72700 1 --2- 172.21.15.2:0/1094773860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a6c0ffd20 0x7f5a6c074890 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5a60004f40 tx=0x7f5a600057c0 comp rx=0 tx=0).stop 2023-11-16T18:16:51.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.581+0000 7f5a72d72700 1 -- 172.21.15.2:0/1094773860 shutdown_connections 2023-11-16T18:16:51.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.581+0000 7f5a72d72700 1 --2- 172.21.15.2:0/1094773860 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5a5c038870 0x7f5a5c03ad20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:51.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.581+0000 7f5a72d72700 1 --2- 172.21.15.2:0/1094773860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a6c0ffd20 0x7f5a6c074890 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:51.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.581+0000 7f5a72d72700 1 -- 172.21.15.2:0/1094773860 >> 172.21.15.2:0/1094773860 conn(0x7f5a6c0fb2b0 msgr2=0x7f5a6c0fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:51.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.581+0000 7f5a72d72700 1 -- 172.21.15.2:0/1094773860 shutdown_connections 2023-11-16T18:16:51.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:51.581+0000 7f5a72d72700 1 -- 172.21.15.2:0/1094773860 wait complete. 2023-11-16T18:16:51.930 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi040 2023-11-16T18:16:51.931 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-11-16T18:16:51.931 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/etc/ceph/ceph.conf 2023-11-16T18:16:51.952 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-11-16T18:16:51.952 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:16:52.013 INFO:tasks.cephadm:Adding host smithi040 to orchestrator... 2023-11-16T18:16:52.013 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi040 2023-11-16T18:16:52.718 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:16:52.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:52 smithi002 ceph-mon[98807]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:16:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:16:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:16:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:16:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:52 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:16:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:52 smithi002 ceph-mon[98807]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T18:16:53.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.915+0000 7f7ebdb55700 1 -- 172.21.15.2:0/521383034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7eb80fef00 msgr2=0x7f7eb80ff2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:53.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.915+0000 7f7ebdb55700 1 --2- 172.21.15.2:0/521383034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7eb80fef00 0x7f7eb80ff2d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8009910 tx=0x7f7ea8009c20 comp rx=0 tx=0).stop 2023-11-16T18:16:53.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.915+0000 7f7ebdb55700 1 -- 172.21.15.2:0/521383034 shutdown_connections 2023-11-16T18:16:53.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.915+0000 7f7ebdb55700 1 --2- 172.21.15.2:0/521383034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7eb80fef00 0x7f7eb80ff2d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:53.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.915+0000 7f7ebdb55700 1 -- 172.21.15.2:0/521383034 >> 172.21.15.2:0/521383034 conn(0x7f7eb8074930 msgr2=0x7f7eb8074d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:53.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.915+0000 7f7ebdb55700 1 -- 172.21.15.2:0/521383034 shutdown_connections 2023-11-16T18:16:53.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.915+0000 7f7ebdb55700 1 -- 172.21.15.2:0/521383034 wait complete. 2023-11-16T18:16:53.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.915+0000 7f7ebdb55700 1 Processor -- start 2023-11-16T18:16:53.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.916+0000 7f7ebdb55700 1 -- start start 2023-11-16T18:16:53.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.916+0000 7f7ebdb55700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7eb80fef00 0x7f7eb8195270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:53.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.916+0000 7f7ebdb55700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb81957b0 con 0x7f7eb80fef00 2023-11-16T18:16:53.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.916+0000 7f7eb77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7eb80fef00 0x7f7eb8195270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:53.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.916+0000 7f7eb77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7eb80fef00 0x7f7eb8195270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46834/0 (socket says 172.21.15.2:46834) 2023-11-16T18:16:53.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.916+0000 7f7eb77fe700 1 -- 172.21.15.2:0/3515406350 learned_addr learned my addr 172.21.15.2:0/3515406350 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:53.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.916+0000 7f7eb77fe700 1 -- 172.21.15.2:0/3515406350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea80095f0 con 0x7f7eb80fef00 2023-11-16T18:16:53.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.916+0000 7f7eb77fe700 1 --2- 172.21.15.2:0/3515406350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7eb80fef00 0x7f7eb8195270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8004f40 tx=0x7f7ea8005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:53.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.916+0000 7f7eb4ff9700 1 -- 172.21.15.2:0/3515406350 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ea801d070 con 0x7f7eb80fef00 2023-11-16T18:16:53.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.917+0000 7f7eb4ff9700 1 -- 172.21.15.2:0/3515406350 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ea8004370 con 0x7f7eb80fef00 2023-11-16T18:16:53.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.917+0000 7f7ebdb55700 1 -- 172.21.15.2:0/3515406350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb8109090 con 0x7f7eb80fef00 2023-11-16T18:16:53.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.917+0000 7f7eb4ff9700 1 -- 172.21.15.2:0/3515406350 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ea80039b0 con 0x7f7eb80fef00 2023-11-16T18:16:53.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.917+0000 7f7ebdb55700 1 -- 172.21.15.2:0/3515406350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb8108f00 con 0x7f7eb80fef00 2023-11-16T18:16:53.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.917+0000 7f7eb4ff9700 1 -- 172.21.15.2:0/3515406350 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f7ea8003c70 con 0x7f7eb80fef00 2023-11-16T18:16:53.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.918+0000 7f7eb4ff9700 1 --2- 172.21.15.2:0/3515406350 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7ea0038870 0x7f7ea003ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:53.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.918+0000 7f7ebdb55700 1 -- 172.21.15.2:0/3515406350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb8109220 con 0x7f7eb80fef00 2023-11-16T18:16:53.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.918+0000 7f7eb4ff9700 1 -- 172.21.15.2:0/3515406350 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7ea804c560 con 0x7f7eb80fef00 2023-11-16T18:16:53.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.918+0000 7f7eb6ffd700 1 --2- 172.21.15.2:0/3515406350 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7ea0038870 0x7f7ea003ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:53.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.918+0000 7f7eb6ffd700 1 --2- 172.21.15.2:0/3515406350 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7ea0038870 0x7f7ea003ad20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7eac006fd0 tx=0x7f7eac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:53.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:53.922+0000 7f7eb4ff9700 1 -- 172.21.15.2:0/3515406350 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7ea8026020 con 0x7f7eb80fef00 2023-11-16T18:16:54.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:53 smithi002 ceph-mon[98807]: Updating smithi002:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:16:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:53 smithi002 ceph-mon[98807]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:16:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:53 smithi002 ceph-mon[98807]: Updating smithi002:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:16:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:53 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:53 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:53 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:54.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:54.125+0000 7f7ebdb55700 1 -- 172.21.15.2:0/3515406350 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi040", "target": ["mon-mgr", ""]}) v1 -- 0x7f7eb8195d60 con 0x7f7ea0038870 2023-11-16T18:16:55.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:54 smithi002 ceph-mon[98807]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi040", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:16:55.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:54 smithi002 ceph-mon[98807]: Deploying cephadm binary to smithi040 2023-11-16T18:16:56.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:56.588+0000 7f7eb4ff9700 1 -- 172.21.15.2:0/3515406350 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f7eb8195d60 con 0x7f7ea0038870 2023-11-16T18:16:56.595 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi040' with addr '172.21.15.40' 2023-11-16T18:16:56.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:56.590+0000 7f7ebdb55700 1 -- 172.21.15.2:0/3515406350 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7ea0038870 msgr2=0x7f7ea003ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:56.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:56.590+0000 7f7ebdb55700 1 --2- 172.21.15.2:0/3515406350 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7ea0038870 0x7f7ea003ad20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7eac006fd0 tx=0x7f7eac006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:56.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:56.590+0000 7f7ebdb55700 1 -- 172.21.15.2:0/3515406350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7eb80fef00 msgr2=0x7f7eb8195270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:56.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:56.590+0000 7f7ebdb55700 1 --2- 172.21.15.2:0/3515406350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7eb80fef00 0x7f7eb8195270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8004f40 tx=0x7f7ea8005020 comp rx=0 tx=0).stop 2023-11-16T18:16:56.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:56.590+0000 7f7ebdb55700 1 -- 172.21.15.2:0/3515406350 shutdown_connections 2023-11-16T18:16:56.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:56.590+0000 7f7ebdb55700 1 --2- 172.21.15.2:0/3515406350 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7ea0038870 0x7f7ea003ad20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:56.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:56.590+0000 7f7ebdb55700 1 --2- 172.21.15.2:0/3515406350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7eb80fef00 0x7f7eb8195270 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:56.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:56.590+0000 7f7ebdb55700 1 -- 172.21.15.2:0/3515406350 >> 172.21.15.2:0/3515406350 conn(0x7f7eb8074930 msgr2=0x7f7eb8067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:56.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:56.591+0000 7f7ebdb55700 1 -- 172.21.15.2:0/3515406350 shutdown_connections 2023-11-16T18:16:56.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:56.591+0000 7f7ebdb55700 1 -- 172.21.15.2:0/3515406350 wait complete. 2023-11-16T18:16:56.989 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-16T18:16:57.645 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:16:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:57 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:57 smithi002 ceph-mon[98807]: Added host smithi040 2023-11-16T18:16:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:57 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:16:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:57 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.830+0000 7efe314f4700 1 -- 172.21.15.2:0/940464068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe2c0fef00 msgr2=0x7efe2c0ff2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.830+0000 7efe314f4700 1 --2- 172.21.15.2:0/940464068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe2c0fef00 0x7efe2c0ff2d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efe1c009910 tx=0x7efe1c009c20 comp rx=0 tx=0).stop 2023-11-16T18:16:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.830+0000 7efe314f4700 1 -- 172.21.15.2:0/940464068 shutdown_connections 2023-11-16T18:16:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.830+0000 7efe314f4700 1 --2- 172.21.15.2:0/940464068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe2c0fef00 0x7efe2c0ff2d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.830+0000 7efe314f4700 1 -- 172.21.15.2:0/940464068 >> 172.21.15.2:0/940464068 conn(0x7efe2c074930 msgr2=0x7efe2c074d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.830+0000 7efe314f4700 1 -- 172.21.15.2:0/940464068 shutdown_connections 2023-11-16T18:16:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.830+0000 7efe314f4700 1 -- 172.21.15.2:0/940464068 wait complete. 2023-11-16T18:16:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.831+0000 7efe314f4700 1 Processor -- start 2023-11-16T18:16:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.831+0000 7efe314f4700 1 -- start start 2023-11-16T18:16:58.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.832+0000 7efe314f4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe2c0fef00 0x7efe2c196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:58.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.832+0000 7efe314f4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c196550 con 0x7efe2c0fef00 2023-11-16T18:16:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.832+0000 7efe2affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe2c0fef00 0x7efe2c196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.832+0000 7efe2affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe2c0fef00 0x7efe2c196010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46860/0 (socket says 172.21.15.2:46860) 2023-11-16T18:16:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.832+0000 7efe2affd700 1 -- 172.21.15.2:0/1150014277 learned_addr learned my addr 172.21.15.2:0/1150014277 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:16:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.832+0000 7efe2affd700 1 -- 172.21.15.2:0/1150014277 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c0095f0 con 0x7efe2c0fef00 2023-11-16T18:16:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.832+0000 7efe2affd700 1 --2- 172.21.15.2:0/1150014277 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe2c0fef00 0x7efe2c196010 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efe1c006010 tx=0x7efe1c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.832+0000 7efe23fff700 1 -- 172.21.15.2:0/1150014277 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe1c01d070 con 0x7efe2c0fef00 2023-11-16T18:16:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.832+0000 7efe314f4700 1 -- 172.21.15.2:0/1150014277 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2c109090 con 0x7efe2c0fef00 2023-11-16T18:16:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.832+0000 7efe23fff700 1 -- 172.21.15.2:0/1150014277 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe1c00bc10 con 0x7efe2c0fef00 2023-11-16T18:16:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.832+0000 7efe23fff700 1 -- 172.21.15.2:0/1150014277 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe1c003910 con 0x7efe2c0fef00 2023-11-16T18:16:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.832+0000 7efe314f4700 1 -- 172.21.15.2:0/1150014277 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2c108f00 con 0x7efe2c0fef00 2023-11-16T18:16:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.833+0000 7efe23fff700 1 -- 172.21.15.2:0/1150014277 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7efe1c003bd0 con 0x7efe2c0fef00 2023-11-16T18:16:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.833+0000 7efe23fff700 1 --2- 172.21.15.2:0/1150014277 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe18038870 0x7efe1803ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:16:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.833+0000 7efe314f4700 1 -- 172.21.15.2:0/1150014277 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe0c005320 con 0x7efe2c0fef00 2023-11-16T18:16:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.833+0000 7efe23fff700 1 -- 172.21.15.2:0/1150014277 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efe1c00bd80 con 0x7efe2c0fef00 2023-11-16T18:16:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.833+0000 7efe2a7fc700 1 --2- 172.21.15.2:0/1150014277 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe18038870 0x7efe1803ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:16:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.834+0000 7efe2a7fc700 1 --2- 172.21.15.2:0/1150014277 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe18038870 0x7efe1803ad20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efe14006fd0 tx=0x7efe14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:16:58.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:58.839+0000 7efe23fff700 1 -- 172.21.15.2:0/1150014277 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efe1c026020 con 0x7efe2c0fef00 2023-11-16T18:16:59.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:59.040+0000 7efe314f4700 1 -- 172.21.15.2:0/1150014277 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe0c000bf0 con 0x7efe18038870 2023-11-16T18:16:59.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:59.042+0000 7efe23fff700 1 -- 172.21.15.2:0/1150014277 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+158 (secure 0 0 0) 0x7efe0c000bf0 con 0x7efe18038870 2023-11-16T18:16:59.044 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:16:59.044 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}] 2023-11-16T18:16:59.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:59.043+0000 7efe314f4700 1 -- 172.21.15.2:0/1150014277 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe18038870 msgr2=0x7efe1803ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:59.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:59.043+0000 7efe314f4700 1 --2- 172.21.15.2:0/1150014277 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe18038870 0x7efe1803ad20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efe14006fd0 tx=0x7efe14006e40 comp rx=0 tx=0).stop 2023-11-16T18:16:59.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:59.043+0000 7efe314f4700 1 -- 172.21.15.2:0/1150014277 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe2c0fef00 msgr2=0x7efe2c196010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:16:59.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:59.043+0000 7efe314f4700 1 --2- 172.21.15.2:0/1150014277 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe2c0fef00 0x7efe2c196010 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efe1c006010 tx=0x7efe1c004ff0 comp rx=0 tx=0).stop 2023-11-16T18:16:59.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:59.044+0000 7efe314f4700 1 -- 172.21.15.2:0/1150014277 shutdown_connections 2023-11-16T18:16:59.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:59.044+0000 7efe314f4700 1 --2- 172.21.15.2:0/1150014277 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe18038870 0x7efe1803ad20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:59.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:59.044+0000 7efe314f4700 1 --2- 172.21.15.2:0/1150014277 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe2c0fef00 0x7efe2c196010 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:16:59.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:59.044+0000 7efe314f4700 1 -- 172.21.15.2:0/1150014277 >> 172.21.15.2:0/1150014277 conn(0x7efe2c074930 msgr2=0x7efe2c103090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:16:59.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:59.044+0000 7efe314f4700 1 -- 172.21.15.2:0/1150014277 shutdown_connections 2023-11-16T18:16:59.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:16:59.044+0000 7efe314f4700 1 -- 172.21.15.2:0/1150014277 wait complete. 2023-11-16T18:16:59.079 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:58 smithi002 ceph-mon[98807]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:16:59.079 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:16:58 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:16:59.492 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi188 2023-11-16T18:16:59.493 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-16T18:16:59.493 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.conf 2023-11-16T18:16:59.514 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-16T18:16:59.514 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:16:59.573 INFO:tasks.cephadm:Adding host smithi188 to orchestrator... 2023-11-16T18:16:59.574 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi188 2023-11-16T18:17:00.228 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:17:00.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:00 smithi002 ceph-mon[98807]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:17:00.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:00 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:00.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:00 smithi002 ceph-mon[98807]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:01.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.413+0000 7faa833f9700 1 -- 172.21.15.2:0/485713601 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa7c0fff90 msgr2=0x7faa7c100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:01.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.413+0000 7faa833f9700 1 --2- 172.21.15.2:0/485713601 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa7c0fff90 0x7faa7c100360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faa78009910 tx=0x7faa78009c20 comp rx=0 tx=0).stop 2023-11-16T18:17:01.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.413+0000 7faa833f9700 1 -- 172.21.15.2:0/485713601 shutdown_connections 2023-11-16T18:17:01.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.413+0000 7faa833f9700 1 --2- 172.21.15.2:0/485713601 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa7c0fff90 0x7faa7c100360 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:01.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.413+0000 7faa833f9700 1 -- 172.21.15.2:0/485713601 >> 172.21.15.2:0/485713601 conn(0x7faa7c0fb870 msgr2=0x7faa7c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:01.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.413+0000 7faa833f9700 1 -- 172.21.15.2:0/485713601 shutdown_connections 2023-11-16T18:17:01.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.413+0000 7faa833f9700 1 -- 172.21.15.2:0/485713601 wait complete. 2023-11-16T18:17:01.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.413+0000 7faa833f9700 1 Processor -- start 2023-11-16T18:17:01.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.414+0000 7faa833f9700 1 -- start start 2023-11-16T18:17:01.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.414+0000 7faa833f9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa7c0fff90 0x7faa7c072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:01.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.414+0000 7faa833f9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c074d70 con 0x7faa7c0fff90 2023-11-16T18:17:01.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.414+0000 7faa81195700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa7c0fff90 0x7faa7c072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:01.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.414+0000 7faa81195700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa7c0fff90 0x7faa7c072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46866/0 (socket says 172.21.15.2:46866) 2023-11-16T18:17:01.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.414+0000 7faa81195700 1 -- 172.21.15.2:0/671630148 learned_addr learned my addr 172.21.15.2:0/671630148 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:17:01.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.414+0000 7faa81195700 1 -- 172.21.15.2:0/671630148 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa780095f0 con 0x7faa7c0fff90 2023-11-16T18:17:01.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.414+0000 7faa81195700 1 --2- 172.21.15.2:0/671630148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa7c0fff90 0x7faa7c072df0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faa78004f40 tx=0x7faa78005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:01.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.414+0000 7faa727fc700 1 -- 172.21.15.2:0/671630148 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa7801d070 con 0x7faa7c0fff90 2023-11-16T18:17:01.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.414+0000 7faa727fc700 1 -- 172.21.15.2:0/671630148 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa78004370 con 0x7faa7c0fff90 2023-11-16T18:17:01.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.414+0000 7faa833f9700 1 -- 172.21.15.2:0/671630148 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7c10a170 con 0x7faa7c0fff90 2023-11-16T18:17:01.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.414+0000 7faa727fc700 1 -- 172.21.15.2:0/671630148 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa780039b0 con 0x7faa7c0fff90 2023-11-16T18:17:01.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.415+0000 7faa833f9700 1 -- 172.21.15.2:0/671630148 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7c109fe0 con 0x7faa7c0fff90 2023-11-16T18:17:01.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.415+0000 7faa727fc700 1 -- 172.21.15.2:0/671630148 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7faa78003c20 con 0x7faa7c0fff90 2023-11-16T18:17:01.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.415+0000 7faa833f9700 1 -- 172.21.15.2:0/671630148 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7c10a300 con 0x7faa7c0fff90 2023-11-16T18:17:01.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.416+0000 7faa727fc700 1 --2- 172.21.15.2:0/671630148 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7faa680388c0 0x7faa6803ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:01.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.416+0000 7faa727fc700 1 -- 172.21.15.2:0/671630148 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faa7804c750 con 0x7faa7c0fff90 2023-11-16T18:17:01.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.416+0000 7faa80994700 1 --2- 172.21.15.2:0/671630148 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7faa680388c0 0x7faa6803ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:01.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.416+0000 7faa80994700 1 --2- 172.21.15.2:0/671630148 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7faa680388c0 0x7faa6803ad70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faa6c006fd0 tx=0x7faa6c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:01.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.420+0000 7faa727fc700 1 -- 172.21.15.2:0/671630148 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faa78029360 con 0x7faa7c0fff90 2023-11-16T18:17:01.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:01.621+0000 7faa833f9700 1 -- 172.21.15.2:0/671630148 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}) v1 -- 0x7faa7c074320 con 0x7faa680388c0 2023-11-16T18:17:02.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:01 smithi002 ceph-mon[98807]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:17:02.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:01 smithi002 ceph-mon[98807]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:03.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:03.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:03.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:03.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:03.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:17:03.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:03.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:03.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:02 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:03.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:02 smithi002 ceph-mon[98807]: Updating smithi040:/etc/ceph/ceph.conf 2023-11-16T18:17:03.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:02 smithi002 ceph-mon[98807]: Deploying cephadm binary to smithi188 2023-11-16T18:17:03.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:02 smithi002 ceph-mon[98807]: Updating smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:03.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:02 smithi002 ceph-mon[98807]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:17:04.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:04.024+0000 7faa727fc700 1 -- 172.21.15.2:0/671630148 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7faa7c074320 con 0x7faa680388c0 2023-11-16T18:17:04.026 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi188' with addr '172.21.15.188' 2023-11-16T18:17:04.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:04.026+0000 7faa833f9700 1 -- 172.21.15.2:0/671630148 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7faa680388c0 msgr2=0x7faa6803ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:04.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:04.026+0000 7faa833f9700 1 --2- 172.21.15.2:0/671630148 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7faa680388c0 0x7faa6803ad70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faa6c006fd0 tx=0x7faa6c006e40 comp rx=0 tx=0).stop 2023-11-16T18:17:04.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:04.026+0000 7faa833f9700 1 -- 172.21.15.2:0/671630148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa7c0fff90 msgr2=0x7faa7c072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:04.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:04.026+0000 7faa833f9700 1 --2- 172.21.15.2:0/671630148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa7c0fff90 0x7faa7c072df0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faa78004f40 tx=0x7faa78005020 comp rx=0 tx=0).stop 2023-11-16T18:17:04.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:04.026+0000 7faa833f9700 1 -- 172.21.15.2:0/671630148 shutdown_connections 2023-11-16T18:17:04.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:04.026+0000 7faa833f9700 1 --2- 172.21.15.2:0/671630148 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7faa680388c0 0x7faa6803ad70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:04.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:04.026+0000 7faa833f9700 1 --2- 172.21.15.2:0/671630148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa7c0fff90 0x7faa7c072df0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:04.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:04.026+0000 7faa833f9700 1 -- 172.21.15.2:0/671630148 >> 172.21.15.2:0/671630148 conn(0x7faa7c0fb870 msgr2=0x7faa7c104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:04.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:04.026+0000 7faa833f9700 1 -- 172.21.15.2:0/671630148 shutdown_connections 2023-11-16T18:17:04.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:04.026+0000 7faa833f9700 1 -- 172.21.15.2:0/671630148 wait complete. 2023-11-16T18:17:04.393 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-16T18:17:05.045 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:17:05.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:04 smithi002 ceph-mon[98807]: Updating smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:17:05.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:04 smithi002 ceph-mon[98807]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:05.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:05.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:05.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:05.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:05.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:05.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:05.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:05 smithi002 ceph-mon[98807]: Added host smithi188 2023-11-16T18:17:05.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:05 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:06.220 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.216+0000 7fd59f8d0700 1 -- 172.21.15.2:0/847963875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd598100df0 msgr2=0x7fd5981011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:06.220 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.216+0000 7fd59f8d0700 1 --2- 172.21.15.2:0/847963875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd598100df0 0x7fd5981011c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd588009910 tx=0x7fd588009c20 comp rx=0 tx=0).stop 2023-11-16T18:17:06.220 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.216+0000 7fd59f8d0700 1 -- 172.21.15.2:0/847963875 shutdown_connections 2023-11-16T18:17:06.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.216+0000 7fd59f8d0700 1 --2- 172.21.15.2:0/847963875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd598100df0 0x7fd5981011c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:06.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.216+0000 7fd59f8d0700 1 -- 172.21.15.2:0/847963875 >> 172.21.15.2:0/847963875 conn(0x7fd598074930 msgr2=0x7fd598074d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:06.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.216+0000 7fd59f8d0700 1 -- 172.21.15.2:0/847963875 shutdown_connections 2023-11-16T18:17:06.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.216+0000 7fd59f8d0700 1 -- 172.21.15.2:0/847963875 wait complete. 2023-11-16T18:17:06.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.217+0000 7fd59f8d0700 1 Processor -- start 2023-11-16T18:17:06.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.217+0000 7fd59f8d0700 1 -- start start 2023-11-16T18:17:06.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.217+0000 7fd59f8d0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd598100df0 0x7fd5981951c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:06.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.217+0000 7fd59f8d0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd598195700 con 0x7fd598100df0 2023-11-16T18:17:06.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.217+0000 7fd59d66c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd598100df0 0x7fd5981951c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:06.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.217+0000 7fd59d66c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd598100df0 0x7fd5981951c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46892/0 (socket says 172.21.15.2:46892) 2023-11-16T18:17:06.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.217+0000 7fd59d66c700 1 -- 172.21.15.2:0/908537331 learned_addr learned my addr 172.21.15.2:0/908537331 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:17:06.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.217+0000 7fd59d66c700 1 -- 172.21.15.2:0/908537331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5880095f0 con 0x7fd598100df0 2023-11-16T18:17:06.222 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.218+0000 7fd59d66c700 1 --2- 172.21.15.2:0/908537331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd598100df0 0x7fd5981951c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd588005850 tx=0x7fd588004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:06.222 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.218+0000 7fd58e7fc700 1 -- 172.21.15.2:0/908537331 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd58801d070 con 0x7fd598100df0 2023-11-16T18:17:06.222 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.218+0000 7fd58e7fc700 1 -- 172.21.15.2:0/908537331 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd588004370 con 0x7fd598100df0 2023-11-16T18:17:06.222 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.218+0000 7fd59f8d0700 1 -- 172.21.15.2:0/908537331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59810a190 con 0x7fd598100df0 2023-11-16T18:17:06.222 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.218+0000 7fd58e7fc700 1 -- 172.21.15.2:0/908537331 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd588003ab0 con 0x7fd598100df0 2023-11-16T18:17:06.222 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.218+0000 7fd59f8d0700 1 -- 172.21.15.2:0/908537331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59810a000 con 0x7fd598100df0 2023-11-16T18:17:06.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.219+0000 7fd58e7fc700 1 -- 172.21.15.2:0/908537331 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fd588003d20 con 0x7fd598100df0 2023-11-16T18:17:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.219+0000 7fd58e7fc700 1 --2- 172.21.15.2:0/908537331 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd5840388c0 0x7fd58403ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.219+0000 7fd58e7fc700 1 -- 172.21.15.2:0/908537331 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd58804c540 con 0x7fd598100df0 2023-11-16T18:17:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.219+0000 7fd59ce6b700 1 --2- 172.21.15.2:0/908537331 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd5840388c0 0x7fd58403ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.219+0000 7fd59f8d0700 1 -- 172.21.15.2:0/908537331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59810a320 con 0x7fd598100df0 2023-11-16T18:17:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.220+0000 7fd59ce6b700 1 --2- 172.21.15.2:0/908537331 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd5840388c0 0x7fd58403ad70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd594006fd0 tx=0x7fd594006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.224+0000 7fd58e7fc700 1 -- 172.21.15.2:0/908537331 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd588026020 con 0x7fd598100df0 2023-11-16T18:17:06.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.426+0000 7fd59f8d0700 1 -- 172.21.15.2:0/908537331 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5981072d0 con 0x7fd5840388c0 2023-11-16T18:17:06.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.427+0000 7fd58e7fc700 1 -- 172.21.15.2:0/908537331 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+238 (secure 0 0 0) 0x7fd5981072d0 con 0x7fd5840388c0 2023-11-16T18:17:06.429 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:17:06.429 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.188", "hostname": "smithi188", "labels": [], "status": ""}] 2023-11-16T18:17:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.429+0000 7fd59f8d0700 1 -- 172.21.15.2:0/908537331 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd5840388c0 msgr2=0x7fd58403ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.429+0000 7fd59f8d0700 1 --2- 172.21.15.2:0/908537331 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd5840388c0 0x7fd58403ad70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd594006fd0 tx=0x7fd594006e40 comp rx=0 tx=0).stop 2023-11-16T18:17:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.429+0000 7fd59f8d0700 1 -- 172.21.15.2:0/908537331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd598100df0 msgr2=0x7fd5981951c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.429+0000 7fd59f8d0700 1 --2- 172.21.15.2:0/908537331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd598100df0 0x7fd5981951c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd588005850 tx=0x7fd588004ff0 comp rx=0 tx=0).stop 2023-11-16T18:17:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.429+0000 7fd59f8d0700 1 -- 172.21.15.2:0/908537331 shutdown_connections 2023-11-16T18:17:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.429+0000 7fd59f8d0700 1 --2- 172.21.15.2:0/908537331 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd5840388c0 0x7fd58403ad70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.429+0000 7fd59f8d0700 1 --2- 172.21.15.2:0/908537331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd598100df0 0x7fd5981951c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.429+0000 7fd59f8d0700 1 -- 172.21.15.2:0/908537331 >> 172.21.15.2:0/908537331 conn(0x7fd598074930 msgr2=0x7fd598104610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.429+0000 7fd59f8d0700 1 -- 172.21.15.2:0/908537331 shutdown_connections 2023-11-16T18:17:06.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:06.429+0000 7fd59f8d0700 1 -- 172.21.15.2:0/908537331 wait complete. 2023-11-16T18:17:06.806 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-16T18:17:06.806 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-16T18:17:07.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:06 smithi002 ceph-mon[98807]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:07.111 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:06 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:07.458 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:17:07.937 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:07 smithi002 ceph-mon[98807]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:17:08.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.537+0000 7fba1863e700 1 -- 172.21.15.2:0/3276634239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba10100df0 msgr2=0x7fba101011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:08.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.537+0000 7fba1863e700 1 --2- 172.21.15.2:0/3276634239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba10100df0 0x7fba101011c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fba00009910 tx=0x7fba00009c20 comp rx=0 tx=0).stop 2023-11-16T18:17:08.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.537+0000 7fba1863e700 1 -- 172.21.15.2:0/3276634239 shutdown_connections 2023-11-16T18:17:08.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.537+0000 7fba1863e700 1 --2- 172.21.15.2:0/3276634239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba10100df0 0x7fba101011c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:08.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.537+0000 7fba1863e700 1 -- 172.21.15.2:0/3276634239 >> 172.21.15.2:0/3276634239 conn(0x7fba10074930 msgr2=0x7fba10074d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:08.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.537+0000 7fba1863e700 1 -- 172.21.15.2:0/3276634239 shutdown_connections 2023-11-16T18:17:08.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.537+0000 7fba1863e700 1 -- 172.21.15.2:0/3276634239 wait complete. 2023-11-16T18:17:08.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.538+0000 7fba1863e700 1 Processor -- start 2023-11-16T18:17:08.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.538+0000 7fba1863e700 1 -- start start 2023-11-16T18:17:08.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.538+0000 7fba1863e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba10100df0 0x7fba101951c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:08.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.538+0000 7fba1863e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba10195700 con 0x7fba10100df0 2023-11-16T18:17:08.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.538+0000 7fba163da700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba10100df0 0x7fba101951c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:08.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.538+0000 7fba163da700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba10100df0 0x7fba101951c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46898/0 (socket says 172.21.15.2:46898) 2023-11-16T18:17:08.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.538+0000 7fba163da700 1 -- 172.21.15.2:0/3929959364 learned_addr learned my addr 172.21.15.2:0/3929959364 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:17:08.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.538+0000 7fba163da700 1 -- 172.21.15.2:0/3929959364 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba000095f0 con 0x7fba10100df0 2023-11-16T18:17:08.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.538+0000 7fba163da700 1 --2- 172.21.15.2:0/3929959364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba10100df0 0x7fba101951c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fba00005850 tx=0x7fba00004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:08.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.539+0000 7fba077fe700 1 -- 172.21.15.2:0/3929959364 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba0001d070 con 0x7fba10100df0 2023-11-16T18:17:08.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.539+0000 7fba1863e700 1 -- 172.21.15.2:0/3929959364 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1010a190 con 0x7fba10100df0 2023-11-16T18:17:08.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.539+0000 7fba077fe700 1 -- 172.21.15.2:0/3929959364 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba00004370 con 0x7fba10100df0 2023-11-16T18:17:08.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.539+0000 7fba077fe700 1 -- 172.21.15.2:0/3929959364 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba00003ab0 con 0x7fba10100df0 2023-11-16T18:17:08.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.539+0000 7fba1863e700 1 -- 172.21.15.2:0/3929959364 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1010a000 con 0x7fba10100df0 2023-11-16T18:17:08.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.539+0000 7fba077fe700 1 -- 172.21.15.2:0/3929959364 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fba0000f460 con 0x7fba10100df0 2023-11-16T18:17:08.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.540+0000 7fba077fe700 1 --2- 172.21.15.2:0/3929959364 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb9fc038870 0x7fb9fc03ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:08.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.540+0000 7fba077fe700 1 -- 172.21.15.2:0/3929959364 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fba0004c600 con 0x7fba10100df0 2023-11-16T18:17:08.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.540+0000 7fba1863e700 1 -- 172.21.15.2:0/3929959364 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1010a320 con 0x7fba10100df0 2023-11-16T18:17:08.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.540+0000 7fba15bd9700 1 --2- 172.21.15.2:0/3929959364 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb9fc038870 0x7fb9fc03ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:08.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.540+0000 7fba15bd9700 1 --2- 172.21.15.2:0/3929959364 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb9fc038870 0x7fb9fc03ad20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fba0c006fd0 tx=0x7fba0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:08.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.545+0000 7fba077fe700 1 -- 172.21.15.2:0/3929959364 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fba00026020 con 0x7fba10100df0 2023-11-16T18:17:08.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:08.747+0000 7fba1863e700 1 -- 172.21.15.2:0/3929959364 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fba10062120 con 0x7fba10100df0 2023-11-16T18:17:09.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:08 smithi002 ceph-mon[98807]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:09.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:09.745+0000 7fba077fe700 1 -- 172.21.15.2:0/3929959364 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fba00029950 con 0x7fba10100df0 2023-11-16T18:17:09.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:09.747+0000 7fba1863e700 1 -- 172.21.15.2:0/3929959364 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb9fc038870 msgr2=0x7fb9fc03ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:09.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:09.747+0000 7fba1863e700 1 --2- 172.21.15.2:0/3929959364 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb9fc038870 0x7fb9fc03ad20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fba0c006fd0 tx=0x7fba0c006e40 comp rx=0 tx=0).stop 2023-11-16T18:17:09.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:09.747+0000 7fba1863e700 1 -- 172.21.15.2:0/3929959364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba10100df0 msgr2=0x7fba101951c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:09.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:09.747+0000 7fba1863e700 1 --2- 172.21.15.2:0/3929959364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba10100df0 0x7fba101951c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fba00005850 tx=0x7fba00004ff0 comp rx=0 tx=0).stop 2023-11-16T18:17:09.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:09.748+0000 7fba1863e700 1 -- 172.21.15.2:0/3929959364 shutdown_connections 2023-11-16T18:17:09.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:09.748+0000 7fba1863e700 1 --2- 172.21.15.2:0/3929959364 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb9fc038870 0x7fb9fc03ad20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:09.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:09.748+0000 7fba1863e700 1 --2- 172.21.15.2:0/3929959364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba10100df0 0x7fba101951c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:09.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:09.748+0000 7fba1863e700 1 -- 172.21.15.2:0/3929959364 >> 172.21.15.2:0/3929959364 conn(0x7fba10074930 msgr2=0x7fba10104610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:09.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:09.748+0000 7fba1863e700 1 -- 172.21.15.2:0/3929959364 shutdown_connections 2023-11-16T18:17:09.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:09.748+0000 7fba1863e700 1 -- 172.21.15.2:0/3929959364 wait complete. 2023-11-16T18:17:09.753 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2023-11-16T18:17:10.045 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:09 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3929959364' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T18:17:10.046 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:10.046 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:10.047 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:10.047 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:10.047 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:17:10.047 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:10.048 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:10.120 INFO:tasks.cephadm:Adding mon.a on smithi002 2023-11-16T18:17:10.120 INFO:tasks.cephadm:Adding mon.b on smithi040 2023-11-16T18:17:10.121 INFO:tasks.cephadm:Adding mon.c on smithi188 2023-11-16T18:17:10.121 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '3;smithi002:172.21.15.2=a;smithi040:172.21.15.40=b;smithi188:172.21.15.188=c' 2023-11-16T18:17:10.791 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:11.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:10 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:11.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:10 smithi002 ceph-mon[98807]: Updating smithi188:/etc/ceph/ceph.conf 2023-11-16T18:17:11.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:10 smithi002 ceph-mon[98807]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:11.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:10 smithi002 ceph-mon[98807]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:11.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:10 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3929959364' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T18:17:11.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:10 smithi002 ceph-mon[98807]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T18:17:11.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:10 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:11.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:10 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:11.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:10 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:11.983 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.979+0000 7f4fe3702700 1 -- 172.21.15.188:0/373848754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fdc0fefd0 msgr2=0x7f4fdc0ff3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:11.983 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.979+0000 7f4fe3702700 1 --2- 172.21.15.188:0/373848754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fdc0fefd0 0x7f4fdc0ff3a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009910 tx=0x7f4fcc009c20 comp rx=0 tx=0).stop 2023-11-16T18:17:11.983 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.979+0000 7f4fe3702700 1 -- 172.21.15.188:0/373848754 shutdown_connections 2023-11-16T18:17:11.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.979+0000 7f4fe3702700 1 --2- 172.21.15.188:0/373848754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fdc0fefd0 0x7f4fdc0ff3a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:11.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.979+0000 7f4fe3702700 1 -- 172.21.15.188:0/373848754 >> 172.21.15.188:0/373848754 conn(0x7f4fdc074350 msgr2=0x7f4fdc074750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:11.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.979+0000 7f4fe3702700 1 -- 172.21.15.188:0/373848754 shutdown_connections 2023-11-16T18:17:11.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.979+0000 7f4fe3702700 1 -- 172.21.15.188:0/373848754 wait complete. 2023-11-16T18:17:11.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.980+0000 7f4fe3702700 1 Processor -- start 2023-11-16T18:17:11.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.980+0000 7f4fe3702700 1 -- start start 2023-11-16T18:17:11.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.980+0000 7f4fe3702700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fdc0fefd0 0x7f4fdc0fbe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:11.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.980+0000 7f4fe3702700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc0fdda0 con 0x7f4fdc0fefd0 2023-11-16T18:17:11.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.980+0000 7f4fe149e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fdc0fefd0 0x7f4fdc0fbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:11.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.980+0000 7f4fe149e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fdc0fefd0 0x7f4fdc0fbe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.188:55252/0 (socket says 172.21.15.188:55252) 2023-11-16T18:17:11.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.980+0000 7f4fe149e700 1 -- 172.21.15.188:0/3666121976 learned_addr learned my addr 172.21.15.188:0/3666121976 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-16T18:17:11.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.981+0000 7f4fe149e700 1 -- 172.21.15.188:0/3666121976 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fcc0095f0 con 0x7f4fdc0fefd0 2023-11-16T18:17:11.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.981+0000 7f4fe149e700 1 --2- 172.21.15.188:0/3666121976 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fdc0fefd0 0x7f4fdc0fbe20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc004f40 tx=0x7f4fcc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:11.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.981+0000 7f4fd27fc700 1 -- 172.21.15.188:0/3666121976 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4fcc01d070 con 0x7f4fdc0fefd0 2023-11-16T18:17:11.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.981+0000 7f4fe3702700 1 -- 172.21.15.188:0/3666121976 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fdc108010 con 0x7f4fdc0fefd0 2023-11-16T18:17:11.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.981+0000 7f4fd27fc700 1 -- 172.21.15.188:0/3666121976 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fcc00be20 con 0x7f4fdc0fefd0 2023-11-16T18:17:11.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.981+0000 7f4fd27fc700 1 -- 172.21.15.188:0/3666121976 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4fcc022d00 con 0x7f4fdc0fefd0 2023-11-16T18:17:11.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.981+0000 7f4fe3702700 1 -- 172.21.15.188:0/3666121976 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fdc107e80 con 0x7f4fdc0fefd0 2023-11-16T18:17:11.995 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.982+0000 7f4fe3702700 1 -- 172.21.15.188:0/3666121976 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc0005320 con 0x7f4fdc0fefd0 2023-11-16T18:17:11.997 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.982+0000 7f4fd27fc700 1 -- 172.21.15.188:0/3666121976 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f4fcc00b960 con 0x7f4fdc0fefd0 2023-11-16T18:17:11.998 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.982+0000 7f4fd27fc700 1 --2- 172.21.15.188:0/3666121976 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4fc8038850 0x7f4fc803ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:11.998 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.982+0000 7f4fd27fc700 1 -- 172.21.15.188:0/3666121976 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4fcc04c930 con 0x7f4fdc0fefd0 2023-11-16T18:17:11.998 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.982+0000 7f4fe0c9d700 1 --2- 172.21.15.188:0/3666121976 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4fc8038850 0x7f4fc803ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:11.998 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.983+0000 7f4fe0c9d700 1 --2- 172.21.15.188:0/3666121976 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4fc8038850 0x7f4fc803ad00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8006fd0 tx=0x7f4fd8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:11.998 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:11.987+0000 7f4fd27fc700 1 -- 172.21.15.188:0/3666121976 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4fcc02a540 con 0x7f4fdc0fefd0 2023-11-16T18:17:12.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:11 smithi002 ceph-mon[98807]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:17:12.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:11 smithi002 ceph-mon[98807]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:17:12.183 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:12.181+0000 7f4fe3702700 1 -- 172.21.15.188:0/3666121976 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi040:172.21.15.40=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fc0000ca0 con 0x7f4fc8038850 2023-11-16T18:17:12.190 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:12.189+0000 7f4fd27fc700 1 -- 172.21.15.188:0/3666121976 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4fc0000ca0 con 0x7f4fc8038850 2023-11-16T18:17:12.191 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mon update... 2023-11-16T18:17:12.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:12.190+0000 7f4fe3702700 1 -- 172.21.15.188:0/3666121976 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4fc8038850 msgr2=0x7f4fc803ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:12.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:12.190+0000 7f4fe3702700 1 --2- 172.21.15.188:0/3666121976 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4fc8038850 0x7f4fc803ad00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8006fd0 tx=0x7f4fd8006e40 comp rx=0 tx=0).stop 2023-11-16T18:17:12.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:12.190+0000 7f4fe3702700 1 -- 172.21.15.188:0/3666121976 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fdc0fefd0 msgr2=0x7f4fdc0fbe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:12.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:12.190+0000 7f4fe3702700 1 --2- 172.21.15.188:0/3666121976 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fdc0fefd0 0x7f4fdc0fbe20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc004f40 tx=0x7f4fcc005020 comp rx=0 tx=0).stop 2023-11-16T18:17:12.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:12.190+0000 7f4fe3702700 1 -- 172.21.15.188:0/3666121976 shutdown_connections 2023-11-16T18:17:12.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:12.190+0000 7f4fe3702700 1 --2- 172.21.15.188:0/3666121976 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4fc8038850 0x7f4fc803ad00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:12.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:12.190+0000 7f4fe3702700 1 --2- 172.21.15.188:0/3666121976 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fdc0fefd0 0x7f4fdc0fbe20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:12.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:12.190+0000 7f4fe3702700 1 -- 172.21.15.188:0/3666121976 >> 172.21.15.188:0/3666121976 conn(0x7f4fdc074350 msgr2=0x7f4fdc101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:12.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:12.190+0000 7f4fe3702700 1 -- 172.21.15.188:0/3666121976 shutdown_connections 2023-11-16T18:17:12.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:12.190+0000 7f4fe3702700 1 -- 172.21.15.188:0/3666121976 wait complete. 2023-11-16T18:17:12.656 DEBUG:teuthology.orchestra.run.smithi040:mon.b> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.b.service 2023-11-16T18:17:12.665 DEBUG:teuthology.orchestra.run.smithi188:mon.c> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.c.service 2023-11-16T18:17:12.670 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-16T18:17:12.676 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T18:17:12.697 INFO:journalctl@ceph.mon.b.smithi040.stdout:-- Logs begin at Thu 2023-11-16 17:59:26 UTC. -- 2023-11-16T18:17:12.697 INFO:journalctl@ceph.mon.c.smithi188.stdout:-- Logs begin at Thu 2023-11-16 17:59:32 UTC. -- 2023-11-16T18:17:13.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:12 smithi002 ceph-mon[98807]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:13.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:13.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:13.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:13.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:13.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:13.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:13.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:13.703 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:14.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:13 smithi002 ceph-mon[98807]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi040:172.21.15.40=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:17:14.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:13 smithi002 ceph-mon[98807]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi040:172.21.15.40=b;smithi188:172.21.15.188=c;count:3 2023-11-16T18:17:14.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:13 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:14.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:13 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:14.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:13 smithi002 ceph-mon[98807]: Deploying daemon mon.c on smithi188 2023-11-16T18:17:15.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:14 smithi002 ceph-mon[98807]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:15.212 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.208+0000 7fd00336c700 1 -- 172.21.15.188:0/1501438848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcffc0fefd0 msgr2=0x7fcffc0ff3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:15.212 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.208+0000 7fd00336c700 1 --2- 172.21.15.188:0/1501438848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcffc0fefd0 0x7fcffc0ff3a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcff800aa20 tx=0x7fcff800ad30 comp rx=0 tx=0).stop 2023-11-16T18:17:15.212 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.208+0000 7fd00336c700 1 -- 172.21.15.188:0/1501438848 shutdown_connections 2023-11-16T18:17:15.212 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.208+0000 7fd00336c700 1 --2- 172.21.15.188:0/1501438848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcffc0fefd0 0x7fcffc0ff3a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:15.212 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.208+0000 7fd00336c700 1 -- 172.21.15.188:0/1501438848 >> 172.21.15.188:0/1501438848 conn(0x7fcffc074350 msgr2=0x7fcffc074750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:15.212 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.208+0000 7fd00336c700 1 -- 172.21.15.188:0/1501438848 shutdown_connections 2023-11-16T18:17:15.212 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.208+0000 7fd00336c700 1 -- 172.21.15.188:0/1501438848 wait complete. 2023-11-16T18:17:15.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.209+0000 7fd00336c700 1 Processor -- start 2023-11-16T18:17:15.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.209+0000 7fd00336c700 1 -- start start 2023-11-16T18:17:15.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.209+0000 7fd00336c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcffc0fefd0 0x7fcffc192f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:15.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.209+0000 7fd00336c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc1934b0 con 0x7fcffc0fefd0 2023-11-16T18:17:15.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.209+0000 7fd001108700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcffc0fefd0 0x7fcffc192f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:15.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.209+0000 7fd001108700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcffc0fefd0 0x7fcffc192f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.188:55258/0 (socket says 172.21.15.188:55258) 2023-11-16T18:17:15.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.209+0000 7fd001108700 1 -- 172.21.15.188:0/402698231 learned_addr learned my addr 172.21.15.188:0/402698231 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-16T18:17:15.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.210+0000 7fd001108700 1 -- 172.21.15.188:0/402698231 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff800a700 con 0x7fcffc0fefd0 2023-11-16T18:17:15.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.210+0000 7fd001108700 1 --2- 172.21.15.188:0/402698231 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcffc0fefd0 0x7fcffc192f70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcff8004c30 tx=0x7fcff8004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:15.214 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.210+0000 7fcff27fc700 1 -- 172.21.15.188:0/402698231 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcff80054b0 con 0x7fcffc0fefd0 2023-11-16T18:17:15.214 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.210+0000 7fd00336c700 1 -- 172.21.15.188:0/402698231 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcffc108010 con 0x7fcffc0fefd0 2023-11-16T18:17:15.214 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.210+0000 7fcff27fc700 1 -- 172.21.15.188:0/402698231 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcff8005610 con 0x7fcffc0fefd0 2023-11-16T18:17:15.214 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.210+0000 7fcff27fc700 1 -- 172.21.15.188:0/402698231 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcff800e700 con 0x7fcffc0fefd0 2023-11-16T18:17:15.214 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.210+0000 7fd00336c700 1 -- 172.21.15.188:0/402698231 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcffc107e80 con 0x7fcffc0fefd0 2023-11-16T18:17:15.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.211+0000 7fcff27fc700 1 -- 172.21.15.188:0/402698231 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fcff800e860 con 0x7fcffc0fefd0 2023-11-16T18:17:15.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.211+0000 7fd00336c700 1 -- 172.21.15.188:0/402698231 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcffc1081a0 con 0x7fcffc0fefd0 2023-11-16T18:17:15.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.211+0000 7fcff27fc700 1 --2- 172.21.15.188:0/402698231 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcfe8038870 0x7fcfe803ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:15.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.211+0000 7fcff27fc700 1 -- 172.21.15.188:0/402698231 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcff8012070 con 0x7fcffc0fefd0 2023-11-16T18:17:15.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.211+0000 7fd000907700 1 --2- 172.21.15.188:0/402698231 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcfe8038870 0x7fcfe803ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:15.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.212+0000 7fd000907700 1 --2- 172.21.15.188:0/402698231 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcfe8038870 0x7fcfe803ad20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcfec006fd0 tx=0x7fcfec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:15.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.217+0000 7fcff27fc700 1 -- 172.21.15.188:0/402698231 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcff8029d30 con 0x7fcffc0fefd0 2023-11-16T18:17:15.512 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.510+0000 7fd00336c700 1 -- 172.21.15.188:0/402698231 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcffc193b10 con 0x7fcffc0fefd0 2023-11-16T18:17:15.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.511+0000 7fcff27fc700 1 -- 172.21.15.188:0/402698231 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+732 (secure 0 0 0) 0x7fcff8026020 con 0x7fcffc0fefd0 2023-11-16T18:17:15.513 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:17:15.513 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T18:15:39.352712Z","created":"2023-11-16T18:15:39.352712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T18:17:15.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.513+0000 7fd00336c700 1 -- 172.21.15.188:0/402698231 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcfe8038870 msgr2=0x7fcfe803ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:15.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.513+0000 7fd00336c700 1 --2- 172.21.15.188:0/402698231 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcfe8038870 0x7fcfe803ad20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcfec006fd0 tx=0x7fcfec006e40 comp rx=0 tx=0).stop 2023-11-16T18:17:15.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.513+0000 7fd00336c700 1 -- 172.21.15.188:0/402698231 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcffc0fefd0 msgr2=0x7fcffc192f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:15.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.513+0000 7fd00336c700 1 --2- 172.21.15.188:0/402698231 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcffc0fefd0 0x7fcffc192f70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcff8004c30 tx=0x7fcff8004ee0 comp rx=0 tx=0).stop 2023-11-16T18:17:15.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.513+0000 7fd00336c700 1 -- 172.21.15.188:0/402698231 shutdown_connections 2023-11-16T18:17:15.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.513+0000 7fd00336c700 1 --2- 172.21.15.188:0/402698231 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcfe8038870 0x7fcfe803ad20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:15.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.513+0000 7fd00336c700 1 --2- 172.21.15.188:0/402698231 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcffc0fefd0 0x7fcffc192f70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:15.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.513+0000 7fd00336c700 1 -- 172.21.15.188:0/402698231 >> 172.21.15.188:0/402698231 conn(0x7fcffc074350 msgr2=0x7fcffc0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:15.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.513+0000 7fd00336c700 1 -- 172.21.15.188:0/402698231 shutdown_connections 2023-11-16T18:17:15.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:15.513+0000 7fd00336c700 1 -- 172.21.15.188:0/402698231 wait complete. 2023-11-16T18:17:15.517 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-16T18:17:15.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T18:17:15.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T18:17:15.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T18:17:15.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-16T18:17:15.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T18:17:15.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-16T18:17:15.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-16T18:17:15.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T18:17:15.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compression_opts.level: 32767 2023-11-16T18:17:15.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compression_opts.strategy: 0 2023-11-16T18:17:15.643 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T18:17:15.643 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T18:17:15.643 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-16T18:17:15.646 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-16T18:17:15.646 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compression_opts.enabled: false 2023-11-16T18:17:15.646 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-16T18:17:15.647 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T18:17:15.647 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T18:17:15.647 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T18:17:15.647 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.target_file_size_base: 67108864 2023-11-16T18:17:15.648 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T18:17:15.648 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T18:17:15.648 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T18:17:15.649 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T18:17:15.649 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T18:17:15.649 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T18:17:15.649 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T18:17:15.649 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T18:17:15.650 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T18:17:15.650 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T18:17:15.650 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T18:17:15.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T18:17:16.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T18:17:16.054 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-16T18:17:16.054 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.arena_block_size: 1048576 2023-11-16T18:17:16.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T18:17:16.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T18:17:16.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.disable_auto_compactions: 0 2023-11-16T18:17:16.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T18:17:16.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T18:17:16.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T18:17:16.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T18:17:16.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T18:17:16.057 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T18:17:16.057 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T18:17:16.057 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T18:17:16.057 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T18:17:16.058 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T18:17:16.058 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.table_properties_collectors: 2023-11-16T18:17:16.058 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.inplace_update_support: 0 2023-11-16T18:17:16.058 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T18:17:16.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T18:17:16.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T18:17:16.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T18:17:16.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.bloom_locality: 0 2023-11-16T18:17:16.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.max_successive_merges: 0 2023-11-16T18:17:16.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T18:17:16.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.paranoid_file_checks: 0 2023-11-16T18:17:16.061 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.force_consistency_checks: 1 2023-11-16T18:17:16.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.report_bg_io_stats: 0 2023-11-16T18:17:16.070 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.ttl: 2592000 2023-11-16T18:17:16.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T18:17:16.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-16T18:17:16.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-16T18:17:16.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.enable_blob_files: false 2023-11-16T18:17:16.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.min_blob_size: 0 2023-11-16T18:17:16.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.blob_file_size: 268435456 2023-11-16T18:17:16.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-16T18:17:16.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-16T18:17:16.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-16T18:17:16.076 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-16T18:17:16.076 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-16T18:17:16.076 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.blob_file_starting_level: 0 2023-11-16T18:17:16.076 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-16T18:17:16.077 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-16T18:17:16.077 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-11-16T18:17:16.077 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 44c5a2d3-f93e-4320-a5eb-019f899a534e 2023-11-16T18:17:16.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700158635635552, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-11-16T18:17:16.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-11-16T18:17:16.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700158635635954, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1700158635, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "44c5a2d3-f93e-4320-a5eb-019f899a534e", "db_session_id": "61324F4145P37IPSWFZZ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-11-16T18:17:16.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700158635636020, "job": 1, "event": "recovery_finished"} 2023-11-16T18:17:16.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-11-16T18:17:16.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-16T18:17:16.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5625108b0000 2023-11-16T18:17:16.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: DB pointer 0x56251089c000 2023-11-16T18:17:16.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-16T18:17:16.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-16T18:17:16.081 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** DB Stats ** 2023-11-16T18:17:16.081 INFO:journalctl@ceph.mon.c.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T18:17:16.081 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T18:17:16.082 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T18:17:16.082 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T18:17:16.082 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T18:17:16.085 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T18:17:16.086 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T18:17:16.086 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2023-11-16T18:17:16.086 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** Compaction Stats [default] ** 2023-11-16T18:17:16.086 INFO:journalctl@ceph.mon.c.smithi188.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T18:17:16.087 INFO:journalctl@ceph.mon.c.smithi188.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-16T18:17:16.087 INFO:journalctl@ceph.mon.c.smithi188.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T18:17:16.087 INFO:journalctl@ceph.mon.c.smithi188.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T18:17:16.090 INFO:journalctl@ceph.mon.c.smithi188.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T18:17:16.090 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2023-11-16T18:17:16.092 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** Compaction Stats [default] ** 2023-11-16T18:17:16.098 INFO:journalctl@ceph.mon.c.smithi188.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T18:17:16.109 INFO:journalctl@ceph.mon.c.smithi188.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T18:17:16.111 INFO:journalctl@ceph.mon.c.smithi188.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T18:17:16.121 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2023-11-16T18:17:16.123 INFO:journalctl@ceph.mon.c.smithi188.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-16T18:17:16.126 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2023-11-16T18:17:16.129 INFO:journalctl@ceph.mon.c.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T18:17:16.129 INFO:journalctl@ceph.mon.c.smithi188.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T18:17:16.129 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T18:17:16.129 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T18:17:16.131 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T18:17:16.132 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-16T18:17:16.133 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T18:17:16.134 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T18:17:16.134 INFO:journalctl@ceph.mon.c.smithi188.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T18:17:16.138 INFO:journalctl@ceph.mon.c.smithi188.stdout: Block cache BinnedLRUCache@0x5625108131f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-11-16T18:17:16.142 INFO:journalctl@ceph.mon.c.smithi188.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-11-16T18:17:16.142 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2023-11-16T18:17:16.142 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-16T18:17:16.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-11-16T18:17:16.143 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: using public_addr v2:172.21.15.188:0/0 -> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2023-11-16T18:17:16.143 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: starting mon.c rank -1 at public addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] at bind addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:17:16.143 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(???) e0 preinit fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:17:16.143 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing).mds e1 new map 2023-11-16T18:17:16.144 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing).mds e1 print_map 2023-11-16T18:17:16.144 INFO:journalctl@ceph.mon.c.smithi188.stdout: e1 2023-11-16T18:17:16.144 INFO:journalctl@ceph.mon.c.smithi188.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T18:17:16.144 INFO:journalctl@ceph.mon.c.smithi188.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T18:17:16.145 INFO:journalctl@ceph.mon.c.smithi188.stdout: legacy client fscid: -1 2023-11-16T18:17:16.148 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2023-11-16T18:17:16.148 INFO:journalctl@ceph.mon.c.smithi188.stdout: No filesystems configured 2023-11-16T18:17:16.158 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-11-16T18:17:16.159 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-11-16T18:17:16.160 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-11-16T18:17:16.160 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-11-16T18:17:16.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-11-16T18:17:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-11-16T18:17:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T18:17:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T18:17:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T18:17:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T18:17:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: Updating smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:17:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: Added host smithi188 2023-11-16T18:17:16.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:16.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:17:16.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:16.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3929959364' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T18:17:16.177 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.178 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.178 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.184 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:17:16.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:16.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:16.188 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:17:16.188 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: Updating smithi188:/etc/ceph/ceph.conf 2023-11-16T18:17:16.196 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:16.197 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:16.207 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3929959364' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T18:17:16.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T18:17:16.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:17:16.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:17:16.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:16.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:16.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:16.216 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:16.217 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:16.217 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:16.218 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:16.218 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi040:172.21.15.40=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:17:16.218 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi040:172.21.15.40=b;smithi188:172.21.15.188=c;count:3 2023-11-16T18:17:16.219 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:17:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:17:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: Deploying daemon mon.c on smithi188 2023-11-16T18:17:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-11-16T18:17:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-11-16T18:17:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: expand_channel_meta expand map: {default=false} 2023-11-16T18:17:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: expand_channel_meta from 'false' to 'false' 2023-11-16T18:17:16.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: expand_channel_meta expanded map: {default=false} 2023-11-16T18:17:16.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: expand_channel_meta expand map: {default=info} 2023-11-16T18:17:16.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: expand_channel_meta from 'info' to 'info' 2023-11-16T18:17:16.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: expand_channel_meta expanded map: {default=info} 2023-11-16T18:17:16.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: expand_channel_meta expand map: {default=daemon} 2023-11-16T18:17:16.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: expand_channel_meta from 'daemon' to 'daemon' 2023-11-16T18:17:16.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: expand_channel_meta expanded map: {default=daemon} 2023-11-16T18:17:16.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: expand_channel_meta expand map: {default=debug} 2023-11-16T18:17:16.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: expand_channel_meta from 'debug' to 'debug' 2023-11-16T18:17:16.231 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 ceph-mon[100639]: expand_channel_meta expanded map: {default=debug} 2023-11-16T18:17:16.231 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:15 smithi188 systemd[1]: Started Ceph mon.c for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:17:17.234 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-16T18:17:17.235 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T18:17:17.885 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:17:21.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:17:21.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: mon.a calling monitor election 2023-11-16T18:17:21.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:21.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:21.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:21.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:21.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: mon.c calling monitor election 2023-11-16T18:17:21.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:21.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:21.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:21.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:21.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:21.053 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T18:17:21.053 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-11-16T18:17:21.053 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: fsmap 2023-11-16T18:17:21.053 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T18:17:21.053 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: mgrmap e13: x(active, since 42s) 2023-11-16T18:17:21.053 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: overall HEALTH_OK 2023-11-16T18:17:21.053 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:21.053 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:21.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:17:21.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: mon.a calling monitor election 2023-11-16T18:17:21.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:21.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:21.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:21.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:21.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: mon.c calling monitor election 2023-11-16T18:17:21.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:21.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:21.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:21.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:21.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:21.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T18:17:21.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-11-16T18:17:21.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: fsmap 2023-11-16T18:17:21.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T18:17:21.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: mgrmap e13: x(active, since 42s) 2023-11-16T18:17:21.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: overall HEALTH_OK 2023-11-16T18:17:21.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:21.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:22.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:21 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:22.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:21 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:22.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:21 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:22.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:21 smithi188 ceph-mon[100639]: Deploying daemon mon.b on smithi040 2023-11-16T18:17:22.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:21 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:22.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:21 smithi188 ceph-mon[100639]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:22.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:21 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:22.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:21 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:22.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:21 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:22.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:21 smithi002 ceph-mon[98807]: Deploying daemon mon.b on smithi040 2023-11-16T18:17:22.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:21 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:22.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:21 smithi002 ceph-mon[98807]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:22.112 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.111+0000 7fc95f68a700 1 -- 172.21.15.188:0/1410774552 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9440057d0 msgr2=0x7fc944007c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:22.112 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.111+0000 7fc95f68a700 1 --2- 172.21.15.188:0/1410774552 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9440057d0 0x7fc944007c50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc9580fe4c0 tx=0x7fc954010ec0 comp rx=0 tx=0).stop 2023-11-16T18:17:22.112 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.111+0000 7fc95f68a700 1 -- 172.21.15.188:0/1410774552 shutdown_connections 2023-11-16T18:17:22.112 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.111+0000 7fc95f68a700 1 --2- 172.21.15.188:0/1410774552 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9440057d0 0x7fc944007c50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:22.113 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.111+0000 7fc95f68a700 1 -- 172.21.15.188:0/1410774552 >> 172.21.15.188:0/1410774552 conn(0x7fc9580742a0 msgr2=0x7fc9580746a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:22.113 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.111+0000 7fc95f68a700 1 -- 172.21.15.188:0/1410774552 shutdown_connections 2023-11-16T18:17:22.113 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.111+0000 7fc95f68a700 1 -- 172.21.15.188:0/1410774552 wait complete. 2023-11-16T18:17:22.115 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.112+0000 7fc95f68a700 1 Processor -- start 2023-11-16T18:17:22.115 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.112+0000 7fc95f68a700 1 -- start start 2023-11-16T18:17:22.115 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95f68a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9440057d0 0x7fc9581973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:22.115 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95f68a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc958197930 0x7fc95819bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:22.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95f68a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc958197eb0 con 0x7fc958197930 2023-11-16T18:17:22.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95f68a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc958198020 con 0x7fc9440057d0 2023-11-16T18:17:22.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95d426700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9440057d0 0x7fc9581973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:22.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95d426700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9440057d0 0x7fc9581973f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:48000/0 (socket says 172.21.15.188:48000) 2023-11-16T18:17:22.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95d426700 1 -- 172.21.15.188:0/756903033 learned_addr learned my addr 172.21.15.188:0/756903033 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-16T18:17:22.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95d426700 1 -- 172.21.15.188:0/756903033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9440057d0 msgr2=0x7fc9581973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-11-16T18:17:22.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95d426700 1 -- 172.21.15.188:0/756903033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9440057d0 msgr2=0x7fc9581973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-16T18:17:22.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95d426700 1 --2- 172.21.15.188:0/756903033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9440057d0 0x7fc9581973f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-16T18:17:22.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95d426700 1 --2- 172.21.15.188:0/756903033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9440057d0 0x7fc9581973f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T18:17:22.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95cc25700 1 --2- 172.21.15.188:0/756903033 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc958197930 0x7fc95819bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:22.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95cc25700 1 -- 172.21.15.188:0/756903033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9440057d0 msgr2=0x7fc9581973f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T18:17:22.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95cc25700 1 --2- 172.21.15.188:0/756903033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9440057d0 0x7fc9581973f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:22.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.113+0000 7fc95cc25700 1 -- 172.21.15.188:0/756903033 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc954010910 con 0x7fc958197930 2023-11-16T18:17:22.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.114+0000 7fc95cc25700 1 --2- 172.21.15.188:0/756903033 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc958197930 0x7fc95819bda0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc94800b4d0 tx=0x7fc94800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:22.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.114+0000 7fc94e7fc700 1 -- 172.21.15.188:0/756903033 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fc948011630 con 0x7fc958197930 2023-11-16T18:17:22.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.114+0000 7fc94e7fc700 1 -- 172.21.15.188:0/756903033 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc948011c50 con 0x7fc958197930 2023-11-16T18:17:22.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.114+0000 7fc95f68a700 1 -- 172.21.15.188:0/756903033 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc958105ba0 con 0x7fc958197930 2023-11-16T18:17:22.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.114+0000 7fc94e7fc700 1 -- 172.21.15.188:0/756903033 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fc948010d90 con 0x7fc958197930 2023-11-16T18:17:22.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.114+0000 7fc95f68a700 1 -- 172.21.15.188:0/756903033 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc958105a10 con 0x7fc958197930 2023-11-16T18:17:22.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.115+0000 7fc94e7fc700 1 -- 172.21.15.188:0/756903033 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fc94801a400 con 0x7fc958197930 2023-11-16T18:17:22.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.115+0000 7fc94e7fc700 1 --2- 172.21.15.188:0/756903033 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc93c0386b0 0x7fc93c03ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:22.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.115+0000 7fc94e7fc700 1 -- 172.21.15.188:0/756903033 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc94804ae40 con 0x7fc958197930 2023-11-16T18:17:22.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.115+0000 7fc95f68a700 1 -- 172.21.15.188:0/756903033 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc958105d30 con 0x7fc958197930 2023-11-16T18:17:22.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.115+0000 7fc95d426700 1 --2- 172.21.15.188:0/756903033 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc93c0386b0 0x7fc93c03ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:22.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.116+0000 7fc95d426700 1 --2- 172.21.15.188:0/756903033 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc93c0386b0 0x7fc93c03ab60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc9540108a0 tx=0x7fc954010830 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:22.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.121+0000 7fc94e7fc700 1 -- 172.21.15.188:0/756903033 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc9480193b0 con 0x7fc958197930 2023-11-16T18:17:22.387 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.386+0000 7fc95f68a700 1 -- 172.21.15.188:0/756903033 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc958198890 con 0x7fc958197930 2023-11-16T18:17:22.387 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.386+0000 7fc94e7fc700 1 -- 172.21.15.188:0/756903033 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1002 (secure 0 0 0) 0x7fc9480172c0 con 0x7fc958197930 2023-11-16T18:17:22.387 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:17:22.388 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":2,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T18:17:15.663363Z","created":"2023-11-16T18:15:39.352712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-11-16T18:17:22.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.388+0000 7fc95f68a700 1 -- 172.21.15.188:0/756903033 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc93c0386b0 msgr2=0x7fc93c03ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:22.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.388+0000 7fc95f68a700 1 --2- 172.21.15.188:0/756903033 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc93c0386b0 0x7fc93c03ab60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc9540108a0 tx=0x7fc954010830 comp rx=0 tx=0).stop 2023-11-16T18:17:22.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.388+0000 7fc95f68a700 1 -- 172.21.15.188:0/756903033 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc958197930 msgr2=0x7fc95819bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:22.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.388+0000 7fc95f68a700 1 --2- 172.21.15.188:0/756903033 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc958197930 0x7fc95819bda0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc94800b4d0 tx=0x7fc94800b890 comp rx=0 tx=0).stop 2023-11-16T18:17:22.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.388+0000 7fc95f68a700 1 -- 172.21.15.188:0/756903033 shutdown_connections 2023-11-16T18:17:22.393 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.388+0000 7fc95f68a700 1 --2- 172.21.15.188:0/756903033 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc93c0386b0 0x7fc93c03ab60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:22.393 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.388+0000 7fc95f68a700 1 --2- 172.21.15.188:0/756903033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9440057d0 0x7fc9581973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:22.393 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.388+0000 7fc95f68a700 1 --2- 172.21.15.188:0/756903033 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc958197930 0x7fc95819bda0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:22.393 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.388+0000 7fc95f68a700 1 -- 172.21.15.188:0/756903033 >> 172.21.15.188:0/756903033 conn(0x7fc9580742a0 msgr2=0x7fc958102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:22.393 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.389+0000 7fc95f68a700 1 -- 172.21.15.188:0/756903033 shutdown_connections 2023-11-16T18:17:22.393 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:22.389+0000 7fc95f68a700 1 -- 172.21.15.188:0/756903033 wait complete. 2023-11-16T18:17:22.393 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 2 2023-11-16T18:17:23.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:22 smithi188 ceph-mon[100639]: from='client.? 172.21.15.188:0/756903033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T18:17:23.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:22 smithi002 ceph-mon[98807]: from='client.? 172.21.15.188:0/756903033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T18:17:23.678 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 systemd[1]: Starting Ceph mon.b for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:17:23.834 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-16T18:17:23.835 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T18:17:23.968 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 podman[100211]: 2023-11-16T18:17:23.970 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 bash[100211]: aa5c1646976771c6591b5d59e57a61aec772ba2680046a09772c98a5aa62f874 2023-11-16T18:17:23.971 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: set uid:gid to 167:167 (ceph:ceph) 2023-11-16T18:17:23.972 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev), process ceph-mon, pid 7 2023-11-16T18:17:23.972 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: pidfile_write: ignore empty --pid-file 2023-11-16T18:17:23.973 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: load: jerasure load: lrc 2023-11-16T18:17:23.976 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: RocksDB version: 7.9.2 2023-11-16T18:17:23.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Git sha 0 2023-11-16T18:17:23.982 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Compile date 2023-11-15 23:53:54 2023-11-16T18:17:23.982 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: DB SUMMARY 2023-11-16T18:17:23.982 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: DB Session ID: 194CSAA4MZ3L9WMKXAUT 2023-11-16T18:17:23.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: CURRENT file: CURRENT 2023-11-16T18:17:23.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: IDENTITY file: IDENTITY 2023-11-16T18:17:23.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-11-16T18:17:23.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-11-16T18:17:23.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-11-16T18:17:23.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.error_if_exists: 0 2023-11-16T18:17:23.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.create_if_missing: 0 2023-11-16T18:17:23.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.paranoid_checks: 1 2023-11-16T18:17:23.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.flush_verify_memtable_count: 1 2023-11-16T18:17:23.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-16T18:17:23.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-11-16T18:17:23.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.env: 0x55eac9d99720 2023-11-16T18:17:23.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.fs: PosixFileSystem 2023-11-16T18:17:23.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.info_log: 0x55eacc9333c0 2023-11-16T18:17:23.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_file_opening_threads: 16 2023-11-16T18:17:23.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.statistics: (nil) 2023-11-16T18:17:23.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.use_fsync: 0 2023-11-16T18:17:23.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_log_file_size: 0 2023-11-16T18:17:23.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T18:17:23.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T18:17:23.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.keep_log_file_num: 1000 2023-11-16T18:17:23.988 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.recycle_log_file_num: 0 2023-11-16T18:17:23.988 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.allow_fallocate: 1 2023-11-16T18:17:23.988 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.allow_mmap_reads: 0 2023-11-16T18:17:23.988 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.allow_mmap_writes: 0 2023-11-16T18:17:23.989 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.use_direct_reads: 0 2023-11-16T18:17:23.989 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T18:17:23.989 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.create_missing_column_families: 0 2023-11-16T18:17:23.990 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.db_log_dir: 2023-11-16T18:17:23.990 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.wal_dir: 2023-11-16T18:17:23.990 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T18:17:23.990 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T18:17:23.991 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T18:17:23.991 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T18:17:23.991 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T18:17:23.991 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T18:17:23.993 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.advise_random_on_open: 1 2023-11-16T18:17:23.993 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.db_write_buffer_size: 0 2023-11-16T18:17:23.993 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.write_buffer_manager: 0x55eacbba43c0 2023-11-16T18:17:23.993 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T18:17:23.994 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T18:17:23.994 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T18:17:23.994 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.rate_limiter: (nil) 2023-11-16T18:17:23.994 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T18:17:23.995 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.wal_recovery_mode: 2 2023-11-16T18:17:23.995 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.enable_thread_tracking: 0 2023-11-16T18:17:23.995 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.enable_pipelined_write: 0 2023-11-16T18:17:23.995 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.unordered_write: 0 2023-11-16T18:17:23.996 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T18:17:23.996 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T18:17:23.996 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T18:17:23.996 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T18:17:23.997 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.row_cache: None 2023-11-16T18:17:23.997 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.wal_filter: None 2023-11-16T18:17:23.998 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T18:17:24.002 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.allow_ingest_behind: 0 2023-11-16T18:17:24.002 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.two_write_queues: 0 2023-11-16T18:17:24.002 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.manual_wal_flush: 0 2023-11-16T18:17:24.003 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.wal_compression: 0 2023-11-16T18:17:24.003 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.atomic_flush: 0 2023-11-16T18:17:24.004 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T18:17:24.004 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T18:17:24.008 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T18:17:24.008 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.log_readahead_size: 0 2023-11-16T18:17:24.010 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-16T18:17:24.010 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.best_efforts_recovery: 0 2023-11-16T18:17:24.011 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-16T18:17:24.011 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-16T18:17:24.012 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.allow_data_in_errors: 0 2023-11-16T18:17:24.013 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.db_host_id: __hostname__ 2023-11-16T18:17:24.013 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.enforce_single_del_contracts: true 2023-11-16T18:17:24.013 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_background_jobs: 2 2023-11-16T18:17:24.014 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_background_compactions: -1 2023-11-16T18:17:24.014 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_subcompactions: 1 2023-11-16T18:17:24.014 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T18:17:24.014 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T18:17:24.015 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T18:17:24.016 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_total_wal_size: 0 2023-11-16T18:17:24.017 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T18:17:24.018 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T18:17:24.048 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T18:17:24.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T18:17:24.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_open_files: -1 2023-11-16T18:17:24.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.bytes_per_sync: 0 2023-11-16T18:17:24.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T18:17:24.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T18:17:24.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compaction_readahead_size: 0 2023-11-16T18:17:24.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_background_flushes: -1 2023-11-16T18:17:24.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Compression algorithms supported: 2023-11-16T18:17:24.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T18:17:24.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: kZSTD supported: 0 2023-11-16T18:17:24.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: kXpressCompression supported: 0 2023-11-16T18:17:24.055 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: kLZ4HCCompression supported: 1 2023-11-16T18:17:24.055 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: kZlibCompression supported: 1 2023-11-16T18:17:24.055 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: kSnappyCompression supported: 1 2023-11-16T18:17:24.106 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: kLZ4Compression supported: 1 2023-11-16T18:17:24.130 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: kBZip2Compression supported: 0 2023-11-16T18:17:24.130 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T18:17:24.130 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: DMutex implementation: pthread_mutex_t 2023-11-16T18:17:24.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-11-16T18:17:24.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-11-16T18:17:24.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T18:17:24.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.merge_operator: 2023-11-16T18:17:24.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compaction_filter: None 2023-11-16T18:17:24.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compaction_filter_factory: None 2023-11-16T18:17:24.131 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.sst_partitioner_factory: None 2023-11-16T18:17:24.132 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T18:17:24.132 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T18:17:24.132 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eacc9334e0) 2023-11-16T18:17:24.132 INFO:journalctl@ceph.mon.b.smithi040.stdout: cache_index_and_filter_blocks: 1 2023-11-16T18:17:24.132 INFO:journalctl@ceph.mon.b.smithi040.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T18:17:24.132 INFO:journalctl@ceph.mon.b.smithi040.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T18:17:24.132 INFO:journalctl@ceph.mon.b.smithi040.stdout: pin_top_level_index_and_filter: 1 2023-11-16T18:17:24.132 INFO:journalctl@ceph.mon.b.smithi040.stdout: index_type: 0 2023-11-16T18:17:24.132 INFO:journalctl@ceph.mon.b.smithi040.stdout: data_block_index_type: 0 2023-11-16T18:17:24.132 INFO:journalctl@ceph.mon.b.smithi040.stdout: index_shortening: 1 2023-11-16T18:17:24.133 INFO:journalctl@ceph.mon.b.smithi040.stdout: data_block_hash_table_util_ratio: 0.750000 2023-11-16T18:17:24.133 INFO:journalctl@ceph.mon.b.smithi040.stdout: checksum: 4 2023-11-16T18:17:24.133 INFO:journalctl@ceph.mon.b.smithi040.stdout: no_block_cache: 0 2023-11-16T18:17:24.133 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_cache: 0x55eacbc45350 2023-11-16T18:17:24.133 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_cache_name: BinnedLRUCache 2023-11-16T18:17:24.133 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_cache_options: 2023-11-16T18:17:24.133 INFO:journalctl@ceph.mon.b.smithi040.stdout: capacity : 536870912 2023-11-16T18:17:24.133 INFO:journalctl@ceph.mon.b.smithi040.stdout: num_shard_bits : 4 2023-11-16T18:17:24.133 INFO:journalctl@ceph.mon.b.smithi040.stdout: strict_capacity_limit : 0 2023-11-16T18:17:24.133 INFO:journalctl@ceph.mon.b.smithi040.stdout: high_pri_pool_ratio: 0.000 2023-11-16T18:17:24.134 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_cache_compressed: (nil) 2023-11-16T18:17:24.134 INFO:journalctl@ceph.mon.b.smithi040.stdout: persistent_cache: (nil) 2023-11-16T18:17:24.134 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_size: 4096 2023-11-16T18:17:24.157 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_size_deviation: 10 2023-11-16T18:17:24.157 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_restart_interval: 16 2023-11-16T18:17:24.157 INFO:journalctl@ceph.mon.b.smithi040.stdout: index_block_restart_interval: 1 2023-11-16T18:17:24.157 INFO:journalctl@ceph.mon.b.smithi040.stdout: metadata_block_size: 4096 2023-11-16T18:17:24.157 INFO:journalctl@ceph.mon.b.smithi040.stdout: partition_filters: 0 2023-11-16T18:17:24.157 INFO:journalctl@ceph.mon.b.smithi040.stdout: use_delta_encoding: 1 2023-11-16T18:17:24.157 INFO:journalctl@ceph.mon.b.smithi040.stdout: filter_policy: bloomfilter 2023-11-16T18:17:24.158 INFO:journalctl@ceph.mon.b.smithi040.stdout: whole_key_filtering: 1 2023-11-16T18:17:24.158 INFO:journalctl@ceph.mon.b.smithi040.stdout: verify_compression: 0 2023-11-16T18:17:24.158 INFO:journalctl@ceph.mon.b.smithi040.stdout: read_amp_bytes_per_bit: 0 2023-11-16T18:17:24.158 INFO:journalctl@ceph.mon.b.smithi040.stdout: format_version: 5 2023-11-16T18:17:24.158 INFO:journalctl@ceph.mon.b.smithi040.stdout: enable_index_compression: 1 2023-11-16T18:17:24.158 INFO:journalctl@ceph.mon.b.smithi040.stdout: block_align: 0 2023-11-16T18:17:24.158 INFO:journalctl@ceph.mon.b.smithi040.stdout: max_auto_readahead_size: 262144 2023-11-16T18:17:24.158 INFO:journalctl@ceph.mon.b.smithi040.stdout: prepopulate_block_cache: 0 2023-11-16T18:17:24.158 INFO:journalctl@ceph.mon.b.smithi040.stdout: initial_auto_readahead_size: 8192 2023-11-16T18:17:24.159 INFO:journalctl@ceph.mon.b.smithi040.stdout: num_file_reads_for_auto_readahead: 2 2023-11-16T18:17:24.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.write_buffer_size: 33554432 2023-11-16T18:17:24.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_write_buffer_number: 2 2023-11-16T18:17:24.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compression: NoCompression 2023-11-16T18:17:24.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.bottommost_compression: Disabled 2023-11-16T18:17:24.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.prefix_extractor: nullptr 2023-11-16T18:17:24.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T18:17:24.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.num_levels: 7 2023-11-16T18:17:24.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T18:17:24.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T18:17:24.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T18:17:24.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T18:17:24.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T18:17:24.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T18:17:24.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T18:17:24.175 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T18:17:24.176 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-16T18:17:24.176 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T18:17:24.176 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-16T18:17:24.176 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-16T18:17:24.176 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T18:17:24.176 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compression_opts.level: 32767 2023-11-16T18:17:24.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compression_opts.strategy: 0 2023-11-16T18:17:24.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T18:17:24.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T18:17:24.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-16T18:17:24.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-16T18:17:24.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compression_opts.enabled: false 2023-11-16T18:17:24.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-16T18:17:24.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T18:17:24.206 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T18:17:24.206 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T18:17:24.207 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.target_file_size_base: 67108864 2023-11-16T18:17:24.207 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T18:17:24.208 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T18:17:24.208 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T18:17:24.208 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T18:17:24.208 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T18:17:24.208 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T18:17:24.208 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T18:17:24.208 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T18:17:24.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T18:17:24.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T18:17:24.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T18:17:24.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T18:17:24.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T18:17:24.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-16T18:17:24.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.arena_block_size: 1048576 2023-11-16T18:17:24.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T18:17:24.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T18:17:24.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.disable_auto_compactions: 0 2023-11-16T18:17:24.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T18:17:24.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T18:17:24.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T18:17:24.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T18:17:24.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T18:17:24.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T18:17:24.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T18:17:24.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T18:17:24.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T18:17:24.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T18:17:24.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.table_properties_collectors: 2023-11-16T18:17:24.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.inplace_update_support: 0 2023-11-16T18:17:24.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T18:17:24.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T18:17:24.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T18:17:24.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T18:17:24.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.bloom_locality: 0 2023-11-16T18:17:24.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.max_successive_merges: 0 2023-11-16T18:17:24.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T18:17:24.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.paranoid_file_checks: 0 2023-11-16T18:17:24.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.force_consistency_checks: 1 2023-11-16T18:17:24.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.report_bg_io_stats: 0 2023-11-16T18:17:24.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.ttl: 2592000 2023-11-16T18:17:24.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T18:17:24.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-16T18:17:24.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-16T18:17:24.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.enable_blob_files: false 2023-11-16T18:17:24.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.min_blob_size: 0 2023-11-16T18:17:24.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.blob_file_size: 268435456 2023-11-16T18:17:24.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-16T18:17:24.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-16T18:17:24.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-16T18:17:24.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-16T18:17:24.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-16T18:17:24.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.blob_file_starting_level: 0 2023-11-16T18:17:24.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-16T18:17:24.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-16T18:17:24.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-11-16T18:17:24.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 05051e50-8bb5-4e14-bf30-f25d1a787108 2023-11-16T18:17:24.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700158643887548, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-11-16T18:17:24.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-11-16T18:17:24.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700158643887961, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1700158643, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "05051e50-8bb5-4e14-bf30-f25d1a787108", "db_session_id": "194CSAA4MZ3L9WMKXAUT", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-11-16T18:17:24.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700158643888032, "job": 1, "event": "recovery_finished"} 2023-11-16T18:17:24.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-11-16T18:17:24.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-16T18:17:24.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55eacbce2000 2023-11-16T18:17:24.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: DB pointer 0x55eacbcce000 2023-11-16T18:17:24.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-16T18:17:24.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-16T18:17:24.215 INFO:journalctl@ceph.mon.b.smithi040.stdout: ** DB Stats ** 2023-11-16T18:17:24.216 INFO:journalctl@ceph.mon.b.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T18:17:24.216 INFO:journalctl@ceph.mon.b.smithi040.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T18:17:24.216 INFO:journalctl@ceph.mon.b.smithi040.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T18:17:24.216 INFO:journalctl@ceph.mon.b.smithi040.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T18:17:24.216 INFO:journalctl@ceph.mon.b.smithi040.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T18:17:24.216 INFO:journalctl@ceph.mon.b.smithi040.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T18:17:24.216 INFO:journalctl@ceph.mon.b.smithi040.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T18:17:24.216 INFO:journalctl@ceph.mon.b.smithi040.stdout: 2023-11-16T18:17:24.216 INFO:journalctl@ceph.mon.b.smithi040.stdout: ** Compaction Stats [default] ** 2023-11-16T18:17:24.217 INFO:journalctl@ceph.mon.b.smithi040.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T18:17:24.217 INFO:journalctl@ceph.mon.b.smithi040.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-16T18:17:24.217 INFO:journalctl@ceph.mon.b.smithi040.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T18:17:24.217 INFO:journalctl@ceph.mon.b.smithi040.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T18:17:24.217 INFO:journalctl@ceph.mon.b.smithi040.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T18:17:24.217 INFO:journalctl@ceph.mon.b.smithi040.stdout: 2023-11-16T18:17:24.217 INFO:journalctl@ceph.mon.b.smithi040.stdout: ** Compaction Stats [default] ** 2023-11-16T18:17:24.217 INFO:journalctl@ceph.mon.b.smithi040.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T18:17:24.217 INFO:journalctl@ceph.mon.b.smithi040.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T18:17:24.217 INFO:journalctl@ceph.mon.b.smithi040.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T18:17:24.218 INFO:journalctl@ceph.mon.b.smithi040.stdout: 2023-11-16T18:17:24.218 INFO:journalctl@ceph.mon.b.smithi040.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-16T18:17:24.218 INFO:journalctl@ceph.mon.b.smithi040.stdout: 2023-11-16T18:17:24.218 INFO:journalctl@ceph.mon.b.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T18:17:24.218 INFO:journalctl@ceph.mon.b.smithi040.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T18:17:24.218 INFO:journalctl@ceph.mon.b.smithi040.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T18:17:24.218 INFO:journalctl@ceph.mon.b.smithi040.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T18:17:24.229 INFO:journalctl@ceph.mon.b.smithi040.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T18:17:24.229 INFO:journalctl@ceph.mon.b.smithi040.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-16T18:17:24.230 INFO:journalctl@ceph.mon.b.smithi040.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T18:17:24.230 INFO:journalctl@ceph.mon.b.smithi040.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T18:17:24.231 INFO:journalctl@ceph.mon.b.smithi040.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T18:17:24.231 INFO:journalctl@ceph.mon.b.smithi040.stdout: Block cache BinnedLRUCache@0x55eacbc45350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-11-16T18:17:24.231 INFO:journalctl@ceph.mon.b.smithi040.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-11-16T18:17:24.232 INFO:journalctl@ceph.mon.b.smithi040.stdout: 2023-11-16T18:17:24.232 INFO:journalctl@ceph.mon.b.smithi040.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-16T18:17:24.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-11-16T18:17:24.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: using public_addr v2:172.21.15.40:0/0 -> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-11-16T18:17:24.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: starting mon.b rank -1 at public addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] at bind addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:17:24.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(???) e0 preinit fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:17:24.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing).mds e1 new map 2023-11-16T18:17:24.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing).mds e1 print_map 2023-11-16T18:17:24.232 INFO:journalctl@ceph.mon.b.smithi040.stdout: e1 2023-11-16T18:17:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T18:17:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T18:17:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout: legacy client fscid: -1 2023-11-16T18:17:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout: 2023-11-16T18:17:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout: No filesystems configured 2023-11-16T18:17:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-11-16T18:17:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-11-16T18:17:24.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-11-16T18:17:24.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-11-16T18:17:24.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-11-16T18:17:24.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-11-16T18:17:24.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T18:17:24.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T18:17:24.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T18:17:24.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T18:17:24.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: Updating smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:17:24.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:24.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.249 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.250 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.251 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.251 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:24.251 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.252 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: Added host smithi188 2023-11-16T18:17:24.252 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.252 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:24.252 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.252 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:17:24.252 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:24.264 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3929959364' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T18:17:24.264 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:17:24.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:24.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:24.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:17:24.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: Updating smithi188:/etc/ceph/ceph.conf 2023-11-16T18:17:24.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:24.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:24.270 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3929959364' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T18:17:24.270 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T18:17:24.270 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.270 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.270 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.270 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:17:24.270 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:17:24.271 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:24.271 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.271 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:24.271 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:24.271 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:24.271 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.271 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:24.272 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:24.272 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi040:172.21.15.40=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:17:24.272 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi040:172.21.15.40=b;smithi188:172.21.15.188=c;count:3 2023-11-16T18:17:24.272 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:17:24.272 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:17:24.272 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: Deploying daemon mon.c on smithi188 2023-11-16T18:17:24.272 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:24.272 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:17:24.272 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.a calling monitor election 2023-11-16T18:17:24.273 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:24.273 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:24.273 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:24.273 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:24.273 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.c calling monitor election 2023-11-16T18:17:24.273 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:24.273 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:24.273 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:24.273 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:24.273 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:24.273 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T18:17:24.274 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-11-16T18:17:24.274 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: fsmap 2023-11-16T18:17:24.284 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T18:17:24.285 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mgrmap e13: x(active, since 42s) 2023-11-16T18:17:24.285 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: overall HEALTH_OK 2023-11-16T18:17:24.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.298 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.298 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:24.298 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:24.299 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:24.299 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: Deploying daemon mon.b on smithi040 2023-11-16T18:17:24.299 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:24.299 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:24.299 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: from='client.? 172.21.15.188:0/756903033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T18:17:24.300 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:24.300 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-11-16T18:17:24.300 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: expand_channel_meta expand map: {default=false} 2023-11-16T18:17:24.301 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: expand_channel_meta from 'false' to 'false' 2023-11-16T18:17:24.301 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: expand_channel_meta expanded map: {default=false} 2023-11-16T18:17:24.301 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: expand_channel_meta expand map: {default=info} 2023-11-16T18:17:24.301 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: expand_channel_meta from 'info' to 'info' 2023-11-16T18:17:24.302 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: expand_channel_meta expanded map: {default=info} 2023-11-16T18:17:24.302 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: expand_channel_meta expand map: {default=daemon} 2023-11-16T18:17:24.302 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: expand_channel_meta from 'daemon' to 'daemon' 2023-11-16T18:17:24.302 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: expand_channel_meta expanded map: {default=daemon} 2023-11-16T18:17:24.303 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: expand_channel_meta expand map: {default=debug} 2023-11-16T18:17:24.303 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: expand_channel_meta from 'debug' to 'debug' 2023-11-16T18:17:24.303 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: expand_channel_meta expanded map: {default=debug} 2023-11-16T18:17:24.303 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 ceph-mon[100251]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-11-16T18:17:24.304 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:23 smithi040 systemd[1]: Started Ceph mon.b for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:17:24.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:23 smithi002 ceph-mon[98807]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:24.305 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:23 smithi188 ceph-mon[100639]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:24.484 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:17:28.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.970+0000 7efc05b55700 1 -- 172.21.15.188:0/3910478776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbec0035c0 msgr2=0x7efbec005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:28.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.970+0000 7efc05b55700 1 --2- 172.21.15.188:0/3910478776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbec0035c0 0x7efbec005a40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efc000ffda0 tx=0x7efbe8013e90 comp rx=0 tx=0).stop 2023-11-16T18:17:28.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.970+0000 7efc05b55700 1 -- 172.21.15.188:0/3910478776 shutdown_connections 2023-11-16T18:17:28.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.970+0000 7efc05b55700 1 --2- 172.21.15.188:0/3910478776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbec0035c0 0x7efbec005a40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:28.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.970+0000 7efc05b55700 1 -- 172.21.15.188:0/3910478776 >> 172.21.15.188:0/3910478776 conn(0x7efc000fb990 msgr2=0x7efc000fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:28.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.971+0000 7efc05b55700 1 -- 172.21.15.188:0/3910478776 shutdown_connections 2023-11-16T18:17:28.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.971+0000 7efc05b55700 1 -- 172.21.15.188:0/3910478776 wait complete. 2023-11-16T18:17:28.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.971+0000 7efc05b55700 1 Processor -- start 2023-11-16T18:17:28.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.972+0000 7efc05b55700 1 -- start start 2023-11-16T18:17:28.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.972+0000 7efc05b55700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efbec0035c0 0x7efc00197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:28.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.972+0000 7efc05b55700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc00197b80 0x7efc0019bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:28.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.972+0000 7efc05b55700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc0019c4e0 0x7efc0019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:28.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.972+0000 7efc05b55700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc00198250 con 0x7efc0019c4e0 2023-11-16T18:17:28.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.972+0000 7efc05b55700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc001a4250 con 0x7efbec0035c0 2023-11-16T18:17:28.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.972+0000 7efc05b55700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc001a4390 con 0x7efc00197b80 2023-11-16T18:17:28.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.972+0000 7efbfeffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc00197b80 0x7efc0019bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:28.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.972+0000 7efbff7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efbec0035c0 0x7efc00197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:28.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.972+0000 7efbfffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc0019c4e0 0x7efc0019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:28.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.972+0000 7efbff7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efbec0035c0 0x7efc00197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.188:57058/0 (socket says 172.21.15.188:57058) 2023-11-16T18:17:28.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.972+0000 7efbfeffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc00197b80 0x7efc0019bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:48024/0 (socket says 172.21.15.188:48024) 2023-11-16T18:17:28.978 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.972+0000 7efbff7fe700 1 -- 172.21.15.188:0/2501089654 learned_addr learned my addr 172.21.15.188:0/2501089654 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-16T18:17:28.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbff7fe700 1 -- 172.21.15.188:0/2501089654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efbec0035c0 msgr2=0x7efc00197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-11-16T18:17:28.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbff7fe700 1 -- 172.21.15.188:0/2501089654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efbec0035c0 msgr2=0x7efc00197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-16T18:17:28.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbff7fe700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efbec0035c0 0x7efc00197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-16T18:17:28.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbfeffd700 1 -- 172.21.15.188:0/2501089654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc00197b80 msgr2=0x7efc0019bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-11-16T18:17:28.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbfeffd700 1 -- 172.21.15.188:0/2501089654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc00197b80 msgr2=0x7efc0019bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-16T18:17:28.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbfeffd700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc00197b80 0x7efc0019bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-16T18:17:28.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbff7fe700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efbec0035c0 0x7efc00197640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T18:17:28.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbfeffd700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc00197b80 0x7efc0019bfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T18:17:28.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbfffff700 1 -- 172.21.15.188:0/2501089654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc00197b80 msgr2=0x7efc0019bfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T18:17:28.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbfffff700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc00197b80 0x7efc0019bfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:28.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbfffff700 1 -- 172.21.15.188:0/2501089654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efbec0035c0 msgr2=0x7efc00197640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T18:17:28.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbfffff700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efbec0035c0 0x7efc00197640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:28.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbfffff700 1 -- 172.21.15.188:0/2501089654 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe80138e0 con 0x7efc0019c4e0 2023-11-16T18:17:28.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbfffff700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc0019c4e0 0x7efc0019c980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efbf800bd80 tx=0x7efbf800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:28.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbfcff9700 1 -- 172.21.15.188:0/2501089654 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf800cd10 con 0x7efc0019c4e0 2023-11-16T18:17:28.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbfcff9700 1 -- 172.21.15.188:0/2501089654 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efbf800ce70 con 0x7efc0019c4e0 2023-11-16T18:17:28.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efc05b55700 1 -- 172.21.15.188:0/2501089654 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc00107cc0 con 0x7efc0019c4e0 2023-11-16T18:17:28.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.973+0000 7efbfcff9700 1 -- 172.21.15.188:0/2501089654 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8012880 con 0x7efc0019c4e0 2023-11-16T18:17:28.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.974+0000 7efc05b55700 1 -- 172.21.15.188:0/2501089654 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc00107b30 con 0x7efc0019c4e0 2023-11-16T18:17:28.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.974+0000 7efbfcff9700 1 -- 172.21.15.188:0/2501089654 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7efbf80129e0 con 0x7efc0019c4e0 2023-11-16T18:17:28.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.974+0000 7efbfcff9700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efbe0038d40 0x7efbe003b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:28.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.975+0000 7efbfcff9700 1 -- 172.21.15.188:0/2501089654 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efbf8014b00 con 0x7efc0019c4e0 2023-11-16T18:17:28.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.975+0000 7efc05b55700 1 -- 172.21.15.188:0/2501089654 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc00107e50 con 0x7efc0019c4e0 2023-11-16T18:17:28.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.975+0000 7efbff7fe700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efbe0038d40 0x7efbe003b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:28.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.975+0000 7efbff7fe700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efbe0038d40 0x7efbe003b1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efbe8004910 tx=0x7efbe8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:28.983 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:28.982+0000 7efbfcff9700 1 -- 172.21.15.188:0/2501089654 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efbf801d3b0 con 0x7efc0019c4e0 2023-11-16T18:17:29.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.263+0000 7efc05b55700 1 -- 172.21.15.188:0/2501089654 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc00198a20 con 0x7efc0019c4e0 2023-11-16T18:17:29.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.264+0000 7efbfcff9700 1 -- 172.21.15.188:0/2501089654 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1268 (secure 0 0 0) 0x7efbf80112a0 con 0x7efc0019c4e0 2023-11-16T18:17:29.265 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-16T18:17:29.265 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":3,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T18:17:23.917300Z","created":"2023-11-16T18:15:39.352712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2023-11-16T18:17:29.267 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.266+0000 7efc05b55700 1 -- 172.21.15.188:0/2501089654 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efbe0038d40 msgr2=0x7efbe003b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:29.267 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.266+0000 7efc05b55700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efbe0038d40 0x7efbe003b1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efbe8004910 tx=0x7efbe8005960 comp rx=0 tx=0).stop 2023-11-16T18:17:29.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.266+0000 7efc05b55700 1 -- 172.21.15.188:0/2501089654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc0019c4e0 msgr2=0x7efc0019c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:29.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.266+0000 7efc05b55700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc0019c4e0 0x7efc0019c980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efbf800bd80 tx=0x7efbf800be60 comp rx=0 tx=0).stop 2023-11-16T18:17:29.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.266+0000 7efc05b55700 1 -- 172.21.15.188:0/2501089654 shutdown_connections 2023-11-16T18:17:29.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.266+0000 7efc05b55700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efbe0038d40 0x7efbe003b1f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:29.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.266+0000 7efc05b55700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc0019c4e0 0x7efc0019c980 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:29.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.266+0000 7efc05b55700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efbec0035c0 0x7efc00197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:29.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.266+0000 7efc05b55700 1 --2- 172.21.15.188:0/2501089654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc00197b80 0x7efc0019bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:29.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.266+0000 7efc05b55700 1 -- 172.21.15.188:0/2501089654 >> 172.21.15.188:0/2501089654 conn(0x7efc000fb990 msgr2=0x7efc0018d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:29.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.267+0000 7efc05b55700 1 -- 172.21.15.188:0/2501089654 shutdown_connections 2023-11-16T18:17:29.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:29.267+0000 7efc05b55700 1 -- 172.21.15.188:0/2501089654 wait complete. 2023-11-16T18:17:29.269 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 3 2023-11-16T18:17:29.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:17:29.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: mon.a calling monitor election 2023-11-16T18:17:29.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: mon.c calling monitor election 2023-11-16T18:17:29.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:29.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:29.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: mon.b calling monitor election 2023-11-16T18:17:29.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:29.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-11-16T18:17:29.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-11-16T18:17:29.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: fsmap 2023-11-16T18:17:29.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T18:17:29.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: mgrmap e13: x(active, since 51s) 2023-11-16T18:17:29.301 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: overall HEALTH_OK 2023-11-16T18:17:29.301 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:29.301 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:29.301 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:29.301 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:29.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:17:29.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: mon.a calling monitor election 2023-11-16T18:17:29.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: mon.c calling monitor election 2023-11-16T18:17:29.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:29.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:29.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: mon.b calling monitor election 2023-11-16T18:17:29.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:29.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-11-16T18:17:29.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-11-16T18:17:29.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: fsmap 2023-11-16T18:17:29.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T18:17:29.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: mgrmap e13: x(active, since 51s) 2023-11-16T18:17:29.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: overall HEALTH_OK 2023-11-16T18:17:29.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:29.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:29.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:29.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:29.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:17:29.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: mon.a calling monitor election 2023-11-16T18:17:29.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: mon.c calling monitor election 2023-11-16T18:17:29.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:17:29.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:29.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: mon.b calling monitor election 2023-11-16T18:17:29.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:29.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:29.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-11-16T18:17:29.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-11-16T18:17:29.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: fsmap 2023-11-16T18:17:29.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T18:17:29.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: mgrmap e13: x(active, since 51s) 2023-11-16T18:17:29.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: overall HEALTH_OK 2023-11-16T18:17:29.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:29.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:29.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:29.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:29.636 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-11-16T18:17:29.636 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph config generate-minimal-conf 2023-11-16T18:17:30.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:30 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:30.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:30 smithi188 ceph-mon[100639]: from='client.? 172.21.15.188:0/2501089654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T18:17:30.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:30 smithi188 ceph-mon[100639]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:30.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:30 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:30.309 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:17:30.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:30 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:30.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:30 smithi002 ceph-mon[98807]: from='client.? 172.21.15.188:0/2501089654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T18:17:30.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:30 smithi002 ceph-mon[98807]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:30.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:30 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:30.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:30 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:30.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:30 smithi040 ceph-mon[100251]: from='client.? 172.21.15.188:0/2501089654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T18:17:30.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:30 smithi040 ceph-mon[100251]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:30.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:30 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:17:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 -- 172.21.15.2:0/1499037325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8da80fff90 msgr2=0x7f8da8100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 --2- 172.21.15.2:0/1499037325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8da80fff90 0x7f8da8100360 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009910 tx=0x7f8d98009c20 comp rx=0 tx=0).stop 2023-11-16T18:17:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 -- 172.21.15.2:0/1499037325 shutdown_connections 2023-11-16T18:17:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 --2- 172.21.15.2:0/1499037325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8da80fff90 0x7f8da8100360 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 -- 172.21.15.2:0/1499037325 >> 172.21.15.2:0/1499037325 conn(0x7f8da80fb870 msgr2=0x7f8da80fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 -- 172.21.15.2:0/1499037325 shutdown_connections 2023-11-16T18:17:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 -- 172.21.15.2:0/1499037325 wait complete. 2023-11-16T18:17:31.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 Processor -- start 2023-11-16T18:17:31.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 -- start start 2023-11-16T18:17:31.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8da80fff90 0x7f8da8197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:31.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8da8197a70 0x7f8da819be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:31.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8da819c3b0 0x7f8da819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:31.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da8198140 con 0x7f8da80fff90 2023-11-16T18:17:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.398+0000 7f8dadb7b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da81a4120 con 0x7f8da8197a70 2023-11-16T18:17:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8dadb7b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da81a4260 con 0x7f8da819c3b0 2023-11-16T18:17:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8da77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8da80fff90 0x7f8da8197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8da77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8da80fff90 0x7f8da8197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46904/0 (socket says 172.21.15.2:46904) 2023-11-16T18:17:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8da77fe700 1 -- 172.21.15.2:0/297550692 learned_addr learned my addr 172.21.15.2:0/297550692 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:17:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8d9ffff700 1 --2- 172.21.15.2:0/297550692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8da8197a70 0x7f8da819be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8da77fe700 1 -- 172.21.15.2:0/297550692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8da819c3b0 msgr2=0x7f8da819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8da77fe700 1 --2- 172.21.15.2:0/297550692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8da819c3b0 0x7f8da819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8da77fe700 1 -- 172.21.15.2:0/297550692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8da8197a70 msgr2=0x7f8da819be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8da77fe700 1 --2- 172.21.15.2:0/297550692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8da8197a70 0x7f8da819be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8da77fe700 1 -- 172.21.15.2:0/297550692 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d980095f0 con 0x7f8da80fff90 2023-11-16T18:17:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8da77fe700 1 --2- 172.21.15.2:0/297550692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8da80fff90 0x7f8da8197530 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8d98004ce0 tx=0x7f8d98004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:31.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8da57fa700 1 -- 172.21.15.2:0/297550692 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9801d070 con 0x7f8da80fff90 2023-11-16T18:17:31.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8da57fa700 1 -- 172.21.15.2:0/297550692 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d9800be20 con 0x7f8da80fff90 2023-11-16T18:17:31.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8dadb7b700 1 -- 172.21.15.2:0/297550692 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da810a120 con 0x7f8da80fff90 2023-11-16T18:17:31.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8da57fa700 1 -- 172.21.15.2:0/297550692 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9800f460 con 0x7f8da80fff90 2023-11-16T18:17:31.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.399+0000 7f8dadb7b700 1 -- 172.21.15.2:0/297550692 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da8109f90 con 0x7f8da80fff90 2023-11-16T18:17:31.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.400+0000 7f8da57fa700 1 -- 172.21.15.2:0/297550692 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f8d98022670 con 0x7f8da80fff90 2023-11-16T18:17:31.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.400+0000 7f8dadb7b700 1 -- 172.21.15.2:0/297550692 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da810a2b0 con 0x7f8da80fff90 2023-11-16T18:17:31.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.400+0000 7f8da57fa700 1 --2- 172.21.15.2:0/297550692 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8d94038cf0 0x7f8d9403b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:31.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.400+0000 7f8da57fa700 1 -- 172.21.15.2:0/297550692 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8d9804c750 con 0x7f8da80fff90 2023-11-16T18:17:31.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.400+0000 7f8d9ffff700 1 --2- 172.21.15.2:0/297550692 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8d94038cf0 0x7f8d9403b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:31.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.401+0000 7f8d9ffff700 1 --2- 172.21.15.2:0/297550692 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8d94038cf0 0x7f8d9403b1a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8d900078e0 tx=0x7f8d90008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:31.412 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.405+0000 7f8da57fa700 1 -- 172.21.15.2:0/297550692 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8d9800e9f0 con 0x7f8da80fff90 2023-11-16T18:17:31.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.612+0000 7f8dadb7b700 1 -- 172.21.15.2:0/297550692 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f8da81a4d30 con 0x7f8da80fff90 2023-11-16T18:17:31.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.613+0000 7f8da57fa700 1 -- 172.21.15.2:0/297550692 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+271 (secure 0 0 0) 0x7f8d98027070 con 0x7f8da80fff90 2023-11-16T18:17:31.616 INFO:teuthology.orchestra.run.smithi002.stdout:# minimal ceph.conf for 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:17:31.616 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2023-11-16T18:17:31.616 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:17:31.616 INFO:teuthology.orchestra.run.smithi002.stdout: mon_host = [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2023-11-16T18:17:31.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.615+0000 7f8dadb7b700 1 -- 172.21.15.2:0/297550692 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8d94038cf0 msgr2=0x7f8d9403b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:31.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.615+0000 7f8dadb7b700 1 --2- 172.21.15.2:0/297550692 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8d94038cf0 0x7f8d9403b1a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8d900078e0 tx=0x7f8d90008040 comp rx=0 tx=0).stop 2023-11-16T18:17:31.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.615+0000 7f8dadb7b700 1 -- 172.21.15.2:0/297550692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8da80fff90 msgr2=0x7f8da8197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:31.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.615+0000 7f8dadb7b700 1 --2- 172.21.15.2:0/297550692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8da80fff90 0x7f8da8197530 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8d98004ce0 tx=0x7f8d98004dc0 comp rx=0 tx=0).stop 2023-11-16T18:17:31.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.615+0000 7f8dadb7b700 1 -- 172.21.15.2:0/297550692 shutdown_connections 2023-11-16T18:17:31.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.615+0000 7f8dadb7b700 1 --2- 172.21.15.2:0/297550692 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8d94038cf0 0x7f8d9403b1a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:31.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.615+0000 7f8dadb7b700 1 --2- 172.21.15.2:0/297550692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8da819c3b0 0x7f8da819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:31.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.615+0000 7f8dadb7b700 1 --2- 172.21.15.2:0/297550692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8da80fff90 0x7f8da8197530 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:31.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.615+0000 7f8dadb7b700 1 --2- 172.21.15.2:0/297550692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8da8197a70 0x7f8da819be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:31.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.615+0000 7f8dadb7b700 1 -- 172.21.15.2:0/297550692 >> 172.21.15.2:0/297550692 conn(0x7f8da80fb870 msgr2=0x7f8da81038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:31.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.615+0000 7f8dadb7b700 1 -- 172.21.15.2:0/297550692 shutdown_connections 2023-11-16T18:17:31.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:31.615+0000 7f8dadb7b700 1 -- 172.21.15.2:0/297550692 wait complete. 2023-11-16T18:17:31.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:31.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:31.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:31.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:31.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:31.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:31.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:17:31.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T18:17:31.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: Updating smithi040:/etc/ceph/ceph.conf 2023-11-16T18:17:31.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: Updating smithi188:/etc/ceph/ceph.conf 2023-11-16T18:17:31.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:31.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:31.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:31.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:31.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:31.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:31.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:31.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:31.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T18:17:31.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:31.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:31 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/297550692' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:32.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:32.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:32.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:17:32.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T18:17:32.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: Updating smithi040:/etc/ceph/ceph.conf 2023-11-16T18:17:32.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: Updating smithi188:/etc/ceph/ceph.conf 2023-11-16T18:17:32.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:32.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T18:17:32.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:32.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:31 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/297550692' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:32.058 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.058 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.058 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.058 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.059 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:32.059 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:32.059 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:32.059 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T18:17:32.059 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: Updating smithi040:/etc/ceph/ceph.conf 2023-11-16T18:17:32.059 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: Updating smithi188:/etc/ceph/ceph.conf 2023-11-16T18:17:32.059 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.059 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.059 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.059 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.059 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.059 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.060 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:32.060 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:32.060 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T18:17:32.060 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:32.060 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:31 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/297550692' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:32.078 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-11-16T18:17:32.079 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:17:32.079 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.conf 2023-11-16T18:17:32.113 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:17:32.114 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:17:32.183 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-11-16T18:17:32.183 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.conf 2023-11-16T18:17:32.222 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-11-16T18:17:32.222 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:17:32.294 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-16T18:17:32.294 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.conf 2023-11-16T18:17:32.327 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-16T18:17:32.328 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:17:32.399 INFO:tasks.cephadm:Adding mgr.x on smithi002 2023-11-16T18:17:32.399 INFO:tasks.cephadm:Adding mgr.y on smithi040 2023-11-16T18:17:32.399 INFO:tasks.cephadm:Adding mgr.z on smithi188 2023-11-16T18:17:32.399 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch apply mgr '3;smithi002=x;smithi040=y;smithi188=z' 2023-11-16T18:17:32.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:32 smithi040 ceph-mon[100251]: Updating smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:32.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:32 smithi040 ceph-mon[100251]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:32.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:32 smithi040 ceph-mon[100251]: Updating smithi002:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:32.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:32 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:17:32.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:32 smithi040 ceph-mon[100251]: Reconfiguring mon.a (unknown last config time)... 2023-11-16T18:17:32.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:32 smithi040 ceph-mon[100251]: Reconfiguring daemon mon.a on smithi002 2023-11-16T18:17:32.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:32 smithi040 ceph-mon[100251]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:33.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:32 smithi188 ceph-mon[100639]: Updating smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:33.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:32 smithi188 ceph-mon[100639]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:33.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:32 smithi188 ceph-mon[100639]: Updating smithi002:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:33.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:32 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:17:33.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:32 smithi188 ceph-mon[100639]: Reconfiguring mon.a (unknown last config time)... 2023-11-16T18:17:33.057 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:32 smithi188 ceph-mon[100639]: Reconfiguring daemon mon.a on smithi002 2023-11-16T18:17:33.057 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:32 smithi188 ceph-mon[100639]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:33.076 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:17:33.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:32 smithi002 ceph-mon[98807]: Updating smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:33.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:32 smithi002 ceph-mon[98807]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:33.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:32 smithi002 ceph-mon[98807]: Updating smithi002:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:17:33.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:32 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:33.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:32 smithi002 ceph-mon[98807]: Reconfiguring mon.a (unknown last config time)... 2023-11-16T18:17:33.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:32 smithi002 ceph-mon[98807]: Reconfiguring daemon mon.a on smithi002 2023-11-16T18:17:33.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:32 smithi002 ceph-mon[98807]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:33.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:33.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:33.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:33.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:33 smithi188 ceph-mon[100639]: Reconfiguring mon.b (monmap changed)... 2023-11-16T18:17:33.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:33.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T18:17:33.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:33.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:33 smithi188 ceph-mon[100639]: Reconfiguring daemon mon.b on smithi040 2023-11-16T18:17:33.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:33 smithi188 ceph-mon[100639]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:33.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:33.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:33.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:33.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:33 smithi040 ceph-mon[100251]: Reconfiguring mon.b (monmap changed)... 2023-11-16T18:17:33.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:33.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T18:17:33.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:33.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:33 smithi040 ceph-mon[100251]: Reconfiguring daemon mon.b on smithi040 2023-11-16T18:17:33.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:33 smithi040 ceph-mon[100251]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:34.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:34.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:34.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:34.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:33 smithi002 ceph-mon[98807]: Reconfiguring mon.b (monmap changed)... 2023-11-16T18:17:34.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:34.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T18:17:34.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:34.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:33 smithi002 ceph-mon[98807]: Reconfiguring daemon mon.b on smithi040 2023-11-16T18:17:34.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:33 smithi002 ceph-mon[98807]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:34.221 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.215+0000 7f44f8237700 1 -- 172.21.15.188:0/625746373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fed50 msgr2=0x7f44f00ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:34.221 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.215+0000 7f44f8237700 1 --2- 172.21.15.188:0/625746373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fed50 0x7f44f00ff160 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009910 tx=0x7f44ec009c20 comp rx=0 tx=0).stop 2023-11-16T18:17:34.221 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.215+0000 7f44f8237700 1 -- 172.21.15.188:0/625746373 shutdown_connections 2023-11-16T18:17:34.221 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.215+0000 7f44f8237700 1 --2- 172.21.15.188:0/625746373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fed50 0x7f44f00ff160 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:34.221 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.215+0000 7f44f8237700 1 -- 172.21.15.188:0/625746373 >> 172.21.15.188:0/625746373 conn(0x7f44f00fa300 msgr2=0x7f44f00fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:34.221 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.215+0000 7f44f8237700 1 -- 172.21.15.188:0/625746373 shutdown_connections 2023-11-16T18:17:34.221 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.215+0000 7f44f8237700 1 -- 172.21.15.188:0/625746373 wait complete. 2023-11-16T18:17:34.221 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f8237700 1 Processor -- start 2023-11-16T18:17:34.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f8237700 1 -- start start 2023-11-16T18:17:34.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f8237700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fed50 0x7f44f00fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:34.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f8237700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f00fa8e0 0x7f44f00fad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:34.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f8237700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44f00fb290 0x7f44f019f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:34.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f8237700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f019f650 con 0x7f44f00fed50 2023-11-16T18:17:34.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f8237700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f019f790 con 0x7f44f00fa8e0 2023-11-16T18:17:34.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f8237700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f019f8d0 con 0x7f44f00fb290 2023-11-16T18:17:34.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f5fd3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fed50 0x7f44f00fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:34.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f5fd3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fed50 0x7f44f00fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.188:55322/0 (socket says 172.21.15.188:55322) 2023-11-16T18:17:34.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f5fd3700 1 -- 172.21.15.188:0/2206395826 learned_addr learned my addr 172.21.15.188:0/2206395826 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-16T18:17:34.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f67d4700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44f00fb290 0x7f44f019f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:34.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f57d2700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f00fa8e0 0x7f44f00fad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:34.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f5fd3700 1 -- 172.21.15.188:0/2206395826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44f00fb290 msgr2=0x7f44f019f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:34.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f5fd3700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44f00fb290 0x7f44f019f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:34.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f5fd3700 1 -- 172.21.15.188:0/2206395826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f00fa8e0 msgr2=0x7f44f00fad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:34.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f5fd3700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f00fa8e0 0x7f44f00fad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:34.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f5fd3700 1 -- 172.21.15.188:0/2206395826 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ec0095f0 con 0x7f44f00fed50 2023-11-16T18:17:34.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.216+0000 7f44f67d4700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44f00fb290 0x7f44f019f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:17:34.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.217+0000 7f44f5fd3700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fed50 0x7f44f00fc290 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f44ec000c00 tx=0x7f44ec004cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:34.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.217+0000 7f44e6ffd700 1 -- 172.21.15.188:0/2206395826 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec01d070 con 0x7f44f00fed50 2023-11-16T18:17:34.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.217+0000 7f44f8237700 1 -- 172.21.15.188:0/2206395826 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f0104bd0 con 0x7f44f00fed50 2023-11-16T18:17:34.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.217+0000 7f44e6ffd700 1 -- 172.21.15.188:0/2206395826 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f44ec00be20 con 0x7f44f00fed50 2023-11-16T18:17:34.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.217+0000 7f44e6ffd700 1 -- 172.21.15.188:0/2206395826 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec00f460 con 0x7f44f00fed50 2023-11-16T18:17:34.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.217+0000 7f44f8237700 1 -- 172.21.15.188:0/2206395826 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f0104a40 con 0x7f44f00fed50 2023-11-16T18:17:34.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.218+0000 7f44e6ffd700 1 -- 172.21.15.188:0/2206395826 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f44ec022600 con 0x7f44f00fed50 2023-11-16T18:17:34.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.218+0000 7f44f8237700 1 -- 172.21.15.188:0/2206395826 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f018e910 con 0x7f44f00fed50 2023-11-16T18:17:34.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.218+0000 7f44e6ffd700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f44dc038cf0 0x7f44dc03b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:34.225 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.218+0000 7f44e6ffd700 1 -- 172.21.15.188:0/2206395826 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f44ec04c9f0 con 0x7f44f00fed50 2023-11-16T18:17:34.225 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.218+0000 7f44f57d2700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f44dc038cf0 0x7f44dc03b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:34.225 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.219+0000 7f44f57d2700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f44dc038cf0 0x7f44dc03b1a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009850 tx=0x7f44e0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:34.225 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.223+0000 7f44e6ffd700 1 -- 172.21.15.188:0/2206395826 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f44ec00edb0 con 0x7f44f00fed50 2023-11-16T18:17:34.436 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.435+0000 7f44f8237700 1 -- 172.21.15.188:0/2206395826 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi040=y;smithi188=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f44f0061bf0 con 0x7f44dc038cf0 2023-11-16T18:17:34.443 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.441+0000 7f44e6ffd700 1 -- 172.21.15.188:0/2206395826 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f44f0061bf0 con 0x7f44dc038cf0 2023-11-16T18:17:34.444 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mgr update... 2023-11-16T18:17:34.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.443+0000 7f44f8237700 1 -- 172.21.15.188:0/2206395826 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f44dc038cf0 msgr2=0x7f44dc03b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:34.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.443+0000 7f44f8237700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f44dc038cf0 0x7f44dc03b1a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009850 tx=0x7f44e0008040 comp rx=0 tx=0).stop 2023-11-16T18:17:34.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.444+0000 7f44f8237700 1 -- 172.21.15.188:0/2206395826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fed50 msgr2=0x7f44f00fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:34.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.444+0000 7f44f8237700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fed50 0x7f44f00fc290 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f44ec000c00 tx=0x7f44ec004cb0 comp rx=0 tx=0).stop 2023-11-16T18:17:34.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.444+0000 7f44f8237700 1 -- 172.21.15.188:0/2206395826 shutdown_connections 2023-11-16T18:17:34.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.444+0000 7f44f8237700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f44dc038cf0 0x7f44dc03b1a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:34.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.444+0000 7f44f8237700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44f00fb290 0x7f44f019f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:34.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.444+0000 7f44f8237700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fed50 0x7f44f00fc290 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:34.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.444+0000 7f44f8237700 1 --2- 172.21.15.188:0/2206395826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f00fa8e0 0x7f44f00fad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:34.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.444+0000 7f44f8237700 1 -- 172.21.15.188:0/2206395826 >> 172.21.15.188:0/2206395826 conn(0x7f44f00fa300 msgr2=0x7f44f0069370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:34.447 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.444+0000 7f44f8237700 1 -- 172.21.15.188:0/2206395826 shutdown_connections 2023-11-16T18:17:34.447 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:17:34.444+0000 7f44f8237700 1 -- 172.21.15.188:0/2206395826 wait complete. 2023-11-16T18:17:34.829 DEBUG:teuthology.orchestra.run.smithi040:mgr.y> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.y.service 2023-11-16T18:17:34.832 DEBUG:teuthology.orchestra.run.smithi188:mgr.z> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.z.service 2023-11-16T18:17:34.842 INFO:tasks.cephadm:Deploying OSDs... 2023-11-16T18:17:34.843 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:17:34.843 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T18:17:34.864 INFO:journalctl@ceph.mgr.y.smithi040.stdout:-- Logs begin at Thu 2023-11-16 17:59:26 UTC. -- 2023-11-16T18:17:34.864 INFO:journalctl@ceph.mgr.z.smithi188.stdout:-- Logs begin at Thu 2023-11-16 17:59:32 UTC. -- 2023-11-16T18:17:34.866 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T18:17:34.866 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2023-11-16T18:17:34.931 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T18:17:34.931 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T18:17:34.931 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 6h/6d Inode: 388400 Links: 1 2023-11-16T18:17:34.931 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T18:17:34.931 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T18:17:34.931 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-16 18:16:48.160106113 +0000 2023-11-16T18:17:34.931 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-16 18:10:13.725909467 +0000 2023-11-16T18:17:34.932 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-16 18:10:13.725909467 +0000 2023-11-16T18:17:34.932 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-11-16T18:17:34.932 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T18:17:35.005 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-11-16T18:17:35.005 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-11-16T18:17:35.005 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.00014007 s, 3.7 MB/s 2023-11-16T18:17:35.006 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T18:17:35.071 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2023-11-16T18:17:35.131 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T18:17:35.132 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T18:17:35.132 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 6h/6d Inode: 395344 Links: 1 2023-11-16T18:17:35.132 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T18:17:35.133 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T18:17:35.133 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-16 18:16:48.160106113 +0000 2023-11-16T18:17:35.133 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-16 18:10:14.834884845 +0000 2023-11-16T18:17:35.134 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-16 18:10:14.834884845 +0000 2023-11-16T18:17:35.134 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-11-16T18:17:35.135 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T18:17:35.204 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-11-16T18:17:35.204 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-11-16T18:17:35.204 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.00010754 s, 4.8 MB/s 2023-11-16T18:17:35.205 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T18:17:35.272 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2023-11-16T18:17:35.332 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T18:17:35.332 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T18:17:35.333 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 6h/6d Inode: 387479 Links: 1 2023-11-16T18:17:35.333 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T18:17:35.333 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T18:17:35.333 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-16 18:16:48.160106113 +0000 2023-11-16T18:17:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-16 18:10:16.033858221 +0000 2023-11-16T18:17:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-16 18:10:16.033858221 +0000 2023-11-16T18:17:35.334 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-11-16T18:17:35.334 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T18:17:35.398 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:35.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:35.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:35 smithi188 ceph-mon[100639]: Reconfiguring mon.c (monmap changed)... 2023-11-16T18:17:35.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:35.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T18:17:35.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:35.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:35 smithi188 ceph-mon[100639]: Reconfiguring daemon mon.c on smithi188 2023-11-16T18:17:35.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:35 smithi188 ceph-mon[100639]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi040=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:17:35.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:35 smithi188 ceph-mon[100639]: Saving service mgr spec with placement smithi002=x;smithi040=y;smithi188=z;count:3 2023-11-16T18:17:35.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:35.407 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-11-16T18:17:35.407 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-11-16T18:17:35.407 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000117762 s, 4.3 MB/s 2023-11-16T18:17:35.407 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T18:17:35.470 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2023-11-16T18:17:35.529 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T18:17:35.529 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T18:17:35.529 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 6h/6d Inode: 388565 Links: 1 2023-11-16T18:17:35.530 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T18:17:35.530 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T18:17:35.530 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-16 18:16:48.160106113 +0000 2023-11-16T18:17:35.530 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-16 18:10:17.086834837 +0000 2023-11-16T18:17:35.530 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-16 18:10:17.086834837 +0000 2023-11-16T18:17:35.530 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-11-16T18:17:35.530 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T18:17:35.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:35.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:35.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:35 smithi002 ceph-mon[98807]: Reconfiguring mon.c (monmap changed)... 2023-11-16T18:17:35.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:35.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T18:17:35.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:35.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:35 smithi002 ceph-mon[98807]: Reconfiguring daemon mon.c on smithi188 2023-11-16T18:17:35.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:35 smithi002 ceph-mon[98807]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi040=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:17:35.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:35 smithi002 ceph-mon[98807]: Saving service mgr spec with placement smithi002=x;smithi040=y;smithi188=z;count:3 2023-11-16T18:17:35.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:35.603 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-11-16T18:17:35.603 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-11-16T18:17:35.603 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000106485 s, 4.8 MB/s 2023-11-16T18:17:35.604 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T18:17:35.664 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-11-16T18:17:35.664 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T18:17:35.686 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T18:17:35.686 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2023-11-16T18:17:35.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:35.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:35.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:35 smithi040 ceph-mon[100251]: Reconfiguring mon.c (monmap changed)... 2023-11-16T18:17:35.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T18:17:35.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T18:17:35.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:35.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:35 smithi040 ceph-mon[100251]: Reconfiguring daemon mon.c on smithi188 2023-11-16T18:17:35.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:35 smithi040 ceph-mon[100251]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi040=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:17:35.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:35 smithi040 ceph-mon[100251]: Saving service mgr spec with placement smithi002=x;smithi040=y;smithi188=z;count:3 2023-11-16T18:17:35.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:35.751 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T18:17:35.751 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T18:17:35.752 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 389821 Links: 1 2023-11-16T18:17:35.752 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T18:17:35.752 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T18:17:35.752 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-11-16 18:17:23.323588547 +0000 2023-11-16T18:17:35.752 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-11-16 18:10:35.741550361 +0000 2023-11-16T18:17:35.752 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-11-16 18:10:35.741550361 +0000 2023-11-16T18:17:35.752 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-11-16T18:17:35.752 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T18:17:35.826 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-11-16T18:17:35.827 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-11-16T18:17:35.827 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000234021 s, 2.2 MB/s 2023-11-16T18:17:35.827 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T18:17:35.888 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2023-11-16T18:17:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T18:17:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T18:17:35.949 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 389910 Links: 1 2023-11-16T18:17:35.949 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T18:17:35.949 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T18:17:35.949 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-11-16 18:17:23.323588547 +0000 2023-11-16T18:17:35.949 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-11-16 18:10:37.152532282 +0000 2023-11-16T18:17:35.949 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-11-16 18:10:37.152532282 +0000 2023-11-16T18:17:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-11-16T18:17:35.950 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T18:17:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-11-16T18:17:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-11-16T18:17:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000236623 s, 2.2 MB/s 2023-11-16T18:17:36.021 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T18:17:36.083 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2023-11-16T18:17:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T18:17:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T18:17:36.148 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 389066 Links: 1 2023-11-16T18:17:36.148 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T18:17:36.149 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T18:17:36.149 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-11-16 18:17:23.323588547 +0000 2023-11-16T18:17:36.149 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-11-16 18:10:38.190518983 +0000 2023-11-16T18:17:36.149 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-11-16 18:10:38.190518983 +0000 2023-11-16T18:17:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-11-16T18:17:36.150 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T18:17:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-11-16T18:17:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-11-16T18:17:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000224852 s, 2.3 MB/s 2023-11-16T18:17:36.219 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T18:17:36.285 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2023-11-16T18:17:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T18:17:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T18:17:36.346 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 396299 Links: 1 2023-11-16T18:17:36.346 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T18:17:36.347 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T18:17:36.347 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-11-16 18:17:23.323588547 +0000 2023-11-16T18:17:36.347 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-11-16 18:10:39.370503864 +0000 2023-11-16T18:17:36.347 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-11-16 18:10:39.370503864 +0000 2023-11-16T18:17:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-11-16T18:17:36.348 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T18:17:36.416 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-11-16T18:17:36.417 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-11-16T18:17:36.417 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000217966 s, 2.3 MB/s 2023-11-16T18:17:36.418 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T18:17:36.479 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-16T18:17:36.479 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T18:17:36.502 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T18:17:36.502 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2023-11-16T18:17:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T18:17:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T18:17:36.562 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 388730 Links: 1 2023-11-16T18:17:36.563 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T18:17:36.564 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T18:17:36.565 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2023-11-16 18:17:14.897303846 +0000 2023-11-16T18:17:36.565 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2023-11-16 18:10:13.100244938 +0000 2023-11-16T18:17:36.565 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2023-11-16 18:10:13.100244938 +0000 2023-11-16T18:17:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2023-11-16T18:17:36.566 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T18:17:36.631 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2023-11-16T18:17:36.632 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2023-11-16T18:17:36.632 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000238421 s, 2.1 MB/s 2023-11-16T18:17:36.633 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T18:17:36.690 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2023-11-16T18:17:36.697 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:36.697 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:36.698 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:36.698 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:36.698 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:36.698 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:17:36.699 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:36.699 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:17:36.699 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T18:17:36.701 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T18:17:36.701 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T18:17:36.701 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:36.701 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: Deploying daemon mgr.z on smithi188 2023-11-16T18:17:36.702 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:36 smithi188 ceph-mon[100639]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:36.719 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T18:17:36.719 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T18:17:36.720 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 388781 Links: 1 2023-11-16T18:17:36.720 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T18:17:36.720 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T18:17:36.720 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2023-11-16 18:17:14.897303846 +0000 2023-11-16T18:17:36.721 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2023-11-16 18:10:14.351222546 +0000 2023-11-16T18:17:36.721 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2023-11-16 18:10:14.351222546 +0000 2023-11-16T18:17:36.721 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2023-11-16T18:17:36.721 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T18:17:36.789 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2023-11-16T18:17:36.789 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2023-11-16T18:17:36.789 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000240526 s, 2.1 MB/s 2023-11-16T18:17:36.789 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T18:17:36.847 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2023-11-16T18:17:36.906 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T18:17:36.906 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T18:17:36.906 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 390835 Links: 1 2023-11-16T18:17:36.907 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T18:17:36.907 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T18:17:36.907 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2023-11-16 18:17:14.897303846 +0000 2023-11-16T18:17:36.907 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2023-11-16 18:10:15.636199545 +0000 2023-11-16T18:17:36.907 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2023-11-16 18:10:15.636199545 +0000 2023-11-16T18:17:36.907 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2023-11-16T18:17:36.908 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T18:17:36.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:36.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:36.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:36.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:36.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:36.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:17:36.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:36.988 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:17:36.988 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T18:17:36.988 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T18:17:36.988 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T18:17:36.989 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:36.989 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: Deploying daemon mgr.z on smithi188 2023-11-16T18:17:36.989 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:36 smithi040 ceph-mon[100251]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:37.040 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2023-11-16T18:17:37.040 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2023-11-16T18:17:37.042 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000233053 s, 2.2 MB/s 2023-11-16T18:17:37.043 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T18:17:37.062 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2023-11-16T18:17:37.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:37.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:37.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:37.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:37.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:37.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:37.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:37.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:17:37.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T18:17:37.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T18:17:37.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T18:17:37.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:37.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: Deploying daemon mgr.z on smithi188 2023-11-16T18:17:37.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:36 smithi002 ceph-mon[98807]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:37.119 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T18:17:37.119 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T18:17:37.119 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 390876 Links: 1 2023-11-16T18:17:37.119 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T18:17:37.119 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T18:17:37.120 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2023-11-16 18:17:36.984914390 +0000 2023-11-16T18:17:37.120 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2023-11-16 18:10:16.793178835 +0000 2023-11-16T18:17:37.120 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2023-11-16 18:10:16.793178835 +0000 2023-11-16T18:17:37.120 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2023-11-16T18:17:37.120 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T18:17:37.188 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2023-11-16T18:17:37.188 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2023-11-16T18:17:37.189 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.00022318 s, 2.3 MB/s 2023-11-16T18:17:37.190 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T18:17:37.249 INFO:tasks.cephadm:Deploying osd.0 on smithi002 with /dev/vg_nvme/lv_4... 2023-11-16T18:17:37.249 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-16T18:17:37.298 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:37 smithi188 systemd[1]: Starting Ceph mgr.z for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:17:37.715 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:37 smithi188 podman[102762]: 2023-11-16T18:17:37.917 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:17:38.006 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:37 smithi188 ceph-mon[100639]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:38.008 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:37 smithi188 bash[102762]: 74ea28b621ecbbf21259d2750d822417193c05fb0a5522c93c4d06461e6427e4 2023-11-16T18:17:38.008 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:37 smithi188 ceph-mgr[102802]: -- 172.21.15.188:0/4012133496 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55693ac7b520 con 0x55693ac51000 2023-11-16T18:17:38.012 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:37 smithi188 systemd[1]: Started Ceph mgr.z for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:17:38.013 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:37 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:37.838+0000 7f75ce7bb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T18:17:38.014 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:37 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:37.917+0000 7f75ce7bb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T18:17:38.014 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:38.003+0000 7f75ce7bb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T18:17:38.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:37 smithi002 ceph-mon[98807]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:38.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:37 smithi040 ceph-mon[100251]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:38.262 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:38.095+0000 7f75ce7bb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T18:17:38.546 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:38.260+0000 7f75ce7bb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T18:17:38.826 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:38.565+0000 7f75ce7bb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T18:17:38.826 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:38.653+0000 7f75ce7bb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T18:17:38.826 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:38.788+0000 7f75ce7bb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T18:17:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T18:17:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T18:17:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T18:17:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:38 smithi040 ceph-mon[100251]: Deploying daemon mgr.y on smithi040 2023-11-16T18:17:39.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:39.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:39.114 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:39.114 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T18:17:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T18:17:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T18:17:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:38 smithi002 ceph-mon[98807]: Deploying daemon mgr.y on smithi040 2023-11-16T18:17:39.295 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-16T18:17:39.297 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-16T18:17:39.297 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2023-11-16T18:17:39.297 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2023-11-16T18:17:39.298 INFO:teuthology.orchestra.run.smithi002.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0132823 s, 789 MB/s 2023-11-16T18:17:39.299 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2023-11-16T18:17:39.300 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:38.882+0000 7f75ce7bb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T18:17:39.301 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:39.301 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:39.301 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:39.301 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T18:17:39.302 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T18:17:39.302 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T18:17:39.302 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:39.303 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:38 smithi188 ceph-mon[100639]: Deploying daemon mgr.y on smithi040 2023-11-16T18:17:39.664 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi002:vg_nvme/lv_4 2023-11-16T18:17:39.733 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:39 smithi040 systemd[1]: Starting Ceph mgr.y for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:17:39.796 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:39.297+0000 7f75ce7bb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T18:17:40.079 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:39 smithi040 ceph-mon[100251]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:40.079 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:39 smithi040 podman[101691]: 2023-11-16T18:17:40.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:39 smithi002 ceph-mon[98807]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:40.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:39 smithi188 ceph-mon[100639]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:40.297 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:39.869+0000 7f75ce7bb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T18:17:40.297 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:40.039+0000 7f75ce7bb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T18:17:40.332 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:17:40.427 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:40 smithi040 ceph-mgr[101731]: -- 172.21.15.40:0/3966436948 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5576e523b520 con 0x5576e5211000 2023-11-16T18:17:40.427 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:40 smithi040 bash[101691]: 2112007917f62265ce2924ab31cfc65a7911feac648691e3cb13deb71bef6a45 2023-11-16T18:17:40.428 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:40 smithi040 systemd[1]: Started Ceph mgr.y for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:17:40.428 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:40 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:40.216+0000 7f54b16ca200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T18:17:40.428 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:40 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:40.316+0000 7f54b16ca200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T18:17:40.428 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:40 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:40.425+0000 7f54b16ca200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T18:17:40.734 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:40 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:40.537+0000 7f54b16ca200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T18:17:41.093 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:40.825+0000 7f75ce7bb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:17:41.094 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:40.918+0000 7f75ce7bb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T18:17:41.163 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:40 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:40.750+0000 7f54b16ca200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T18:17:41.164 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:41 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:41.160+0000 7f54b16ca200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T18:17:41.393 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:41.091+0000 7f75ce7bb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:17:41.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:41 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:41.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:41 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:41.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:41 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:41.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:41 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:41.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:41 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:17:41.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:41 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:41.429 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:41 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:41.272+0000 7f54b16ca200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T18:17:41.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:41 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:41.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:41 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:41.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:41 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:41.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:41 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:41.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:41 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:17:41.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:41 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:41.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.513+0000 7f967c18b700 1 -- 172.21.15.2:0/1265260458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9674101ea0 msgr2=0x7f9674108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:41.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.513+0000 7f967c18b700 1 --2- 172.21.15.2:0/1265260458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9674101ea0 0x7f9674108760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f966c009f90 tx=0x7f966c00b330 comp rx=0 tx=0).stop 2023-11-16T18:17:41.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.513+0000 7f967c18b700 1 -- 172.21.15.2:0/1265260458 shutdown_connections 2023-11-16T18:17:41.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.513+0000 7f967c18b700 1 --2- 172.21.15.2:0/1265260458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9674101ea0 0x7f9674108760 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:41.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.513+0000 7f967c18b700 1 --2- 172.21.15.2:0/1265260458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674100310 0x7f9674100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:41.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.513+0000 7f967c18b700 1 --2- 172.21.15.2:0/1265260458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9674101510 0x7f9674101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:41.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.513+0000 7f967c18b700 1 -- 172.21.15.2:0/1265260458 >> 172.21.15.2:0/1265260458 conn(0x7f96740fb960 msgr2=0x7f96740fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:41.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.513+0000 7f967c18b700 1 -- 172.21.15.2:0/1265260458 shutdown_connections 2023-11-16T18:17:41.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.513+0000 7f967c18b700 1 -- 172.21.15.2:0/1265260458 wait complete. 2023-11-16T18:17:41.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.514+0000 7f967c18b700 1 Processor -- start 2023-11-16T18:17:41.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.514+0000 7f967c18b700 1 -- start start 2023-11-16T18:17:41.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.514+0000 7f967c18b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9674100310 0x7f9674196100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:41.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.514+0000 7f967c18b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9674101510 0x7f9674196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:41.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.514+0000 7f967c18b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674101ea0 0x7f967419d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:41.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.514+0000 7f967c18b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967419dc90 con 0x7f9674101510 2023-11-16T18:17:41.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.514+0000 7f967c18b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967419ddd0 con 0x7f9674101ea0 2023-11-16T18:17:41.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.514+0000 7f967c18b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967419df60 con 0x7f9674100310 2023-11-16T18:17:41.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.514+0000 7f9679726700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9674101510 0x7f9674196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:41.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.514+0000 7f9679f27700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9674100310 0x7f9674196100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:41.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.514+0000 7f9679726700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9674101510 0x7f9674196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46918/0 (socket says 172.21.15.2:46918) 2023-11-16T18:17:41.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.514+0000 7f9679726700 1 -- 172.21.15.2:0/488074527 learned_addr learned my addr 172.21.15.2:0/488074527 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:17:41.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.514+0000 7f967a728700 1 --2- 172.21.15.2:0/488074527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674101ea0 0x7f967419d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:41.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.515+0000 7f9679726700 1 -- 172.21.15.2:0/488074527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9674100310 msgr2=0x7f9674196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:41.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.515+0000 7f9679726700 1 --2- 172.21.15.2:0/488074527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9674100310 0x7f9674196100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:41.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.515+0000 7f9679726700 1 -- 172.21.15.2:0/488074527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674101ea0 msgr2=0x7f967419d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:41.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.515+0000 7f9679726700 1 --2- 172.21.15.2:0/488074527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674101ea0 0x7f967419d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:41.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.515+0000 7f9679726700 1 -- 172.21.15.2:0/488074527 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c009c40 con 0x7f9674101510 2023-11-16T18:17:41.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.515+0000 7f9679726700 1 --2- 172.21.15.2:0/488074527 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9674101510 0x7f9674196640 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f967000e9d0 tx=0x7f967000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:41.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.515+0000 7f966affd700 1 -- 172.21.15.2:0/488074527 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670009790 con 0x7f9674101510 2023-11-16T18:17:41.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.515+0000 7f966affd700 1 -- 172.21.15.2:0/488074527 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f967000ee60 con 0x7f9674101510 2023-11-16T18:17:41.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.515+0000 7f967c18b700 1 -- 172.21.15.2:0/488074527 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967410bc70 con 0x7f9674101510 2023-11-16T18:17:41.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.515+0000 7f966affd700 1 -- 172.21.15.2:0/488074527 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670018680 con 0x7f9674101510 2023-11-16T18:17:41.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.515+0000 7f967c18b700 1 -- 172.21.15.2:0/488074527 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967410bae0 con 0x7f9674101510 2023-11-16T18:17:41.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.516+0000 7f966affd700 1 -- 172.21.15.2:0/488074527 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f9670010430 con 0x7f9674101510 2023-11-16T18:17:41.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.516+0000 7f967c18b700 1 -- 172.21.15.2:0/488074527 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9674107810 con 0x7f9674101510 2023-11-16T18:17:41.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.516+0000 7f966affd700 1 --2- 172.21.15.2:0/488074527 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f96600389e0 0x7f966003ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:17:41.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.516+0000 7f966affd700 1 -- 172.21.15.2:0/488074527 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9670014070 con 0x7f9674101510 2023-11-16T18:17:41.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.519+0000 7f9679f27700 1 --2- 172.21.15.2:0/488074527 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f96600389e0 0x7f966003ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:17:41.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.519+0000 7f9679f27700 1 --2- 172.21.15.2:0/488074527 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f96600389e0 0x7f966003ae90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9674101370 tx=0x7f9664009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:17:41.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.521+0000 7f966affd700 1 -- 172.21.15.2:0/488074527 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f96700188d0 con 0x7f9674101510 2023-11-16T18:17:41.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:41 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:41.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:41 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:41.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:41 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:41.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:41 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:41.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:41 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:41.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:41 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:41.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:41.727+0000 7f967c18b700 1 -- 172.21.15.2:0/488074527 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9674061cb0 con 0x7f96600389e0 2023-11-16T18:17:41.733 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:41 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:41.427+0000 7f54b16ca200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T18:17:41.733 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:41 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:41.547+0000 7f54b16ca200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T18:17:41.793 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:41.389+0000 7f75ce7bb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T18:17:41.793 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:41.631+0000 7f75ce7bb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T18:17:42.046 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:41.790+0000 7f75ce7bb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T18:17:42.486 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:42.074+0000 7f54b16ca200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T18:17:42.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-mon[100251]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:17:42.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:17:42.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:17:42.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:42.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-mon[100251]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:42.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:42.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:42.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.546 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:42.058+0000 7f75ce7bb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T18:17:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-mon[100639]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:17:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:17:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:17:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-mon[100639]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:42.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:42.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:42 smithi002 ceph-mon[98807]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:17:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:17:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:17:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:42 smithi002 ceph-mon[98807]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:42.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:43.046 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:42.761+0000 7f75ce7bb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T18:17:43.047 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:42.890+0000 7f75ce7bb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T18:17:43.233 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:42.771+0000 7f54b16ca200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T18:17:43.233 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:42 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:42.962+0000 7f54b16ca200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T18:17:43.287 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:43 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:43.287 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:43 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:17:43.288 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:43 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:43.288 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:43 smithi002 ceph-mon[98807]: Reconfiguring mgr.x (unknown last config time)... 2023-11-16T18:17:43.288 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:43 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T18:17:43.288 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:43 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T18:17:43.288 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:43 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:43.288 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:43 smithi002 ceph-mon[98807]: Reconfiguring daemon mgr.x on smithi002 2023-11-16T18:17:43.288 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:43 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:43.403 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:43.077+0000 7f75ce7bb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T18:17:43.403 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:43.188+0000 7f75ce7bb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T18:17:43.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:43 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:17:43.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:43 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:17:43.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:43 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:17:43.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:43 smithi188 ceph-mon[100639]: Reconfiguring mgr.x (unknown last config time)... 2023-11-16T18:17:43.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:43 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T18:17:43.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:43 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T18:17:43.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:43 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:43.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:43 smithi188 ceph-mon[100639]: Reconfiguring daemon mgr.x on smithi002 2023-11-16T18:17:43.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:43 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:43.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:43 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:17:43.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:43 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:17:43.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:43 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:17:43.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:43 smithi040 ceph-mon[100251]: Reconfiguring mgr.x (unknown last config time)... 2023-11-16T18:17:43.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:43 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T18:17:43.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:43 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T18:17:43.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:43 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:43.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:43 smithi040 ceph-mon[100251]: Reconfiguring daemon mgr.x on smithi002 2023-11-16T18:17:43.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:43 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:44.233 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:43 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:43.920+0000 7f54b16ca200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:17:44.233 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:44.030+0000 7f54b16ca200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T18:17:44.546 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:43.401+0000 7f75ce7bb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T18:17:44.547 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:17:44.250+0000 7f75ce7bb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T18:17:44.596 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:44.249+0000 7f54b16ca200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:17:44.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:44.754+0000 7f966affd700 1 -- 172.21.15.2:0/488074527 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 14) v1 ==== 90598+0+0 (secure 0 0 0) 0x7f967000acf0 con 0x7f9674101510 2023-11-16T18:17:44.849 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:44.594+0000 7f54b16ca200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T18:17:44.849 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:44.849 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:44.849 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:44.849 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:44.849 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:44.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:44.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:17:44.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:44.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:17:44.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:17:44.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: Standby manager daemon z started 2023-11-16T18:17:44.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.188:0/3437015850' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T18:17:44.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.188:0/3437015850' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T18:17:44.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.188:0/3437015850' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T18:17:44.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.188:0/3437015850' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T18:17:45.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:45.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:45.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:45.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:45.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:17:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:17:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:17:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: Standby manager daemon z started 2023-11-16T18:17:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.188:0/3437015850' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T18:17:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.188:0/3437015850' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T18:17:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.188:0/3437015850' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T18:17:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:44 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.188:0/3437015850' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T18:17:45.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:45.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:45.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:45.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:45.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:45.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:45.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:45.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:45.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:17:45.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:45.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: Standby manager daemon z started 2023-11-16T18:17:45.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.188:0/3437015850' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T18:17:45.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.188:0/3437015850' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T18:17:45.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.188:0/3437015850' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T18:17:45.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:44 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.188:0/3437015850' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T18:17:45.234 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:44 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:44.847+0000 7f54b16ca200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T18:17:45.234 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:45 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:45.028+0000 7f54b16ca200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T18:17:45.711 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:45 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:45.354+0000 7f54b16ca200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T18:17:45.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:45 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/590230001' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d70b5c0-08db-4d20-879d-c61212c48535"}]: dispatch 2023-11-16T18:17:45.988 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:45 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/590230001' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d70b5c0-08db-4d20-879d-c61212c48535"}]': finished 2023-11-16T18:17:45.989 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:45 smithi040 ceph-mon[100251]: osdmap e5: 1 total, 0 up, 1 in 2023-11-16T18:17:45.989 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:17:45.989 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:45 smithi040 ceph-mon[100251]: mgrmap e14: x(active, since 67s), standbys: z 2023-11-16T18:17:45.990 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T18:17:45.990 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:45 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2563662150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:17:46.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:45 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/590230001' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d70b5c0-08db-4d20-879d-c61212c48535"}]: dispatch 2023-11-16T18:17:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:45 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/590230001' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d70b5c0-08db-4d20-879d-c61212c48535"}]': finished 2023-11-16T18:17:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:45 smithi188 ceph-mon[100639]: osdmap e5: 1 total, 0 up, 1 in 2023-11-16T18:17:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:17:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:45 smithi188 ceph-mon[100639]: mgrmap e14: x(active, since 67s), standbys: z 2023-11-16T18:17:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T18:17:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:45 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2563662150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:17:46.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:45 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/590230001' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d70b5c0-08db-4d20-879d-c61212c48535"}]: dispatch 2023-11-16T18:17:46.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:45 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/590230001' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d70b5c0-08db-4d20-879d-c61212c48535"}]': finished 2023-11-16T18:17:46.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:45 smithi002 ceph-mon[98807]: osdmap e5: 1 total, 0 up, 1 in 2023-11-16T18:17:46.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:17:46.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:45 smithi002 ceph-mon[98807]: mgrmap e14: x(active, since 67s), standbys: z 2023-11-16T18:17:46.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T18:17:46.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:45 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2563662150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:17:46.483 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:46 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:46.181+0000 7f54b16ca200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T18:17:46.483 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:46 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:46.340+0000 7f54b16ca200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T18:17:46.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:46 smithi040 ceph-mon[100251]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:46.983 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:46 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:46.580+0000 7f54b16ca200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T18:17:46.983 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:46 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:46.717+0000 7f54b16ca200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T18:17:47.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:46 smithi188 ceph-mon[100639]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:47.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:46 smithi002 ceph-mon[98807]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:47.483 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:46 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:46.987+0000 7f54b16ca200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T18:17:48.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:47 smithi002 ceph-mon[98807]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:48.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:47 smithi040 ceph-mon[100251]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:48.233 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:17:48 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:17:48.033+0000 7f54b16ca200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T18:17:48.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:47 smithi188 ceph-mon[100639]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:48.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:48.823+0000 7f966affd700 1 -- 172.21.15.2:0/488074527 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7f967000fc10 con 0x7f9674101510 2023-11-16T18:17:49.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:48 smithi002 ceph-mon[98807]: Standby manager daemon y started 2023-11-16T18:17:49.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:48 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.40:0/3754157282' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T18:17:49.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:48 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.40:0/3754157282' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T18:17:49.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:48 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.40:0/3754157282' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T18:17:49.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:48 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.40:0/3754157282' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T18:17:49.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:48 smithi040 ceph-mon[100251]: Standby manager daemon y started 2023-11-16T18:17:49.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:48 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.40:0/3754157282' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T18:17:49.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:48 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.40:0/3754157282' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T18:17:49.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:48 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.40:0/3754157282' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T18:17:49.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:48 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.40:0/3754157282' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T18:17:49.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:48 smithi188 ceph-mon[100639]: Standby manager daemon y started 2023-11-16T18:17:49.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:48 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.40:0/3754157282' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T18:17:49.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:48 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.40:0/3754157282' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T18:17:49.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:48 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.40:0/3754157282' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T18:17:49.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:48 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.40:0/3754157282' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T18:17:49.965 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:49 smithi002 ceph-mon[98807]: mgrmap e15: x(active, since 71s), standbys: z, y 2023-11-16T18:17:49.965 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:49 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T18:17:49.965 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:49 smithi002 ceph-mon[98807]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:50.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:49 smithi040 ceph-mon[100251]: mgrmap e15: x(active, since 71s), standbys: z, y 2023-11-16T18:17:50.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:49 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T18:17:50.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:49 smithi040 ceph-mon[100251]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:50.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:49 smithi188 ceph-mon[100639]: mgrmap e15: x(active, since 71s), standbys: z, y 2023-11-16T18:17:50.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:49 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T18:17:50.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:49 smithi188 ceph-mon[100639]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:51.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:51 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-16T18:17:51.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:51 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:52.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:51 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-16T18:17:52.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:51 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:52.063 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:51 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-16T18:17:52.064 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:51 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:52.967 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:52 smithi002 ceph-mon[98807]: Deploying daemon osd.0 on smithi002 2023-11-16T18:17:52.967 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:52 smithi002 ceph-mon[98807]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:52.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:52 smithi040 ceph-mon[100251]: Deploying daemon osd.0 on smithi002 2023-11-16T18:17:52.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:52 smithi040 ceph-mon[100251]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:53.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:52 smithi188 ceph-mon[100639]: Deploying daemon osd.0 on smithi002 2023-11-16T18:17:53.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:52 smithi188 ceph-mon[100639]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:54.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:53 smithi002 ceph-mon[98807]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:54.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:53 smithi040 ceph-mon[100251]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:54.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:53 smithi188 ceph-mon[100639]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:56 smithi188 ceph-mon[100639]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:56.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:56.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:56.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:56.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:56 smithi002 ceph-mon[98807]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:56.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:56.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:56.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:56.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:56 smithi040 ceph-mon[100251]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:58.579+0000 7f966affd700 1 -- 172.21.15.2:0/488074527 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9674061cb0 con 0x7f96600389e0 2023-11-16T18:17:58.585 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 0 on host 'smithi002' 2023-11-16T18:17:58.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:58.582+0000 7f967c18b700 1 -- 172.21.15.2:0/488074527 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f96600389e0 msgr2=0x7f966003ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:58.582+0000 7f967c18b700 1 --2- 172.21.15.2:0/488074527 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f96600389e0 0x7f966003ae90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9674101370 tx=0x7f9664009310 comp rx=0 tx=0).stop 2023-11-16T18:17:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:58.582+0000 7f967c18b700 1 -- 172.21.15.2:0/488074527 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9674101510 msgr2=0x7f9674196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:17:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:58.582+0000 7f967c18b700 1 --2- 172.21.15.2:0/488074527 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9674101510 0x7f9674196640 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f967000e9d0 tx=0x7f967000ece0 comp rx=0 tx=0).stop 2023-11-16T18:17:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:58.582+0000 7f967c18b700 1 -- 172.21.15.2:0/488074527 shutdown_connections 2023-11-16T18:17:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:58.582+0000 7f967c18b700 1 --2- 172.21.15.2:0/488074527 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f96600389e0 0x7f966003ae90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:58.582+0000 7f967c18b700 1 --2- 172.21.15.2:0/488074527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9674100310 0x7f9674196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:58.582+0000 7f967c18b700 1 --2- 172.21.15.2:0/488074527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674101ea0 0x7f967419d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:58.582+0000 7f967c18b700 1 --2- 172.21.15.2:0/488074527 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9674101510 0x7f9674196640 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:17:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:58.582+0000 7f967c18b700 1 -- 172.21.15.2:0/488074527 >> 172.21.15.2:0/488074527 conn(0x7f96740fb960 msgr2=0x7f9674104950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:17:58.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:58.582+0000 7f967c18b700 1 -- 172.21.15.2:0/488074527 shutdown_connections 2023-11-16T18:17:58.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:17:58.582+0000 7f967c18b700 1 -- 172.21.15.2:0/488074527 wait complete. 2023-11-16T18:17:58.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:58 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:58.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:58 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:58.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:58 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:58.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:58 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:58.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:58 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:58.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:58 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:58.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:58 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:17:58.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:58 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:17:58.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:58 smithi002 ceph-mon[98807]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:58.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:58 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:58.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:58 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:58.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:58 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:58.935 DEBUG:teuthology.orchestra.run.smithi002:osd.0> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.0.service 2023-11-16T18:17:58.943 INFO:tasks.cephadm:Deploying osd.1 on smithi040 with /dev/vg_nvme/lv_4... 2023-11-16T18:17:58.944 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-16T18:17:58.968 INFO:journalctl@ceph.osd.0.smithi002.stdout:-- Logs begin at Thu 2023-11-16 17:59:05 UTC. -- 2023-11-16T18:17:58.976 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:58 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:58.977 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:58 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:58.977 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:58 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:58.977 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:58 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:58.977 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:58 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:17:58.977 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:58 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:58.977 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:58 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:17:58.977 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:58 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:17:58.977 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:58 smithi040 ceph-mon[100251]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:58.977 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:58 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:58.977 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:58 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:58.978 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:58 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:59.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:58 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:59.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:58 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:59.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:58 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:17:59.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:58 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:17:59.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:58 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:17:59.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:58 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:59.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:58 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:17:59.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:58 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:17:59.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:58 smithi188 ceph-mon[100639]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:17:59.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:58 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:59.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:58 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:17:59.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:58 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:17:59.479 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:17:59 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-0[111877]: 2023-11-16T18:17:59.208+0000 7fcef8cc1640 -1 osd.0 0 log_to_monitors true 2023-11-16T18:17:59.607 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T18:17:59.788 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:17:59 smithi002 ceph-mon[98807]: from='osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-16T18:17:59.900 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:17:59 smithi040 ceph-mon[100251]: from='osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-16T18:18:00.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:17:59 smithi188 ceph-mon[100639]: from='osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-16T18:18:00.718 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:00 smithi040 ceph-mon[100251]: from='osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-16T18:18:00.719 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:00 smithi040 ceph-mon[100251]: osdmap e6: 1 total, 0 up, 1 in 2023-11-16T18:18:00.719 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:00 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:00.719 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:00 smithi040 ceph-mon[100251]: from='osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T18:18:00.719 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:00 smithi040 ceph-mon[100251]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:18:00.955 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:18:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-0[111877]: 2023-11-16T18:18:00.654+0000 7fceee338700 -1 osd.0 0 waiting for initial osdmap 2023-11-16T18:18:00.955 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:18:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-0[111877]: 2023-11-16T18:18:00.657+0000 7fcee7127700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T18:18:00.955 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:00 smithi002 ceph-mon[98807]: from='osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-16T18:18:00.955 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:00 smithi002 ceph-mon[98807]: osdmap e6: 1 total, 0 up, 1 in 2023-11-16T18:18:00.955 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:00 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:00.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:00 smithi002 ceph-mon[98807]: from='osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T18:18:00.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:00 smithi002 ceph-mon[98807]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:18:01.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:00 smithi188 ceph-mon[100639]: from='osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-16T18:18:01.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:00 smithi188 ceph-mon[100639]: osdmap e6: 1 total, 0 up, 1 in 2023-11-16T18:18:01.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:00 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:01.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:00 smithi188 ceph-mon[100639]: from='osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T18:18:01.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:00 smithi188 ceph-mon[100639]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:18:01.232 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-16T18:18:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-16T18:18:01.233 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2023-11-16T18:18:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2023-11-16T18:18:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0147112 s, 713 MB/s 2023-11-16T18:18:01.233 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 2023-11-16T18:18:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2023-11-16T18:18:01.752 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi040:vg_nvme/lv_4 2023-11-16T18:18:01.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:01 smithi040 ceph-mon[100251]: from='osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T18:18:01.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:01 smithi040 ceph-mon[100251]: osdmap e7: 1 total, 0 up, 1 in 2023-11-16T18:18:01.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:01.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:01.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:01.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:01.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:18:01.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:01.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:01.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:01.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:02.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:01 smithi188 ceph-mon[100639]: from='osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T18:18:02.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:01 smithi188 ceph-mon[100639]: osdmap e7: 1 total, 0 up, 1 in 2023-11-16T18:18:02.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:02.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:02.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:02.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:02.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:18:02.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:02.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:02.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:02.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:02.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:01 smithi002 ceph-mon[98807]: from='osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T18:18:02.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:01 smithi002 ceph-mon[98807]: osdmap e7: 1 total, 0 up, 1 in 2023-11-16T18:18:02.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:02.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:02.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:02.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:02.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:18:02.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:02.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:02.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:02.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:02.443 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T18:18:02.928 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:02 smithi040 ceph-mon[100251]: purged_snaps scrub starts 2023-11-16T18:18:02.928 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:02 smithi040 ceph-mon[100251]: purged_snaps scrub ok 2023-11-16T18:18:02.928 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:02 smithi040 ceph-mon[100251]: Detected new or changed devices on smithi002 2023-11-16T18:18:02.929 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:02 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi002 to 17173M 2023-11-16T18:18:02.929 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:02 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:18:02.929 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:02 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:18:02.929 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:02 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:18:02.929 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:02 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:02.929 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:02 smithi040 ceph-mon[100251]: osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884] boot 2023-11-16T18:18:02.929 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:02 smithi040 ceph-mon[100251]: osdmap e8: 1 total, 1 up, 1 in 2023-11-16T18:18:02.929 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:02 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:02.929 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:02 smithi040 ceph-mon[100251]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:18:03.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:02 smithi188 ceph-mon[100639]: purged_snaps scrub starts 2023-11-16T18:18:03.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:02 smithi188 ceph-mon[100639]: purged_snaps scrub ok 2023-11-16T18:18:03.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:02 smithi188 ceph-mon[100639]: Detected new or changed devices on smithi002 2023-11-16T18:18:03.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:02 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi002 to 17173M 2023-11-16T18:18:03.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:02 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:18:03.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:02 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:18:03.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:02 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:18:03.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:02 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:03.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:02 smithi188 ceph-mon[100639]: osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884] boot 2023-11-16T18:18:03.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:02 smithi188 ceph-mon[100639]: osdmap e8: 1 total, 1 up, 1 in 2023-11-16T18:18:03.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:02 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:03.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:02 smithi188 ceph-mon[100639]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:18:03.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:02 smithi002 ceph-mon[98807]: purged_snaps scrub starts 2023-11-16T18:18:03.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:02 smithi002 ceph-mon[98807]: purged_snaps scrub ok 2023-11-16T18:18:03.111 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:02 smithi002 ceph-mon[98807]: Detected new or changed devices on smithi002 2023-11-16T18:18:03.111 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:02 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi002 to 17173M 2023-11-16T18:18:03.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:02 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:18:03.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:02 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:18:03.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:02 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:18:03.114 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:03.114 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:02 smithi002 ceph-mon[98807]: osd.0 [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884] boot 2023-11-16T18:18:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:02 smithi002 ceph-mon[98807]: osdmap e8: 1 total, 1 up, 1 in 2023-11-16T18:18:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:18:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:02 smithi002 ceph-mon[98807]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T18:18:03.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.678+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/115998292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c14100d70 msgr2=0x7f8c14103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:18:03.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.678+0000 7f8c1ab8c700 1 --2- 172.21.15.40:0/115998292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c14100d70 0x7f8c14103150 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400b0a0 tx=0x7f8c0400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:18:03.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.678+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/115998292 shutdown_connections 2023-11-16T18:18:03.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.678+0000 7f8c1ab8c700 1 --2- 172.21.15.40:0/115998292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c14105fb0 0x7f8c14108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:03.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.678+0000 7f8c1ab8c700 1 --2- 172.21.15.40:0/115998292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c14100d70 0x7f8c14103150 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:03.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.678+0000 7f8c1ab8c700 1 --2- 172.21.15.40:0/115998292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c14103690 0x7f8c14105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:03.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.678+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/115998292 >> 172.21.15.40:0/115998292 conn(0x7f8c140fa7f0 msgr2=0x7f8c140fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:18:03.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.678+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/115998292 shutdown_connections 2023-11-16T18:18:03.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.678+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/115998292 wait complete. 2023-11-16T18:18:03.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.679+0000 7f8c1ab8c700 1 Processor -- start 2023-11-16T18:18:03.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c1ab8c700 1 -- start start 2023-11-16T18:18:03.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c1ab8c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c14100d70 0x7f8c14196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:18:03.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c1ab8c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c14103690 0x7f8c14196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:18:03.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c1ab8c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c14105fb0 0x7f8c1419d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:18:03.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c1ab8c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1419dcc0 con 0x7f8c14103690 2023-11-16T18:18:03.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c1ab8c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1419de00 con 0x7f8c14105fb0 2023-11-16T18:18:03.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c1ab8c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1419df90 con 0x7f8c14100d70 2023-11-16T18:18:03.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c19129700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c14105fb0 0x7f8c1419d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:18:03.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c19129700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c14105fb0 0x7f8c1419d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48762/0 (socket says 172.21.15.40:48762) 2023-11-16T18:18:03.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c19129700 1 -- 172.21.15.40:0/621745704 learned_addr learned my addr 172.21.15.40:0/621745704 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-11-16T18:18:03.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c18928700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c14100d70 0x7f8c14196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:18:03.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c13fff700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c14103690 0x7f8c14196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:18:03.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c19129700 1 -- 172.21.15.40:0/621745704 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c14100d70 msgr2=0x7f8c14196130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:18:03.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c19129700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c14100d70 0x7f8c14196130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:03.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c19129700 1 -- 172.21.15.40:0/621745704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c14103690 msgr2=0x7f8c14196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:18:03.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c19129700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c14103690 0x7f8c14196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:03.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c19129700 1 -- 172.21.15.40:0/621745704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c04009d00 con 0x7f8c14105fb0 2023-11-16T18:18:03.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.680+0000 7f8c13fff700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c14103690 0x7f8c14196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:18:03.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.681+0000 7f8c19129700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c14105fb0 0x7f8c1419d6f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c00d950 tx=0x7f8c0c00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:18:03.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.681+0000 7f8c11ffb700 1 -- 172.21.15.40:0/621745704 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c004a30 con 0x7f8c14105fb0 2023-11-16T18:18:03.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.681+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/621745704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1410bc80 con 0x7f8c14105fb0 2023-11-16T18:18:03.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.681+0000 7f8c11ffb700 1 -- 172.21.15.40:0/621745704 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c0c00b5c0 con 0x7f8c14105fb0 2023-11-16T18:18:03.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.681+0000 7f8c11ffb700 1 -- 172.21.15.40:0/621745704 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c009ab0 con 0x7f8c14105fb0 2023-11-16T18:18:03.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.681+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/621745704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1410baf0 con 0x7f8c14105fb0 2023-11-16T18:18:03.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.682+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/621745704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c140fc360 con 0x7f8c14105fb0 2023-11-16T18:18:03.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.683+0000 7f8c11ffb700 1 -- 172.21.15.40:0/621745704 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7f8c0c009c50 con 0x7f8c14105fb0 2023-11-16T18:18:03.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.683+0000 7f8c11ffb700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8bfc0a1140 0x7f8bfc0a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:18:03.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.683+0000 7f8c11ffb700 1 -- 172.21.15.40:0/621745704 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f8c0c0ca640 con 0x7f8c14105fb0 2023-11-16T18:18:03.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.684+0000 7f8c18928700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8bfc0a1140 0x7f8bfc0a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:18:03.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.684+0000 7f8c18928700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8bfc0a1140 0x7f8bfc0a35f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8c04007ab0 tx=0x7f8c0400b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:18:03.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.688+0000 7f8c11ffb700 1 -- 172.21.15.40:0/621745704 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8c0c095800 con 0x7f8c14105fb0 2023-11-16T18:18:03.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:03 smithi040 ceph-mon[100251]: osdmap e9: 1 total, 1 up, 1 in 2023-11-16T18:18:03.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:03.892+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/621745704 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c14061be0 con 0x7f8bfc0a1140 2023-11-16T18:18:04.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:03 smithi188 ceph-mon[100639]: osdmap e9: 1 total, 1 up, 1 in 2023-11-16T18:18:04.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:03 smithi002 ceph-mon[98807]: osdmap e9: 1 total, 1 up, 1 in 2023-11-16T18:18:04.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:04 smithi040 ceph-mon[100251]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:04.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:04 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:18:04.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:04 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:18:04.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:04 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:05.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:04 smithi188 ceph-mon[100639]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:05.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:04 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:18:05.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:04 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:18:05.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:04 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:05.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:04 smithi002 ceph-mon[98807]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:05.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:18:05.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:18:05.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:05.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:05 smithi040 ceph-mon[100251]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:18:06.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:05 smithi188 ceph-mon[100639]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:18:06.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:05 smithi002 ceph-mon[98807]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:18:06.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:06 smithi040 ceph-mon[100251]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:06.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:06 smithi040 ceph-mon[100251]: from='client.? 172.21.15.40:0/98433405' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69bb34ad-0d6c-4e75-911b-e366bb104d4c"}]: dispatch 2023-11-16T18:18:06.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:06 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69bb34ad-0d6c-4e75-911b-e366bb104d4c"}]: dispatch 2023-11-16T18:18:06.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:06 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69bb34ad-0d6c-4e75-911b-e366bb104d4c"}]': finished 2023-11-16T18:18:06.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:06 smithi040 ceph-mon[100251]: osdmap e10: 2 total, 1 up, 2 in 2023-11-16T18:18:06.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:06 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:07.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:06 smithi188 ceph-mon[100639]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:07.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:06 smithi188 ceph-mon[100639]: from='client.? 172.21.15.40:0/98433405' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69bb34ad-0d6c-4e75-911b-e366bb104d4c"}]: dispatch 2023-11-16T18:18:07.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:06 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69bb34ad-0d6c-4e75-911b-e366bb104d4c"}]: dispatch 2023-11-16T18:18:07.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:06 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69bb34ad-0d6c-4e75-911b-e366bb104d4c"}]': finished 2023-11-16T18:18:07.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:06 smithi188 ceph-mon[100639]: osdmap e10: 2 total, 1 up, 2 in 2023-11-16T18:18:07.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:06 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:07.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:06 smithi002 ceph-mon[98807]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:07.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:06 smithi002 ceph-mon[98807]: from='client.? 172.21.15.40:0/98433405' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69bb34ad-0d6c-4e75-911b-e366bb104d4c"}]: dispatch 2023-11-16T18:18:07.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:06 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69bb34ad-0d6c-4e75-911b-e366bb104d4c"}]: dispatch 2023-11-16T18:18:07.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:06 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69bb34ad-0d6c-4e75-911b-e366bb104d4c"}]': finished 2023-11-16T18:18:07.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:06 smithi002 ceph-mon[98807]: osdmap e10: 2 total, 1 up, 2 in 2023-11-16T18:18:07.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:06 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:07.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:07 smithi040 ceph-mon[100251]: from='client.? 172.21.15.40:0/803330768' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:18:08.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:07 smithi188 ceph-mon[100639]: from='client.? 172.21.15.40:0/803330768' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:18:08.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:07 smithi002 ceph-mon[98807]: from='client.? 172.21.15.40:0/803330768' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:18:08.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:08 smithi040 ceph-mon[100251]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:09.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:08 smithi188 ceph-mon[100639]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:09.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:08 smithi002 ceph-mon[98807]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:10.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:09 smithi002 ceph-mon[98807]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:10.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:09 smithi040 ceph-mon[100251]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:10.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:09 smithi188 ceph-mon[100639]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:12.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:11 smithi002 ceph-mon[98807]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:12.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:11 smithi040 ceph-mon[100251]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:12.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:11 smithi188 ceph-mon[100639]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:13.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:13 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-16T18:18:13.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:13 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:13.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:13 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-16T18:18:13.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:13 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:13.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:13 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-16T18:18:13.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:13 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:14 smithi040 ceph-mon[100251]: Deploying daemon osd.1 on smithi040 2023-11-16T18:18:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:14 smithi040 ceph-mon[100251]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:14.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:14 smithi188 ceph-mon[100639]: Deploying daemon osd.1 on smithi040 2023-11-16T18:18:14.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:14 smithi188 ceph-mon[100639]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:14.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:14 smithi002 ceph-mon[98807]: Deploying daemon osd.1 on smithi040 2023-11-16T18:18:14.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:14 smithi002 ceph-mon[98807]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:16.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:16 smithi040 ceph-mon[100251]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:17.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:16 smithi188 ceph-mon[100639]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:17.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:16 smithi002 ceph-mon[98807]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:18.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:18 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:18.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:18 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:18:18.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:18 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:18.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:18 smithi040 ceph-mon[100251]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:19.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:18 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:19.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:18 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:18:19.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:18 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:19.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:18 smithi188 ceph-mon[100639]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:19.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:18 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:19.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:18 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:18:19.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:18 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:19.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:18 smithi002 ceph-mon[98807]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:20.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:19 smithi002 ceph-mon[98807]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:19 smithi040 ceph-mon[100251]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:20.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:19 smithi188 ceph-mon[100639]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:20.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:20.914+0000 7f8c11ffb700 1 -- 172.21.15.40:0/621745704 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8c14061be0 con 0x7f8bfc0a1140 2023-11-16T18:18:20.917 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 1 on host 'smithi040' 2023-11-16T18:18:20.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:20.916+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/621745704 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8bfc0a1140 msgr2=0x7f8bfc0a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:18:20.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:20.916+0000 7f8c1ab8c700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8bfc0a1140 0x7f8bfc0a35f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8c04007ab0 tx=0x7f8c0400b7b0 comp rx=0 tx=0).stop 2023-11-16T18:18:20.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:20.916+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/621745704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c14105fb0 msgr2=0x7f8c1419d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:18:20.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:20.916+0000 7f8c1ab8c700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c14105fb0 0x7f8c1419d6f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c00d950 tx=0x7f8c0c00dc60 comp rx=0 tx=0).stop 2023-11-16T18:18:20.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:20.916+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/621745704 shutdown_connections 2023-11-16T18:18:20.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:20.916+0000 7f8c1ab8c700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8bfc0a1140 0x7f8bfc0a35f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:20.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:20.916+0000 7f8c1ab8c700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c14100d70 0x7f8c14196130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:20.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:20.916+0000 7f8c1ab8c700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c14105fb0 0x7f8c1419d6f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:20.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:20.916+0000 7f8c1ab8c700 1 --2- 172.21.15.40:0/621745704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c14103690 0x7f8c14196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:20.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:20.916+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/621745704 >> 172.21.15.40:0/621745704 conn(0x7f8c140fa7f0 msgr2=0x7f8c141019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:18:20.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:20.916+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/621745704 shutdown_connections 2023-11-16T18:18:20.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:18:20.916+0000 7f8c1ab8c700 1 -- 172.21.15.40:0/621745704 wait complete. 2023-11-16T18:18:21.050 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:20 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.050 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:20 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.050 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:20 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:21.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:20 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:21.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:20 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:18:21.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:20 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:20 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:18:21.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:20 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:18:21.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:20 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:20 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:20 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:18:21.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:21.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:21.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:20 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:18:21.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:20 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:18:21.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:20 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:18:21.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:18:21.331 DEBUG:teuthology.orchestra.run.smithi040:osd.1> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.1.service 2023-11-16T18:18:21.334 INFO:tasks.cephadm:Deploying osd.2 on smithi188 with /dev/vg_nvme/lv_4... 2023-11-16T18:18:21.334 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-16T18:18:21.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:21.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:21.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:20 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:18:21.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:20 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:18:21.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:20 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:18:21.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:21.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:18:21.366 INFO:journalctl@ceph.osd.1.smithi040.stdout:-- Logs begin at Thu 2023-11-16 17:59:26 UTC. -- 2023-11-16T18:18:21.834 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:18:21 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-1[105211]: 2023-11-16T18:18:21.577+0000 7f7c97129640 -1 osd.1 0 log_to_monitors true 2023-11-16T18:18:21.983 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:18:22.132 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:21 smithi040 ceph-mon[100251]: from='osd.1 [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T18:18:22.133 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:21 smithi040 ceph-mon[100251]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T18:18:22.133 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:21 smithi040 ceph-mon[100251]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:22.192 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:21 smithi188 ceph-mon[100639]: from='osd.1 [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T18:18:22.192 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:21 smithi188 ceph-mon[100639]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T18:18:22.192 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:21 smithi188 ceph-mon[100639]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:22.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:21 smithi002 ceph-mon[98807]: from='osd.1 [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T18:18:22.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:21 smithi002 ceph-mon[98807]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T18:18:22.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:21 smithi002 ceph-mon[98807]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:23.233 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:18:22 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-1[105211]: 2023-11-16T18:18:22.935+0000 7f7c8c7a0700 -1 osd.1 0 waiting for initial osdmap 2023-11-16T18:18:23.234 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:18:22 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-1[105211]: 2023-11-16T18:18:22.939+0000 7f7c8558f700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T18:18:23.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:22 smithi040 ceph-mon[100251]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-16T18:18:23.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:22 smithi040 ceph-mon[100251]: osdmap e11: 2 total, 1 up, 2 in 2023-11-16T18:18:23.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:22 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:23.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:22 smithi040 ceph-mon[100251]: from='osd.1 [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:18:23.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:22 smithi040 ceph-mon[100251]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:18:23.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:22 smithi188 ceph-mon[100639]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-16T18:18:23.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:22 smithi188 ceph-mon[100639]: osdmap e11: 2 total, 1 up, 2 in 2023-11-16T18:18:23.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:22 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:23.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:22 smithi188 ceph-mon[100639]: from='osd.1 [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:18:23.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:22 smithi188 ceph-mon[100639]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:18:23.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:22 smithi002 ceph-mon[98807]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-16T18:18:23.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:22 smithi002 ceph-mon[98807]: osdmap e11: 2 total, 1 up, 2 in 2023-11-16T18:18:23.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:22 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:23.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:22 smithi002 ceph-mon[98807]: from='osd.1 [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:18:23.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:22 smithi002 ceph-mon[98807]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:18:23.442 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-16T18:18:23.443 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-16T18:18:23.443 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2023-11-16T18:18:23.443 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2023-11-16T18:18:23.443 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114787 s, 913 MB/s 2023-11-16T18:18:23.443 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2023-11-16T18:18:23.835 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi188:vg_nvme/lv_4 2023-11-16T18:18:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-11-16T18:18:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: osdmap e12: 2 total, 1 up, 2 in 2023-11-16T18:18:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:24.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:24.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: Detected new or changed devices on smithi040 2023-11-16T18:18:24.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:24.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:24.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:18:24.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi040 to 17173M 2023-11-16T18:18:24.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:24.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:24.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:24.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:18:24.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:24.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:18:24.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:18:24.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:23 smithi040 ceph-mon[100251]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:24.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-11-16T18:18:24.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: osdmap e12: 2 total, 1 up, 2 in 2023-11-16T18:18:24.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:24.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:24.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: Detected new or changed devices on smithi040 2023-11-16T18:18:24.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:24.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:24.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:18:24.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi040 to 17173M 2023-11-16T18:18:24.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:24.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:24.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:24.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:18:24.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:24.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:18:24.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:18:24.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:23 smithi188 ceph-mon[100639]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:24.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-11-16T18:18:24.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: osdmap e12: 2 total, 1 up, 2 in 2023-11-16T18:18:24.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:24.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:24.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: Detected new or changed devices on smithi040 2023-11-16T18:18:24.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:24.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:24.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:18:24.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi040 to 17173M 2023-11-16T18:18:24.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:24.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:24.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:24.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:18:24.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:24.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:18:24.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:18:24.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:23 smithi002 ceph-mon[98807]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T18:18:24.477 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:18:25.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:24 smithi188 ceph-mon[100639]: purged_snaps scrub starts 2023-11-16T18:18:25.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:24 smithi188 ceph-mon[100639]: purged_snaps scrub ok 2023-11-16T18:18:25.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:24 smithi188 ceph-mon[100639]: osd.1 [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247] boot 2023-11-16T18:18:25.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:24 smithi188 ceph-mon[100639]: osdmap e13: 2 total, 2 up, 2 in 2023-11-16T18:18:25.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:24 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:25.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:24 smithi040 ceph-mon[100251]: purged_snaps scrub starts 2023-11-16T18:18:25.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:24 smithi040 ceph-mon[100251]: purged_snaps scrub ok 2023-11-16T18:18:25.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:24 smithi040 ceph-mon[100251]: osd.1 [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247] boot 2023-11-16T18:18:25.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:24 smithi040 ceph-mon[100251]: osdmap e13: 2 total, 2 up, 2 in 2023-11-16T18:18:25.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:24 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:25.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:24 smithi002 ceph-mon[98807]: purged_snaps scrub starts 2023-11-16T18:18:25.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:24 smithi002 ceph-mon[98807]: purged_snaps scrub ok 2023-11-16T18:18:25.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:24 smithi002 ceph-mon[98807]: osd.1 [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247] boot 2023-11-16T18:18:25.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:24 smithi002 ceph-mon[98807]: osdmap e13: 2 total, 2 up, 2 in 2023-11-16T18:18:25.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:24 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:18:25.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.647+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/760596478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1cc100db0 msgr2=0x7fd1cc103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:18:25.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.647+0000 7fd1d1c3a700 1 --2- 172.21.15.188:0/760596478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1cc100db0 0x7fd1cc103190 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc009990 tx=0x7fd1bc009ca0 comp rx=0 tx=0).stop 2023-11-16T18:18:25.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.647+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/760596478 shutdown_connections 2023-11-16T18:18:25.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.647+0000 7fd1d1c3a700 1 --2- 172.21.15.188:0/760596478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1cc100db0 0x7fd1cc103190 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:25.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.647+0000 7fd1d1c3a700 1 --2- 172.21.15.188:0/760596478 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1cc105ff0 0x7fd1cc108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:25.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.647+0000 7fd1d1c3a700 1 --2- 172.21.15.188:0/760596478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc1036d0 0x7fd1cc105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:25.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.647+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/760596478 >> 172.21.15.188:0/760596478 conn(0x7fd1cc0fa7f0 msgr2=0x7fd1cc0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:18:25.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.647+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/760596478 shutdown_connections 2023-11-16T18:18:25.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.647+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/760596478 wait complete. 2023-11-16T18:18:25.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.648+0000 7fd1d1c3a700 1 Processor -- start 2023-11-16T18:18:25.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.648+0000 7fd1d1c3a700 1 -- start start 2023-11-16T18:18:25.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.648+0000 7fd1d1c3a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1cc100db0 0x7fd1cc100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:18:25.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.648+0000 7fd1d1c3a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc1036d0 0x7fd1cc0fed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:18:25.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.648+0000 7fd1d1c3a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1cc105ff0 0x7fd1cc0ff2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:18:25.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.648+0000 7fd1d1c3a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc0ffa20 con 0x7fd1cc100db0 2023-11-16T18:18:25.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.648+0000 7fd1d1c3a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc0ffb90 con 0x7fd1cc1036d0 2023-11-16T18:18:25.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.648+0000 7fd1d1c3a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc19ffc0 con 0x7fd1cc105ff0 2023-11-16T18:18:25.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.648+0000 7fd1cbfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1cc105ff0 0x7fd1cc0ff2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:18:25.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.648+0000 7fd1cbfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1cc105ff0 0x7fd1cc0ff2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:48068/0 (socket says 172.21.15.188:48068) 2023-11-16T18:18:25.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.648+0000 7fd1cbfff700 1 -- 172.21.15.188:0/2761590032 learned_addr learned my addr 172.21.15.188:0/2761590032 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-16T18:18:25.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.649+0000 7fd1caffd700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc1036d0 0x7fd1cc0fed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:18:25.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.649+0000 7fd1cb7fe700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1cc100db0 0x7fd1cc100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:18:25.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.649+0000 7fd1cbfff700 1 -- 172.21.15.188:0/2761590032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc1036d0 msgr2=0x7fd1cc0fed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:18:25.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.649+0000 7fd1cbfff700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc1036d0 0x7fd1cc0fed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:25.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.649+0000 7fd1cbfff700 1 -- 172.21.15.188:0/2761590032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1cc100db0 msgr2=0x7fd1cc100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:18:25.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.649+0000 7fd1cbfff700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1cc100db0 0x7fd1cc100710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:25.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.649+0000 7fd1cbfff700 1 -- 172.21.15.188:0/2761590032 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc0095f0 con 0x7fd1cc105ff0 2023-11-16T18:18:25.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.649+0000 7fd1cbfff700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1cc105ff0 0x7fd1cc0ff2a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40094f0 tx=0x7fd1c400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:18:25.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.649+0000 7fd1c8ff9700 1 -- 172.21.15.188:0/2761590032 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4004db0 con 0x7fd1cc105ff0 2023-11-16T18:18:25.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.649+0000 7fd1c8ff9700 1 -- 172.21.15.188:0/2761590032 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1c400f040 con 0x7fd1cc105ff0 2023-11-16T18:18:25.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.649+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/2761590032 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1cc10bc80 con 0x7fd1cc105ff0 2023-11-16T18:18:25.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.649+0000 7fd1c8ff9700 1 -- 172.21.15.188:0/2761590032 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4005980 con 0x7fd1cc105ff0 2023-11-16T18:18:25.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.649+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/2761590032 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1cc10baf0 con 0x7fd1cc105ff0 2023-11-16T18:18:25.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.650+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/2761590032 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1cc0fc360 con 0x7fd1cc105ff0 2023-11-16T18:18:25.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.651+0000 7fd1c8ff9700 1 -- 172.21.15.188:0/2761590032 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7fd1c4005000 con 0x7fd1cc105ff0 2023-11-16T18:18:25.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.652+0000 7fd1c8ff9700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd1b40a10c0 0x7fd1b40a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:18:25.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.652+0000 7fd1c8ff9700 1 -- 172.21.15.188:0/2761590032 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fd1c40cb220 con 0x7fd1cc105ff0 2023-11-16T18:18:25.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.652+0000 7fd1cb7fe700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd1b40a10c0 0x7fd1b40a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:18:25.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.653+0000 7fd1cb7fe700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd1b40a10c0 0x7fd1b40a3570 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00b5c0 tx=0x7fd1bc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:18:25.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.656+0000 7fd1c8ff9700 1 -- 172.21.15.188:0/2761590032 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd1c40961c0 con 0x7fd1cc105ff0 2023-11-16T18:18:25.850 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:25.849+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/2761590032 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1cc061c70 con 0x7fd1b40a10c0 2023-11-16T18:18:26.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:25 smithi040 ceph-mon[100251]: osdmap e14: 2 total, 2 up, 2 in 2023-11-16T18:18:26.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:25 smithi040 ceph-mon[100251]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:26.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:25 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:18:26.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:25 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:18:26.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:25 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:26.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:25 smithi188 ceph-mon[100639]: osdmap e14: 2 total, 2 up, 2 in 2023-11-16T18:18:26.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:25 smithi188 ceph-mon[100639]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:26.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:25 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:18:26.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:25 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:18:26.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:25 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:26.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:25 smithi002 ceph-mon[98807]: osdmap e14: 2 total, 2 up, 2 in 2023-11-16T18:18:26.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:25 smithi002 ceph-mon[98807]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:26.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:25 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:18:26.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:25 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:18:26.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:25 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:27.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:27 smithi040 ceph-mon[100251]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:18:28.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:27 smithi188 ceph-mon[100639]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:18:28.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:27 smithi002 ceph-mon[98807]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:18:28.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:28 smithi040 ceph-mon[100251]: pgmap v59: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:28.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:28 smithi040 ceph-mon[100251]: from='client.? 172.21.15.188:0/726262821' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80f9c468-7dd5-494a-a6e2-98bea69fbe4b"}]: dispatch 2023-11-16T18:18:28.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:28 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80f9c468-7dd5-494a-a6e2-98bea69fbe4b"}]: dispatch 2023-11-16T18:18:28.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:28 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80f9c468-7dd5-494a-a6e2-98bea69fbe4b"}]': finished 2023-11-16T18:18:28.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:28 smithi040 ceph-mon[100251]: osdmap e15: 3 total, 2 up, 3 in 2023-11-16T18:18:28.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:28 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:28 smithi188 ceph-mon[100639]: pgmap v59: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:28 smithi188 ceph-mon[100639]: from='client.? 172.21.15.188:0/726262821' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80f9c468-7dd5-494a-a6e2-98bea69fbe4b"}]: dispatch 2023-11-16T18:18:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:28 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80f9c468-7dd5-494a-a6e2-98bea69fbe4b"}]: dispatch 2023-11-16T18:18:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:28 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80f9c468-7dd5-494a-a6e2-98bea69fbe4b"}]': finished 2023-11-16T18:18:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:28 smithi188 ceph-mon[100639]: osdmap e15: 3 total, 2 up, 3 in 2023-11-16T18:18:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:28 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:28 smithi002 ceph-mon[98807]: pgmap v59: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:28 smithi002 ceph-mon[98807]: from='client.? 172.21.15.188:0/726262821' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80f9c468-7dd5-494a-a6e2-98bea69fbe4b"}]: dispatch 2023-11-16T18:18:29.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:28 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80f9c468-7dd5-494a-a6e2-98bea69fbe4b"}]: dispatch 2023-11-16T18:18:29.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:28 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80f9c468-7dd5-494a-a6e2-98bea69fbe4b"}]': finished 2023-11-16T18:18:29.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:28 smithi002 ceph-mon[98807]: osdmap e15: 3 total, 2 up, 3 in 2023-11-16T18:18:29.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:28 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:29.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:29 smithi040 ceph-mon[100251]: from='client.? 172.21.15.188:0/3546514928' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:18:30.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:29 smithi188 ceph-mon[100639]: from='client.? 172.21.15.188:0/3546514928' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:18:30.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:29 smithi002 ceph-mon[98807]: from='client.? 172.21.15.188:0/3546514928' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:18:30.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:30 smithi040 ceph-mon[100251]: pgmap v61: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:31.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:30 smithi188 ceph-mon[100639]: pgmap v61: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:31.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:30 smithi002 ceph-mon[98807]: pgmap v61: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:32.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:31 smithi188 ceph-mon[100639]: pgmap v62: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:32.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:31 smithi002 ceph-mon[98807]: pgmap v62: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:32.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:31 smithi040 ceph-mon[100251]: pgmap v62: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:34.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:33 smithi002 ceph-mon[98807]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:34.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:33 smithi040 ceph-mon[100251]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:34.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:33 smithi188 ceph-mon[100639]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:35.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-16T18:18:35.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-16T18:18:35.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-16T18:18:36.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:36.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:35 smithi002 ceph-mon[98807]: Deploying daemon osd.2 on smithi188 2023-11-16T18:18:36.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:35 smithi002 ceph-mon[98807]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:36.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:36.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:35 smithi188 ceph-mon[100639]: Deploying daemon osd.2 on smithi188 2023-11-16T18:18:36.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:35 smithi188 ceph-mon[100639]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:36.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:36.239 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:35 smithi040 ceph-mon[100251]: Deploying daemon osd.2 on smithi188 2023-11-16T18:18:36.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:35 smithi040 ceph-mon[100251]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:38.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:37 smithi002 ceph-mon[98807]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:38.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:37 smithi040 ceph-mon[100251]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:38.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:37 smithi188 ceph-mon[100639]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:40.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:40.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:40.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:18:40.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:40 smithi188 ceph-mon[100639]: pgmap v66: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:40.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:40.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:40.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:18:40.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:40 smithi002 ceph-mon[98807]: pgmap v66: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:40.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:40.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:40.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:18:40.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:40 smithi040 ceph-mon[100251]: pgmap v66: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:42.014 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:42.012+0000 7fd1c8ff9700 1 -- 172.21.15.188:0/2761590032 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd1cc061c70 con 0x7fd1b40a10c0 2023-11-16T18:18:42.014 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 2 on host 'smithi188' 2023-11-16T18:18:42.016 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:42.014+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/2761590032 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd1b40a10c0 msgr2=0x7fd1b40a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:18:42.016 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:42.014+0000 7fd1d1c3a700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd1b40a10c0 0x7fd1b40a3570 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00b5c0 tx=0x7fd1bc005960 comp rx=0 tx=0).stop 2023-11-16T18:18:42.016 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:42.014+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/2761590032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1cc105ff0 msgr2=0x7fd1cc0ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:18:42.016 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:42.014+0000 7fd1d1c3a700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1cc105ff0 0x7fd1cc0ff2a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40094f0 tx=0x7fd1c400d3b0 comp rx=0 tx=0).stop 2023-11-16T18:18:42.016 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:42.014+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/2761590032 shutdown_connections 2023-11-16T18:18:42.017 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:42.014+0000 7fd1d1c3a700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd1b40a10c0 0x7fd1b40a3570 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:42.017 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:42.014+0000 7fd1d1c3a700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1cc105ff0 0x7fd1cc0ff2a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:42.017 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:42.014+0000 7fd1d1c3a700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1cc100db0 0x7fd1cc100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:42.017 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:42.014+0000 7fd1d1c3a700 1 --2- 172.21.15.188:0/2761590032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc1036d0 0x7fd1cc0fed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:42.018 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:42.014+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/2761590032 >> 172.21.15.188:0/2761590032 conn(0x7fd1cc0fa7f0 msgr2=0x7fd1cc101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:18:42.018 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:42.014+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/2761590032 shutdown_connections 2023-11-16T18:18:42.018 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:18:42.014+0000 7fd1d1c3a700 1 -- 172.21.15.188:0/2761590032 wait complete. 2023-11-16T18:18:42.477 DEBUG:teuthology.orchestra.run.smithi188:osd.2> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.2.service 2023-11-16T18:18:42.487 INFO:tasks.cephadm:Deploying osd.3 on smithi002 with /dev/vg_nvme/lv_3... 2023-11-16T18:18:42.487 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-16T18:18:42.518 INFO:journalctl@ceph.osd.2.smithi188.stdout:-- Logs begin at Thu 2023-11-16 17:59:32 UTC. -- 2023-11-16T18:18:42.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:42.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:42.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:42 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:18:42.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:42 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:18:42.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:42 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:18:42.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:42 smithi188 ceph-mon[100639]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:42.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:18:42.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:42.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:42.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:42 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:18:42.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:42 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:18:42.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:42 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:18:42.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:42 smithi002 ceph-mon[98807]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:42.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:18:42.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:42.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:42.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:42 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:18:42.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:42 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:18:42.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:42 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:18:42.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:42 smithi040 ceph-mon[100251]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:42.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:42.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:18:42.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:43.139 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:18:43.549 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:18:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-2[106458]: 2023-11-16T18:18:43.296+0000 7f95e4c75640 -1 osd.2 0 log_to_monitors true 2023-11-16T18:18:43.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:43 smithi002 ceph-mon[98807]: from='osd.2 [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T18:18:43.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:43 smithi002 ceph-mon[98807]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T18:18:43.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:43 smithi040 ceph-mon[100251]: from='osd.2 [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T18:18:43.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:43 smithi040 ceph-mon[100251]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T18:18:44.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:43 smithi188 ceph-mon[100639]: from='osd.2 [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T18:18:44.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:43 smithi188 ceph-mon[100639]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T18:18:44.611 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-16T18:18:44.611 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-16T18:18:44.611 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2023-11-16T18:18:44.612 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2023-11-16T18:18:44.612 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00875219 s, 1.2 GB/s 2023-11-16T18:18:44.612 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2023-11-16T18:18:44.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:44 smithi002 ceph-mon[98807]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-16T18:18:44.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:44 smithi002 ceph-mon[98807]: osdmap e16: 3 total, 2 up, 3 in 2023-11-16T18:18:44.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:44 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:44.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:44 smithi002 ceph-mon[98807]: from='osd.2 [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:18:44.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:44 smithi002 ceph-mon[98807]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:18:44.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:44 smithi002 ceph-mon[98807]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:44.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:44 smithi040 ceph-mon[100251]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-16T18:18:44.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:44 smithi040 ceph-mon[100251]: osdmap e16: 3 total, 2 up, 3 in 2023-11-16T18:18:44.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:44 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:44.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:44 smithi040 ceph-mon[100251]: from='osd.2 [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:18:44.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:44 smithi040 ceph-mon[100251]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:18:44.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:44 smithi040 ceph-mon[100251]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:44.997 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi002:vg_nvme/lv_3 2023-11-16T18:18:45.047 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:18:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-2[106458]: 2023-11-16T18:18:44.564+0000 7f95da2ec700 -1 osd.2 0 waiting for initial osdmap 2023-11-16T18:18:45.047 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:18:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-2[106458]: 2023-11-16T18:18:44.568+0000 7f95d58e0700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T18:18:45.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:44 smithi188 ceph-mon[100639]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-16T18:18:45.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:44 smithi188 ceph-mon[100639]: osdmap e16: 3 total, 2 up, 3 in 2023-11-16T18:18:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:44 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:44 smithi188 ceph-mon[100639]: from='osd.2 [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:18:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:44 smithi188 ceph-mon[100639]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:18:45.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:44 smithi188 ceph-mon[100639]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T18:18:45.651 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:18:45.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-11-16T18:18:45.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: osdmap e17: 3 total, 2 up, 3 in 2023-11-16T18:18:45.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:45.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:45.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: Detected new or changed devices on smithi188 2023-11-16T18:18:45.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:45.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:45.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:18:45.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi188 to 17168M 2023-11-16T18:18:45.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:45.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:45.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:45.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:18:45.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:45.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:18:45.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:45 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:18:45.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-11-16T18:18:45.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: osdmap e17: 3 total, 2 up, 3 in 2023-11-16T18:18:45.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:45.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:45.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: Detected new or changed devices on smithi188 2023-11-16T18:18:45.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:45.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:45.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:18:45.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi188 to 17168M 2023-11-16T18:18:45.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:45.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:45.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:45.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:18:45.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:45.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:18:45.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:45 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:18:46.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-11-16T18:18:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: osdmap e17: 3 total, 2 up, 3 in 2023-11-16T18:18:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: Detected new or changed devices on smithi188 2023-11-16T18:18:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:18:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi188 to 17168M 2023-11-16T18:18:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:46.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:18:46.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:18:46.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:46.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:18:46.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:45 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:18:46.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:46 smithi002 ceph-mon[98807]: purged_snaps scrub starts 2023-11-16T18:18:46.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:46 smithi002 ceph-mon[98807]: purged_snaps scrub ok 2023-11-16T18:18:46.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:46 smithi002 ceph-mon[98807]: osd.2 [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401] boot 2023-11-16T18:18:46.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:46 smithi002 ceph-mon[98807]: osdmap e18: 3 total, 3 up, 3 in 2023-11-16T18:18:46.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:46 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:46.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:46 smithi002 ceph-mon[98807]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:46.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:46 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-16T18:18:46.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.913+0000 7f06791ca700 1 -- 172.21.15.2:0/3103968992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06741003e0 msgr2=0x7f0674100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:18:46.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.913+0000 7f06791ca700 1 --2- 172.21.15.2:0/3103968992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06741003e0 0x7f0674100830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0664009990 tx=0x7f0664009ca0 comp rx=0 tx=0).stop 2023-11-16T18:18:46.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.913+0000 7f06791ca700 1 -- 172.21.15.2:0/3103968992 shutdown_connections 2023-11-16T18:18:46.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.913+0000 7f06791ca700 1 --2- 172.21.15.2:0/3103968992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0674100d70 0x7f0674107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:46.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.913+0000 7f06791ca700 1 --2- 172.21.15.2:0/3103968992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06740ff1e0 0x7f06740ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:46.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.913+0000 7f06791ca700 1 --2- 172.21.15.2:0/3103968992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06741003e0 0x7f0674100830 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:46.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.913+0000 7f06791ca700 1 -- 172.21.15.2:0/3103968992 >> 172.21.15.2:0/3103968992 conn(0x7f06740fa7f0 msgr2=0x7f06740fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:18:46.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.913+0000 7f06791ca700 1 -- 172.21.15.2:0/3103968992 shutdown_connections 2023-11-16T18:18:46.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.913+0000 7f06791ca700 1 -- 172.21.15.2:0/3103968992 wait complete. 2023-11-16T18:18:46.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.914+0000 7f06791ca700 1 Processor -- start 2023-11-16T18:18:46.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.914+0000 7f06791ca700 1 -- start start 2023-11-16T18:18:46.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.914+0000 7f06791ca700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06740ff1e0 0x7f06740699c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:18:46.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.914+0000 7f06791ca700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06741003e0 0x7f067419ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:18:46.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.914+0000 7f06791ca700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0674100d70 0x7f067419d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:18:46.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.914+0000 7f06791ca700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067419db20 con 0x7f06741003e0 2023-11-16T18:18:46.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.914+0000 7f06791ca700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067419dc90 con 0x7f06740ff1e0 2023-11-16T18:18:46.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.914+0000 7f06791ca700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067419de50 con 0x7f0674100d70 2023-11-16T18:18:46.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.914+0000 7f067259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06741003e0 0x7f067419ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:18:46.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.914+0000 7f067259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06741003e0 0x7f067419ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46990/0 (socket says 172.21.15.2:46990) 2023-11-16T18:18:46.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.914+0000 7f067259c700 1 -- 172.21.15.2:0/2038977069 learned_addr learned my addr 172.21.15.2:0/2038977069 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:18:46.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.914+0000 7f0672d9d700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06740ff1e0 0x7f06740699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:18:46.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.914+0000 7f067359e700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0674100d70 0x7f067419d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:18:46.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.915+0000 7f067259c700 1 -- 172.21.15.2:0/2038977069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0674100d70 msgr2=0x7f067419d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:18:46.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.915+0000 7f067259c700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0674100d70 0x7f067419d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:46.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.915+0000 7f067259c700 1 -- 172.21.15.2:0/2038977069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06740ff1e0 msgr2=0x7f06740699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:18:46.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.915+0000 7f067259c700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06740ff1e0 0x7f06740699c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:18:46.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.915+0000 7f067259c700 1 -- 172.21.15.2:0/2038977069 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06640095f0 con 0x7f06741003e0 2023-11-16T18:18:46.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.915+0000 7f067259c700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06741003e0 0x7f067419ad30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0664000c00 tx=0x7f066400f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:18:46.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.915+0000 7f065bfff700 1 -- 172.21.15.2:0/2038977069 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066401d070 con 0x7f06741003e0 2023-11-16T18:18:46.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.915+0000 7f06791ca700 1 -- 172.21.15.2:0/2038977069 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067410ab40 con 0x7f06741003e0 2023-11-16T18:18:46.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.915+0000 7f065bfff700 1 -- 172.21.15.2:0/2038977069 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f066400fb50 con 0x7f06741003e0 2023-11-16T18:18:46.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.915+0000 7f065bfff700 1 -- 172.21.15.2:0/2038977069 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06640173f0 con 0x7f06741003e0 2023-11-16T18:18:46.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.915+0000 7f06791ca700 1 -- 172.21.15.2:0/2038977069 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067410a9b0 con 0x7f06741003e0 2023-11-16T18:18:46.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.916+0000 7f06791ca700 1 -- 172.21.15.2:0/2038977069 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06741067d0 con 0x7f06741003e0 2023-11-16T18:18:46.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.917+0000 7f065bfff700 1 -- 172.21.15.2:0/2038977069 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7f066400fcc0 con 0x7f06741003e0 2023-11-16T18:18:46.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.918+0000 7f065bfff700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f065c0a1140 0x7f065c0a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:18:46.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.918+0000 7f0672d9d700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f065c0a1140 0x7f065c0a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:18:46.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.918+0000 7f065bfff700 1 -- 172.21.15.2:0/2038977069 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7f06640cc670 con 0x7f06741003e0 2023-11-16T18:18:46.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.918+0000 7f0672d9d700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f065c0a1140 0x7f065c0a35f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0674100240 tx=0x7f066800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:18:46.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:46.922+0000 7f065bfff700 1 -- 172.21.15.2:0/2038977069 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0664097270 con 0x7f06741003e0 2023-11-16T18:18:46.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:46 smithi040 ceph-mon[100251]: purged_snaps scrub starts 2023-11-16T18:18:46.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:46 smithi040 ceph-mon[100251]: purged_snaps scrub ok 2023-11-16T18:18:46.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:46 smithi040 ceph-mon[100251]: osd.2 [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401] boot 2023-11-16T18:18:46.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:46 smithi040 ceph-mon[100251]: osdmap e18: 3 total, 3 up, 3 in 2023-11-16T18:18:46.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:46 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:46.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:46 smithi040 ceph-mon[100251]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:46.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:46 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-16T18:18:47.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:46 smithi188 ceph-mon[100639]: purged_snaps scrub starts 2023-11-16T18:18:47.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:46 smithi188 ceph-mon[100639]: purged_snaps scrub ok 2023-11-16T18:18:47.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:46 smithi188 ceph-mon[100639]: osd.2 [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401] boot 2023-11-16T18:18:47.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:46 smithi188 ceph-mon[100639]: osdmap e18: 3 total, 3 up, 3 in 2023-11-16T18:18:47.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:46 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:18:47.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:46 smithi188 ceph-mon[100639]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:47.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:46 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-16T18:18:47.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:47.126+0000 7f06791ca700 1 -- 172.21.15.2:0/2038977069 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0674060f30 con 0x7f065c0a1140 2023-11-16T18:18:47.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:47 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-16T18:18:47.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:47 smithi040 ceph-mon[100251]: osdmap e19: 3 total, 3 up, 3 in 2023-11-16T18:18:47.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:47 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T18:18:47.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:47 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:18:47.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:47 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:18:47.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:47 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:48.007 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:47 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-16T18:18:48.007 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:47 smithi002 ceph-mon[98807]: osdmap e19: 3 total, 3 up, 3 in 2023-11-16T18:18:48.007 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:47 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T18:18:48.007 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:47 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:18:48.007 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:47 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:18:48.007 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:47 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:48.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:47 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-16T18:18:48.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:47 smithi188 ceph-mon[100639]: osdmap e19: 3 total, 3 up, 3 in 2023-11-16T18:18:48.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:47 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T18:18:48.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:47 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:18:48.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:47 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:18:48.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:47 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:48.605 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:18:48 smithi002 sudo[115054]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T18:18:48.605 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:18:48 smithi002 sudo[115054]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T18:18:48.606 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:18:48 smithi002 sudo[115054]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T18:18:48.606 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:18:48 smithi002 sudo[115054]: pam_unix(sudo:session): session closed for user root 2023-11-16T18:18:48.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:18:48.734+0000 7f065bfff700 1 -- 172.21.15.2:0/2038977069 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f0664097640 con 0x7f06741003e0 2023-11-16T18:18:48.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:48 smithi040 ceph-mon[100251]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:18:48.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-16T18:18:48.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:48 smithi040 ceph-mon[100251]: osdmap e20: 3 total, 3 up, 3 in 2023-11-16T18:18:48.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:48 smithi040 ceph-mon[100251]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:49.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:48 smithi188 ceph-mon[100639]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:18:49.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-16T18:18:49.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:48 smithi188 ceph-mon[100639]: osdmap e20: 3 total, 3 up, 3 in 2023-11-16T18:18:49.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:48 smithi188 ceph-mon[100639]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:49.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:48 smithi002 ceph-mon[98807]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:18:49.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-16T18:18:49.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:48 smithi002 ceph-mon[98807]: osdmap e20: 3 total, 3 up, 3 in 2023-11-16T18:18:49.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:48 smithi002 ceph-mon[98807]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:49.107 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:18:48 smithi002 sudo[115087]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T18:18:49.107 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:18:48 smithi002 sudo[115087]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T18:18:49.107 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:18:48 smithi002 sudo[115087]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T18:18:49.107 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:18:48 smithi002 sudo[115087]: pam_unix(sudo:session): session closed for user root 2023-11-16T18:18:49.483 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:18:49 smithi040 sudo[107514]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T18:18:49.484 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:18:49 smithi040 sudo[107514]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T18:18:49.484 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:18:49 smithi040 sudo[107514]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T18:18:49.484 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:18:49 smithi040 sudo[107514]: pam_unix(sudo:session): session closed for user root 2023-11-16T18:18:49.983 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:18:49 smithi040 sudo[107517]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T18:18:49.983 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:18:49 smithi040 sudo[107517]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T18:18:49.983 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:18:49 smithi040 sudo[107517]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T18:18:49.984 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:18:49 smithi040 sudo[107517]: pam_unix(sudo:session): session closed for user root 2023-11-16T18:18:49.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:49 smithi040 ceph-mon[100251]: osdmap e21: 3 total, 3 up, 3 in 2023-11-16T18:18:49.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:49 smithi040 ceph-mon[100251]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-11-16T18:18:49.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:49 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:49.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:49 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2732674227' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a13117e-ad7f-4576-8dcd-dee08b526577"}]: dispatch 2023-11-16T18:18:49.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:49 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2732674227' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a13117e-ad7f-4576-8dcd-dee08b526577"}]': finished 2023-11-16T18:18:49.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:49 smithi040 ceph-mon[100251]: osdmap e22: 4 total, 3 up, 4 in 2023-11-16T18:18:49.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:49 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:18:49.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:49 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:50.047 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:18:49 smithi188 sudo[108707]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T18:18:50.047 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:18:49 smithi188 sudo[108707]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T18:18:50.047 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:18:49 smithi188 sudo[108707]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T18:18:50.047 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:18:49 smithi188 sudo[108707]: pam_unix(sudo:session): session closed for user root 2023-11-16T18:18:50.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:49 smithi188 ceph-mon[100639]: osdmap e21: 3 total, 3 up, 3 in 2023-11-16T18:18:50.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:49 smithi188 ceph-mon[100639]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-11-16T18:18:50.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:49 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:50.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:49 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2732674227' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a13117e-ad7f-4576-8dcd-dee08b526577"}]: dispatch 2023-11-16T18:18:50.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:49 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2732674227' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a13117e-ad7f-4576-8dcd-dee08b526577"}]': finished 2023-11-16T18:18:50.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:49 smithi188 ceph-mon[100639]: osdmap e22: 4 total, 3 up, 4 in 2023-11-16T18:18:50.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:49 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:18:50.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:49 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:50.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:49 smithi002 ceph-mon[98807]: osdmap e21: 3 total, 3 up, 3 in 2023-11-16T18:18:50.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:49 smithi002 ceph-mon[98807]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-11-16T18:18:50.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:49 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:50.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:49 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2732674227' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a13117e-ad7f-4576-8dcd-dee08b526577"}]: dispatch 2023-11-16T18:18:50.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:49 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2732674227' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a13117e-ad7f-4576-8dcd-dee08b526577"}]': finished 2023-11-16T18:18:50.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:49 smithi002 ceph-mon[98807]: osdmap e22: 4 total, 3 up, 4 in 2023-11-16T18:18:50.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:49 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:18:50.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:49 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:50.548 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:18:50 smithi188 sudo[108710]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T18:18:50.548 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:18:50 smithi188 sudo[108710]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T18:18:50.548 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:18:50 smithi188 sudo[108710]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T18:18:50.548 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:18:50 smithi188 sudo[108710]: pam_unix(sudo:session): session closed for user root 2023-11-16T18:18:51.102 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:50 smithi002 sudo[115189]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T18:18:51.102 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:50 smithi002 sudo[115189]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T18:18:51.102 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:50 smithi002 sudo[115189]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T18:18:51.102 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:50 smithi002 sudo[115189]: pam_unix(sudo:session): session closed for user root 2023-11-16T18:18:51.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:51 smithi002 sudo[115192]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T18:18:51.360 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:51 smithi002 sudo[115192]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T18:18:51.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:51 smithi002 sudo[115192]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T18:18:51.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:51 smithi002 sudo[115192]: pam_unix(sudo:session): session closed for user root 2023-11-16T18:18:51.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:51 smithi002 ceph-mon[98807]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:51.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:51 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:51.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:51 smithi002 ceph-mon[98807]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T18:18:51.368 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:51 smithi188 ceph-mon[100639]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:51.369 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:51 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:51.369 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:51 smithi188 ceph-mon[100639]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T18:18:51.370 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:51 smithi188 sudo[108713]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T18:18:51.370 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:51 smithi188 sudo[108713]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T18:18:51.370 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:51 smithi188 sudo[108713]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T18:18:51.371 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:51 smithi188 sudo[108713]: pam_unix(sudo:session): session closed for user root 2023-11-16T18:18:51.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:51 smithi040 ceph-mon[100251]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:51.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:51 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:18:51.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:51 smithi040 ceph-mon[100251]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T18:18:51.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:51 smithi188 sudo[108716]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T18:18:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:51 smithi188 sudo[108716]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T18:18:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:51 smithi188 sudo[108716]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T18:18:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:51 smithi188 sudo[108716]: pam_unix(sudo:session): session closed for user root 2023-11-16T18:18:52.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 sudo[107521]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T18:18:52.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 sudo[107521]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T18:18:52.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 sudo[107521]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T18:18:52.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 sudo[107521]: pam_unix(sudo:session): session closed for user root 2023-11-16T18:18:52.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T18:18:52.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:18:52.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3351187833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:18:52.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:18:52.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:18:52.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T18:18:52.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:18:52.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:18:52.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:18:52.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T18:18:52.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:18:52.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:18:52.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:18:52.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:52 smithi188 ceph-mon[100639]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T18:18:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:18:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3351187833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:18:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:18:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:18:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T18:18:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:18:52.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:18:52.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:18:52.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T18:18:52.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:18:52.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:18:52.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:18:52.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:52 smithi002 ceph-mon[98807]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:52.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 sudo[107524]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T18:18:52.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 sudo[107524]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T18:18:52.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 sudo[107524]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T18:18:52.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 sudo[107524]: pam_unix(sudo:session): session closed for user root 2023-11-16T18:18:52.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T18:18:52.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:18:52.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3351187833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:18:52.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:18:52.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:18:52.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T18:18:52.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:18:52.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:18:52.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:18:52.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T18:18:52.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:18:52.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:18:52.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:18:52.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:52 smithi040 ceph-mon[100251]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:53.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:53 smithi188 ceph-mon[100639]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T18:18:53.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:53 smithi188 ceph-mon[100639]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T18:18:53.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:53 smithi002 ceph-mon[98807]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T18:18:53.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:53 smithi002 ceph-mon[98807]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T18:18:53.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:53 smithi040 ceph-mon[100251]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T18:18:53.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:53 smithi040 ceph-mon[100251]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T18:18:54.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:54 smithi188 ceph-mon[100639]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:54.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:54 smithi002 ceph-mon[98807]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:54.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:54 smithi040 ceph-mon[100251]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:18:56.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:55 smithi002 ceph-mon[98807]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 196 KiB/s wr, 14 op/s 2023-11-16T18:18:56.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:55 smithi040 ceph-mon[100251]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 196 KiB/s wr, 14 op/s 2023-11-16T18:18:56.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:55 smithi188 ceph-mon[100639]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 196 KiB/s wr, 14 op/s 2023-11-16T18:18:57.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:57 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-16T18:18:57.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:57 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:57.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:57 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-16T18:18:57.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:57 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:57.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:57 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-16T18:18:57.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:57 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:18:58.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:58 smithi002 ceph-mon[98807]: Deploying daemon osd.3 on smithi002 2023-11-16T18:18:58.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:58 smithi002 ceph-mon[98807]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-11-16T18:18:58.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:58 smithi040 ceph-mon[100251]: Deploying daemon osd.3 on smithi002 2023-11-16T18:18:58.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:58 smithi040 ceph-mon[100251]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-11-16T18:18:58.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:58 smithi188 ceph-mon[100639]: Deploying daemon osd.3 on smithi002 2023-11-16T18:18:58.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:58 smithi188 ceph-mon[100639]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-11-16T18:19:00.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:18:59 smithi002 ceph-mon[98807]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-11-16T18:19:00.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:18:59 smithi040 ceph-mon[100251]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-11-16T18:19:00.302 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:18:59 smithi188 ceph-mon[100639]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-11-16T18:19:02.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:02 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:02.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:02 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:02.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:02 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:02.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:02 smithi188 ceph-mon[100639]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-11-16T18:19:02.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:02.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:02.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:02.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:02 smithi002 ceph-mon[98807]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-11-16T18:19:02.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:02 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:02.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:02 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:02.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:02 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:02.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:02 smithi040 ceph-mon[100251]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-11-16T18:19:04.078 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 3 on host 'smithi002' 2023-11-16T18:19:04.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:04.076+0000 7f065bfff700 1 -- 172.21.15.2:0/2038977069 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0674060f30 con 0x7f065c0a1140 2023-11-16T18:19:04.080 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:04.079+0000 7f06791ca700 1 -- 172.21.15.2:0/2038977069 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f065c0a1140 msgr2=0x7f065c0a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:04.080 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:04.079+0000 7f06791ca700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f065c0a1140 0x7f065c0a35f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0674100240 tx=0x7f066800b410 comp rx=0 tx=0).stop 2023-11-16T18:19:04.080 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:04.079+0000 7f06791ca700 1 -- 172.21.15.2:0/2038977069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06741003e0 msgr2=0x7f067419ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:04.080 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:04.079+0000 7f06791ca700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06741003e0 0x7f067419ad30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0664000c00 tx=0x7f066400f710 comp rx=0 tx=0).stop 2023-11-16T18:19:04.080 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:04.079+0000 7f06791ca700 1 -- 172.21.15.2:0/2038977069 shutdown_connections 2023-11-16T18:19:04.080 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:04.079+0000 7f06791ca700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f065c0a1140 0x7f065c0a35f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:04.080 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:04.079+0000 7f06791ca700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0674100d70 0x7f067419d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:04.080 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:04.079+0000 7f06791ca700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06740ff1e0 0x7f06740699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:04.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:04.079+0000 7f06791ca700 1 --2- 172.21.15.2:0/2038977069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06741003e0 0x7f067419ad30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:04.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:04.079+0000 7f06791ca700 1 -- 172.21.15.2:0/2038977069 >> 172.21.15.2:0/2038977069 conn(0x7f06740fa7f0 msgr2=0x7f0674103910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:19:04.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:04.079+0000 7f06791ca700 1 -- 172.21.15.2:0/2038977069 shutdown_connections 2023-11-16T18:19:04.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:04.079+0000 7f06791ca700 1 -- 172.21.15.2:0/2038977069 wait complete. 2023-11-16T18:19:04.549 DEBUG:teuthology.orchestra.run.smithi002:osd.3> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.3.service 2023-11-16T18:19:04.553 INFO:tasks.cephadm:Deploying osd.4 on smithi040 with /dev/vg_nvme/lv_3... 2023-11-16T18:19:04.553 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-16T18:19:04.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:04.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:04.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:04 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:19:04.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:04 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:19:04.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:04 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:19:04.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:04 smithi002 ceph-mon[98807]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-11-16T18:19:04.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:04 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:04.599 INFO:journalctl@ceph.osd.3.smithi002.stdout:-- Logs begin at Thu 2023-11-16 17:59:05 UTC. -- 2023-11-16T18:19:04.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:04 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:04 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:04 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:04.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:04 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:04.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:04 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:19:04.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:04 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:04 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:19:04.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:04 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:19:04.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:04 smithi188 ceph-mon[100639]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-11-16T18:19:04.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:04 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:04 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:04 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:04.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:04 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:04 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:04 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:04.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:04 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:04.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:04 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:19:04.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:04 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:04 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:19:04.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:04 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:19:04.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:04 smithi040 ceph-mon[100251]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-11-16T18:19:04.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:04 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:04 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:04.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:04 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:05.231 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T18:19:05.356 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 18:19:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-3[117440]: 2023-11-16T18:19:05.221+0000 7f117180f640 -1 osd.3 0 log_to_monitors true 2023-11-16T18:19:05.718 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:05 smithi040 ceph-mon[100251]: from='osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T18:19:05.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:05 smithi002 ceph-mon[98807]: from='osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T18:19:05.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:05 smithi188 ceph-mon[100639]: from='osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T18:19:06.691 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:06 smithi002 ceph-mon[98807]: from='osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-16T18:19:06.694 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:06 smithi002 ceph-mon[98807]: osdmap e23: 4 total, 3 up, 4 in 2023-11-16T18:19:06.694 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:06 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:06.696 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:06 smithi002 ceph-mon[98807]: from='osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T18:19:06.698 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:06 smithi002 ceph-mon[98807]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:19:06.699 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 18:19:06 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-3[117440]: 2023-11-16T18:19:06.545+0000 7f1166e86700 -1 osd.3 0 waiting for initial osdmap 2023-11-16T18:19:06.700 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 18:19:06 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-3[117440]: 2023-11-16T18:19:06.551+0000 7f116247a700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T18:19:06.756 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-16T18:19:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-16T18:19:06.757 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2023-11-16T18:19:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2023-11-16T18:19:06.757 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123608 s, 848 MB/s 2023-11-16T18:19:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2023-11-16T18:19:06.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:06 smithi040 ceph-mon[100251]: from='osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-16T18:19:06.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:06 smithi040 ceph-mon[100251]: osdmap e23: 4 total, 3 up, 4 in 2023-11-16T18:19:06.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:06 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:06.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:06 smithi040 ceph-mon[100251]: from='osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T18:19:06.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:06 smithi040 ceph-mon[100251]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:19:07.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:06 smithi188 ceph-mon[100639]: from='osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-16T18:19:07.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:06 smithi188 ceph-mon[100639]: osdmap e23: 4 total, 3 up, 4 in 2023-11-16T18:19:07.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:06 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:07.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:06 smithi188 ceph-mon[100639]: from='osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T18:19:07.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:06 smithi188 ceph-mon[100639]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:19:07.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi040:vg_nvme/lv_3 2023-11-16T18:19:07.828 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T18:19:07.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: from='osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T18:19:07.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: osdmap e24: 4 total, 3 up, 4 in 2023-11-16T18:19:07.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:07.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:07.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: Detected new or changed devices on smithi002 2023-11-16T18:19:07.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:07.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:07.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:07.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:07.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi002 to 8586M 2023-11-16T18:19:07.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:07.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:07.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:07.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:07.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:07 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:07.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: from='osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T18:19:07.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: osdmap e24: 4 total, 3 up, 4 in 2023-11-16T18:19:07.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:07.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:07.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: Detected new or changed devices on smithi002 2023-11-16T18:19:07.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:07.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:07.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:07.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:07.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi002 to 8586M 2023-11-16T18:19:07.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:07.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:07.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:07.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:07.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:07 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:08.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: from='osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T18:19:08.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: osdmap e24: 4 total, 3 up, 4 in 2023-11-16T18:19:08.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:08.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:08.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: Detected new or changed devices on smithi002 2023-11-16T18:19:08.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:08.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:08.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:08.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:08.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi002 to 8586M 2023-11-16T18:19:08.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:08.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:08.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:08.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:08.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:07 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:08.631 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:08 smithi040 ceph-mon[100251]: purged_snaps scrub starts 2023-11-16T18:19:08.631 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:08 smithi040 ceph-mon[100251]: purged_snaps scrub ok 2023-11-16T18:19:08.631 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:08 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:19:08.632 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:08 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:19:08.632 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:08 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:19:08.632 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:08 smithi040 ceph-mon[100251]: osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732] boot 2023-11-16T18:19:08.632 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:08 smithi040 ceph-mon[100251]: osdmap e25: 4 total, 4 up, 4 in 2023-11-16T18:19:08.632 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:08 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:08.632 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:08 smithi040 ceph-mon[100251]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:19:08.633 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:08 smithi040 ceph-mon[100251]: osdmap e26: 4 total, 4 up, 4 in 2023-11-16T18:19:08.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:08 smithi002 ceph-mon[98807]: purged_snaps scrub starts 2023-11-16T18:19:08.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:08 smithi002 ceph-mon[98807]: purged_snaps scrub ok 2023-11-16T18:19:08.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:08 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:19:08.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:08 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:19:08.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:08 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:19:08.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:08 smithi002 ceph-mon[98807]: osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732] boot 2023-11-16T18:19:08.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:08 smithi002 ceph-mon[98807]: osdmap e25: 4 total, 4 up, 4 in 2023-11-16T18:19:08.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:08 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:08.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:08 smithi002 ceph-mon[98807]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:19:08.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:08 smithi002 ceph-mon[98807]: osdmap e26: 4 total, 4 up, 4 in 2023-11-16T18:19:09.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.026+0000 7f84444f9700 1 -- 172.21.15.40:0/1529262839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c1003e0 msgr2=0x7f843c100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:09.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.026+0000 7f84444f9700 1 --2- 172.21.15.40:0/1529262839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c1003e0 0x7f843c100830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8430009990 tx=0x7f8430009ca0 comp rx=0 tx=0).stop 2023-11-16T18:19:09.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 -- 172.21.15.40:0/1529262839 shutdown_connections 2023-11-16T18:19:09.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 --2- 172.21.15.40:0/1529262839 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f843c100d70 0x7f843c107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:09.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 --2- 172.21.15.40:0/1529262839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f843c0ff1e0 0x7f843c0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:09.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 --2- 172.21.15.40:0/1529262839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c1003e0 0x7f843c100830 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:09.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 -- 172.21.15.40:0/1529262839 >> 172.21.15.40:0/1529262839 conn(0x7f843c0fa7f0 msgr2=0x7f843c0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:19:09.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 -- 172.21.15.40:0/1529262839 shutdown_connections 2023-11-16T18:19:09.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 -- 172.21.15.40:0/1529262839 wait complete. 2023-11-16T18:19:09.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 Processor -- start 2023-11-16T18:19:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 -- start start 2023-11-16T18:19:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f843c0ff1e0 0x7f843c069130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:19:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c1003e0 0x7f843c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:19:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f843c100d70 0x7f843c19d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:19:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843c19dbb0 con 0x7f843c100d70 2023-11-16T18:19:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843c19dd20 con 0x7f843c1003e0 2023-11-16T18:19:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.027+0000 7f84444f9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843c19dee0 con 0x7f843c0ff1e0 2023-11-16T18:19:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f8441a94700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c1003e0 0x7f843c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:19:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f8441a94700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c1003e0 0x7f843c19adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48838/0 (socket says 172.21.15.40:48838) 2023-11-16T18:19:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f8441a94700 1 -- 172.21.15.40:0/886522495 learned_addr learned my addr 172.21.15.40:0/886522495 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-11-16T18:19:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f8442295700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f843c0ff1e0 0x7f843c069130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:19:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f8442a96700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f843c100d70 0x7f843c19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:19:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f8441a94700 1 -- 172.21.15.40:0/886522495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f843c0ff1e0 msgr2=0x7f843c069130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f8441a94700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f843c0ff1e0 0x7f843c069130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f8441a94700 1 -- 172.21.15.40:0/886522495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f843c100d70 msgr2=0x7f843c19d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:09.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f8441a94700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f843c100d70 0x7f843c19d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:09.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f8441a94700 1 -- 172.21.15.40:0/886522495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84300095f0 con 0x7f843c1003e0 2023-11-16T18:19:09.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f8442a96700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f843c100d70 0x7f843c19d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:19:09.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f8441a94700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c1003e0 0x7f843c19adc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8430006010 tx=0x7f843000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:19:09.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f842f7fe700 1 -- 172.21.15.40:0/886522495 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843001d070 con 0x7f843c1003e0 2023-11-16T18:19:09.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f842f7fe700 1 -- 172.21.15.40:0/886522495 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8430022470 con 0x7f843c1003e0 2023-11-16T18:19:09.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f842f7fe700 1 -- 172.21.15.40:0/886522495 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8430017650 con 0x7f843c1003e0 2023-11-16T18:19:09.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f84444f9700 1 -- 172.21.15.40:0/886522495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f843c10ab50 con 0x7f843c1003e0 2023-11-16T18:19:09.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.028+0000 7f84444f9700 1 -- 172.21.15.40:0/886522495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f843c10a9c0 con 0x7f843c1003e0 2023-11-16T18:19:09.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.029+0000 7f84444f9700 1 -- 172.21.15.40:0/886522495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f843c19e420 con 0x7f843c1003e0 2023-11-16T18:19:09.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.030+0000 7f842f7fe700 1 -- 172.21.15.40:0/886522495 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f8430022890 con 0x7f843c1003e0 2023-11-16T18:19:09.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.031+0000 7f842f7fe700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f84280a1150 0x7f84280a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:19:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.031+0000 7f842f7fe700 1 -- 172.21.15.40:0/886522495 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f84300cc640 con 0x7f843c1003e0 2023-11-16T18:19:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.031+0000 7f8442295700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f84280a1150 0x7f84280a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:19:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.032+0000 7f8442295700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f84280a1150 0x7f84280a3600 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f843c100240 tx=0x7f843800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:19:09.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:08 smithi188 ceph-mon[100639]: purged_snaps scrub starts 2023-11-16T18:19:09.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:08 smithi188 ceph-mon[100639]: purged_snaps scrub ok 2023-11-16T18:19:09.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:08 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:19:09.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:08 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:19:09.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:08 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:19:09.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:08 smithi188 ceph-mon[100639]: osd.3 [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732] boot 2023-11-16T18:19:09.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:08 smithi188 ceph-mon[100639]: osdmap e25: 4 total, 4 up, 4 in 2023-11-16T18:19:09.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:08 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:19:09.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:08 smithi188 ceph-mon[100639]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T18:19:09.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:08 smithi188 ceph-mon[100639]: osdmap e26: 4 total, 4 up, 4 in 2023-11-16T18:19:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.035+0000 7f842f7fe700 1 -- 172.21.15.40:0/886522495 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8430097020 con 0x7f843c1003e0 2023-11-16T18:19:09.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:09.243+0000 7f84444f9700 1 -- 172.21.15.40:0/886522495 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f843c060f30 con 0x7f84280a1150 2023-11-16T18:19:09.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:19:09.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:19:09.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:09.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:09 smithi002 ceph-mon[98807]: osdmap e27: 4 total, 4 up, 4 in 2023-11-16T18:19:09.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:09 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:19:09.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:09 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:19:09.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:09 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:09.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:09 smithi040 ceph-mon[100251]: osdmap e27: 4 total, 4 up, 4 in 2023-11-16T18:19:10.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:09 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:19:10.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:09 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:19:10.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:09 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:10.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:09 smithi188 ceph-mon[100639]: osdmap e27: 4 total, 4 up, 4 in 2023-11-16T18:19:10.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:10 smithi040 ceph-mon[100251]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:19:10.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:10 smithi040 ceph-mon[100251]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T18:19:10.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:10 smithi002 ceph-mon[98807]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:19:10.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:10 smithi002 ceph-mon[98807]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T18:19:11.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:10 smithi188 ceph-mon[100639]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:19:11.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:10 smithi188 ceph-mon[100639]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T18:19:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:12 smithi040 ceph-mon[100251]: osdmap e28: 4 total, 4 up, 4 in 2023-11-16T18:19:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:12 smithi040 ceph-mon[100251]: pgmap v95: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-11-16T18:19:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:12 smithi040 ceph-mon[100251]: from='client.? 172.21.15.40:0/971668420' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0453e4fa-a79a-4aea-a701-aedd106bdb08"}]: dispatch 2023-11-16T18:19:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:12 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0453e4fa-a79a-4aea-a701-aedd106bdb08"}]: dispatch 2023-11-16T18:19:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:12 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0453e4fa-a79a-4aea-a701-aedd106bdb08"}]': finished 2023-11-16T18:19:12.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:12 smithi040 ceph-mon[100251]: osdmap e29: 5 total, 4 up, 5 in 2023-11-16T18:19:12.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:12.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:12 smithi188 ceph-mon[100639]: osdmap e28: 4 total, 4 up, 4 in 2023-11-16T18:19:12.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:12 smithi188 ceph-mon[100639]: pgmap v95: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-11-16T18:19:12.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:12 smithi188 ceph-mon[100639]: from='client.? 172.21.15.40:0/971668420' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0453e4fa-a79a-4aea-a701-aedd106bdb08"}]: dispatch 2023-11-16T18:19:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:12 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0453e4fa-a79a-4aea-a701-aedd106bdb08"}]: dispatch 2023-11-16T18:19:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:12 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0453e4fa-a79a-4aea-a701-aedd106bdb08"}]': finished 2023-11-16T18:19:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:12 smithi188 ceph-mon[100639]: osdmap e29: 5 total, 4 up, 5 in 2023-11-16T18:19:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:12 smithi002 ceph-mon[98807]: osdmap e28: 4 total, 4 up, 4 in 2023-11-16T18:19:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:12 smithi002 ceph-mon[98807]: pgmap v95: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-11-16T18:19:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:12 smithi002 ceph-mon[98807]: from='client.? 172.21.15.40:0/971668420' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0453e4fa-a79a-4aea-a701-aedd106bdb08"}]: dispatch 2023-11-16T18:19:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:12 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0453e4fa-a79a-4aea-a701-aedd106bdb08"}]: dispatch 2023-11-16T18:19:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:12 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0453e4fa-a79a-4aea-a701-aedd106bdb08"}]': finished 2023-11-16T18:19:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:12 smithi002 ceph-mon[98807]: osdmap e29: 5 total, 4 up, 5 in 2023-11-16T18:19:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:13.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:13 smithi040 ceph-mon[100251]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-11-16T18:19:13.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:13 smithi040 ceph-mon[100251]: from='client.? 172.21.15.40:0/2624818618' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:19:13.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:13 smithi188 ceph-mon[100639]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-11-16T18:19:13.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:13 smithi188 ceph-mon[100639]: from='client.? 172.21.15.40:0/2624818618' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:19:13.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:13 smithi002 ceph-mon[98807]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-11-16T18:19:13.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:13 smithi002 ceph-mon[98807]: from='client.? 172.21.15.40:0/2624818618' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:19:14.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:14 smithi040 ceph-mon[100251]: pgmap v97: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-11-16T18:19:14.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:14 smithi188 ceph-mon[100639]: pgmap v97: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-11-16T18:19:14.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:14 smithi002 ceph-mon[98807]: pgmap v97: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-11-16T18:19:15.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:15 smithi040 ceph-mon[100251]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-11-16T18:19:15.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:15 smithi040 ceph-mon[100251]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-11-16T18:19:15.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:15 smithi040 ceph-mon[100251]: Cluster is now healthy 2023-11-16T18:19:16.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:15 smithi002 ceph-mon[98807]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-11-16T18:19:16.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:15 smithi002 ceph-mon[98807]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-11-16T18:19:16.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:15 smithi002 ceph-mon[98807]: Cluster is now healthy 2023-11-16T18:19:16.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:15 smithi188 ceph-mon[100639]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-11-16T18:19:16.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:15 smithi188 ceph-mon[100639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-11-16T18:19:16.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:15 smithi188 ceph-mon[100639]: Cluster is now healthy 2023-11-16T18:19:18.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:17 smithi002 ceph-mon[98807]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T18:19:18.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:17 smithi040 ceph-mon[100251]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T18:19:18.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:17 smithi188 ceph-mon[100639]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T18:19:18.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:18 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-16T18:19:18.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:18 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:18.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:18 smithi040 ceph-mon[100251]: Deploying daemon osd.4 on smithi040 2023-11-16T18:19:19.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:18 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-16T18:19:19.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:18 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:19.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:18 smithi002 ceph-mon[98807]: Deploying daemon osd.4 on smithi040 2023-11-16T18:19:19.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:18 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-16T18:19:19.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:18 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:19.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:18 smithi188 ceph-mon[100639]: Deploying daemon osd.4 on smithi040 2023-11-16T18:19:19.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:19 smithi040 ceph-mon[100251]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2023-11-16T18:19:20.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:19 smithi002 ceph-mon[98807]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2023-11-16T18:19:20.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:19 smithi188 ceph-mon[100639]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2023-11-16T18:19:22.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:22 smithi040 ceph-mon[100251]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-16T18:19:22.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:22 smithi188 ceph-mon[100639]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-16T18:19:22.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:22 smithi002 ceph-mon[98807]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-16T18:19:24.424 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:24 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:24.425 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:24 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:24.425 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:24 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:24.425 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:24 smithi040 ceph-mon[100251]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-11-16T18:19:24.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:24 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:24.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:24 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:24.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:24 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:24.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:24 smithi188 ceph-mon[100639]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-11-16T18:19:24.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:24 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:24.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:24 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:24.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:24 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:24.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:24 smithi002 ceph-mon[98807]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-11-16T18:19:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:26.209+0000 7f842f7fe700 1 -- 172.21.15.40:0/886522495 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f843c060f30 con 0x7f84280a1150 2023-11-16T18:19:26.212 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 4 on host 'smithi040' 2023-11-16T18:19:26.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:26.211+0000 7f84444f9700 1 -- 172.21.15.40:0/886522495 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f84280a1150 msgr2=0x7f84280a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:26.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:26.211+0000 7f84444f9700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f84280a1150 0x7f84280a3600 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f843c100240 tx=0x7f843800b410 comp rx=0 tx=0).stop 2023-11-16T18:19:26.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:26.211+0000 7f84444f9700 1 -- 172.21.15.40:0/886522495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c1003e0 msgr2=0x7f843c19adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:26.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:26.211+0000 7f84444f9700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c1003e0 0x7f843c19adc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8430006010 tx=0x7f843000f690 comp rx=0 tx=0).stop 2023-11-16T18:19:26.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:26.211+0000 7f84444f9700 1 -- 172.21.15.40:0/886522495 shutdown_connections 2023-11-16T18:19:26.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:26.211+0000 7f84444f9700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f84280a1150 0x7f84280a3600 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:26.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:26.211+0000 7f84444f9700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f843c0ff1e0 0x7f843c069130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:26.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:26.211+0000 7f84444f9700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f843c100d70 0x7f843c19d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:26.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:26.211+0000 7f84444f9700 1 --2- 172.21.15.40:0/886522495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c1003e0 0x7f843c19adc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:26.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:26.211+0000 7f84444f9700 1 -- 172.21.15.40:0/886522495 >> 172.21.15.40:0/886522495 conn(0x7f843c0fa7f0 msgr2=0x7f843c0fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:19:26.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:26.211+0000 7f84444f9700 1 -- 172.21.15.40:0/886522495 shutdown_connections 2023-11-16T18:19:26.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:19:26.211+0000 7f84444f9700 1 -- 172.21.15.40:0/886522495 wait complete. 2023-11-16T18:19:26.618 DEBUG:teuthology.orchestra.run.smithi040:osd.4> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.4.service 2023-11-16T18:19:26.621 INFO:tasks.cephadm:Deploying osd.5 on smithi188 with /dev/vg_nvme/lv_3... 2023-11-16T18:19:26.622 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-16T18:19:26.652 INFO:journalctl@ceph.osd.4.smithi040.stdout:-- Logs begin at Thu 2023-11-16 17:59:26 UTC. -- 2023-11-16T18:19:26.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:26 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:26.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:26 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:26.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:26 smithi040 ceph-mon[100251]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-11-16T18:19:26.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:26 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:26.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:26 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:27.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:26 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:27.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:26 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:27.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:26 smithi188 ceph-mon[100639]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-11-16T18:19:27.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:26 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:27.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:26 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:27.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:26 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:27.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:26 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:27.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:26 smithi002 ceph-mon[98807]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-11-16T18:19:27.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:26 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:27.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:26 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:27.276 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:19:27.411 INFO:journalctl@ceph.osd.4.smithi040.stdout:Nov 16 18:19:27 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-4[110650]: 2023-11-16T18:19:27.144+0000 7fad11a91640 -1 osd.4 0 log_to_monitors true 2023-11-16T18:19:27.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:27 smithi040 ceph-mon[100251]: from='osd.4 [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T18:19:27.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:27 smithi040 ceph-mon[100251]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T18:19:28.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:27 smithi188 ceph-mon[100639]: from='osd.4 [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T18:19:28.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:27 smithi188 ceph-mon[100639]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T18:19:28.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:27 smithi002 ceph-mon[98807]: from='osd.4 [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T18:19:28.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:27 smithi002 ceph-mon[98807]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T18:19:28.750 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-16T18:19:28.750 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-16T18:19:28.750 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2023-11-16T18:19:28.750 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2023-11-16T18:19:28.750 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109996 s, 953 MB/s 2023-11-16T18:19:28.750 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2023-11-16T18:19:28.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:28 smithi040 ceph-mon[100251]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-16T18:19:28.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:28 smithi040 ceph-mon[100251]: osdmap e30: 5 total, 4 up, 5 in 2023-11-16T18:19:28.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:28 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:28.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:28 smithi040 ceph-mon[100251]: from='osd.4 [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:19:28.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:28 smithi040 ceph-mon[100251]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:19:28.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:28 smithi040 ceph-mon[100251]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T18:19:28.985 INFO:journalctl@ceph.osd.4.smithi040.stdout:Nov 16 18:19:28 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-4[110650]: 2023-11-16T18:19:28.733+0000 7fad07108700 -1 osd.4 0 waiting for initial osdmap 2023-11-16T18:19:28.985 INFO:journalctl@ceph.osd.4.smithi040.stdout:Nov 16 18:19:28 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-4[110650]: 2023-11-16T18:19:28.741+0000 7fad00ef9700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T18:19:29.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:28 smithi188 ceph-mon[100639]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-16T18:19:29.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:28 smithi188 ceph-mon[100639]: osdmap e30: 5 total, 4 up, 5 in 2023-11-16T18:19:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:28 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:28 smithi188 ceph-mon[100639]: from='osd.4 [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:19:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:28 smithi188 ceph-mon[100639]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:19:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:28 smithi188 ceph-mon[100639]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T18:19:29.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:28 smithi002 ceph-mon[98807]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-16T18:19:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:28 smithi002 ceph-mon[98807]: osdmap e30: 5 total, 4 up, 5 in 2023-11-16T18:19:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:28 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:28 smithi002 ceph-mon[98807]: from='osd.4 [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:19:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:28 smithi002 ceph-mon[98807]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:19:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:28 smithi002 ceph-mon[98807]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T18:19:29.218 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi188:vg_nvme/lv_3 2023-11-16T18:19:29.862 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:19:40.739 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-11-16T18:19:40.739 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: osdmap e31: 5 total, 4 up, 5 in 2023-11-16T18:19:40.739 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:40.739 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:40.739 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.739 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: purged_snaps scrub starts 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: purged_snaps scrub ok 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: Detected new or changed devices on smithi040 2023-11-16T18:19:40.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi040 to 8586M 2023-11-16T18:19:40.741 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:19:40.741 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:19:40.741 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:19:40.741 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:19:40.741 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:19:40.741 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:19:40.741 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: osd.4 [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130] boot 2023-11-16T18:19:40.741 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: osdmap e32: 5 total, 5 up, 5 in 2023-11-16T18:19:40.741 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:40.741 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:30 smithi188 ceph-mon[100639]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T18:19:40.745 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-11-16T18:19:40.745 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: osdmap e31: 5 total, 4 up, 5 in 2023-11-16T18:19:40.745 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:40.745 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:40.745 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.746 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.746 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:40.746 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:40.746 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.746 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:40.746 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:40.746 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.746 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:40.746 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:40.746 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:40.746 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.746 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: purged_snaps scrub starts 2023-11-16T18:19:40.746 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: purged_snaps scrub ok 2023-11-16T18:19:40.747 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: Detected new or changed devices on smithi040 2023-11-16T18:19:40.747 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi040 to 8586M 2023-11-16T18:19:40.747 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:19:40.747 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:19:40.747 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:19:40.747 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:19:40.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:19:40.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:19:40.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: osd.4 [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130] boot 2023-11-16T18:19:40.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: osdmap e32: 5 total, 5 up, 5 in 2023-11-16T18:19:40.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:40.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:30 smithi040 ceph-mon[100251]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T18:19:40.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:31 smithi040 ceph-mon[100251]: osdmap e33: 5 total, 5 up, 5 in 2023-11-16T18:19:40.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:19:40.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:19:40.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:40.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:32 smithi040 ceph-mon[100251]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:19:40.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:32 smithi040 ceph-mon[100251]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:33 smithi040 ceph-mon[100251]: from='client.? 172.21.15.188:0/2110287587' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30994739-f13b-4840-b14d-af1553319d21"}]: dispatch 2023-11-16T18:19:40.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:33 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30994739-f13b-4840-b14d-af1553319d21"}]: dispatch 2023-11-16T18:19:40.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:33 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30994739-f13b-4840-b14d-af1553319d21"}]': finished 2023-11-16T18:19:40.773 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:33 smithi040 ceph-mon[100251]: osdmap e34: 6 total, 5 up, 6 in 2023-11-16T18:19:40.774 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:40.774 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:34 smithi040 ceph-mon[100251]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.774 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:34 smithi040 ceph-mon[100251]: from='client.? 172.21.15.188:0/66861090' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:19:40.774 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:35 smithi040 ceph-mon[100251]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.774 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:37 smithi040 ceph-mon[100251]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.774 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:39 smithi040 ceph-mon[100251]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.774 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-11-16T18:19:40.774 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: osdmap e31: 5 total, 4 up, 5 in 2023-11-16T18:19:40.774 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:40.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:40.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:40.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:40.837 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.837 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:40.837 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:40.837 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:40.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:40.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:40.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:40.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: purged_snaps scrub starts 2023-11-16T18:19:40.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: purged_snaps scrub ok 2023-11-16T18:19:40.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: Detected new or changed devices on smithi040 2023-11-16T18:19:40.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi040 to 8586M 2023-11-16T18:19:40.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:19:40.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:19:40.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:19:40.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:19:40.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: osd.4 [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130] boot 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: osdmap e32: 5 total, 5 up, 5 in 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:30 smithi002 ceph-mon[98807]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:31 smithi002 ceph-mon[98807]: osdmap e33: 5 total, 5 up, 5 in 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:32 smithi002 ceph-mon[98807]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:32 smithi002 ceph-mon[98807]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:33 smithi002 ceph-mon[98807]: from='client.? 172.21.15.188:0/2110287587' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30994739-f13b-4840-b14d-af1553319d21"}]: dispatch 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:33 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30994739-f13b-4840-b14d-af1553319d21"}]: dispatch 2023-11-16T18:19:40.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:33 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30994739-f13b-4840-b14d-af1553319d21"}]': finished 2023-11-16T18:19:40.840 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:33 smithi002 ceph-mon[98807]: osdmap e34: 6 total, 5 up, 6 in 2023-11-16T18:19:40.840 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:40.840 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:34 smithi002 ceph-mon[98807]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.840 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:34 smithi002 ceph-mon[98807]: from='client.? 172.21.15.188:0/66861090' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:19:40.840 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:35 smithi002 ceph-mon[98807]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.840 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:37 smithi002 ceph-mon[98807]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.840 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:39 smithi002 ceph-mon[98807]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.842 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.067+0000 7f58de0a7700 1 -- 172.21.15.188:0/3562432094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58d8100310 msgr2=0x7f58d8100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:40.842 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.067+0000 7f58de0a7700 1 --2- 172.21.15.188:0/3562432094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58d8100310 0x7f58d8100720 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f58c8009990 tx=0x7f58c8009ca0 comp rx=0 tx=0).stop 2023-11-16T18:19:40.842 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.068+0000 7f58de0a7700 1 -- 172.21.15.188:0/3562432094 shutdown_connections 2023-11-16T18:19:40.842 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.068+0000 7f58de0a7700 1 --2- 172.21.15.188:0/3562432094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58d8100310 0x7f58d8100720 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:40.842 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.068+0000 7f58de0a7700 1 --2- 172.21.15.188:0/3562432094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58d8101ea0 0x7f58d8108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:40.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.068+0000 7f58de0a7700 1 --2- 172.21.15.188:0/3562432094 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58d8101510 0x7f58d8101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:40.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.068+0000 7f58de0a7700 1 -- 172.21.15.188:0/3562432094 >> 172.21.15.188:0/3562432094 conn(0x7f58d80fb960 msgr2=0x7f58d80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:19:40.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.068+0000 7f58de0a7700 1 -- 172.21.15.188:0/3562432094 shutdown_connections 2023-11-16T18:19:40.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.068+0000 7f58de0a7700 1 -- 172.21.15.188:0/3562432094 wait complete. 2023-11-16T18:19:40.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.068+0000 7f58de0a7700 1 Processor -- start 2023-11-16T18:19:40.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.068+0000 7f58de0a7700 1 -- start start 2023-11-16T18:19:40.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.068+0000 7f58de0a7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58d8100310 0x7f58d8195fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:19:40.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.068+0000 7f58de0a7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58d8101510 0x7f58d8196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:19:40.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58de0a7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58d8101ea0 0x7f58d819d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:19:40.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58de0a7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d819db50 con 0x7f58d8101ea0 2023-11-16T18:19:40.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58de0a7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d819dc90 con 0x7f58d8100310 2023-11-16T18:19:40.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58de0a7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d819de20 con 0x7f58d8101510 2023-11-16T18:19:40.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d6ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58d8101510 0x7f58d8196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:19:40.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d6ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58d8101510 0x7f58d8196500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:48148/0 (socket says 172.21.15.188:48148) 2023-11-16T18:19:40.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d6ffd700 1 -- 172.21.15.188:0/2496182611 learned_addr learned my addr 172.21.15.188:0/2496182611 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-16T18:19:40.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d77fe700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58d8100310 0x7f58d8195fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:19:40.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d7fff700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58d8101ea0 0x7f58d819d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:19:40.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d6ffd700 1 -- 172.21.15.188:0/2496182611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58d8100310 msgr2=0x7f58d8195fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:40.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d6ffd700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58d8100310 0x7f58d8195fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:40.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d6ffd700 1 -- 172.21.15.188:0/2496182611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58d8101ea0 msgr2=0x7f58d819d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:40.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d6ffd700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58d8101ea0 0x7f58d819d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:40.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d6ffd700 1 -- 172.21.15.188:0/2496182611 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c80095f0 con 0x7f58d8101510 2023-11-16T18:19:40.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d7fff700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58d8101ea0 0x7f58d819d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:19:40.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d6ffd700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58d8101510 0x7f58d8196500 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00d9b0 tx=0x7f58cc00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:19:40.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d4ff9700 1 -- 172.21.15.188:0/2496182611 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc004220 con 0x7f58d8101510 2023-11-16T18:19:40.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d4ff9700 1 -- 172.21.15.188:0/2496182611 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58cc007640 con 0x7f58d8101510 2023-11-16T18:19:40.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58de0a7700 1 -- 172.21.15.188:0/2496182611 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d810bc60 con 0x7f58d8101510 2023-11-16T18:19:40.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58d4ff9700 1 -- 172.21.15.188:0/2496182611 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc005650 con 0x7f58d8101510 2023-11-16T18:19:40.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.069+0000 7f58de0a7700 1 -- 172.21.15.188:0/2496182611 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d810bad0 con 0x7f58d8101510 2023-11-16T18:19:40.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.070+0000 7f58de0a7700 1 -- 172.21.15.188:0/2496182611 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d8073290 con 0x7f58d8101510 2023-11-16T18:19:40.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.071+0000 7f58d4ff9700 1 -- 172.21.15.188:0/2496182611 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f58cc004750 con 0x7f58d8101510 2023-11-16T18:19:40.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.072+0000 7f58d4ff9700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58c00a11d0 0x7f58c00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:19:40.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.072+0000 7f58d4ff9700 1 -- 172.21.15.188:0/2496182611 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f58cc0cbc10 con 0x7f58d8101510 2023-11-16T18:19:40.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.072+0000 7f58d77fe700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58c00a11d0 0x7f58c00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:19:40.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.073+0000 7f58d77fe700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58c00a11d0 0x7f58c00a3680 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f58c800b5c0 tx=0x7f58c8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:19:40.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.075+0000 7f58d4ff9700 1 -- 172.21.15.188:0/2496182611 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f58cc0963d0 con 0x7f58d8101510 2023-11-16T18:19:40.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:31.267+0000 7f58de0a7700 1 -- 172.21.15.188:0/2496182611 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f58d8073ff0 con 0x7f58c00a11d0 2023-11-16T18:19:40.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:31 smithi188 ceph-mon[100639]: osdmap e33: 5 total, 5 up, 5 in 2023-11-16T18:19:40.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:19:40.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:19:40.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:40.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:32 smithi188 ceph-mon[100639]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:19:40.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:32 smithi188 ceph-mon[100639]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:33 smithi188 ceph-mon[100639]: from='client.? 172.21.15.188:0/2110287587' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30994739-f13b-4840-b14d-af1553319d21"}]: dispatch 2023-11-16T18:19:40.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:33 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30994739-f13b-4840-b14d-af1553319d21"}]: dispatch 2023-11-16T18:19:40.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:33 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30994739-f13b-4840-b14d-af1553319d21"}]': finished 2023-11-16T18:19:40.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:33 smithi188 ceph-mon[100639]: osdmap e34: 6 total, 5 up, 6 in 2023-11-16T18:19:40.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:40.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:34 smithi188 ceph-mon[100639]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:34 smithi188 ceph-mon[100639]: from='client.? 172.21.15.188:0/66861090' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:19:40.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:35 smithi188 ceph-mon[100639]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:37 smithi188 ceph-mon[100639]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:40.904 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:39 smithi188 ceph-mon[100639]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:41.096 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-16T18:19:41.096 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:41.096 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:40 smithi188 ceph-mon[100639]: Deploying daemon osd.5 on smithi188 2023-11-16T18:19:41.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-16T18:19:41.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:41.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:40 smithi002 ceph-mon[98807]: Deploying daemon osd.5 on smithi188 2023-11-16T18:19:41.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-16T18:19:41.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:41.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:40 smithi040 ceph-mon[100251]: Deploying daemon osd.5 on smithi188 2023-11-16T18:19:42.245 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:42 smithi188 ceph-mon[100639]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:42 smithi040 ceph-mon[100251]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:42.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:42 smithi002 ceph-mon[98807]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:44.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:43 smithi002 ceph-mon[98807]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:44.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:43 smithi040 ceph-mon[100251]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:44.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:43 smithi188 ceph-mon[100639]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:46.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:46.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:46.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:46.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:46.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:46.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:46.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:47.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:46 smithi188 ceph-mon[100639]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:47.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:46 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:47.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:46 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:47.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:46 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:47.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:46 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:47.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:46 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:47.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:46 smithi002 ceph-mon[98807]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:47.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:46 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:47.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:46 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:47.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:46 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:47.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:46 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:47.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:46 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:47.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:46 smithi040 ceph-mon[100251]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:47.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:46 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:47.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:46 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:47.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:46 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:47.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:46 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:47.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:46 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:47.781 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:47.779+0000 7f58d4ff9700 1 -- 172.21.15.188:0/2496182611 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f58d8073ff0 con 0x7f58c00a11d0 2023-11-16T18:19:47.782 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 5 on host 'smithi188' 2023-11-16T18:19:47.783 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:47.782+0000 7f58de0a7700 1 -- 172.21.15.188:0/2496182611 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58c00a11d0 msgr2=0x7f58c00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:47.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:47.782+0000 7f58de0a7700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58c00a11d0 0x7f58c00a3680 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f58c800b5c0 tx=0x7f58c8005fd0 comp rx=0 tx=0).stop 2023-11-16T18:19:47.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:47.782+0000 7f58de0a7700 1 -- 172.21.15.188:0/2496182611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58d8101510 msgr2=0x7f58d8196500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:47.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:47.782+0000 7f58de0a7700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58d8101510 0x7f58d8196500 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00d9b0 tx=0x7f58cc00dcc0 comp rx=0 tx=0).stop 2023-11-16T18:19:47.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:47.782+0000 7f58de0a7700 1 -- 172.21.15.188:0/2496182611 shutdown_connections 2023-11-16T18:19:47.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:47.782+0000 7f58de0a7700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58c00a11d0 0x7f58c00a3680 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:47.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:47.782+0000 7f58de0a7700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58d8101ea0 0x7f58d819d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:47.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:47.782+0000 7f58de0a7700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58d8100310 0x7f58d8195fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:47.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:47.782+0000 7f58de0a7700 1 --2- 172.21.15.188:0/2496182611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58d8101510 0x7f58d8196500 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:47.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:47.782+0000 7f58de0a7700 1 -- 172.21.15.188:0/2496182611 >> 172.21.15.188:0/2496182611 conn(0x7f58d80fb960 msgr2=0x7f58d80fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:19:47.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:47.782+0000 7f58de0a7700 1 -- 172.21.15.188:0/2496182611 shutdown_connections 2023-11-16T18:19:47.785 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:19:47.782+0000 7f58de0a7700 1 -- 172.21.15.188:0/2496182611 wait complete. 2023-11-16T18:19:48.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:47 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:19:48.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:47 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:19:48.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:47 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:19:48.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:47 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:19:48.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:47 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:19:48.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:47 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:19:48.141 DEBUG:teuthology.orchestra.run.smithi188:osd.5> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.5.service 2023-11-16T18:19:48.143 INFO:tasks.cephadm:Deploying osd.6 on smithi002 with /dev/vg_nvme/lv_2... 2023-11-16T18:19:48.144 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-16T18:19:48.173 INFO:journalctl@ceph.osd.5.smithi188.stdout:-- Logs begin at Thu 2023-11-16 17:59:32 UTC. -- 2023-11-16T18:19:48.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:47 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:19:48.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:47 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:19:48.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:47 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:19:48.807 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:19:49.008 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:19:48 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:19:48.747+0000 7f53d6d27640 -1 osd.5 0 log_to_monitors true 2023-11-16T18:19:49.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:48 smithi188 ceph-mon[100639]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:49.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:49.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:49.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:49.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:48 smithi188 ceph-mon[100639]: from='osd.5 [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T18:19:49.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:48 smithi188 ceph-mon[100639]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T18:19:49.033 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:48 smithi002 ceph-mon[98807]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:49.034 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:49.034 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:49.034 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:49.035 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:48 smithi002 ceph-mon[98807]: from='osd.5 [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T18:19:49.035 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:48 smithi002 ceph-mon[98807]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T18:19:49.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:48 smithi040 ceph-mon[100251]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:49.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:49.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:49.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:19:49.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:48 smithi040 ceph-mon[100251]: from='osd.5 [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T18:19:49.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:48 smithi040 ceph-mon[100251]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T18:19:50.310 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-16T18:19:50.310 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-16T18:19:50.311 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2023-11-16T18:19:50.311 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2023-11-16T18:19:50.311 INFO:teuthology.orchestra.run.smithi002.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00849797 s, 1.2 GB/s 2023-11-16T18:19:50.311 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2023-11-16T18:19:50.663 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi002:vg_nvme/lv_2 2023-11-16T18:19:51.046 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:19:50.761+0000 7f53cc39e700 -1 osd.5 0 waiting for initial osdmap 2023-11-16T18:19:51.047 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:19:50.771+0000 7f53c418b700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T18:19:51.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-16T18:19:51.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: osdmap e35: 6 total, 5 up, 6 in 2023-11-16T18:19:51.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:51.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: from='osd.5 [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:19:51.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:19:51.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:51.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:51.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:51.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:51.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:51.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:51.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:51.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:51.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:50 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:51.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-16T18:19:51.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: osdmap e35: 6 total, 5 up, 6 in 2023-11-16T18:19:51.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:51.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: from='osd.5 [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:19:51.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:19:51.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:51.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:51.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:51.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:51.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:51.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:51.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:51.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:51.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:50 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:51.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-16T18:19:51.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: osdmap e35: 6 total, 5 up, 6 in 2023-11-16T18:19:51.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:51.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: from='osd.5 [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:19:51.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:19:51.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:51.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:51.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:51.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:51.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:19:51.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:51.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:51.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:19:51.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:50 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:19:51.334 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:19:51.779 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: purged_snaps scrub starts 2023-11-16T18:19:51.779 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: purged_snaps scrub ok 2023-11-16T18:19:51.779 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: Detected new or changed devices on smithi188 2023-11-16T18:19:51.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi188 to 8584M 2023-11-16T18:19:51.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:19:51.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:19:51.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:19:51.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-11-16T18:19:51.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: osdmap e36: 6 total, 5 up, 6 in 2023-11-16T18:19:51.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:51.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:51.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: osd.5 [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840] boot 2023-11-16T18:19:51.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: osdmap e37: 6 total, 6 up, 6 in 2023-11-16T18:19:51.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:51 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:52.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: purged_snaps scrub starts 2023-11-16T18:19:52.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: purged_snaps scrub ok 2023-11-16T18:19:52.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: Detected new or changed devices on smithi188 2023-11-16T18:19:52.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi188 to 8584M 2023-11-16T18:19:52.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:19:52.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:19:52.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:19:52.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-11-16T18:19:52.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: osdmap e36: 6 total, 5 up, 6 in 2023-11-16T18:19:52.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:52.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:52.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: osd.5 [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840] boot 2023-11-16T18:19:52.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: osdmap e37: 6 total, 6 up, 6 in 2023-11-16T18:19:52.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:51 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:52.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: purged_snaps scrub starts 2023-11-16T18:19:52.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: purged_snaps scrub ok 2023-11-16T18:19:52.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: Detected new or changed devices on smithi188 2023-11-16T18:19:52.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi188 to 8584M 2023-11-16T18:19:52.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:19:52.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:19:52.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:19:52.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-11-16T18:19:52.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: osdmap e36: 6 total, 5 up, 6 in 2023-11-16T18:19:52.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:52.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:52.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: osd.5 [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840] boot 2023-11-16T18:19:52.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: osdmap e37: 6 total, 6 up, 6 in 2023-11-16T18:19:52.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:51 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:19:52.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.519+0000 7efeecdd1700 1 -- 172.21.15.2:0/1449679280 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee8106010 msgr2=0x7efee8108410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:52.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.519+0000 7efeecdd1700 1 --2- 172.21.15.2:0/1449679280 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee8106010 0x7efee8108410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efee000b0a0 tx=0x7efee000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:19:52.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.519+0000 7efeecdd1700 1 -- 172.21.15.2:0/1449679280 shutdown_connections 2023-11-16T18:19:52.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.519+0000 7efeecdd1700 1 --2- 172.21.15.2:0/1449679280 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee8106010 0x7efee8108410 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:52.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.519+0000 7efeecdd1700 1 --2- 172.21.15.2:0/1449679280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efee8100dd0 0x7efee81031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:52.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.519+0000 7efeecdd1700 1 --2- 172.21.15.2:0/1449679280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efee81036f0 0x7efee8105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:52.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.519+0000 7efeecdd1700 1 -- 172.21.15.2:0/1449679280 >> 172.21.15.2:0/1449679280 conn(0x7efee80fa7f0 msgr2=0x7efee80fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:19:52.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.519+0000 7efeecdd1700 1 -- 172.21.15.2:0/1449679280 shutdown_connections 2023-11-16T18:19:52.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.519+0000 7efeecdd1700 1 -- 172.21.15.2:0/1449679280 wait complete. 2023-11-16T18:19:52.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.520+0000 7efeecdd1700 1 Processor -- start 2023-11-16T18:19:52.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.520+0000 7efeecdd1700 1 -- start start 2023-11-16T18:19:52.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.520+0000 7efeecdd1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efee8100dd0 0x7efee81961e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:19:52.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.520+0000 7efeecdd1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee81036f0 0x7efee8196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:19:52.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.520+0000 7efeecdd1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efee8106010 0x7efee819d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:19:52.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.520+0000 7efeecdd1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee819dd70 con 0x7efee81036f0 2023-11-16T18:19:52.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.520+0000 7efeecdd1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee819deb0 con 0x7efee8106010 2023-11-16T18:19:52.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.520+0000 7efeecdd1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee819e040 con 0x7efee8100dd0 2023-11-16T18:19:52.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.520+0000 7efee5d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee81036f0 0x7efee8196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:19:52.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.520+0000 7efee5d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee81036f0 0x7efee8196720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47118/0 (socket says 172.21.15.2:47118) 2023-11-16T18:19:52.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.520+0000 7efee5d9b700 1 -- 172.21.15.2:0/1954708031 learned_addr learned my addr 172.21.15.2:0/1954708031 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:19:52.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.521+0000 7efee659c700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efee8100dd0 0x7efee81961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:19:52.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.521+0000 7efee6d9d700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efee8106010 0x7efee819d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:19:52.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.521+0000 7efee5d9b700 1 -- 172.21.15.2:0/1954708031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efee8100dd0 msgr2=0x7efee81961e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:52.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.521+0000 7efee5d9b700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efee8100dd0 0x7efee81961e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:52.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.521+0000 7efee5d9b700 1 -- 172.21.15.2:0/1954708031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efee8106010 msgr2=0x7efee819d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:19:52.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.521+0000 7efee5d9b700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efee8106010 0x7efee819d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:19:52.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.521+0000 7efee5d9b700 1 -- 172.21.15.2:0/1954708031 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee0009d00 con 0x7efee81036f0 2023-11-16T18:19:52.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.521+0000 7efee5d9b700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee81036f0 0x7efee8196720 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efed800b560 tx=0x7efed800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:19:52.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.521+0000 7efed77fe700 1 -- 172.21.15.2:0/1954708031 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed8011830 con 0x7efee81036f0 2023-11-16T18:19:52.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.521+0000 7efeecdd1700 1 -- 172.21.15.2:0/1954708031 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee810bcb0 con 0x7efee81036f0 2023-11-16T18:19:52.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.521+0000 7efed77fe700 1 -- 172.21.15.2:0/1954708031 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efed8011e50 con 0x7efee81036f0 2023-11-16T18:19:52.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.521+0000 7efed77fe700 1 -- 172.21.15.2:0/1954708031 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed8010e50 con 0x7efee81036f0 2023-11-16T18:19:52.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.521+0000 7efeecdd1700 1 -- 172.21.15.2:0/1954708031 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee810bb20 con 0x7efee81036f0 2023-11-16T18:19:52.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.522+0000 7efeecdd1700 1 -- 172.21.15.2:0/1954708031 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee80fc360 con 0x7efee81036f0 2023-11-16T18:19:52.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.523+0000 7efed77fe700 1 -- 172.21.15.2:0/1954708031 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7efed80104f0 con 0x7efee81036f0 2023-11-16T18:19:52.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.524+0000 7efed77fe700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efed00a1150 0x7efed00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:19:52.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.524+0000 7efee659c700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efed00a1150 0x7efed00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:19:52.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.524+0000 7efed77fe700 1 -- 172.21.15.2:0/1954708031 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4562+0+0 (secure 0 0 0) 0x7efed80cb430 con 0x7efee81036f0 2023-11-16T18:19:52.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.524+0000 7efee659c700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efed00a1150 0x7efed00a3600 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efedc005950 tx=0x7efedc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:19:52.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.528+0000 7efed77fe700 1 -- 172.21.15.2:0/1954708031 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efed8095a20 con 0x7efee81036f0 2023-11-16T18:19:52.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:19:52.736+0000 7efeecdd1700 1 -- 172.21.15.2:0/1954708031 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7efee8061c60 con 0x7efed00a1150 2023-11-16T18:19:53.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:52 smithi188 ceph-mon[100639]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:53.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:52 smithi188 ceph-mon[100639]: osdmap e38: 6 total, 6 up, 6 in 2023-11-16T18:19:53.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:19:53.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:19:53.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:53.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:52 smithi002 ceph-mon[98807]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:53.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:52 smithi002 ceph-mon[98807]: osdmap e38: 6 total, 6 up, 6 in 2023-11-16T18:19:53.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:19:53.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:19:53.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:53.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:52 smithi040 ceph-mon[100251]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T18:19:53.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:52 smithi040 ceph-mon[100251]: osdmap e38: 6 total, 6 up, 6 in 2023-11-16T18:19:53.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:19:53.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:19:53.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:19:53.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:53 smithi002 ceph-mon[98807]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:19:53.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:53 smithi002 ceph-mon[98807]: osdmap e39: 6 total, 6 up, 6 in 2023-11-16T18:19:54.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:53 smithi188 ceph-mon[100639]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:19:54.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:53 smithi188 ceph-mon[100639]: osdmap e39: 6 total, 6 up, 6 in 2023-11-16T18:19:54.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:53 smithi040 ceph-mon[100251]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:19:54.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:53 smithi040 ceph-mon[100251]: osdmap e39: 6 total, 6 up, 6 in 2023-11-16T18:19:55.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:54 smithi188 ceph-mon[100639]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:19:55.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:54 smithi188 ceph-mon[100639]: osdmap e40: 6 total, 6 up, 6 in 2023-11-16T18:19:55.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:54 smithi002 ceph-mon[98807]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:19:55.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:54 smithi002 ceph-mon[98807]: osdmap e40: 6 total, 6 up, 6 in 2023-11-16T18:19:55.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:54 smithi040 ceph-mon[100251]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:19:55.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:54 smithi040 ceph-mon[100251]: osdmap e40: 6 total, 6 up, 6 in 2023-11-16T18:19:56.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:55 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3902170575' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d97c575-96ae-46ad-a165-36cc16633818"}]: dispatch 2023-11-16T18:19:56.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:55 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3902170575' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d97c575-96ae-46ad-a165-36cc16633818"}]': finished 2023-11-16T18:19:56.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:55 smithi188 ceph-mon[100639]: osdmap e41: 7 total, 6 up, 7 in 2023-11-16T18:19:56.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:55 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:19:56.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:55 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2335359307' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:19:56.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:55 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3902170575' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d97c575-96ae-46ad-a165-36cc16633818"}]: dispatch 2023-11-16T18:19:56.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:55 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3902170575' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d97c575-96ae-46ad-a165-36cc16633818"}]': finished 2023-11-16T18:19:56.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:55 smithi002 ceph-mon[98807]: osdmap e41: 7 total, 6 up, 7 in 2023-11-16T18:19:56.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:55 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:19:56.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:55 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2335359307' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:19:56.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:55 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3902170575' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d97c575-96ae-46ad-a165-36cc16633818"}]: dispatch 2023-11-16T18:19:56.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:55 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3902170575' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d97c575-96ae-46ad-a165-36cc16633818"}]': finished 2023-11-16T18:19:56.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:55 smithi040 ceph-mon[100251]: osdmap e41: 7 total, 6 up, 7 in 2023-11-16T18:19:56.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:55 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:19:56.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:55 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2335359307' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:19:57.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:56 smithi188 ceph-mon[100639]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T18:19:57.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:56 smithi002 ceph-mon[98807]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T18:19:57.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:56 smithi040 ceph-mon[100251]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T18:19:59.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:19:58 smithi188 ceph-mon[100639]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-11-16T18:19:59.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:19:58 smithi002 ceph-mon[98807]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-11-16T18:19:59.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:19:58 smithi040 ceph-mon[100251]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-11-16T18:20:01.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:00 smithi188 ceph-mon[100639]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2023-11-16T18:20:01.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:00 smithi188 ceph-mon[100639]: overall HEALTH_OK 2023-11-16T18:20:01.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:00 smithi002 ceph-mon[98807]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2023-11-16T18:20:01.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:00 smithi002 ceph-mon[98807]: overall HEALTH_OK 2023-11-16T18:20:01.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:00 smithi040 ceph-mon[100251]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2023-11-16T18:20:01.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:00 smithi040 ceph-mon[100251]: overall HEALTH_OK 2023-11-16T18:20:02.385 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-16T18:20:02.385 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:02.385 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:02 smithi002 ceph-mon[98807]: Deploying daemon osd.6 on smithi002 2023-11-16T18:20:02.385 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:02 smithi002 ceph-mon[98807]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T18:20:02.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:02 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-16T18:20:02.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:02 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:02.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:02 smithi040 ceph-mon[100251]: Deploying daemon osd.6 on smithi002 2023-11-16T18:20:02.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:02 smithi040 ceph-mon[100251]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T18:20:02.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:02 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-16T18:20:02.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:02 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:02.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:02 smithi188 ceph-mon[100639]: Deploying daemon osd.6 on smithi002 2023-11-16T18:20:02.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:02 smithi188 ceph-mon[100639]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T18:20:03.964 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:03 smithi002 ceph-mon[98807]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2023-11-16T18:20:04.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:03 smithi040 ceph-mon[100251]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2023-11-16T18:20:04.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:03 smithi188 ceph-mon[100639]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2023-11-16T18:20:06.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:05 smithi002 ceph-mon[98807]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:20:06.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:05 smithi040 ceph-mon[100251]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:20:06.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:05 smithi188 ceph-mon[100639]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:20:07.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:06 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:07.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:06 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:07.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:06 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:07.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:06 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:07.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:06 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:07.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:06 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:07.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:06 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:07.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:06 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:07.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:06 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:08.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:07 smithi002 ceph-mon[98807]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:20:08.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:07 smithi040 ceph-mon[100251]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:20:08.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:07 smithi188 ceph-mon[100639]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:20:08.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:08.734+0000 7efed77fe700 1 -- 172.21.15.2:0/1954708031 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efee8061c60 con 0x7efed00a1150 2023-11-16T18:20:08.737 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 6 on host 'smithi002' 2023-11-16T18:20:08.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:08.736+0000 7efeecdd1700 1 -- 172.21.15.2:0/1954708031 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efed00a1150 msgr2=0x7efed00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:08.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:08.736+0000 7efeecdd1700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efed00a1150 0x7efed00a3600 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efedc005950 tx=0x7efedc00b410 comp rx=0 tx=0).stop 2023-11-16T18:20:08.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:08.736+0000 7efeecdd1700 1 -- 172.21.15.2:0/1954708031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee81036f0 msgr2=0x7efee8196720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:08.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:08.736+0000 7efeecdd1700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee81036f0 0x7efee8196720 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efed800b560 tx=0x7efed800b870 comp rx=0 tx=0).stop 2023-11-16T18:20:08.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:08.737+0000 7efeecdd1700 1 -- 172.21.15.2:0/1954708031 shutdown_connections 2023-11-16T18:20:08.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:08.737+0000 7efeecdd1700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efed00a1150 0x7efed00a3600 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:08.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:08.737+0000 7efeecdd1700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efee8100dd0 0x7efee81961e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:08.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:08.737+0000 7efeecdd1700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efee8106010 0x7efee819d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:08.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:08.737+0000 7efeecdd1700 1 --2- 172.21.15.2:0/1954708031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee81036f0 0x7efee8196720 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:08.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:08.737+0000 7efeecdd1700 1 -- 172.21.15.2:0/1954708031 >> 172.21.15.2:0/1954708031 conn(0x7efee80fa7f0 msgr2=0x7efee8101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:20:08.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:08.737+0000 7efeecdd1700 1 -- 172.21.15.2:0/1954708031 shutdown_connections 2023-11-16T18:20:08.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:08.737+0000 7efeecdd1700 1 -- 172.21.15.2:0/1954708031 wait complete. 2023-11-16T18:20:09.227 DEBUG:teuthology.orchestra.run.smithi002:osd.6> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.6.service 2023-11-16T18:20:09.230 INFO:tasks.cephadm:Deploying osd.7 on smithi040 with /dev/vg_nvme/lv_2... 2023-11-16T18:20:09.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-16T18:20:09.259 INFO:journalctl@ceph.osd.6.smithi002.stdout:-- Logs begin at Thu 2023-11-16 17:59:05 UTC. -- 2023-11-16T18:20:09.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:09 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:09 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:09 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:09.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:09 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:09.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:09 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:20:09.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:09 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:09 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:20:09.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:09 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:20:09.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:09 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:09 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:09 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:09.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:09 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:09 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:09 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:09.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:09 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:09.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:09 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:20:09.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:09 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:09 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:20:09.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:09 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:20:09.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:09 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:09 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.551 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:09 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:09.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:09.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:09.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:09 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:20:09.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:09 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:20:09.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:09 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:20:09.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:09.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:09 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T18:20:09.993 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:20:09 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:20:09.829+0000 7fe2432e1640 -1 osd.6 0 log_to_monitors true 2023-11-16T18:20:10.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:10 smithi002 ceph-mon[98807]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:20:10.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:10 smithi002 ceph-mon[98807]: from='osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T18:20:10.459 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:10 smithi040 ceph-mon[100251]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:20:10.459 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:10 smithi040 ceph-mon[100251]: from='osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T18:20:10.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:10 smithi188 ceph-mon[100639]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:20:10.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:10 smithi188 ceph-mon[100639]: from='osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T18:20:11.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:11 smithi002 ceph-mon[98807]: from='osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-16T18:20:11.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:11 smithi002 ceph-mon[98807]: osdmap e42: 7 total, 6 up, 7 in 2023-11-16T18:20:11.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:11 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:11.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:11 smithi002 ceph-mon[98807]: from='osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T18:20:11.357 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:20:11 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:20:11.210+0000 7fe238958700 -1 osd.6 0 waiting for initial osdmap 2023-11-16T18:20:11.357 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:20:11 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:20:11.221+0000 7fe230745700 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T18:20:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:11 smithi040 ceph-mon[100251]: from='osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-16T18:20:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:11 smithi040 ceph-mon[100251]: osdmap e42: 7 total, 6 up, 7 in 2023-11-16T18:20:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:11 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:11.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:11 smithi040 ceph-mon[100251]: from='osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T18:20:11.500 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-16T18:20:11.500 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-16T18:20:11.500 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2023-11-16T18:20:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2023-11-16T18:20:11.501 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108672 s, 965 MB/s 2023-11-16T18:20:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2023-11-16T18:20:11.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:11 smithi188 ceph-mon[100639]: from='osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-16T18:20:11.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:11 smithi188 ceph-mon[100639]: osdmap e42: 7 total, 6 up, 7 in 2023-11-16T18:20:11.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:11 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:11.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:11 smithi188 ceph-mon[100639]: from='osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T18:20:11.964 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi040:vg_nvme/lv_2 2023-11-16T18:20:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: from='osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T18:20:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: osdmap e43: 7 total, 6 up, 7 in 2023-11-16T18:20:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: Detected new or changed devices on smithi002 2023-11-16T18:20:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi002 to 5724M 2023-11-16T18:20:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:20:12.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:12.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:20:12.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:20:12.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:20:12.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: from='osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T18:20:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: osdmap e43: 7 total, 6 up, 7 in 2023-11-16T18:20:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: Detected new or changed devices on smithi002 2023-11-16T18:20:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:12.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi002 to 5724M 2023-11-16T18:20:12.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:12.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:12.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:12.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:20:12.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:12.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:20:12.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:20:12.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:20:12.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: from='osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T18:20:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: osdmap e43: 7 total, 6 up, 7 in 2023-11-16T18:20:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: Detected new or changed devices on smithi002 2023-11-16T18:20:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:12.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:12.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:12.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:12.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:12.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi002 to 5724M 2023-11-16T18:20:12.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:12.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:12.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:12.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:20:12.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:12.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:20:12.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:20:12.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T18:20:12.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:12.676 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T18:20:13.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:13 smithi040 ceph-mon[100251]: purged_snaps scrub starts 2023-11-16T18:20:13.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:13 smithi040 ceph-mon[100251]: purged_snaps scrub ok 2023-11-16T18:20:13.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:13 smithi040 ceph-mon[100251]: osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812] boot 2023-11-16T18:20:13.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:13 smithi040 ceph-mon[100251]: osdmap e44: 7 total, 7 up, 7 in 2023-11-16T18:20:13.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:13 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:13.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:13 smithi188 ceph-mon[100639]: purged_snaps scrub starts 2023-11-16T18:20:13.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:13 smithi188 ceph-mon[100639]: purged_snaps scrub ok 2023-11-16T18:20:13.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:13 smithi188 ceph-mon[100639]: osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812] boot 2023-11-16T18:20:13.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:13 smithi188 ceph-mon[100639]: osdmap e44: 7 total, 7 up, 7 in 2023-11-16T18:20:13.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:13 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:13.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:13 smithi002 ceph-mon[98807]: purged_snaps scrub starts 2023-11-16T18:20:13.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:13 smithi002 ceph-mon[98807]: purged_snaps scrub ok 2023-11-16T18:20:13.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:13 smithi002 ceph-mon[98807]: osd.6 [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812] boot 2023-11-16T18:20:13.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:13 smithi002 ceph-mon[98807]: osdmap e44: 7 total, 7 up, 7 in 2023-11-16T18:20:13.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:13 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:20:14.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.076+0000 7f9852767700 1 -- 172.21.15.40:0/2973928935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f984c1003e0 msgr2=0x7f984c100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:14.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.076+0000 7f9852767700 1 --2- 172.21.15.40:0/2973928935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f984c1003e0 0x7f984c100830 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f98400098f0 tx=0x7f9840009c00 comp rx=0 tx=0).stop 2023-11-16T18:20:14.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.076+0000 7f9852767700 1 -- 172.21.15.40:0/2973928935 shutdown_connections 2023-11-16T18:20:14.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.076+0000 7f9852767700 1 --2- 172.21.15.40:0/2973928935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f984c100d70 0x7f984c107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:14.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.076+0000 7f9852767700 1 --2- 172.21.15.40:0/2973928935 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f984c0ff1e0 0x7f984c0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.076+0000 7f9852767700 1 --2- 172.21.15.40:0/2973928935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f984c1003e0 0x7f984c100830 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.076+0000 7f9852767700 1 -- 172.21.15.40:0/2973928935 >> 172.21.15.40:0/2973928935 conn(0x7f984c0fa7f0 msgr2=0x7f984c0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:20:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.076+0000 7f9852767700 1 -- 172.21.15.40:0/2973928935 shutdown_connections 2023-11-16T18:20:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.076+0000 7f9852767700 1 -- 172.21.15.40:0/2973928935 wait complete. 2023-11-16T18:20:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.077+0000 7f9852767700 1 Processor -- start 2023-11-16T18:20:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.077+0000 7f9852767700 1 -- start start 2023-11-16T18:20:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.077+0000 7f9852767700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f984c0ff1e0 0x7f984c196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:20:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.077+0000 7f9852767700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f984c1003e0 0x7f984c196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:20:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.077+0000 7f9852767700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f984c100d70 0x7f984c19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:20:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.077+0000 7f9852767700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984c19dbd0 con 0x7f984c1003e0 2023-11-16T18:20:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.077+0000 7f9852767700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984c19dd10 con 0x7f984c100d70 2023-11-16T18:20:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.077+0000 7f9852767700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984c19dea0 con 0x7f984c0ff1e0 2023-11-16T18:20:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.077+0000 7f9850d04700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f984c100d70 0x7f984c19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:20:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.077+0000 7f984bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f984c0ff1e0 0x7f984c196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:20:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.077+0000 7f9850d04700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f984c100d70 0x7f984c19d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48948/0 (socket says 172.21.15.40:48948) 2023-11-16T18:20:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.077+0000 7f9850d04700 1 -- 172.21.15.40:0/2251729295 learned_addr learned my addr 172.21.15.40:0/2251729295 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-11-16T18:20:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.078+0000 7f9850d04700 1 -- 172.21.15.40:0/2251729295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f984c0ff1e0 msgr2=0x7f984c196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.078+0000 7f984b7fe700 1 --2- 172.21.15.40:0/2251729295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f984c1003e0 0x7f984c196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:20:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.078+0000 7f9850d04700 1 --2- 172.21.15.40:0/2251729295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f984c0ff1e0 0x7f984c196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.078+0000 7f9850d04700 1 -- 172.21.15.40:0/2251729295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f984c1003e0 msgr2=0x7f984c196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.078+0000 7f9850d04700 1 --2- 172.21.15.40:0/2251729295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f984c1003e0 0x7f984c196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.078+0000 7f9850d04700 1 -- 172.21.15.40:0/2251729295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9844009d00 con 0x7f984c100d70 2023-11-16T18:20:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.078+0000 7f9850d04700 1 --2- 172.21.15.40:0/2251729295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f984c100d70 0x7f984c19d600 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f984400e3c0 tx=0x7f984400e6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:20:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.078+0000 7f98497fa700 1 -- 172.21.15.40:0/2251729295 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98440092d0 con 0x7f984c100d70 2023-11-16T18:20:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.078+0000 7f98497fa700 1 -- 172.21.15.40:0/2251729295 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f984400f040 con 0x7f984c100d70 2023-11-16T18:20:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.078+0000 7f9852767700 1 -- 172.21.15.40:0/2251729295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9840009520 con 0x7f984c100d70 2023-11-16T18:20:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.078+0000 7f98497fa700 1 -- 172.21.15.40:0/2251729295 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844007d30 con 0x7f984c100d70 2023-11-16T18:20:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.078+0000 7f9852767700 1 -- 172.21.15.40:0/2251729295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f984c10ab40 con 0x7f984c100d70 2023-11-16T18:20:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.079+0000 7f9852767700 1 -- 172.21.15.40:0/2251729295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f984c066ea0 con 0x7f984c100d70 2023-11-16T18:20:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.080+0000 7f98497fa700 1 -- 172.21.15.40:0/2251729295 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f984401d400 con 0x7f984c100d70 2023-11-16T18:20:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.081+0000 7f98497fa700 1 --2- 172.21.15.40:0/2251729295 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f98340a1150 0x7f98340a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:20:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.081+0000 7f98497fa700 1 -- 172.21.15.40:0/2251729295 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f98440cb9b0 con 0x7f984c100d70 2023-11-16T18:20:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.081+0000 7f984bfff700 1 --2- 172.21.15.40:0/2251729295 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f98340a1150 0x7f98340a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:20:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.082+0000 7f984bfff700 1 --2- 172.21.15.40:0/2251729295 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f98340a1150 0x7f98340a3600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f984c100240 tx=0x7f983c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:20:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.085+0000 7f98497fa700 1 -- 172.21.15.40:0/2251729295 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9844095ea0 con 0x7f984c100d70 2023-11-16T18:20:14.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:14.289+0000 7f9852767700 1 -- 172.21.15.40:0/2251729295 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f984c19e950 con 0x7f98340a1150 2023-11-16T18:20:14.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:14 smithi040 ceph-mon[100251]: osdmap e45: 7 total, 7 up, 7 in 2023-11-16T18:20:14.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:14 smithi040 ceph-mon[100251]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:14.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:14 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:20:14.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:14 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:20:14.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:14 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:14.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:14 smithi188 ceph-mon[100639]: osdmap e45: 7 total, 7 up, 7 in 2023-11-16T18:20:14.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:14 smithi188 ceph-mon[100639]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:14.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:14 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:20:14.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:14 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:20:14.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:14 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:14.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:14 smithi002 ceph-mon[98807]: osdmap e45: 7 total, 7 up, 7 in 2023-11-16T18:20:14.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:14 smithi002 ceph-mon[98807]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:14.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:14 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:20:14.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:14 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:20:14.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:14 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:15.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:15 smithi040 ceph-mon[100251]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:20:15.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:15 smithi188 ceph-mon[100639]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:20:15.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:15 smithi002 ceph-mon[98807]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:20:16.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:16 smithi040 ceph-mon[100251]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:16.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:16 smithi188 ceph-mon[100639]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:16.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:16 smithi002 ceph-mon[98807]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:17 smithi040 ceph-mon[100251]: from='client.? 172.21.15.40:0/2433200834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36"}]: dispatch 2023-11-16T18:20:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:17 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36"}]: dispatch 2023-11-16T18:20:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:17 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36"}]': finished 2023-11-16T18:20:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:17 smithi040 ceph-mon[100251]: osdmap e46: 8 total, 7 up, 8 in 2023-11-16T18:20:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:17.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:17 smithi188 ceph-mon[100639]: from='client.? 172.21.15.40:0/2433200834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36"}]: dispatch 2023-11-16T18:20:17.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:17 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36"}]: dispatch 2023-11-16T18:20:17.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:17 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36"}]': finished 2023-11-16T18:20:17.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:17 smithi188 ceph-mon[100639]: osdmap e46: 8 total, 7 up, 8 in 2023-11-16T18:20:17.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:17.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:17 smithi002 ceph-mon[98807]: from='client.? 172.21.15.40:0/2433200834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36"}]: dispatch 2023-11-16T18:20:17.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:17 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36"}]: dispatch 2023-11-16T18:20:17.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:17 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36"}]': finished 2023-11-16T18:20:17.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:17 smithi002 ceph-mon[98807]: osdmap e46: 8 total, 7 up, 8 in 2023-11-16T18:20:17.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:18.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:18 smithi188 ceph-mon[100639]: from='client.? 172.21.15.40:0/2221216400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:20:18.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:18 smithi188 ceph-mon[100639]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:18.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:18 smithi002 ceph-mon[98807]: from='client.? 172.21.15.40:0/2221216400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:20:18.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:18 smithi002 ceph-mon[98807]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:18.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:18 smithi040 ceph-mon[100251]: from='client.? 172.21.15.40:0/2221216400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:20:18.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:18 smithi040 ceph-mon[100251]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:20.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:19 smithi002 ceph-mon[98807]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:19 smithi040 ceph-mon[100251]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:20.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:19 smithi188 ceph-mon[100639]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:22.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:22 smithi040 ceph-mon[100251]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:22.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:22 smithi188 ceph-mon[100639]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:22.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:22 smithi002 ceph-mon[98807]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:24.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:23 smithi002 ceph-mon[98807]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:24.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:23 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-16T18:20:24.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:23 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:23 smithi040 ceph-mon[100251]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-16T18:20:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:24.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:23 smithi188 ceph-mon[100639]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:24.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:23 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-16T18:20:24.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:23 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:25.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:24 smithi002 ceph-mon[98807]: Deploying daemon osd.7 on smithi040 2023-11-16T18:20:25.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:24 smithi040 ceph-mon[100251]: Deploying daemon osd.7 on smithi040 2023-11-16T18:20:25.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:24 smithi188 ceph-mon[100639]: Deploying daemon osd.7 on smithi040 2023-11-16T18:20:26.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:25 smithi002 ceph-mon[98807]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:26.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:25 smithi040 ceph-mon[100251]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:26.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:25 smithi188 ceph-mon[100639]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:28.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:27 smithi002 ceph-mon[98807]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:28.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:27 smithi040 ceph-mon[100251]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:28.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:27 smithi188 ceph-mon[100639]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:29.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:29.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:29.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:29.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:29.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:29.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:30.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:30.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:30.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:30.831 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:30 smithi040 ceph-mon[100251]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:30.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:30 smithi002 ceph-mon[98807]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:31.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:30 smithi188 ceph-mon[100639]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:31.919+0000 7f98497fa700 1 -- 172.21.15.40:0/2251729295 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f984c19e950 con 0x7f98340a1150 2023-11-16T18:20:31.921 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 7 on host 'smithi040' 2023-11-16T18:20:31.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:31.921+0000 7f9852767700 1 -- 172.21.15.40:0/2251729295 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f98340a1150 msgr2=0x7f98340a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:31.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:31.921+0000 7f9852767700 1 --2- 172.21.15.40:0/2251729295 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f98340a1150 0x7f98340a3600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f984c100240 tx=0x7f983c009260 comp rx=0 tx=0).stop 2023-11-16T18:20:31.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:31.921+0000 7f9852767700 1 -- 172.21.15.40:0/2251729295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f984c100d70 msgr2=0x7f984c19d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:31.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:31.921+0000 7f9852767700 1 --2- 172.21.15.40:0/2251729295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f984c100d70 0x7f984c19d600 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f984400e3c0 tx=0x7f984400e6d0 comp rx=0 tx=0).stop 2023-11-16T18:20:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:31.921+0000 7f9852767700 1 -- 172.21.15.40:0/2251729295 shutdown_connections 2023-11-16T18:20:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:31.921+0000 7f9852767700 1 --2- 172.21.15.40:0/2251729295 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f98340a1150 0x7f98340a3600 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:31.921+0000 7f9852767700 1 --2- 172.21.15.40:0/2251729295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f984c0ff1e0 0x7f984c196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:31.921+0000 7f9852767700 1 --2- 172.21.15.40:0/2251729295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f984c100d70 0x7f984c19d600 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:31.921+0000 7f9852767700 1 --2- 172.21.15.40:0/2251729295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f984c1003e0 0x7f984c196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:31.921+0000 7f9852767700 1 -- 172.21.15.40:0/2251729295 >> 172.21.15.40:0/2251729295 conn(0x7f984c0fa7f0 msgr2=0x7f984c068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:20:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:31.922+0000 7f9852767700 1 -- 172.21.15.40:0/2251729295 shutdown_connections 2023-11-16T18:20:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:20:31.922+0000 7f9852767700 1 -- 172.21.15.40:0/2251729295 wait complete. 2023-11-16T18:20:32.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:32.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:32.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:31 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:20:32.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:31 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:20:32.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:31 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:20:32.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:31 smithi040 ceph-mon[100251]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:32.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:32.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:32.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:32.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:31 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:20:32.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:31 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:20:32.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:31 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:20:32.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:31 smithi188 ceph-mon[100639]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:32.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:32.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:32.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:32.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:31 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:20:32.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:31 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:20:32.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:31 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:20:32.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:31 smithi002 ceph-mon[98807]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:32.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:32.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:32.366 DEBUG:teuthology.orchestra.run.smithi040:osd.7> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.7.service 2023-11-16T18:20:32.368 INFO:tasks.cephadm:Deploying osd.8 on smithi188 with /dev/vg_nvme/lv_2... 2023-11-16T18:20:32.368 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-16T18:20:32.398 INFO:journalctl@ceph.osd.7.smithi040.stdout:-- Logs begin at Thu 2023-11-16 17:59:26 UTC. -- 2023-11-16T18:20:32.734 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:20:32 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:20:32.503+0000 7f35fbde4640 -1 osd.7 0 log_to_monitors true 2023-11-16T18:20:33.033 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:20:33.221 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:32 smithi040 ceph-mon[100251]: from='osd.7 [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T18:20:33.222 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:32 smithi040 ceph-mon[100251]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T18:20:33.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:32 smithi188 ceph-mon[100639]: from='osd.7 [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T18:20:33.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:32 smithi188 ceph-mon[100639]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T18:20:33.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:32 smithi002 ceph-mon[98807]: from='osd.7 [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T18:20:33.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:32 smithi002 ceph-mon[98807]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T18:20:34.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:33 smithi040 ceph-mon[100251]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-16T18:20:34.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:33 smithi040 ceph-mon[100251]: osdmap e47: 8 total, 7 up, 8 in 2023-11-16T18:20:34.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:34.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:33 smithi040 ceph-mon[100251]: from='osd.7 [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:20:34.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:33 smithi040 ceph-mon[100251]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:20:34.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:33 smithi040 ceph-mon[100251]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:34.236 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:20:33 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:20:33.965+0000 7f35f145b700 -1 osd.7 0 waiting for initial osdmap 2023-11-16T18:20:34.236 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:20:33 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:20:33.975+0000 7f35ea24a700 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T18:20:34.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:33 smithi188 ceph-mon[100639]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-16T18:20:34.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:33 smithi188 ceph-mon[100639]: osdmap e47: 8 total, 7 up, 8 in 2023-11-16T18:20:34.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:34.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:33 smithi188 ceph-mon[100639]: from='osd.7 [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:20:34.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:33 smithi188 ceph-mon[100639]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:20:34.302 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:33 smithi188 ceph-mon[100639]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:34.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:33 smithi002 ceph-mon[98807]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-16T18:20:34.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:33 smithi002 ceph-mon[98807]: osdmap e47: 8 total, 7 up, 8 in 2023-11-16T18:20:34.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:34.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:33 smithi002 ceph-mon[98807]: from='osd.7 [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:20:34.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:33 smithi002 ceph-mon[98807]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:20:34.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:33 smithi002 ceph-mon[98807]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:34.809 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-16T18:20:34.809 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-16T18:20:34.809 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2023-11-16T18:20:34.809 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2023-11-16T18:20:34.809 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110913 s, 945 MB/s 2023-11-16T18:20:34.809 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2023-11-16T18:20:35.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-11-16T18:20:35.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: osdmap e48: 8 total, 7 up, 8 in 2023-11-16T18:20:35.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:35.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:35.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: Detected new or changed devices on smithi040 2023-11-16T18:20:35.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:35.176 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:35.176 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:35.176 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:35.176 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:35.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi040 to 5724M 2023-11-16T18:20:35.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:35.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:35.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:35.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:20:35.180 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:35.180 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:20:35.180 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:34 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:20:35.181 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi188:vg_nvme/lv_2 2023-11-16T18:20:35.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-11-16T18:20:35.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: osdmap e48: 8 total, 7 up, 8 in 2023-11-16T18:20:35.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:35.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:35.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: Detected new or changed devices on smithi040 2023-11-16T18:20:35.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:35.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:35.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:35.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:35.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:35.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi040 to 5724M 2023-11-16T18:20:35.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:35.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:35.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:35.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:20:35.242 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:35.243 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:20:35.243 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:34 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:20:35.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-11-16T18:20:35.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: osdmap e48: 8 total, 7 up, 8 in 2023-11-16T18:20:35.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:35.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:35.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: Detected new or changed devices on smithi040 2023-11-16T18:20:35.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:35.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:35.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:35.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:35.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:35.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi040 to 5724M 2023-11-16T18:20:35.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:35.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:35.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:35.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:20:35.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:35.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:20:35.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:34 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:20:35.826 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:20:36.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:35 smithi040 ceph-mon[100251]: purged_snaps scrub starts 2023-11-16T18:20:36.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:35 smithi040 ceph-mon[100251]: purged_snaps scrub ok 2023-11-16T18:20:36.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:35 smithi040 ceph-mon[100251]: osd.7 [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472] boot 2023-11-16T18:20:36.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:35 smithi040 ceph-mon[100251]: osdmap e49: 8 total, 8 up, 8 in 2023-11-16T18:20:36.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:36.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:35 smithi040 ceph-mon[100251]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:36.238 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:35 smithi188 ceph-mon[100639]: purged_snaps scrub starts 2023-11-16T18:20:36.238 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:35 smithi188 ceph-mon[100639]: purged_snaps scrub ok 2023-11-16T18:20:36.238 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:35 smithi188 ceph-mon[100639]: osd.7 [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472] boot 2023-11-16T18:20:36.238 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:35 smithi188 ceph-mon[100639]: osdmap e49: 8 total, 8 up, 8 in 2023-11-16T18:20:36.238 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:36.238 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:35 smithi188 ceph-mon[100639]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:36.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:35 smithi002 ceph-mon[98807]: purged_snaps scrub starts 2023-11-16T18:20:36.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:35 smithi002 ceph-mon[98807]: purged_snaps scrub ok 2023-11-16T18:20:36.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:35 smithi002 ceph-mon[98807]: osd.7 [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472] boot 2023-11-16T18:20:36.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:35 smithi002 ceph-mon[98807]: osdmap e49: 8 total, 8 up, 8 in 2023-11-16T18:20:36.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:20:36.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:35 smithi002 ceph-mon[98807]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T18:20:37.120 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.116+0000 7f1c21c7f700 1 -- 172.21.15.188:0/2647331150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c1c1003e0 msgr2=0x7f1c1c100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:37.120 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.116+0000 7f1c21c7f700 1 --2- 172.21.15.188:0/2647331150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c1c1003e0 0x7f1c1c100830 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1c10009990 tx=0x7f1c10009ca0 comp rx=0 tx=0).stop 2023-11-16T18:20:37.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.116+0000 7f1c21c7f700 1 -- 172.21.15.188:0/2647331150 shutdown_connections 2023-11-16T18:20:37.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.116+0000 7f1c21c7f700 1 --2- 172.21.15.188:0/2647331150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c1c100d70 0x7f1c1c107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:37.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.116+0000 7f1c21c7f700 1 --2- 172.21.15.188:0/2647331150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c1c0ff1e0 0x7f1c1c0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:37.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.116+0000 7f1c21c7f700 1 --2- 172.21.15.188:0/2647331150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c1c1003e0 0x7f1c1c100830 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:37.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.116+0000 7f1c21c7f700 1 -- 172.21.15.188:0/2647331150 >> 172.21.15.188:0/2647331150 conn(0x7f1c1c0fa7f0 msgr2=0x7f1c1c0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:20:37.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.116+0000 7f1c21c7f700 1 -- 172.21.15.188:0/2647331150 shutdown_connections 2023-11-16T18:20:37.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.117+0000 7f1c21c7f700 1 -- 172.21.15.188:0/2647331150 wait complete. 2023-11-16T18:20:37.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.117+0000 7f1c21c7f700 1 Processor -- start 2023-11-16T18:20:37.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.117+0000 7f1c21c7f700 1 -- start start 2023-11-16T18:20:37.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.117+0000 7f1c21c7f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c1c0ff1e0 0x7f1c1c196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:20:37.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.117+0000 7f1c21c7f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c1c1003e0 0x7f1c1c196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:20:37.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.117+0000 7f1c21c7f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c1c100d70 0x7f1c1c19d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:20:37.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.117+0000 7f1c21c7f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c19dbe0 con 0x7f1c1c100d70 2023-11-16T18:20:37.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.117+0000 7f1c21c7f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c19dd20 con 0x7f1c1c0ff1e0 2023-11-16T18:20:37.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.117+0000 7f1c21c7f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c19deb0 con 0x7f1c1c1003e0 2023-11-16T18:20:37.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.117+0000 7f1c1affd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c1c1003e0 0x7f1c1c196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:20:37.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c1affd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c1c1003e0 0x7f1c1c196590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:48258/0 (socket says 172.21.15.188:48258) 2023-11-16T18:20:37.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c1affd700 1 -- 172.21.15.188:0/469478591 learned_addr learned my addr 172.21.15.188:0/469478591 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-16T18:20:37.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c1bfff700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c1c100d70 0x7f1c1c19d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:20:37.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c1b7fe700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c1c0ff1e0 0x7f1c1c196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:20:37.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c1affd700 1 -- 172.21.15.188:0/469478591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c1c0ff1e0 msgr2=0x7f1c1c196050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:37.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c1affd700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c1c0ff1e0 0x7f1c1c196050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:37.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c1affd700 1 -- 172.21.15.188:0/469478591 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c1c100d70 msgr2=0x7f1c1c19d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:37.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c1affd700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c1c100d70 0x7f1c1c19d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:37.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c1affd700 1 -- 172.21.15.188:0/469478591 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c100095f0 con 0x7f1c1c1003e0 2023-11-16T18:20:37.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c1b7fe700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c1c0ff1e0 0x7f1c1c196050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:20:37.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c1bfff700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c1c100d70 0x7f1c1c19d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:20:37.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c1affd700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c1c1003e0 0x7f1c1c196590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1c100059d0 tx=0x7f1c1000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:20:37.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c18ff9700 1 -- 172.21.15.188:0/469478591 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1001d070 con 0x7f1c1c1003e0 2023-11-16T18:20:37.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c18ff9700 1 -- 172.21.15.188:0/469478591 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c10022470 con 0x7f1c1c1003e0 2023-11-16T18:20:37.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c21c7f700 1 -- 172.21.15.188:0/469478591 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c1c10ab40 con 0x7f1c1c1003e0 2023-11-16T18:20:37.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c18ff9700 1 -- 172.21.15.188:0/469478591 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10017650 con 0x7f1c1c1003e0 2023-11-16T18:20:37.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.118+0000 7f1c21c7f700 1 -- 172.21.15.188:0/469478591 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c1c10a9b0 con 0x7f1c1c1003e0 2023-11-16T18:20:37.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.120+0000 7f1c21c7f700 1 -- 172.21.15.188:0/469478591 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c1c19e440 con 0x7f1c1c1003e0 2023-11-16T18:20:37.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.120+0000 7f1c18ff9700 1 -- 172.21.15.188:0/469478591 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f1c100225e0 con 0x7f1c1c1003e0 2023-11-16T18:20:37.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.121+0000 7f1c18ff9700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1c040a11d0 0x7f1c040a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:20:37.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.121+0000 7f1c18ff9700 1 -- 172.21.15.188:0/469478591 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f1c100ccde0 con 0x7f1c1c1003e0 2023-11-16T18:20:37.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.121+0000 7f1c1b7fe700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1c040a11d0 0x7f1c040a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:20:37.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.122+0000 7f1c1b7fe700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1c040a11d0 0x7f1c040a3680 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c005fd0 tx=0x7f1c0c005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:20:37.127 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.125+0000 7f1c18ff9700 1 -- 172.21.15.188:0/469478591 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1c100970f0 con 0x7f1c1c1003e0 2023-11-16T18:20:37.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:36 smithi040 ceph-mon[100251]: osdmap e50: 8 total, 8 up, 8 in 2023-11-16T18:20:37.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:36 smithi188 ceph-mon[100639]: osdmap e50: 8 total, 8 up, 8 in 2023-11-16T18:20:37.322 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:37.320+0000 7f1c21c7f700 1 -- 172.21.15.188:0/469478591 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c1c19eab0 con 0x7f1c040a11d0 2023-11-16T18:20:37.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:36 smithi002 ceph-mon[98807]: osdmap e50: 8 total, 8 up, 8 in 2023-11-16T18:20:38.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:37 smithi040 ceph-mon[100251]: osdmap e51: 8 total, 8 up, 8 in 2023-11-16T18:20:38.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:37 smithi040 ceph-mon[100251]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:20:38.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:20:38.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:20:38.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:38.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:37 smithi040 ceph-mon[100251]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:38.250 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:37 smithi188 ceph-mon[100639]: osdmap e51: 8 total, 8 up, 8 in 2023-11-16T18:20:38.250 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:37 smithi188 ceph-mon[100639]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:20:38.250 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:20:38.250 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:20:38.250 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:38.250 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:37 smithi188 ceph-mon[100639]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:38.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:37 smithi002 ceph-mon[98807]: osdmap e51: 8 total, 8 up, 8 in 2023-11-16T18:20:38.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:37 smithi002 ceph-mon[98807]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:20:38.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:20:38.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:20:38.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:38.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:37 smithi002 ceph-mon[98807]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:39.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:38 smithi188 ceph-mon[100639]: osdmap e52: 8 total, 8 up, 8 in 2023-11-16T18:20:39.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:38 smithi002 ceph-mon[98807]: osdmap e52: 8 total, 8 up, 8 in 2023-11-16T18:20:39.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:38 smithi040 ceph-mon[100251]: osdmap e52: 8 total, 8 up, 8 in 2023-11-16T18:20:40.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:39 smithi188 ceph-mon[100639]: osdmap e53: 8 total, 8 up, 8 in 2023-11-16T18:20:40.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:39 smithi188 ceph-mon[100639]: from='client.? 172.21.15.188:0/2344538113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2589fd33-ad12-47e5-a70d-172ea07d61bc"}]: dispatch 2023-11-16T18:20:40.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:39 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2589fd33-ad12-47e5-a70d-172ea07d61bc"}]: dispatch 2023-11-16T18:20:40.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:39 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2589fd33-ad12-47e5-a70d-172ea07d61bc"}]': finished 2023-11-16T18:20:40.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:39 smithi188 ceph-mon[100639]: osdmap e54: 9 total, 8 up, 9 in 2023-11-16T18:20:40.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:40.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:39 smithi188 ceph-mon[100639]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:40.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:39 smithi002 ceph-mon[98807]: osdmap e53: 8 total, 8 up, 8 in 2023-11-16T18:20:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:39 smithi002 ceph-mon[98807]: from='client.? 172.21.15.188:0/2344538113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2589fd33-ad12-47e5-a70d-172ea07d61bc"}]: dispatch 2023-11-16T18:20:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:39 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2589fd33-ad12-47e5-a70d-172ea07d61bc"}]: dispatch 2023-11-16T18:20:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:39 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2589fd33-ad12-47e5-a70d-172ea07d61bc"}]': finished 2023-11-16T18:20:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:39 smithi002 ceph-mon[98807]: osdmap e54: 9 total, 8 up, 9 in 2023-11-16T18:20:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:39 smithi002 ceph-mon[98807]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:39 smithi040 ceph-mon[100251]: osdmap e53: 8 total, 8 up, 8 in 2023-11-16T18:20:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:39 smithi040 ceph-mon[100251]: from='client.? 172.21.15.188:0/2344538113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2589fd33-ad12-47e5-a70d-172ea07d61bc"}]: dispatch 2023-11-16T18:20:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:39 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2589fd33-ad12-47e5-a70d-172ea07d61bc"}]: dispatch 2023-11-16T18:20:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:39 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2589fd33-ad12-47e5-a70d-172ea07d61bc"}]': finished 2023-11-16T18:20:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:39 smithi040 ceph-mon[100251]: osdmap e54: 9 total, 8 up, 9 in 2023-11-16T18:20:40.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:40.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:39 smithi040 ceph-mon[100251]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:41.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:40 smithi188 ceph-mon[100639]: from='client.? 172.21.15.188:0/1118998740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:20:41.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:40 smithi002 ceph-mon[98807]: from='client.? 172.21.15.188:0/1118998740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:20:41.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:40 smithi040 ceph-mon[100251]: from='client.? 172.21.15.188:0/1118998740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:20:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:42 smithi040 ceph-mon[100251]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:42 smithi188 ceph-mon[100639]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:42 smithi002 ceph-mon[98807]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:44.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:43 smithi188 ceph-mon[100639]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:44.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:43 smithi002 ceph-mon[98807]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:44.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:43 smithi040 ceph-mon[100251]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:46.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:45 smithi002 ceph-mon[98807]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:46.186 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:45 smithi188 ceph-mon[100639]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:46.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:45 smithi040 ceph-mon[100251]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:47.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:47 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-11-16T18:20:47.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:47 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:47.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:47 smithi040 ceph-mon[100251]: Deploying daemon osd.8 on smithi188 2023-11-16T18:20:47.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:47 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-11-16T18:20:47.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:47 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:47.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:47 smithi188 ceph-mon[100639]: Deploying daemon osd.8 on smithi188 2023-11-16T18:20:47.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:47 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-11-16T18:20:47.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:47 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:47.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:47 smithi002 ceph-mon[98807]: Deploying daemon osd.8 on smithi188 2023-11-16T18:20:48.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:48 smithi188 ceph-mon[100639]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:48.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:48 smithi040 ceph-mon[100251]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:48.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:48 smithi002 ceph-mon[98807]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:50.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:49 smithi188 ceph-mon[100639]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:50.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:49 smithi002 ceph-mon[98807]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:50.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:49 smithi040 ceph-mon[100251]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:51 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:51 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:51 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:51.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:51 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:51.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:51 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:51.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:51 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:51.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:51 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:51.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:51 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:51.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:51 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:52.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:52 smithi188 ceph-mon[100639]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:52.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:52 smithi002 ceph-mon[98807]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:52.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:52 smithi040 ceph-mon[100251]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:53.389 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:53.387+0000 7f1c18ff9700 1 -- 172.21.15.188:0/469478591 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1c1c19eab0 con 0x7f1c040a11d0 2023-11-16T18:20:53.389 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 8 on host 'smithi188' 2023-11-16T18:20:53.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:53.389+0000 7f1c21c7f700 1 -- 172.21.15.188:0/469478591 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1c040a11d0 msgr2=0x7f1c040a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:53.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:53.389+0000 7f1c21c7f700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1c040a11d0 0x7f1c040a3680 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c005fd0 tx=0x7f1c0c005f00 comp rx=0 tx=0).stop 2023-11-16T18:20:53.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:53.389+0000 7f1c21c7f700 1 -- 172.21.15.188:0/469478591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c1c1003e0 msgr2=0x7f1c1c196590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:53.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:53.389+0000 7f1c21c7f700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c1c1003e0 0x7f1c1c196590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1c100059d0 tx=0x7f1c1000f690 comp rx=0 tx=0).stop 2023-11-16T18:20:53.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:53.389+0000 7f1c21c7f700 1 -- 172.21.15.188:0/469478591 shutdown_connections 2023-11-16T18:20:53.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:53.389+0000 7f1c21c7f700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1c040a11d0 0x7f1c040a3680 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:53.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:53.389+0000 7f1c21c7f700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c1c100d70 0x7f1c1c19d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:53.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:53.389+0000 7f1c21c7f700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c1c0ff1e0 0x7f1c1c196050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:53.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:53.389+0000 7f1c21c7f700 1 --2- 172.21.15.188:0/469478591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c1c1003e0 0x7f1c1c196590 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:53.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:53.389+0000 7f1c21c7f700 1 -- 172.21.15.188:0/469478591 >> 172.21.15.188:0/469478591 conn(0x7f1c1c0fa7f0 msgr2=0x7f1c1c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:20:53.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:53.390+0000 7f1c21c7f700 1 -- 172.21.15.188:0/469478591 shutdown_connections 2023-11-16T18:20:53.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:20:53.390+0000 7f1c21c7f700 1 -- 172.21.15.188:0/469478591 wait complete. 2023-11-16T18:20:53.814 DEBUG:teuthology.orchestra.run.smithi188:osd.8> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.8.service 2023-11-16T18:20:53.816 INFO:tasks.cephadm:Deploying osd.9 on smithi002 with /dev/vg_nvme/lv_1... 2023-11-16T18:20:53.816 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-16T18:20:53.848 INFO:journalctl@ceph.osd.8.smithi188.stdout:-- Logs begin at Thu 2023-11-16 17:59:32 UTC. -- 2023-11-16T18:20:54.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:54 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:54 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:54 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:54 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:54 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:20:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:54 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:54 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:20:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:54 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:20:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:54 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:54 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:54 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:54 smithi002 ceph-mon[98807]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:54.320 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:54 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.320 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:54 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.320 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:54 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:54.320 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:54 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:54.320 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:54 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:20:54.320 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:54 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.321 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:54 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:20:54.321 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:54 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:20:54.321 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:54 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.321 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:54 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.321 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:54 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:54.321 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:54 smithi188 ceph-mon[100639]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:54.483 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:20:54.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:54 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:54 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:54 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:54.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:54 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:54.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:54 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:20:54.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:54 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:54 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:20:54.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:54 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:20:54.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:54 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:54 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:54.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:54 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:20:54.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:54 smithi040 ceph-mon[100251]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:54.584 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:20:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:20:54.488+0000 7fcdcd1df640 -1 osd.8 0 log_to_monitors true 2023-11-16T18:20:55.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:55 smithi002 ceph-mon[98807]: from='osd.8 [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T18:20:55.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:55 smithi002 ceph-mon[98807]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T18:20:55.637 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:55 smithi188 ceph-mon[100639]: from='osd.8 [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T18:20:55.637 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:55 smithi188 ceph-mon[100639]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T18:20:55.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:55 smithi040 ceph-mon[100251]: from='osd.8 [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T18:20:55.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:55 smithi040 ceph-mon[100251]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T18:20:55.975 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-16T18:20:55.975 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-16T18:20:55.975 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2023-11-16T18:20:55.975 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2023-11-16T18:20:55.975 INFO:teuthology.orchestra.run.smithi002.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00935158 s, 1.1 GB/s 2023-11-16T18:20:55.975 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2023-11-16T18:20:56.438 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi002:vg_nvme/lv_1 2023-11-16T18:20:56.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-11-16T18:20:56.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: osdmap e55: 9 total, 8 up, 9 in 2023-11-16T18:20:56.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:56.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: from='osd.8 [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:20:56.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:20:56.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:56.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:56.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:56.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:56.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:56.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:56.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:56.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:56.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:56.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:56.796 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:20:56.363+0000 7fcdc2856700 -1 osd.8 0 waiting for initial osdmap 2023-11-16T18:20:56.797 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:20:56.374+0000 7fcdbc647700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T18:20:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-11-16T18:20:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: osdmap e55: 9 total, 8 up, 9 in 2023-11-16T18:20:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: from='osd.8 [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:20:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:20:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:56.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-11-16T18:20:56.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: osdmap e55: 9 total, 8 up, 9 in 2023-11-16T18:20:56.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:56.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: from='osd.8 [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:20:56.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:20:56.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:56.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:56.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:56.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:56.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:56.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:20:56.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:56.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:56.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:20:56.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:20:57.097 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:20:57.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:57 smithi002 ceph-mon[98807]: Detected new or changed devices on smithi188 2023-11-16T18:20:57.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:57 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi188 to 5722M 2023-11-16T18:20:57.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:57 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:20:57.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:57 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:20:57.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:57 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:20:57.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:57 smithi002 ceph-mon[98807]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-11-16T18:20:57.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:57 smithi002 ceph-mon[98807]: osdmap e56: 9 total, 8 up, 9 in 2023-11-16T18:20:57.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:57 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:57.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:57 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:57 smithi040 ceph-mon[100251]: Detected new or changed devices on smithi188 2023-11-16T18:20:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:57 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi188 to 5722M 2023-11-16T18:20:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:57 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:20:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:57 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:20:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:57 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:20:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:57 smithi040 ceph-mon[100251]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-11-16T18:20:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:57 smithi040 ceph-mon[100251]: osdmap e56: 9 total, 8 up, 9 in 2023-11-16T18:20:57.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:57 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:57.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:57 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:57.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:57 smithi188 ceph-mon[100639]: Detected new or changed devices on smithi188 2023-11-16T18:20:57.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:57 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi188 to 5722M 2023-11-16T18:20:57.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:57 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:20:57.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:57 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:20:57.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:57 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:20:57.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:57 smithi188 ceph-mon[100639]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-11-16T18:20:57.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:57 smithi188 ceph-mon[100639]: osdmap e56: 9 total, 8 up, 9 in 2023-11-16T18:20:57.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:57 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:57.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:57 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:58.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.287+0000 7f581d19f700 1 -- 172.21.15.2:0/1960214674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58180ff080 msgr2=0x7f58180ff490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:58.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.287+0000 7f581d19f700 1 --2- 172.21.15.2:0/1960214674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58180ff080 0x7f58180ff490 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5808009990 tx=0x7f5808009ca0 comp rx=0 tx=0).stop 2023-11-16T18:20:58.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.287+0000 7f581d19f700 1 -- 172.21.15.2:0/1960214674 shutdown_connections 2023-11-16T18:20:58.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.287+0000 7f581d19f700 1 --2- 172.21.15.2:0/1960214674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5818100c10 0x7f58181074d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:58.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.287+0000 7f581d19f700 1 --2- 172.21.15.2:0/1960214674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58180ff080 0x7f58180ff490 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:58.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.287+0000 7f581d19f700 1 --2- 172.21.15.2:0/1960214674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5818100280 0x7f58181006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:58.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.287+0000 7f581d19f700 1 -- 172.21.15.2:0/1960214674 >> 172.21.15.2:0/1960214674 conn(0x7f58180fa6b0 msgr2=0x7f58180fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:20:58.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.287+0000 7f581d19f700 1 -- 172.21.15.2:0/1960214674 shutdown_connections 2023-11-16T18:20:58.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.287+0000 7f581d19f700 1 -- 172.21.15.2:0/1960214674 wait complete. 2023-11-16T18:20:58.291 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.288+0000 7f581d19f700 1 Processor -- start 2023-11-16T18:20:58.291 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.288+0000 7f581d19f700 1 -- start start 2023-11-16T18:20:58.291 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.288+0000 7f581d19f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58180ff080 0x7f5818195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:20:58.291 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.288+0000 7f581d19f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5818100280 0x7f5818196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:20:58.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.288+0000 7f581d19f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5818100c10 0x7f581819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:20:58.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.288+0000 7f581d19f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581819d9e0 con 0x7f5818100c10 2023-11-16T18:20:58.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.288+0000 7f581d19f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581819db20 con 0x7f5818100280 2023-11-16T18:20:58.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.288+0000 7f581d19f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581819dcb0 con 0x7f58180ff080 2023-11-16T18:20:58.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.288+0000 7f581c99e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5818100c10 0x7f581819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:20:58.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.288+0000 7f581c99e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5818100c10 0x7f581819d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47258/0 (socket says 172.21.15.2:47258) 2023-11-16T18:20:58.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.288+0000 7f581c99e700 1 -- 172.21.15.2:0/307164399 learned_addr learned my addr 172.21.15.2:0/307164399 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:20:58.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.288+0000 7f5817fff700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58180ff080 0x7f5818195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:20:58.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.289+0000 7f58177fe700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5818100280 0x7f5818196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:20:58.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.289+0000 7f581c99e700 1 -- 172.21.15.2:0/307164399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58180ff080 msgr2=0x7f5818195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:58.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.289+0000 7f581c99e700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58180ff080 0x7f5818195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:58.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.289+0000 7f581c99e700 1 -- 172.21.15.2:0/307164399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5818100280 msgr2=0x7f5818196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:20:58.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.289+0000 7f581c99e700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5818100280 0x7f5818196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:20:58.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.289+0000 7f581c99e700 1 -- 172.21.15.2:0/307164399 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58080095f0 con 0x7f5818100c10 2023-11-16T18:20:58.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.289+0000 7f58177fe700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5818100280 0x7f5818196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:20:58.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.289+0000 7f581c99e700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5818100c10 0x7f581819d410 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f581000add0 tx=0x7f581000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:20:58.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.289+0000 7f58157fa700 1 -- 172.21.15.2:0/307164399 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581000f070 con 0x7f5818100c10 2023-11-16T18:20:58.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.289+0000 7f58157fa700 1 -- 172.21.15.2:0/307164399 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5810007b30 con 0x7f5818100c10 2023-11-16T18:20:58.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.289+0000 7f581d19f700 1 -- 172.21.15.2:0/307164399 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581810a9e0 con 0x7f5818100c10 2023-11-16T18:20:58.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.289+0000 7f58157fa700 1 -- 172.21.15.2:0/307164399 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5810005310 con 0x7f5818100c10 2023-11-16T18:20:58.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.289+0000 7f581d19f700 1 -- 172.21.15.2:0/307164399 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581810a850 con 0x7f5818100c10 2023-11-16T18:20:58.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.290+0000 7f581d19f700 1 -- 172.21.15.2:0/307164399 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5818107270 con 0x7f5818100c10 2023-11-16T18:20:58.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.291+0000 7f58157fa700 1 -- 172.21.15.2:0/307164399 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f581000c940 con 0x7f5818100c10 2023-11-16T18:20:58.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.292+0000 7f58157fa700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58000a1150 0x7f58000a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:20:58.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.292+0000 7f5817fff700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58000a1150 0x7f58000a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:20:58.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.292+0000 7f58157fa700 1 -- 172.21.15.2:0/307164399 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5862+0+0 (secure 0 0 0) 0x7f58100cb5e0 con 0x7f5818100c10 2023-11-16T18:20:58.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.292+0000 7f5817fff700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58000a1150 0x7f58000a3600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f580800b5c0 tx=0x7f5808005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:20:58.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.296+0000 7f58157fa700 1 -- 172.21.15.2:0/307164399 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f58100956b0 con 0x7f5818100c10 2023-11-16T18:20:58.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:20:58.507+0000 7f581d19f700 1 -- 172.21.15.2:0/307164399 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5818061c10 con 0x7f58000a1150 2023-11-16T18:20:58.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:58 smithi040 ceph-mon[100251]: purged_snaps scrub starts 2023-11-16T18:20:58.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:58 smithi040 ceph-mon[100251]: purged_snaps scrub ok 2023-11-16T18:20:58.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:58 smithi040 ceph-mon[100251]: osd.8 [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264] boot 2023-11-16T18:20:58.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:58 smithi040 ceph-mon[100251]: osdmap e57: 9 total, 9 up, 9 in 2023-11-16T18:20:58.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:58 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:58.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:58 smithi040 ceph-mon[100251]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:58.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:58 smithi188 ceph-mon[100639]: purged_snaps scrub starts 2023-11-16T18:20:58.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:58 smithi188 ceph-mon[100639]: purged_snaps scrub ok 2023-11-16T18:20:58.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:58 smithi188 ceph-mon[100639]: osd.8 [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264] boot 2023-11-16T18:20:58.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:58 smithi188 ceph-mon[100639]: osdmap e57: 9 total, 9 up, 9 in 2023-11-16T18:20:58.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:58 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:58.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:58 smithi188 ceph-mon[100639]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:58.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:58 smithi002 ceph-mon[98807]: purged_snaps scrub starts 2023-11-16T18:20:58.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:58 smithi002 ceph-mon[98807]: purged_snaps scrub ok 2023-11-16T18:20:58.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:58 smithi002 ceph-mon[98807]: osd.8 [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264] boot 2023-11-16T18:20:58.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:58 smithi002 ceph-mon[98807]: osdmap e57: 9 total, 9 up, 9 in 2023-11-16T18:20:58.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:58 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:20:58.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:58 smithi002 ceph-mon[98807]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T18:20:59.693 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:59 smithi002 ceph-mon[98807]: osdmap e58: 9 total, 9 up, 9 in 2023-11-16T18:20:59.694 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:59 smithi002 ceph-mon[98807]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:20:59.694 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:59 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:20:59.694 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:59 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:20:59.694 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:20:59 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:59.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:59 smithi040 ceph-mon[100251]: osdmap e58: 9 total, 9 up, 9 in 2023-11-16T18:20:59.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:59 smithi040 ceph-mon[100251]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:20:59.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:59 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:20:59.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:59 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:20:59.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:20:59 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:20:59.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:59 smithi188 ceph-mon[100639]: osdmap e58: 9 total, 9 up, 9 in 2023-11-16T18:20:59.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:59 smithi188 ceph-mon[100639]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:20:59.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:59 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:20:59.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:59 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:20:59.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:20:59 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:00.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:00 smithi002 ceph-mon[98807]: osdmap e59: 9 total, 9 up, 9 in 2023-11-16T18:21:00.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:00 smithi002 ceph-mon[98807]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:00.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:00 smithi040 ceph-mon[100251]: osdmap e59: 9 total, 9 up, 9 in 2023-11-16T18:21:00.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:00 smithi040 ceph-mon[100251]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:00.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:00 smithi188 ceph-mon[100639]: osdmap e59: 9 total, 9 up, 9 in 2023-11-16T18:21:00.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:00 smithi188 ceph-mon[100639]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:01.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:01 smithi040 ceph-mon[100251]: osdmap e60: 9 total, 9 up, 9 in 2023-11-16T18:21:01.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:01 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1344721888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b081d7a-d6be-433c-9989-d5159ee895da"}]: dispatch 2023-11-16T18:21:01.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:01 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1344721888' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b081d7a-d6be-433c-9989-d5159ee895da"}]': finished 2023-11-16T18:21:01.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:01 smithi040 ceph-mon[100251]: osdmap e61: 10 total, 9 up, 10 in 2023-11-16T18:21:01.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:01.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:01 smithi188 ceph-mon[100639]: osdmap e60: 9 total, 9 up, 9 in 2023-11-16T18:21:01.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:01 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1344721888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b081d7a-d6be-433c-9989-d5159ee895da"}]: dispatch 2023-11-16T18:21:01.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:01 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1344721888' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b081d7a-d6be-433c-9989-d5159ee895da"}]': finished 2023-11-16T18:21:01.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:01 smithi188 ceph-mon[100639]: osdmap e61: 10 total, 9 up, 10 in 2023-11-16T18:21:01.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:01.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:01 smithi002 ceph-mon[98807]: osdmap e60: 9 total, 9 up, 9 in 2023-11-16T18:21:01.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:01 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1344721888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b081d7a-d6be-433c-9989-d5159ee895da"}]: dispatch 2023-11-16T18:21:01.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:01 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1344721888' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b081d7a-d6be-433c-9989-d5159ee895da"}]': finished 2023-11-16T18:21:01.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:01 smithi002 ceph-mon[98807]: osdmap e61: 10 total, 9 up, 10 in 2023-11-16T18:21:01.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:02.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:02 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2549156329' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:21:02.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:02 smithi040 ceph-mon[100251]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T18:21:02.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:02 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2549156329' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:21:02.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:02 smithi188 ceph-mon[100639]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T18:21:02.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:02 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2549156329' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:21:02.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:02 smithi002 ceph-mon[98807]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T18:21:04.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:03 smithi002 ceph-mon[98807]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-11-16T18:21:04.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:03 smithi040 ceph-mon[100251]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-11-16T18:21:04.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:03 smithi188 ceph-mon[100639]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-11-16T18:21:06.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:05 smithi002 ceph-mon[98807]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-11-16T18:21:06.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:05 smithi040 ceph-mon[100251]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-11-16T18:21:06.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:05 smithi188 ceph-mon[100639]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-11-16T18:21:08.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:07 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-11-16T18:21:08.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:07 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:08.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:07 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-11-16T18:21:08.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:07 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:08.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:07 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-11-16T18:21:08.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:07 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:09.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:08 smithi188 ceph-mon[100639]: Deploying daemon osd.9 on smithi002 2023-11-16T18:21:09.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:08 smithi188 ceph-mon[100639]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T18:21:09.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:08 smithi002 ceph-mon[98807]: Deploying daemon osd.9 on smithi002 2023-11-16T18:21:09.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:08 smithi002 ceph-mon[98807]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T18:21:09.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:08 smithi040 ceph-mon[100251]: Deploying daemon osd.9 on smithi002 2023-11-16T18:21:09.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:08 smithi040 ceph-mon[100251]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T18:21:10.103 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:09 smithi002 ceph-mon[98807]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-11-16T18:21:10.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:09 smithi040 ceph-mon[100251]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-11-16T18:21:10.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:09 smithi188 ceph-mon[100639]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-11-16T18:21:12.452 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:12 smithi002 ceph-mon[98807]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-11-16T18:21:12.452 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:12.452 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:12.452 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:12 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:21:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:12 smithi040 ceph-mon[100251]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-11-16T18:21:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:12 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:21:12.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:12 smithi188 ceph-mon[100639]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-11-16T18:21:12.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:12.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:12.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:12 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:21:13.913 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:13 smithi002 ceph-mon[98807]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:14.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:13 smithi040 ceph-mon[100251]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:14.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:13 smithi188 ceph-mon[100639]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 9 on host 'smithi002' 2023-11-16T18:21:14.809 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:21:14.806+0000 7f58157fa700 1 -- 172.21.15.2:0/307164399 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5818061c10 con 0x7f58000a1150 2023-11-16T18:21:14.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:21:14.810+0000 7f581d19f700 1 -- 172.21.15.2:0/307164399 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58000a1150 msgr2=0x7f58000a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:21:14.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:21:14.810+0000 7f581d19f700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58000a1150 0x7f58000a3600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f580800b5c0 tx=0x7f5808005fd0 comp rx=0 tx=0).stop 2023-11-16T18:21:14.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:21:14.810+0000 7f581d19f700 1 -- 172.21.15.2:0/307164399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5818100c10 msgr2=0x7f581819d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:21:14.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:21:14.810+0000 7f581d19f700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5818100c10 0x7f581819d410 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f581000add0 tx=0x7f581000c330 comp rx=0 tx=0).stop 2023-11-16T18:21:14.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:21:14.810+0000 7f581d19f700 1 -- 172.21.15.2:0/307164399 shutdown_connections 2023-11-16T18:21:14.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:21:14.810+0000 7f581d19f700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58000a1150 0x7f58000a3600 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:14.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:21:14.810+0000 7f581d19f700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58180ff080 0x7f5818195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:14.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:21:14.810+0000 7f581d19f700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5818100c10 0x7f581819d410 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:14.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:21:14.810+0000 7f581d19f700 1 --2- 172.21.15.2:0/307164399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5818100280 0x7f5818196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:14.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:21:14.810+0000 7f581d19f700 1 -- 172.21.15.2:0/307164399 >> 172.21.15.2:0/307164399 conn(0x7f58180fa6b0 msgr2=0x7f58180fc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:21:14.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:21:14.810+0000 7f581d19f700 1 -- 172.21.15.2:0/307164399 shutdown_connections 2023-11-16T18:21:14.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:21:14.810+0000 7f581d19f700 1 -- 172.21.15.2:0/307164399 wait complete. 2023-11-16T18:21:15.284 DEBUG:teuthology.orchestra.run.smithi002:osd.9> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.9.service 2023-11-16T18:21:15.286 INFO:tasks.cephadm:Deploying osd.10 on smithi040 with /dev/vg_nvme/lv_1... 2023-11-16T18:21:15.287 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-16T18:21:15.320 INFO:journalctl@ceph.osd.9.smithi002.stdout:-- Logs begin at Thu 2023-11-16 17:59:05 UTC. -- 2023-11-16T18:21:15.954 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T18:21:16.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:16.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:16.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:16.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:16.083 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:15 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:16.083 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:15 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:16.083 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:15 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:16.083 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:15 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:16.083 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:21:15 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:21:15.979+0000 7f2891685640 -1 osd.9 0 log_to_monitors true 2023-11-16T18:21:16.115 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:15 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:16.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:15 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:16.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:15 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:16.116 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:15 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:16.928 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:16 smithi040 ceph-mon[100251]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:16.928 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:16 smithi040 ceph-mon[100251]: from='osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T18:21:17.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:16 smithi188 ceph-mon[100639]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:17.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:16 smithi188 ceph-mon[100639]: from='osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T18:21:17.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:16 smithi002 ceph-mon[98807]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:17.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:16 smithi002 ceph-mon[98807]: from='osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T18:21:17.393 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-16T18:21:17.393 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-16T18:21:17.393 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2023-11-16T18:21:17.393 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2023-11-16T18:21:17.393 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127654 s, 821 MB/s 2023-11-16T18:21:17.393 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 2023-11-16T18:21:17.393 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2023-11-16T18:21:17.804 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi040:vg_nvme/lv_1 2023-11-16T18:21:17.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-11-16T18:21:17.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: osdmap e62: 10 total, 9 up, 10 in 2023-11-16T18:21:17.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:17.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T18:21:17.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: Detected new or changed devices on smithi002 2023-11-16T18:21:17.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:17.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:17.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:17.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:17.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:17.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:17.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi002 to 4293M 2023-11-16T18:21:17.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:17.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:17.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:21:17.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:21:17.886 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:17.886 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:21:17.887 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:21:17.887 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:17.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:21:17.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:17.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:21:17.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:17.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-11-16T18:21:17.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: osdmap e62: 10 total, 9 up, 10 in 2023-11-16T18:21:17.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:17.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T18:21:17.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: Detected new or changed devices on smithi002 2023-11-16T18:21:17.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:17.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:17.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:17.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:17.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:17.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:17.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi002 to 4293M 2023-11-16T18:21:17.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:17.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:17.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:21:17.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:21:17.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:17.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:21:17.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:21:17.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:17.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:21:17.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:17.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:21:17.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:18.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-11-16T18:21:18.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: osdmap e62: 10 total, 9 up, 10 in 2023-11-16T18:21:18.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:18.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T18:21:18.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: Detected new or changed devices on smithi002 2023-11-16T18:21:18.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:18.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:18.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:18.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:18.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:18.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:18.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi002 to 4293M 2023-11-16T18:21:18.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:18.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:18.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:21:18.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:21:18.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:18.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:21:18.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:21:18.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:18.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:21:18.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:18.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:21:18.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:18.356 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:21:17.875+0000 7f2886cfc700 -1 osd.9 0 waiting for initial osdmap 2023-11-16T18:21:18.356 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:21:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:21:17.889+0000 7f2880aed700 -1 osd.9 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T18:21:18.456 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T18:21:18.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:18 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:21:18.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:18 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:21:18.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:18 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:21:18.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:18 smithi040 ceph-mon[100251]: from='osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T18:21:18.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:18 smithi040 ceph-mon[100251]: osdmap e63: 10 total, 9 up, 10 in 2023-11-16T18:21:18.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:18 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:18.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:18 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:18.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:18 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:19.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:18 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:21:19.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:18 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:21:19.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:18 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:21:19.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:18 smithi188 ceph-mon[100639]: from='osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T18:21:19.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:18 smithi188 ceph-mon[100639]: osdmap e63: 10 total, 9 up, 10 in 2023-11-16T18:21:19.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:18 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:19.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:18 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:19.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:18 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:19.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:18 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:21:19.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:18 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:21:19.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:18 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:21:19.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:18 smithi002 ceph-mon[98807]: from='osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T18:21:19.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:18 smithi002 ceph-mon[98807]: osdmap e63: 10 total, 9 up, 10 in 2023-11-16T18:21:19.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:18 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:19.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:18 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:19.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:18 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:19.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.642+0000 7f74c8b53700 1 -- 172.21.15.40:0/2055929833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c4100d70 msgr2=0x7f74c4107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:21:19.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.642+0000 7f74c8b53700 1 --2- 172.21.15.40:0/2055929833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c4100d70 0x7f74c4107630 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00b0a0 tx=0x7f74bc00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:21:19.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.642+0000 7f74c8b53700 1 -- 172.21.15.40:0/2055929833 shutdown_connections 2023-11-16T18:21:19.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.642+0000 7f74c8b53700 1 --2- 172.21.15.40:0/2055929833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c4100d70 0x7f74c4107630 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:19.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.642+0000 7f74c8b53700 1 --2- 172.21.15.40:0/2055929833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c40ff1e0 0x7f74c40ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:19.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.642+0000 7f74c8b53700 1 --2- 172.21.15.40:0/2055929833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c41003e0 0x7f74c4100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:19.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.642+0000 7f74c8b53700 1 -- 172.21.15.40:0/2055929833 >> 172.21.15.40:0/2055929833 conn(0x7f74c40fa7f0 msgr2=0x7f74c40fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:21:19.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.642+0000 7f74c8b53700 1 -- 172.21.15.40:0/2055929833 shutdown_connections 2023-11-16T18:21:19.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.642+0000 7f74c8b53700 1 -- 172.21.15.40:0/2055929833 wait complete. 2023-11-16T18:21:19.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c8b53700 1 Processor -- start 2023-11-16T18:21:19.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c8b53700 1 -- start start 2023-11-16T18:21:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c8b53700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c40ff1e0 0x7f74c40699a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:21:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c8b53700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c41003e0 0x7f74c419ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:21:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c8b53700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c4100d70 0x7f74c419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:21:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c8b53700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c419dc00 con 0x7f74c41003e0 2023-11-16T18:21:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c8b53700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c419dd70 con 0x7f74c40ff1e0 2023-11-16T18:21:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c8b53700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c419df30 con 0x7f74c4100d70 2023-11-16T18:21:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c40ff1e0 0x7f74c40699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:21:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c40ff1e0 0x7f74c40699a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49084/0 (socket says 172.21.15.40:49084) 2023-11-16T18:21:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c259c700 1 -- 172.21.15.40:0/3307696104 learned_addr learned my addr 172.21.15.40:0/3307696104 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-11-16T18:21:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c1d9b700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c41003e0 0x7f74c419ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:21:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c2d9d700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c4100d70 0x7f74c419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:21:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c259c700 1 -- 172.21.15.40:0/3307696104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c4100d70 msgr2=0x7f74c419d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:21:19.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c259c700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c4100d70 0x7f74c419d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:19.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c259c700 1 -- 172.21.15.40:0/3307696104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c41003e0 msgr2=0x7f74c419ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:21:19.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c259c700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c41003e0 0x7f74c419ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:19.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.643+0000 7f74c259c700 1 -- 172.21.15.40:0/3307696104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc009d00 con 0x7f74c40ff1e0 2023-11-16T18:21:19.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.644+0000 7f74c259c700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c40ff1e0 0x7f74c40699a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f74b400d980 tx=0x7f74b400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:21:19.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.644+0000 7f74b37fe700 1 -- 172.21.15.40:0/3307696104 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b4004020 con 0x7f74c40ff1e0 2023-11-16T18:21:19.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.644+0000 7f74b37fe700 1 -- 172.21.15.40:0/3307696104 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74b4009730 con 0x7f74c40ff1e0 2023-11-16T18:21:19.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.644+0000 7f74c8b53700 1 -- 172.21.15.40:0/3307696104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c410ab30 con 0x7f74c40ff1e0 2023-11-16T18:21:19.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.644+0000 7f74b37fe700 1 -- 172.21.15.40:0/3307696104 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b4003a80 con 0x7f74c40ff1e0 2023-11-16T18:21:19.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.644+0000 7f74c8b53700 1 -- 172.21.15.40:0/3307696104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c410a9a0 con 0x7f74c40ff1e0 2023-11-16T18:21:19.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.645+0000 7f74c8b53700 1 -- 172.21.15.40:0/3307696104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c4066ea0 con 0x7f74c40ff1e0 2023-11-16T18:21:19.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.646+0000 7f74b37fe700 1 -- 172.21.15.40:0/3307696104 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f74b40098a0 con 0x7f74c40ff1e0 2023-11-16T18:21:19.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.647+0000 7f74b37fe700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f74ac0a1150 0x7f74ac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:21:19.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.647+0000 7f74b37fe700 1 -- 172.21.15.40:0/3307696104 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f74b40cc150 con 0x7f74c40ff1e0 2023-11-16T18:21:19.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.647+0000 7f74c1d9b700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f74ac0a1150 0x7f74ac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:21:19.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.647+0000 7f74c1d9b700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f74ac0a1150 0x7f74ac0a3600 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f74c4069f20 tx=0x7f74b800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:21:19.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.650+0000 7f74b37fe700 1 -- 172.21.15.40:0/3307696104 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f74b40960a0 con 0x7f74c40ff1e0 2023-11-16T18:21:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:19.853+0000 7f74c8b53700 1 -- 172.21.15.40:0/3307696104 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f74c419ea90 con 0x7f74ac0a1150 2023-11-16T18:21:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:19 smithi040 ceph-mon[100251]: purged_snaps scrub starts 2023-11-16T18:21:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:19 smithi040 ceph-mon[100251]: purged_snaps scrub ok 2023-11-16T18:21:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:19 smithi040 ceph-mon[100251]: osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180] boot 2023-11-16T18:21:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:19 smithi040 ceph-mon[100251]: osdmap e64: 10 total, 10 up, 10 in 2023-11-16T18:21:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:19 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:19 smithi040 ceph-mon[100251]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:20.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:19 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:21:20.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:19 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:21:20.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:19 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:20.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:19 smithi188 ceph-mon[100639]: purged_snaps scrub starts 2023-11-16T18:21:20.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:19 smithi188 ceph-mon[100639]: purged_snaps scrub ok 2023-11-16T18:21:20.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:19 smithi188 ceph-mon[100639]: osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180] boot 2023-11-16T18:21:20.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:19 smithi188 ceph-mon[100639]: osdmap e64: 10 total, 10 up, 10 in 2023-11-16T18:21:20.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:19 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:20.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:19 smithi188 ceph-mon[100639]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:20.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:19 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:21:20.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:19 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:21:20.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:19 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:20.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:19 smithi002 ceph-mon[98807]: purged_snaps scrub starts 2023-11-16T18:21:20.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:19 smithi002 ceph-mon[98807]: purged_snaps scrub ok 2023-11-16T18:21:20.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:19 smithi002 ceph-mon[98807]: osd.9 [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180] boot 2023-11-16T18:21:20.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:19 smithi002 ceph-mon[98807]: osdmap e64: 10 total, 10 up, 10 in 2023-11-16T18:21:20.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:19 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:21:20.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:19 smithi002 ceph-mon[98807]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T18:21:20.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:19 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:21:20.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:19 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:21:20.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:19 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:20.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:20 smithi040 ceph-mon[100251]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:21:20.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:20 smithi040 ceph-mon[100251]: osdmap e65: 10 total, 10 up, 10 in 2023-11-16T18:21:21.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:20 smithi188 ceph-mon[100639]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:21:21.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:20 smithi188 ceph-mon[100639]: osdmap e65: 10 total, 10 up, 10 in 2023-11-16T18:21:21.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:20 smithi002 ceph-mon[98807]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:21:21.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:20 smithi002 ceph-mon[98807]: osdmap e65: 10 total, 10 up, 10 in 2023-11-16T18:21:22.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:22 smithi040 ceph-mon[100251]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:22.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:22 smithi188 ceph-mon[100639]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:22.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:22 smithi002 ceph-mon[98807]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:23.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:23 smithi040 ceph-mon[100251]: from='client.? 172.21.15.40:0/552476339' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a79f904a-1214-42bf-b36a-8a7709eb0c4a"}]: dispatch 2023-11-16T18:21:23.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:23 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a79f904a-1214-42bf-b36a-8a7709eb0c4a"}]: dispatch 2023-11-16T18:21:23.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:23 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a79f904a-1214-42bf-b36a-8a7709eb0c4a"}]': finished 2023-11-16T18:21:23.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:23 smithi040 ceph-mon[100251]: osdmap e66: 11 total, 10 up, 11 in 2023-11-16T18:21:23.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:23 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:23.488 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:23 smithi040 ceph-mon[100251]: from='client.? 172.21.15.40:0/4253618500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:21:23.558 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:23 smithi188 ceph-mon[100639]: from='client.? 172.21.15.40:0/552476339' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a79f904a-1214-42bf-b36a-8a7709eb0c4a"}]: dispatch 2023-11-16T18:21:23.560 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:23 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a79f904a-1214-42bf-b36a-8a7709eb0c4a"}]: dispatch 2023-11-16T18:21:23.561 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:23 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a79f904a-1214-42bf-b36a-8a7709eb0c4a"}]': finished 2023-11-16T18:21:23.572 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:23 smithi188 ceph-mon[100639]: osdmap e66: 11 total, 10 up, 11 in 2023-11-16T18:21:23.574 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:23 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:23.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:23 smithi188 ceph-mon[100639]: from='client.? 172.21.15.40:0/4253618500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:21:23.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:23 smithi002 ceph-mon[98807]: from='client.? 172.21.15.40:0/552476339' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a79f904a-1214-42bf-b36a-8a7709eb0c4a"}]: dispatch 2023-11-16T18:21:23.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:23 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a79f904a-1214-42bf-b36a-8a7709eb0c4a"}]: dispatch 2023-11-16T18:21:23.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:23 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a79f904a-1214-42bf-b36a-8a7709eb0c4a"}]': finished 2023-11-16T18:21:23.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:23 smithi002 ceph-mon[98807]: osdmap e66: 11 total, 10 up, 11 in 2023-11-16T18:21:23.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:23 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:23.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:23 smithi002 ceph-mon[98807]: from='client.? 172.21.15.40:0/4253618500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:21:24.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:24 smithi040 ceph-mon[100251]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:24.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:24 smithi188 ceph-mon[100639]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:24.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:24 smithi002 ceph-mon[98807]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:26.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:25 smithi002 ceph-mon[98807]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:26.174 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:25 smithi040 ceph-mon[100251]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:26.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:25 smithi188 ceph-mon[100639]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:27.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:27 smithi040 ceph-mon[100251]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:28.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:27 smithi002 ceph-mon[98807]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:28.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:27 smithi188 ceph-mon[100639]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:29.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:28 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-11-16T18:21:29.239 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:28 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:29.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:28 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-11-16T18:21:29.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:28 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:29.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:28 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-11-16T18:21:29.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:28 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:30.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:29 smithi040 ceph-mon[100251]: Deploying daemon osd.10 on smithi040 2023-11-16T18:21:30.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:29 smithi040 ceph-mon[100251]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:30.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:29 smithi188 ceph-mon[100639]: Deploying daemon osd.10 on smithi040 2023-11-16T18:21:30.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:29 smithi188 ceph-mon[100639]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:30.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:29 smithi002 ceph-mon[98807]: Deploying daemon osd.10 on smithi040 2023-11-16T18:21:30.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:29 smithi002 ceph-mon[98807]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:32.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:32 smithi040 ceph-mon[100251]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:32.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:32 smithi188 ceph-mon[100639]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:32.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:32 smithi002 ceph-mon[98807]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:34.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:34.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:34.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:34 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:21:34.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:34 smithi002 ceph-mon[98807]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:34.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:34.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:34.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:34 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:21:34.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:34 smithi040 ceph-mon[100251]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:35.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:35.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:35.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:34 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:21:35.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:34 smithi188 ceph-mon[100639]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:36.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:35 smithi002 ceph-mon[98807]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:36.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:35 smithi040 ceph-mon[100251]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:36.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:35 smithi188 ceph-mon[100639]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:36.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:36.720+0000 7f74b37fe700 1 -- 172.21.15.40:0/3307696104 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f74c419ea90 con 0x7f74ac0a1150 2023-11-16T18:21:36.722 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 10 on host 'smithi040' 2023-11-16T18:21:36.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:36.722+0000 7f74c8b53700 1 -- 172.21.15.40:0/3307696104 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f74ac0a1150 msgr2=0x7f74ac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:21:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:36.722+0000 7f74c8b53700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f74ac0a1150 0x7f74ac0a3600 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f74c4069f20 tx=0x7f74b800a300 comp rx=0 tx=0).stop 2023-11-16T18:21:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:36.723+0000 7f74c8b53700 1 -- 172.21.15.40:0/3307696104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c40ff1e0 msgr2=0x7f74c40699a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:21:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:36.723+0000 7f74c8b53700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c40ff1e0 0x7f74c40699a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f74b400d980 tx=0x7f74b400dc90 comp rx=0 tx=0).stop 2023-11-16T18:21:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:36.723+0000 7f74c8b53700 1 -- 172.21.15.40:0/3307696104 shutdown_connections 2023-11-16T18:21:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:36.723+0000 7f74c8b53700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f74ac0a1150 0x7f74ac0a3600 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:36.723+0000 7f74c8b53700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c4100d70 0x7f74c419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:36.723+0000 7f74c8b53700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c40ff1e0 0x7f74c40699a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:36.723+0000 7f74c8b53700 1 --2- 172.21.15.40:0/3307696104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c41003e0 0x7f74c419ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:36.723+0000 7f74c8b53700 1 -- 172.21.15.40:0/3307696104 >> 172.21.15.40:0/3307696104 conn(0x7f74c40fa7f0 msgr2=0x7f74c40fca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:21:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:36.723+0000 7f74c8b53700 1 -- 172.21.15.40:0/3307696104 shutdown_connections 2023-11-16T18:21:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-11-16T18:21:36.723+0000 7f74c8b53700 1 -- 172.21.15.40:0/3307696104 wait complete. 2023-11-16T18:21:37.154 DEBUG:teuthology.orchestra.run.smithi040:osd.10> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.10.service 2023-11-16T18:21:37.156 INFO:tasks.cephadm:Deploying osd.11 on smithi188 with /dev/vg_nvme/lv_1... 2023-11-16T18:21:37.157 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-16T18:21:37.186 INFO:journalctl@ceph.osd.10.smithi040.stdout:-- Logs begin at Thu 2023-11-16 17:59:26 UTC. -- 2023-11-16T18:21:37.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:37.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:37.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:37.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:37.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:37.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:37.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:37.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:37.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:37.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:37.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:37.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:37.809 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:21:37.887 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:21:37 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:21:37.561+0000 7f369aa71640 -1 osd.10 0 log_to_monitors true 2023-11-16T18:21:38.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:38 smithi040 ceph-mon[100251]: from='osd.10 [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T18:21:38.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:38 smithi040 ceph-mon[100251]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T18:21:38.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:38 smithi040 ceph-mon[100251]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:38.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:38 smithi188 ceph-mon[100639]: from='osd.10 [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T18:21:38.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:38 smithi188 ceph-mon[100639]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T18:21:38.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:38 smithi188 ceph-mon[100639]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:38.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:38 smithi002 ceph-mon[98807]: from='osd.10 [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T18:21:38.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:38 smithi002 ceph-mon[98807]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T18:21:38.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:38 smithi002 ceph-mon[98807]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:39.331 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-16T18:21:39.331 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-16T18:21:39.331 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2023-11-16T18:21:39.332 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2023-11-16T18:21:39.332 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111677 s, 939 MB/s 2023-11-16T18:21:39.333 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2023-11-16T18:21:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:39 smithi188 ceph-mon[100639]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-11-16T18:21:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:39 smithi188 ceph-mon[100639]: osdmap e67: 11 total, 10 up, 11 in 2023-11-16T18:21:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:39 smithi188 ceph-mon[100639]: from='osd.10 [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:21:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:39 smithi188 ceph-mon[100639]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:21:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:39 smithi002 ceph-mon[98807]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-11-16T18:21:39.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:39 smithi002 ceph-mon[98807]: osdmap e67: 11 total, 10 up, 11 in 2023-11-16T18:21:39.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:39.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:39 smithi002 ceph-mon[98807]: from='osd.10 [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:21:39.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:39 smithi002 ceph-mon[98807]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:21:39.656 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:21:39 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:21:39.254+0000 7f36900e8700 -1 osd.10 0 waiting for initial osdmap 2023-11-16T18:21:39.656 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:21:39 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:21:39.269+0000 7f368aedb700 -1 osd.10 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T18:21:39.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:39 smithi040 ceph-mon[100251]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-11-16T18:21:39.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:39 smithi040 ceph-mon[100251]: osdmap e67: 11 total, 10 up, 11 in 2023-11-16T18:21:39.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:39.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:39 smithi040 ceph-mon[100251]: from='osd.10 [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:21:39.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:39 smithi040 ceph-mon[100251]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-11-16T18:21:39.764 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi188:vg_nvme/lv_1 2023-11-16T18:21:40.408 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:21:40.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-11-16T18:21:40.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: osdmap e68: 11 total, 10 up, 11 in 2023-11-16T18:21:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: Detected new or changed devices on smithi040 2023-11-16T18:21:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi040 to 4293M 2023-11-16T18:21:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:21:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:21:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:21:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:21:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:21:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:21:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:40 smithi188 ceph-mon[100639]: osd.10 [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944] boot 2023-11-16T18:21:40.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-11-16T18:21:40.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: osdmap e68: 11 total, 10 up, 11 in 2023-11-16T18:21:40.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:40.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:40.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: Detected new or changed devices on smithi040 2023-11-16T18:21:40.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:40.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:40.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:40.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:40.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi040 to 4293M 2023-11-16T18:21:40.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:40.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:21:40.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:21:40.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:21:40.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:21:40.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:40.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:21:40.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:40.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:21:40.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:40 smithi002 ceph-mon[98807]: osd.10 [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944] boot 2023-11-16T18:21:40.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-11-16T18:21:40.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: osdmap e68: 11 total, 10 up, 11 in 2023-11-16T18:21:40.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:40.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:40.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: Detected new or changed devices on smithi040 2023-11-16T18:21:40.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:40.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:40.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:40.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:21:40.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi040 to 4293M 2023-11-16T18:21:40.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:40.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:21:40.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:21:40.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:21:40.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:21:40.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T18:21:40.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:21:40.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:40.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:21:40.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:21:40.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:40 smithi040 ceph-mon[100251]: osd.10 [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944] boot 2023-11-16T18:21:41.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:41 smithi188 ceph-mon[100639]: purged_snaps scrub starts 2023-11-16T18:21:41.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:41 smithi188 ceph-mon[100639]: purged_snaps scrub ok 2023-11-16T18:21:41.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:41 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:21:41.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:41 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:21:41.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:41 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:21:41.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:41 smithi188 ceph-mon[100639]: osdmap e69: 11 total, 11 up, 11 in 2023-11-16T18:21:41.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:41 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:41.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:41 smithi188 ceph-mon[100639]: osdmap e70: 11 total, 11 up, 11 in 2023-11-16T18:21:41.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:41 smithi002 ceph-mon[98807]: purged_snaps scrub starts 2023-11-16T18:21:41.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:41 smithi002 ceph-mon[98807]: purged_snaps scrub ok 2023-11-16T18:21:41.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:41 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:21:41.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:41 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:21:41.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:41 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:21:41.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:41 smithi002 ceph-mon[98807]: osdmap e69: 11 total, 11 up, 11 in 2023-11-16T18:21:41.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:41 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:41.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:41 smithi002 ceph-mon[98807]: osdmap e70: 11 total, 11 up, 11 in 2023-11-16T18:21:41.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.632+0000 7fc964ee3700 1 -- 172.21.15.188:0/1307510849 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9601003e0 msgr2=0x7fc960100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:21:41.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.632+0000 7fc964ee3700 1 --2- 172.21.15.188:0/1307510849 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9601003e0 0x7fc960100830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc950009990 tx=0x7fc950009ca0 comp rx=0 tx=0).stop 2023-11-16T18:21:41.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.632+0000 7fc964ee3700 1 -- 172.21.15.188:0/1307510849 shutdown_connections 2023-11-16T18:21:41.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.632+0000 7fc964ee3700 1 --2- 172.21.15.188:0/1307510849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960100d70 0x7fc960107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:41.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.632+0000 7fc964ee3700 1 --2- 172.21.15.188:0/1307510849 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9600ff1e0 0x7fc9600ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:41.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.632+0000 7fc964ee3700 1 --2- 172.21.15.188:0/1307510849 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9601003e0 0x7fc960100830 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:41.635 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.632+0000 7fc964ee3700 1 -- 172.21.15.188:0/1307510849 >> 172.21.15.188:0/1307510849 conn(0x7fc9600fa7f0 msgr2=0x7fc9600fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:21:41.635 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.632+0000 7fc964ee3700 1 -- 172.21.15.188:0/1307510849 shutdown_connections 2023-11-16T18:21:41.635 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.632+0000 7fc964ee3700 1 -- 172.21.15.188:0/1307510849 wait complete. 2023-11-16T18:21:41.637 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc964ee3700 1 Processor -- start 2023-11-16T18:21:41.637 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc964ee3700 1 -- start start 2023-11-16T18:21:41.637 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc964ee3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9600ff1e0 0x7fc9600699c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:21:41.637 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc964ee3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9601003e0 0x7fc96019adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:21:41.637 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc964ee3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960100d70 0x7fc96019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:21:41.637 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc964ee3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96019dbe0 con 0x7fc9601003e0 2023-11-16T18:21:41.638 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc964ee3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96019dd50 con 0x7fc960100d70 2023-11-16T18:21:41.638 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc964ee3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96019df10 con 0x7fc9600ff1e0 2023-11-16T18:21:41.638 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc95e59c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9600ff1e0 0x7fc9600699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:21:41.638 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc95e59c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9600ff1e0 0x7fc9600699c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:48402/0 (socket says 172.21.15.188:48402) 2023-11-16T18:21:41.640 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc95e59c700 1 -- 172.21.15.188:0/3765981795 learned_addr learned my addr 172.21.15.188:0/3765981795 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-16T18:21:41.640 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc95dd9b700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9601003e0 0x7fc96019adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:21:41.640 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc95ed9d700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960100d70 0x7fc96019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:21:41.640 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc95e59c700 1 -- 172.21.15.188:0/3765981795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960100d70 msgr2=0x7fc96019d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:21:41.640 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc95e59c700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960100d70 0x7fc96019d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:41.640 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc95e59c700 1 -- 172.21.15.188:0/3765981795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9601003e0 msgr2=0x7fc96019adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:21:41.640 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc95e59c700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9601003e0 0x7fc96019adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:21:41.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc95e59c700 1 -- 172.21.15.188:0/3765981795 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9500095f0 con 0x7fc9600ff1e0 2023-11-16T18:21:41.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.633+0000 7fc95e59c700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9600ff1e0 0x7fc9600699c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc95400b500 tx=0x7fc95400b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:21:41.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.634+0000 7fc94f7fe700 1 -- 172.21.15.188:0/3765981795 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954011830 con 0x7fc9600ff1e0 2023-11-16T18:21:41.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.634+0000 7fc94f7fe700 1 -- 172.21.15.188:0/3765981795 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc954011e50 con 0x7fc9600ff1e0 2023-11-16T18:21:41.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.634+0000 7fc94f7fe700 1 -- 172.21.15.188:0/3765981795 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95400f2e0 con 0x7fc9600ff1e0 2023-11-16T18:21:41.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.634+0000 7fc964ee3700 1 -- 172.21.15.188:0/3765981795 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc96010abd0 con 0x7fc9600ff1e0 2023-11-16T18:21:41.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.634+0000 7fc964ee3700 1 -- 172.21.15.188:0/3765981795 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96010aa40 con 0x7fc9600ff1e0 2023-11-16T18:21:41.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.635+0000 7fc964ee3700 1 -- 172.21.15.188:0/3765981795 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96019e600 con 0x7fc9600ff1e0 2023-11-16T18:21:41.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.636+0000 7fc94f7fe700 1 -- 172.21.15.188:0/3765981795 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc954010580 con 0x7fc9600ff1e0 2023-11-16T18:21:41.642 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.636+0000 7fc94f7fe700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc9480a11d0 0x7fc9480a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:21:41.642 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.636+0000 7fc94f7fe700 1 -- 172.21.15.188:0/3765981795 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6723+0+0 (secure 0 0 0) 0x7fc9540cc250 con 0x7fc9600ff1e0 2023-11-16T18:21:41.643 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.636+0000 7fc95dd9b700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc9480a11d0 0x7fc9480a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:21:41.643 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.637+0000 7fc95dd9b700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc9480a11d0 0x7fc9480a3680 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc960069f20 tx=0x7fc950005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:21:41.643 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.640+0000 7fc94f7fe700 1 -- 172.21.15.188:0/3765981795 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc954095fd0 con 0x7fc9600ff1e0 2023-11-16T18:21:41.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:41 smithi040 ceph-mon[100251]: purged_snaps scrub starts 2023-11-16T18:21:41.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:41 smithi040 ceph-mon[100251]: purged_snaps scrub ok 2023-11-16T18:21:41.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:41 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:21:41.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:41 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:21:41.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:41 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:21:41.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:41 smithi040 ceph-mon[100251]: osdmap e69: 11 total, 11 up, 11 in 2023-11-16T18:21:41.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:41 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:21:41.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:41 smithi040 ceph-mon[100251]: osdmap e70: 11 total, 11 up, 11 in 2023-11-16T18:21:41.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:41.833+0000 7fc964ee3700 1 -- 172.21.15.188:0/3765981795 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc96019ebb0 con 0x7fc9480a11d0 2023-11-16T18:21:42.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:42 smithi188 ceph-mon[100639]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:21:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:21:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:21:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:42 smithi188 ceph-mon[100639]: osdmap e71: 11 total, 11 up, 11 in 2023-11-16T18:21:42.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:42 smithi002 ceph-mon[98807]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:21:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:21:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:21:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:42 smithi002 ceph-mon[98807]: osdmap e71: 11 total, 11 up, 11 in 2023-11-16T18:21:42.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:42 smithi040 ceph-mon[100251]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:21:42.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T18:21:42.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T18:21:42.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:42.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:42 smithi040 ceph-mon[100251]: osdmap e71: 11 total, 11 up, 11 in 2023-11-16T18:21:43.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:43 smithi188 ceph-mon[100639]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:21:43.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:43 smithi002 ceph-mon[98807]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:21:43.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:43 smithi040 ceph-mon[100251]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:21:44.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:44 smithi188 ceph-mon[100639]: osdmap e72: 11 total, 11 up, 11 in 2023-11-16T18:21:44.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:44 smithi188 ceph-mon[100639]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:21:44.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:44 smithi002 ceph-mon[98807]: osdmap e72: 11 total, 11 up, 11 in 2023-11-16T18:21:44.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:44 smithi002 ceph-mon[98807]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:21:44.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:44 smithi040 ceph-mon[100251]: osdmap e72: 11 total, 11 up, 11 in 2023-11-16T18:21:44.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:44 smithi040 ceph-mon[100251]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:21:45.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:45 smithi188 ceph-mon[100639]: from='client.? 172.21.15.188:0/1895566649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f54ca334-9757-4c8b-90e3-cce7daa3b1b1"}]: dispatch 2023-11-16T18:21:45.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:45 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f54ca334-9757-4c8b-90e3-cce7daa3b1b1"}]: dispatch 2023-11-16T18:21:45.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:45 smithi188 ceph-mon[100639]: osdmap e73: 11 total, 11 up, 11 in 2023-11-16T18:21:45.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:45 smithi188 ceph-mon[100639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f54ca334-9757-4c8b-90e3-cce7daa3b1b1"}]': finished 2023-11-16T18:21:45.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:45 smithi188 ceph-mon[100639]: osdmap e74: 12 total, 11 up, 12 in 2023-11-16T18:21:45.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:45 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:21:45.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:45 smithi188 ceph-mon[100639]: from='client.? 172.21.15.188:0/1509395355' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:21:45.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:45 smithi002 ceph-mon[98807]: from='client.? 172.21.15.188:0/1895566649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f54ca334-9757-4c8b-90e3-cce7daa3b1b1"}]: dispatch 2023-11-16T18:21:45.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:45 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f54ca334-9757-4c8b-90e3-cce7daa3b1b1"}]: dispatch 2023-11-16T18:21:45.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:45 smithi002 ceph-mon[98807]: osdmap e73: 11 total, 11 up, 11 in 2023-11-16T18:21:45.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:45 smithi002 ceph-mon[98807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f54ca334-9757-4c8b-90e3-cce7daa3b1b1"}]': finished 2023-11-16T18:21:45.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:45 smithi002 ceph-mon[98807]: osdmap e74: 12 total, 11 up, 12 in 2023-11-16T18:21:45.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:45 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:21:45.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:45 smithi002 ceph-mon[98807]: from='client.? 172.21.15.188:0/1509395355' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:21:45.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:45 smithi040 ceph-mon[100251]: from='client.? 172.21.15.188:0/1895566649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f54ca334-9757-4c8b-90e3-cce7daa3b1b1"}]: dispatch 2023-11-16T18:21:45.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:45 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f54ca334-9757-4c8b-90e3-cce7daa3b1b1"}]: dispatch 2023-11-16T18:21:45.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:45 smithi040 ceph-mon[100251]: osdmap e73: 11 total, 11 up, 11 in 2023-11-16T18:21:45.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:45 smithi040 ceph-mon[100251]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f54ca334-9757-4c8b-90e3-cce7daa3b1b1"}]': finished 2023-11-16T18:21:45.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:45 smithi040 ceph-mon[100251]: osdmap e74: 12 total, 11 up, 12 in 2023-11-16T18:21:45.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:45 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:21:45.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:45 smithi040 ceph-mon[100251]: from='client.? 172.21.15.188:0/1509395355' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T18:21:46.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:46 smithi188 ceph-mon[100639]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:21:46.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:46 smithi002 ceph-mon[98807]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:21:46.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:46 smithi040 ceph-mon[100251]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:21:48.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:47 smithi002 ceph-mon[98807]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-11-16T18:21:48.131 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:47 smithi188 ceph-mon[100639]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-11-16T18:21:48.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:47 smithi040 ceph-mon[100251]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-11-16T18:21:50.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:49 smithi002 ceph-mon[98807]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2023-11-16T18:21:50.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:49 smithi040 ceph-mon[100251]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2023-11-16T18:21:50.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:49 smithi188 ceph-mon[100639]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2023-11-16T18:21:51.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:50 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-11-16T18:21:51.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:50 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:51.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:50 smithi002 ceph-mon[98807]: Deploying daemon osd.11 on smithi188 2023-11-16T18:21:51.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:50 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-11-16T18:21:51.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:50 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:51.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:50 smithi040 ceph-mon[100251]: Deploying daemon osd.11 on smithi188 2023-11-16T18:21:51.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:50 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-11-16T18:21:51.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:50 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:21:51.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:50 smithi188 ceph-mon[100639]: Deploying daemon osd.11 on smithi188 2023-11-16T18:21:52.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:52 smithi040 ceph-mon[100251]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T18:21:52.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:52 smithi188 ceph-mon[100639]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T18:21:52.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:52 smithi002 ceph-mon[98807]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T18:21:54.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:53 smithi188 ceph-mon[100639]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-11-16T18:21:54.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:53 smithi002 ceph-mon[98807]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-11-16T18:21:54.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:53 smithi040 ceph-mon[100251]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-11-16T18:21:56.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:56 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:05.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:56 smithi188 ceph-mon[100639]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-11-16T18:22:05.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:57 smithi188 ceph-mon[100639]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-11-16T18:22:05.049 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 11 on host 'smithi188' 2023-11-16T18:22:05.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:56 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:05.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:56 smithi040 ceph-mon[100251]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-11-16T18:22:05.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:57 smithi040 ceph-mon[100251]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-11-16T18:22:05.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:59 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:59 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:59 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:05.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:59 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:05.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:59 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:22:05.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:59 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:59 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:22:05.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:59 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:22:05.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:59 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:59 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:05.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:21:59 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:00 smithi040 ceph-mon[100251]: from='osd.11 [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T18:22:05.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:00 smithi040 ceph-mon[100251]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T18:22:05.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:00 smithi040 ceph-mon[100251]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:22:05.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-11-16T18:22:05.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: osdmap e75: 12 total, 11 up, 12 in 2023-11-16T18:22:05.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:22:05.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='osd.11 [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:22:05.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:22:05.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:22:05.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:22:05.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:22:05.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:22:05.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:05.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:05.110 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:01 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.110 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:02 smithi040 ceph-mon[100251]: Detected new or changed devices on smithi188 2023-11-16T18:22:05.110 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:02 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi188 to 4292M 2023-11-16T18:22:05.110 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:02 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:22:05.110 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:02 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:22:05.110 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:02 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:22:05.110 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:02 smithi040 ceph-mon[100251]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-11-16T18:22:05.110 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:02 smithi040 ceph-mon[100251]: osdmap e76: 12 total, 11 up, 12 in 2023-11-16T18:22:05.110 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:02 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:22:05.111 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:02 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:22:05.111 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:02 smithi040 ceph-mon[100251]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:22:05.111 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:03 smithi040 ceph-mon[100251]: purged_snaps scrub starts 2023-11-16T18:22:05.111 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:03 smithi040 ceph-mon[100251]: purged_snaps scrub ok 2023-11-16T18:22:05.111 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:03 smithi040 ceph-mon[100251]: osd.11 [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675] boot 2023-11-16T18:22:05.111 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:03 smithi040 ceph-mon[100251]: osdmap e77: 12 total, 12 up, 12 in 2023-11-16T18:22:05.111 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:03 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:22:05.111 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:04 smithi040 ceph-mon[100251]: osdmap e78: 12 total, 12 up, 12 in 2023-11-16T18:22:05.111 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:04 smithi040 ceph-mon[100251]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:05.111 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.111 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.111 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:56 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:05.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:56 smithi002 ceph-mon[98807]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-11-16T18:22:05.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:57 smithi002 ceph-mon[98807]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-11-16T18:22:05.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:59 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:59 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:59 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:05.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:59 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:05.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:59 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:22:05.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:59 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:59 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:22:05.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:59 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:22:05.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:59 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:59 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:05.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:21:59 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:00 smithi002 ceph-mon[98807]: from='osd.11 [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T18:22:05.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:00 smithi002 ceph-mon[98807]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T18:22:05.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:00 smithi002 ceph-mon[98807]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:22:05.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-11-16T18:22:05.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: osdmap e75: 12 total, 11 up, 12 in 2023-11-16T18:22:05.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:22:05.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='osd.11 [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:22:05.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:22:05.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:22:05.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:22:05.114 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:22:05.114 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:22:05.152 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.152 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:05.152 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:05.152 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:01 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.152 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:02 smithi002 ceph-mon[98807]: Detected new or changed devices on smithi188 2023-11-16T18:22:05.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:02 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi188 to 4292M 2023-11-16T18:22:05.158 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:02 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:22:05.158 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:02 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:22:05.158 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:02 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:22:05.158 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:02 smithi002 ceph-mon[98807]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-11-16T18:22:05.158 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:02 smithi002 ceph-mon[98807]: osdmap e76: 12 total, 11 up, 12 in 2023-11-16T18:22:05.158 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:22:05.158 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:02 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:22:05.158 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:02 smithi002 ceph-mon[98807]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:22:05.158 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:03 smithi002 ceph-mon[98807]: purged_snaps scrub starts 2023-11-16T18:22:05.158 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:03 smithi002 ceph-mon[98807]: purged_snaps scrub ok 2023-11-16T18:22:05.158 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:03 smithi002 ceph-mon[98807]: osd.11 [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675] boot 2023-11-16T18:22:05.158 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:03 smithi002 ceph-mon[98807]: osdmap e77: 12 total, 12 up, 12 in 2023-11-16T18:22:05.159 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:03 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:22:05.159 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:04 smithi002 ceph-mon[98807]: osdmap e78: 12 total, 12 up, 12 in 2023-11-16T18:22:05.159 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:04 smithi002 ceph-mon[98807]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:05.160 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:58.299+0000 7fc94f7fe700 1 -- 172.21.15.188:0/3765981795 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fc96019ebb0 con 0x7fc9480a11d0 2023-11-16T18:22:05.160 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:58.301+0000 7fc964ee3700 1 -- 172.21.15.188:0/3765981795 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc9480a11d0 msgr2=0x7fc9480a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:05.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:58.301+0000 7fc964ee3700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc9480a11d0 0x7fc9480a3680 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc960069f20 tx=0x7fc950005fd0 comp rx=0 tx=0).stop 2023-11-16T18:22:05.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:58.301+0000 7fc964ee3700 1 -- 172.21.15.188:0/3765981795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9600ff1e0 msgr2=0x7fc9600699c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:05.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:58.301+0000 7fc964ee3700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9600ff1e0 0x7fc9600699c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc95400b500 tx=0x7fc95400b810 comp rx=0 tx=0).stop 2023-11-16T18:22:05.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:58.301+0000 7fc964ee3700 1 -- 172.21.15.188:0/3765981795 shutdown_connections 2023-11-16T18:22:05.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:58.301+0000 7fc964ee3700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc9480a11d0 0x7fc9480a3680 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:05.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:58.302+0000 7fc964ee3700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9600ff1e0 0x7fc9600699c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:05.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:58.302+0000 7fc964ee3700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960100d70 0x7fc96019d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:05.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:58.302+0000 7fc964ee3700 1 --2- 172.21.15.188:0/3765981795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9601003e0 0x7fc96019adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:05.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:58.302+0000 7fc964ee3700 1 -- 172.21.15.188:0/3765981795 >> 172.21.15.188:0/3765981795 conn(0x7fc9600fa7f0 msgr2=0x7fc9600fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:05.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:58.302+0000 7fc964ee3700 1 -- 172.21.15.188:0/3765981795 shutdown_connections 2023-11-16T18:22:05.209 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:21:58.302+0000 7fc964ee3700 1 -- 172.21.15.188:0/3765981795 wait complete. 2023-11-16T18:22:05.209 DEBUG:teuthology.orchestra.run.smithi188:osd.11> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.11.service 2023-11-16T18:22:05.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:59 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:59 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:59 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:05.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:59 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:05.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:59 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:22:05.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:59 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:59 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:22:05.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:59 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:22:05.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:59 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:59 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:05.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:21:59 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:00 smithi188 ceph-mon[100639]: from='osd.11 [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T18:22:05.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:00 smithi188 ceph-mon[100639]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T18:22:05.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:00 smithi188 ceph-mon[100639]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:22:05.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-11-16T18:22:05.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: osdmap e75: 12 total, 11 up, 12 in 2023-11-16T18:22:05.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:22:05.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='osd.11 [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:22:05.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-11-16T18:22:05.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:22:05.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:22:05.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:22:05.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:22:05.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:05.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:05.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:01 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:05.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:02 smithi188 ceph-mon[100639]: Detected new or changed devices on smithi188 2023-11-16T18:22:05.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:02 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi188 to 4292M 2023-11-16T18:22:05.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:02 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:22:05.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:02 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:22:05.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:02 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:02 smithi188 ceph-mon[100639]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-11-16T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:02 smithi188 ceph-mon[100639]: osdmap e76: 12 total, 11 up, 12 in 2023-11-16T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:02 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:02 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:02 smithi188 ceph-mon[100639]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:03 smithi188 ceph-mon[100639]: purged_snaps scrub starts 2023-11-16T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:03 smithi188 ceph-mon[100639]: purged_snaps scrub ok 2023-11-16T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:03 smithi188 ceph-mon[100639]: osd.11 [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675] boot 2023-11-16T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:03 smithi188 ceph-mon[100639]: osdmap e77: 12 total, 12 up, 12 in 2023-11-16T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:03 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:04 smithi188 ceph-mon[100639]: osdmap e78: 12 total, 12 up, 12 in 2023-11-16T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:04 smithi188 ceph-mon[100639]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:05.220 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-11-16T18:22:05.221 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-16T18:22:05.253 INFO:journalctl@ceph.osd.11.smithi188.stdout:-- Logs begin at Thu 2023-11-16 17:59:32 UTC. -- 2023-11-16T18:22:05.884 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:22:06.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:05 smithi002 ceph-mon[98807]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:06.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:05 smithi040 ceph-mon[100251]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:06.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:05 smithi188 ceph-mon[100639]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:07.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.079+0000 7fc376d5d700 1 -- 172.21.15.2:0/1732079214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc370100d70 msgr2=0x7fc370103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:07.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.079+0000 7fc376d5d700 1 --2- 172.21.15.2:0/1732079214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc370100d70 0x7fc370103150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc364009990 tx=0x7fc364009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.079+0000 7fc376d5d700 1 -- 172.21.15.2:0/1732079214 shutdown_connections 2023-11-16T18:22:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.079+0000 7fc376d5d700 1 --2- 172.21.15.2:0/1732079214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc370105fb0 0x7fc370108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.079+0000 7fc376d5d700 1 --2- 172.21.15.2:0/1732079214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc370100d70 0x7fc370103150 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.079+0000 7fc376d5d700 1 --2- 172.21.15.2:0/1732079214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc370103690 0x7fc370105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.079+0000 7fc376d5d700 1 -- 172.21.15.2:0/1732079214 >> 172.21.15.2:0/1732079214 conn(0x7fc3700fa7f0 msgr2=0x7fc3700fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.079+0000 7fc376d5d700 1 -- 172.21.15.2:0/1732079214 shutdown_connections 2023-11-16T18:22:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.079+0000 7fc376d5d700 1 -- 172.21.15.2:0/1732079214 wait complete. 2023-11-16T18:22:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc376d5d700 1 Processor -- start 2023-11-16T18:22:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc376d5d700 1 -- start start 2023-11-16T18:22:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc376d5d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc370100d70 0x7fc370196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc376d5d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc370103690 0x7fc370196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc376d5d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc370105fb0 0x7fc37019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc376d5d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37019dbd0 con 0x7fc370103690 2023-11-16T18:22:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc376d5d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37019dd10 con 0x7fc370105fb0 2023-11-16T18:22:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc376d5d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37019dea0 con 0x7fc370100d70 2023-11-16T18:22:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc36ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc370103690 0x7fc370196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc36ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc370103690 0x7fc370196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47432/0 (socket says 172.21.15.2:47432) 2023-11-16T18:22:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc36ffff700 1 -- 172.21.15.2:0/2111653747 learned_addr learned my addr 172.21.15.2:0/2111653747 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc374af9700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc370100d70 0x7fc370196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc3752fa700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc370105fb0 0x7fc37019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc36ffff700 1 -- 172.21.15.2:0/2111653747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc370100d70 msgr2=0x7fc370196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc36ffff700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc370100d70 0x7fc370196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc36ffff700 1 -- 172.21.15.2:0/2111653747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc370105fb0 msgr2=0x7fc37019d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc36ffff700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc370105fb0 0x7fc37019d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc36ffff700 1 -- 172.21.15.2:0/2111653747 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3640095f0 con 0x7fc370103690 2023-11-16T18:22:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.080+0000 7fc36ffff700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc370103690 0x7fc370196580 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc35c00d980 tx=0x7fc35c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.081+0000 7fc36dffb700 1 -- 172.21.15.2:0/2111653747 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c009b40 con 0x7fc370103690 2023-11-16T18:22:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.081+0000 7fc376d5d700 1 -- 172.21.15.2:0/2111653747 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37010bc80 con 0x7fc370103690 2023-11-16T18:22:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.081+0000 7fc36dffb700 1 -- 172.21.15.2:0/2111653747 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc35c00b6e0 con 0x7fc370103690 2023-11-16T18:22:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.081+0000 7fc36dffb700 1 -- 172.21.15.2:0/2111653747 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c00f620 con 0x7fc370103690 2023-11-16T18:22:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.081+0000 7fc376d5d700 1 -- 172.21.15.2:0/2111653747 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37010baf0 con 0x7fc370103690 2023-11-16T18:22:07.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.082+0000 7fc376d5d700 1 -- 172.21.15.2:0/2111653747 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3700fc360 con 0x7fc370103690 2023-11-16T18:22:07.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.083+0000 7fc36dffb700 1 -- 172.21.15.2:0/2111653747 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc35c010460 con 0x7fc370103690 2023-11-16T18:22:07.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.083+0000 7fc36dffb700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc3600a1150 0x7fc3600a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:07.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.083+0000 7fc36dffb700 1 -- 172.21.15.2:0/2111653747 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc35c0cc370 con 0x7fc370103690 2023-11-16T18:22:07.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.084+0000 7fc374af9700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc3600a1150 0x7fc3600a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:07.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.084+0000 7fc374af9700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc3600a1150 0x7fc3600a3600 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc36400b5c0 tx=0x7fc364005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:07.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.088+0000 7fc36dffb700 1 -- 172.21.15.2:0/2111653747 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc35c095f60 con 0x7fc370103690 2023-11-16T18:22:07.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.286+0000 7fc376d5d700 1 -- 172.21.15.2:0/2111653747 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fc370061700 con 0x7fc370103690 2023-11-16T18:22:07.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.286+0000 7fc36dffb700 1 -- 172.21.15.2:0/2111653747 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v78) v1 ==== 74+0+133 (secure 0 0 0) 0x7fc35c099580 con 0x7fc370103690 2023-11-16T18:22:07.288 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:07.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.288+0000 7fc376d5d700 1 -- 172.21.15.2:0/2111653747 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc3600a1150 msgr2=0x7fc3600a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:07.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.288+0000 7fc376d5d700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc3600a1150 0x7fc3600a3600 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc36400b5c0 tx=0x7fc364005fb0 comp rx=0 tx=0).stop 2023-11-16T18:22:07.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.288+0000 7fc376d5d700 1 -- 172.21.15.2:0/2111653747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc370103690 msgr2=0x7fc370196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:07.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.288+0000 7fc376d5d700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc370103690 0x7fc370196580 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc35c00d980 tx=0x7fc35c00dd40 comp rx=0 tx=0).stop 2023-11-16T18:22:07.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.288+0000 7fc376d5d700 1 -- 172.21.15.2:0/2111653747 shutdown_connections 2023-11-16T18:22:07.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.288+0000 7fc376d5d700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc3600a1150 0x7fc3600a3600 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:07.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.288+0000 7fc376d5d700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc370100d70 0x7fc370196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:07.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.288+0000 7fc376d5d700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc370105fb0 0x7fc37019d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:07.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.288+0000 7fc376d5d700 1 --2- 172.21.15.2:0/2111653747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc370103690 0x7fc370196580 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:07.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.288+0000 7fc376d5d700 1 -- 172.21.15.2:0/2111653747 >> 172.21.15.2:0/2111653747 conn(0x7fc3700fa7f0 msgr2=0x7fc3701019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:07.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.288+0000 7fc376d5d700 1 -- 172.21.15.2:0/2111653747 shutdown_connections 2023-11-16T18:22:07.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:07.288+0000 7fc376d5d700 1 -- 172.21.15.2:0/2111653747 wait complete. 2023-11-16T18:22:07.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:07 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2111653747' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T18:22:07.644 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":78,"num_osds":12,"num_up_osds":12,"osd_up_since":1700158922,"num_in_osds":12,"osd_in_since":1700158904,"num_remapped_pgs":0} 2023-11-16T18:22:07.645 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-16T18:22:07.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:07 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2111653747' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T18:22:07.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:07 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2111653747' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T18:22:08.297 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:22:08.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:08 smithi002 ceph-mon[98807]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:08.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:08 smithi040 ceph-mon[100251]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:08.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:08 smithi188 ceph-mon[100639]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:09.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.370+0000 7f002bba1700 1 -- 172.21.15.2:0/1750523271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00240ff460 msgr2=0x7f00240ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:09.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.370+0000 7f002bba1700 1 --2- 172.21.15.2:0/1750523271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00240ff460 0x7f00240ff830 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00200099c0 tx=0x7f0020009cd0 comp rx=0 tx=0).stop 2023-11-16T18:22:09.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.370+0000 7f002bba1700 1 -- 172.21.15.2:0/1750523271 shutdown_connections 2023-11-16T18:22:09.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.370+0000 7f002bba1700 1 --2- 172.21.15.2:0/1750523271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0024104500 0x7f002410c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:09.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.370+0000 7f002bba1700 1 --2- 172.21.15.2:0/1750523271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00240ff460 0x7f00240ff830 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:09.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.370+0000 7f002bba1700 1 --2- 172.21.15.2:0/1750523271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00240ffe00 0x7f0024103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:09.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.370+0000 7f002bba1700 1 -- 172.21.15.2:0/1750523271 >> 172.21.15.2:0/1750523271 conn(0x7f00240770a0 msgr2=0x7f00240774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:09.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.370+0000 7f002bba1700 1 -- 172.21.15.2:0/1750523271 shutdown_connections 2023-11-16T18:22:09.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.370+0000 7f002bba1700 1 -- 172.21.15.2:0/1750523271 wait complete. 2023-11-16T18:22:09.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.371+0000 7f002bba1700 1 Processor -- start 2023-11-16T18:22:09.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.371+0000 7f002bba1700 1 -- start start 2023-11-16T18:22:09.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.371+0000 7f002bba1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00240ff460 0x7f0024197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:09.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.371+0000 7f002bba1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00240ffe00 0x7f0024197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:09.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.371+0000 7f002bba1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0024104500 0x7f00241a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:09.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.371+0000 7f002bba1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0024198190 con 0x7f0024104500 2023-11-16T18:22:09.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.371+0000 7f002bba1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00241a2900 con 0x7f00240ff460 2023-11-16T18:22:09.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.371+0000 7f002bba1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00241a2a70 con 0x7f00240ffe00 2023-11-16T18:22:09.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.371+0000 7f002a13e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0024104500 0x7f00241a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:09.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.371+0000 7f002a13e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0024104500 0x7f00241a23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47448/0 (socket says 172.21.15.2:47448) 2023-11-16T18:22:09.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.371+0000 7f002a13e700 1 -- 172.21.15.2:0/4269789620 learned_addr learned my addr 172.21.15.2:0/4269789620 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:09.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.371+0000 7f002913c700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00240ffe00 0x7f0024197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:09.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.371+0000 7f002993d700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00240ff460 0x7f0024197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:09.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.372+0000 7f002a13e700 1 -- 172.21.15.2:0/4269789620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00240ffe00 msgr2=0x7f0024197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:09.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.372+0000 7f002a13e700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00240ffe00 0x7f0024197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:09.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.372+0000 7f002a13e700 1 -- 172.21.15.2:0/4269789620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00240ff460 msgr2=0x7f0024197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:09.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.372+0000 7f002a13e700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00240ff460 0x7f0024197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:09.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.372+0000 7f002a13e700 1 -- 172.21.15.2:0/4269789620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00200095f0 con 0x7f0024104500 2023-11-16T18:22:09.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.372+0000 7f002a13e700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0024104500 0x7f00241a23c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f001c00b970 tx=0x7f001c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:09.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.372+0000 7f0016ffd700 1 -- 172.21.15.2:0/4269789620 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c00d860 con 0x7f0024104500 2023-11-16T18:22:09.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.372+0000 7f0016ffd700 1 -- 172.21.15.2:0/4269789620 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f001c00de80 con 0x7f0024104500 2023-11-16T18:22:09.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.372+0000 7f0016ffd700 1 -- 172.21.15.2:0/4269789620 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c017400 con 0x7f0024104500 2023-11-16T18:22:09.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.372+0000 7f002bba1700 1 -- 172.21.15.2:0/4269789620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002410f330 con 0x7f0024104500 2023-11-16T18:22:09.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.372+0000 7f002bba1700 1 -- 172.21.15.2:0/4269789620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002410f1a0 con 0x7f0024104500 2023-11-16T18:22:09.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.373+0000 7f002bba1700 1 -- 172.21.15.2:0/4269789620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f002410f4c0 con 0x7f0024104500 2023-11-16T18:22:09.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.374+0000 7f0016ffd700 1 -- 172.21.15.2:0/4269789620 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f001c00e630 con 0x7f0024104500 2023-11-16T18:22:09.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.374+0000 7f0016ffd700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f00100a11d0 0x7f00100a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:09.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.375+0000 7f002993d700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f00100a11d0 0x7f00100a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:09.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.375+0000 7f0016ffd700 1 -- 172.21.15.2:0/4269789620 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f001c006e40 con 0x7f0024104500 2023-11-16T18:22:09.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.375+0000 7f002993d700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f00100a11d0 0x7f00100a3680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0020000c00 tx=0x7f0020005040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:09.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.379+0000 7f0016ffd700 1 -- 172.21.15.2:0/4269789620 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f001c095c00 con 0x7f0024104500 2023-11-16T18:22:09.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.577+0000 7f002bba1700 1 -- 172.21.15.2:0/4269789620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0024066ea0 con 0x7f0024104500 2023-11-16T18:22:09.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.578+0000 7f0016ffd700 1 -- 172.21.15.2:0/4269789620 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18315 (secure 0 0 0) 0x7f001c099220 con 0x7f0024104500 2023-11-16T18:22:09.579 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:09.579 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":78,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:22:03.153518+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:22:09.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.580+0000 7f002bba1700 1 -- 172.21.15.2:0/4269789620 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f00100a11d0 msgr2=0x7f00100a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:09.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.580+0000 7f002bba1700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f00100a11d0 0x7f00100a3680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0020000c00 tx=0x7f0020005040 comp rx=0 tx=0).stop 2023-11-16T18:22:09.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.580+0000 7f002bba1700 1 -- 172.21.15.2:0/4269789620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0024104500 msgr2=0x7f00241a23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:09.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.580+0000 7f002bba1700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0024104500 0x7f00241a23c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f001c00b970 tx=0x7f001c00bc80 comp rx=0 tx=0).stop 2023-11-16T18:22:09.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.580+0000 7f002bba1700 1 -- 172.21.15.2:0/4269789620 shutdown_connections 2023-11-16T18:22:09.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.580+0000 7f002bba1700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f00100a11d0 0x7f00100a3680 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:09.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.580+0000 7f002bba1700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0024104500 0x7f00241a23c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:09.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.580+0000 7f002bba1700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00240ff460 0x7f0024197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:09.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.580+0000 7f002bba1700 1 --2- 172.21.15.2:0/4269789620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00240ffe00 0x7f0024197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:09.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.580+0000 7f002bba1700 1 -- 172.21.15.2:0/4269789620 >> 172.21.15.2:0/4269789620 conn(0x7f00240770a0 msgr2=0x7f00240fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:09.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.580+0000 7f002bba1700 1 -- 172.21.15.2:0/4269789620 shutdown_connections 2023-11-16T18:22:09.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:09.580+0000 7f002bba1700 1 -- 172.21.15.2:0/4269789620 wait complete. 2023-11-16T18:22:09.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:09 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/4269789620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:09.946 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-16T18:22:09.947 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd pool get .mgr pg_num 2023-11-16T18:22:09.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:09 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/4269789620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:10.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:09 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/4269789620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:10.599 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:22:10.976 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:10 smithi002 ceph-mon[98807]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:10.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:10 smithi040 ceph-mon[100251]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:11.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:10 smithi188 ceph-mon[100639]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:11.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.813+0000 7f25e8e4b700 1 -- 172.21.15.2:0/3640262851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25e40ff470 msgr2=0x7f25e40ff840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:11.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.813+0000 7f25e8e4b700 1 --2- 172.21.15.2:0/3640262851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25e40ff470 0x7f25e40ff840 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f25d40099c0 tx=0x7f25d4009cd0 comp rx=0 tx=0).stop 2023-11-16T18:22:11.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.813+0000 7f25e8e4b700 1 -- 172.21.15.2:0/3640262851 shutdown_connections 2023-11-16T18:22:11.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.813+0000 7f25e8e4b700 1 --2- 172.21.15.2:0/3640262851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e4104510 0x7f25e410c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:11.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.813+0000 7f25e8e4b700 1 --2- 172.21.15.2:0/3640262851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25e40ff470 0x7f25e40ff840 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:11.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.813+0000 7f25e8e4b700 1 --2- 172.21.15.2:0/3640262851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e40ffe10 0x7f25e4103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:11.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.813+0000 7f25e8e4b700 1 -- 172.21.15.2:0/3640262851 >> 172.21.15.2:0/3640262851 conn(0x7f25e40770d0 msgr2=0x7f25e40774d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:11.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.813+0000 7f25e8e4b700 1 -- 172.21.15.2:0/3640262851 shutdown_connections 2023-11-16T18:22:11.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.813+0000 7f25e8e4b700 1 -- 172.21.15.2:0/3640262851 wait complete. 2023-11-16T18:22:11.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.814+0000 7f25e8e4b700 1 Processor -- start 2023-11-16T18:22:11.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.814+0000 7f25e8e4b700 1 -- start start 2023-11-16T18:22:11.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e8e4b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e40ff470 0x7f25e4195260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:11.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e8e4b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25e40ffe10 0x7f25e41957a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:11.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e8e4b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e4104510 0x7f25e4199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:11.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e8e4b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e419a220 con 0x7f25e40ffe10 2023-11-16T18:22:11.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e8e4b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e419a390 con 0x7f25e40ff470 2023-11-16T18:22:11.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e8e4b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e419a550 con 0x7f25e4104510 2023-11-16T18:22:11.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e1d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25e40ffe10 0x7f25e41957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:11.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e1d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25e40ffe10 0x7f25e41957a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47456/0 (socket says 172.21.15.2:47456) 2023-11-16T18:22:11.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e1d9b700 1 -- 172.21.15.2:0/199300854 learned_addr learned my addr 172.21.15.2:0/199300854 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:11.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e2d9d700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e4104510 0x7f25e4199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:11.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e259c700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e40ff470 0x7f25e4195260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e1d9b700 1 -- 172.21.15.2:0/199300854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e4104510 msgr2=0x7f25e4199c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e1d9b700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e4104510 0x7f25e4199c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e1d9b700 1 -- 172.21.15.2:0/199300854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e40ff470 msgr2=0x7f25e4195260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e1d9b700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e40ff470 0x7f25e4195260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e1d9b700 1 -- 172.21.15.2:0/199300854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d40095f0 con 0x7f25e40ffe10 2023-11-16T18:22:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e259c700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e40ff470 0x7f25e4195260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:22:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e1d9b700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25e40ffe10 0x7f25e41957a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00c9e0 tx=0x7f25cc00cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25db7fe700 1 -- 172.21.15.2:0/199300854 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc007aa0 con 0x7f25e40ffe10 2023-11-16T18:22:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25db7fe700 1 -- 172.21.15.2:0/199300854 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25cc007c00 con 0x7f25e40ffe10 2023-11-16T18:22:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25db7fe700 1 -- 172.21.15.2:0/199300854 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc018660 con 0x7f25e40ffe10 2023-11-16T18:22:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.815+0000 7f25e8e4b700 1 -- 172.21.15.2:0/199300854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e410f250 con 0x7f25e40ffe10 2023-11-16T18:22:11.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.816+0000 7f25e8e4b700 1 -- 172.21.15.2:0/199300854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e410f0c0 con 0x7f25e40ffe10 2023-11-16T18:22:11.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.816+0000 7f25e8e4b700 1 -- 172.21.15.2:0/199300854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e410f3e0 con 0x7f25e40ffe10 2023-11-16T18:22:11.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.817+0000 7f25db7fe700 1 -- 172.21.15.2:0/199300854 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f25cc00f450 con 0x7f25e40ffe10 2023-11-16T18:22:11.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.818+0000 7f25db7fe700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f25d00a11d0 0x7f25d00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:11.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.818+0000 7f25e259c700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f25d00a11d0 0x7f25d00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:11.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.819+0000 7f25db7fe700 1 -- 172.21.15.2:0/199300854 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f25cc0cc430 con 0x7f25e40ffe10 2023-11-16T18:22:11.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.819+0000 7f25e259c700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f25d00a11d0 0x7f25d00a3680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f25d4006010 tx=0x7f25d4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:11.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:11.823+0000 7f25db7fe700 1 -- 172.21.15.2:0/199300854 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f25cc095f80 con 0x7f25e40ffe10 2023-11-16T18:22:12.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.018+0000 7f25e8e4b700 1 -- 172.21.15.2:0/199300854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f25e41a23e0 con 0x7f25e40ffe10 2023-11-16T18:22:12.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.018+0000 7f25db7fe700 1 -- 172.21.15.2:0/199300854 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f25cc0995a0 con 0x7f25e40ffe10 2023-11-16T18:22:12.020 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:22:12.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.020+0000 7f25e8e4b700 1 -- 172.21.15.2:0/199300854 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f25d00a11d0 msgr2=0x7f25d00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:12.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.020+0000 7f25e8e4b700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f25d00a11d0 0x7f25d00a3680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f25d4006010 tx=0x7f25d4000bc0 comp rx=0 tx=0).stop 2023-11-16T18:22:12.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.020+0000 7f25e8e4b700 1 -- 172.21.15.2:0/199300854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25e40ffe10 msgr2=0x7f25e41957a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:12.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.020+0000 7f25e8e4b700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25e40ffe10 0x7f25e41957a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00c9e0 tx=0x7f25cc00cda0 comp rx=0 tx=0).stop 2023-11-16T18:22:12.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.020+0000 7f25e8e4b700 1 -- 172.21.15.2:0/199300854 shutdown_connections 2023-11-16T18:22:12.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.020+0000 7f25e8e4b700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f25d00a11d0 0x7f25d00a3680 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:12.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.020+0000 7f25e8e4b700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e4104510 0x7f25e4199c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:12.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.020+0000 7f25e8e4b700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e40ff470 0x7f25e4195260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:12.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.020+0000 7f25e8e4b700 1 --2- 172.21.15.2:0/199300854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25e40ffe10 0x7f25e41957a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:12.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.020+0000 7f25e8e4b700 1 -- 172.21.15.2:0/199300854 >> 172.21.15.2:0/199300854 conn(0x7f25e40770d0 msgr2=0x7f25e40fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:12.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.020+0000 7f25e8e4b700 1 -- 172.21.15.2:0/199300854 shutdown_connections 2023-11-16T18:22:12.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:12.020+0000 7f25e8e4b700 1 -- 172.21.15.2:0/199300854 wait complete. 2023-11-16T18:22:12.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:12 smithi002 ceph-mon[98807]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:12.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:12 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/199300854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:22:12.356 INFO:tasks.cephadm:Adding mds.a on smithi002 2023-11-16T18:22:12.356 INFO:tasks.cephadm:Adding mds.d on smithi002 2023-11-16T18:22:12.356 INFO:tasks.cephadm:Adding mds.g on smithi002 2023-11-16T18:22:12.356 INFO:tasks.cephadm:Adding mds.j on smithi002 2023-11-16T18:22:12.356 INFO:tasks.cephadm:Adding mds.b on smithi040 2023-11-16T18:22:12.357 INFO:tasks.cephadm:Adding mds.e on smithi040 2023-11-16T18:22:12.357 INFO:tasks.cephadm:Adding mds.h on smithi040 2023-11-16T18:22:12.357 INFO:tasks.cephadm:Adding mds.k on smithi040 2023-11-16T18:22:12.357 INFO:tasks.cephadm:Adding mds.c on smithi188 2023-11-16T18:22:12.357 INFO:tasks.cephadm:Adding mds.f on smithi188 2023-11-16T18:22:12.357 INFO:tasks.cephadm:Adding mds.i on smithi188 2023-11-16T18:22:12.357 INFO:tasks.cephadm:Adding mds.l on smithi188 2023-11-16T18:22:12.357 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph orch apply mds all '12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l' 2023-11-16T18:22:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:12 smithi040 ceph-mon[100251]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:12 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/199300854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:22:12.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:12 smithi188 ceph-mon[100639]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:12.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:12 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/199300854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:22:13.009 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:22:14.117 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:13 smithi188 ceph-mon[100639]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:14.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:13 smithi040 ceph-mon[100251]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:14.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.314+0000 7f0b3ff77700 1 -- 172.21.15.188:0/2653966645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b38100d70 msgr2=0x7f0b38107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:14.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.314+0000 7f0b3ff77700 1 --2- 172.21.15.188:0/2653966645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b38100d70 0x7f0b38107630 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000b0a0 tx=0x7f0b3000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:14.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.314+0000 7f0b3ff77700 1 -- 172.21.15.188:0/2653966645 shutdown_connections 2023-11-16T18:22:14.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.314+0000 7f0b3ff77700 1 --2- 172.21.15.188:0/2653966645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b38100d70 0x7f0b38107630 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:14.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.314+0000 7f0b3ff77700 1 --2- 172.21.15.188:0/2653966645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b380ff1e0 0x7f0b380ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:14.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.314+0000 7f0b3ff77700 1 --2- 172.21.15.188:0/2653966645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b381003e0 0x7f0b38100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:14.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.314+0000 7f0b3ff77700 1 -- 172.21.15.188:0/2653966645 >> 172.21.15.188:0/2653966645 conn(0x7f0b380fa7f0 msgr2=0x7f0b380fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:14.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.314+0000 7f0b3ff77700 1 -- 172.21.15.188:0/2653966645 shutdown_connections 2023-11-16T18:22:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.315+0000 7f0b3ff77700 1 -- 172.21.15.188:0/2653966645 wait complete. 2023-11-16T18:22:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.315+0000 7f0b3ff77700 1 Processor -- start 2023-11-16T18:22:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.315+0000 7f0b3ff77700 1 -- start start 2023-11-16T18:22:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.315+0000 7f0b3ff77700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b380ff1e0 0x7f0b38069a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.315+0000 7f0b3ff77700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b381003e0 0x7f0b3819adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.315+0000 7f0b3ff77700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b38100d70 0x7f0b3819d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.315+0000 7f0b3ff77700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3819dc20 con 0x7f0b380ff1e0 2023-11-16T18:22:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.315+0000 7f0b3ff77700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3819dd90 con 0x7f0b38100d70 2023-11-16T18:22:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.315+0000 7f0b3ff77700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3819df50 con 0x7f0b381003e0 2023-11-16T18:22:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.315+0000 7f0b3d512700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b381003e0 0x7f0b3819adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.315+0000 7f0b3d512700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b381003e0 0x7f0b3819adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:48522/0 (socket says 172.21.15.188:48522) 2023-11-16T18:22:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.315+0000 7f0b3d512700 1 -- 172.21.15.188:0/3341520075 learned_addr learned my addr 172.21.15.188:0/3341520075 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-16T18:22:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.315+0000 7f0b3dd13700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b380ff1e0 0x7f0b38069a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:14.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.316+0000 7f0b3e514700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b38100d70 0x7f0b3819d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:14.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.316+0000 7f0b3d512700 1 -- 172.21.15.188:0/3341520075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b38100d70 msgr2=0x7f0b3819d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:14.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.316+0000 7f0b3d512700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b38100d70 0x7f0b3819d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:14.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.316+0000 7f0b3d512700 1 -- 172.21.15.188:0/3341520075 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b380ff1e0 msgr2=0x7f0b38069a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:14.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.316+0000 7f0b3d512700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b380ff1e0 0x7f0b38069a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:14.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.316+0000 7f0b3d512700 1 -- 172.21.15.188:0/3341520075 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b30009d00 con 0x7f0b381003e0 2023-11-16T18:22:14.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.316+0000 7f0b3d512700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b381003e0 0x7f0b3819adf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0b3400c920 tx=0x7f0b3400cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:14.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.316+0000 7f0b2effd700 1 -- 172.21.15.188:0/3341520075 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b34007b00 con 0x7f0b381003e0 2023-11-16T18:22:14.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.316+0000 7f0b2effd700 1 -- 172.21.15.188:0/3341520075 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b3400ce70 con 0x7f0b381003e0 2023-11-16T18:22:14.320 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.316+0000 7f0b3ff77700 1 -- 172.21.15.188:0/3341520075 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3810ab30 con 0x7f0b381003e0 2023-11-16T18:22:14.321 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.316+0000 7f0b2effd700 1 -- 172.21.15.188:0/3341520075 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b340054e0 con 0x7f0b381003e0 2023-11-16T18:22:14.321 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.316+0000 7f0b3ff77700 1 -- 172.21.15.188:0/3341520075 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3810a9a0 con 0x7f0b381003e0 2023-11-16T18:22:14.322 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.317+0000 7f0b3ff77700 1 -- 172.21.15.188:0/3341520075 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b38106740 con 0x7f0b381003e0 2023-11-16T18:22:14.322 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.319+0000 7f0b2effd700 1 -- 172.21.15.188:0/3341520075 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f0b34004750 con 0x7f0b381003e0 2023-11-16T18:22:14.322 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.319+0000 7f0b2effd700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0b240a1150 0x7f0b240a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:14.322 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.320+0000 7f0b2effd700 1 -- 172.21.15.188:0/3341520075 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0b340cc4f0 con 0x7f0b381003e0 2023-11-16T18:22:14.322 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.320+0000 7f0b3dd13700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0b240a1150 0x7f0b240a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:14.322 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.320+0000 7f0b3dd13700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0b240a1150 0x7f0b240a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0b38100240 tx=0x7f0b2800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:14.323 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.323+0000 7f0b2effd700 1 -- 172.21.15.188:0/3341520075 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0b34096040 con 0x7f0b381003e0 2023-11-16T18:22:14.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:13 smithi002 ceph-mon[98807]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:14.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.517+0000 7f0b3ff77700 1 -- 172.21.15.188:0/3341520075 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b38061d80 con 0x7f0b240a1150 2023-11-16T18:22:14.527 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.526+0000 7f0b2effd700 1 -- 172.21.15.188:0/3341520075 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f0b38061d80 con 0x7f0b240a1150 2023-11-16T18:22:14.527 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mds.all update... 2023-11-16T18:22:14.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.528+0000 7f0b3ff77700 1 -- 172.21.15.188:0/3341520075 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0b240a1150 msgr2=0x7f0b240a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:14.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.528+0000 7f0b3ff77700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0b240a1150 0x7f0b240a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0b38100240 tx=0x7f0b2800a380 comp rx=0 tx=0).stop 2023-11-16T18:22:14.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.528+0000 7f0b3ff77700 1 -- 172.21.15.188:0/3341520075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b381003e0 msgr2=0x7f0b3819adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:14.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.528+0000 7f0b3ff77700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b381003e0 0x7f0b3819adf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0b3400c920 tx=0x7f0b3400cc30 comp rx=0 tx=0).stop 2023-11-16T18:22:14.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.528+0000 7f0b3ff77700 1 -- 172.21.15.188:0/3341520075 shutdown_connections 2023-11-16T18:22:14.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.528+0000 7f0b3ff77700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0b240a1150 0x7f0b240a3600 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:14.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.528+0000 7f0b3ff77700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b38100d70 0x7f0b3819d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:14.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.528+0000 7f0b3ff77700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b380ff1e0 0x7f0b38069a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:14.530 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.528+0000 7f0b3ff77700 1 --2- 172.21.15.188:0/3341520075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b381003e0 0x7f0b3819adf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:14.530 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.528+0000 7f0b3ff77700 1 -- 172.21.15.188:0/3341520075 >> 172.21.15.188:0/3341520075 conn(0x7f0b380fa7f0 msgr2=0x7f0b38103880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:14.530 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.528+0000 7f0b3ff77700 1 -- 172.21.15.188:0/3341520075 shutdown_connections 2023-11-16T18:22:14.530 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-16T18:22:14.528+0000 7f0b3ff77700 1 -- 172.21.15.188:0/3341520075 wait complete. 2023-11-16T18:22:14.954 DEBUG:teuthology.orchestra.run.smithi002:mds.a> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.a.service 2023-11-16T18:22:14.957 DEBUG:teuthology.orchestra.run.smithi002:mds.d> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.d.service 2023-11-16T18:22:14.995 INFO:journalctl@ceph.mds.a.smithi002.stdout:-- Logs begin at Thu 2023-11-16 17:59:05 UTC. -- 2023-11-16T18:22:14.996 DEBUG:teuthology.orchestra.run.smithi002:mds.g> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.g.service 2023-11-16T18:22:15.027 INFO:journalctl@ceph.mds.d.smithi002.stdout:-- Logs begin at Thu 2023-11-16 17:59:05 UTC. -- 2023-11-16T18:22:15.029 DEBUG:teuthology.orchestra.run.smithi002:mds.j> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.j.service 2023-11-16T18:22:15.055 INFO:journalctl@ceph.mds.g.smithi002.stdout:-- Logs begin at Thu 2023-11-16 17:59:05 UTC. -- 2023-11-16T18:22:15.056 DEBUG:teuthology.orchestra.run.smithi040:mds.b> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.b.service 2023-11-16T18:22:15.059 DEBUG:teuthology.orchestra.run.smithi040:mds.e> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.e.service 2023-11-16T18:22:15.085 INFO:journalctl@ceph.mds.j.smithi002.stdout:-- Logs begin at Thu 2023-11-16 17:59:05 UTC. -- 2023-11-16T18:22:15.098 INFO:journalctl@ceph.mds.b.smithi040.stdout:-- Logs begin at Thu 2023-11-16 17:59:26 UTC. -- 2023-11-16T18:22:15.099 DEBUG:teuthology.orchestra.run.smithi040:mds.h> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.h.service 2023-11-16T18:22:15.131 INFO:journalctl@ceph.mds.e.smithi040.stdout:-- Logs begin at Thu 2023-11-16 17:59:26 UTC. -- 2023-11-16T18:22:15.131 DEBUG:teuthology.orchestra.run.smithi040:mds.k> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.k.service 2023-11-16T18:22:15.161 INFO:journalctl@ceph.mds.h.smithi040.stdout:-- Logs begin at Thu 2023-11-16 17:59:26 UTC. -- 2023-11-16T18:22:15.162 DEBUG:teuthology.orchestra.run.smithi188:mds.c> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.c.service 2023-11-16T18:22:15.169 DEBUG:teuthology.orchestra.run.smithi188:mds.f> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.f.service 2023-11-16T18:22:15.189 INFO:journalctl@ceph.mds.k.smithi040.stdout:-- Logs begin at Thu 2023-11-16 17:59:26 UTC. -- 2023-11-16T18:22:15.193 INFO:journalctl@ceph.mds.c.smithi188.stdout:-- Logs begin at Thu 2023-11-16 17:59:32 UTC. -- 2023-11-16T18:22:15.195 DEBUG:teuthology.orchestra.run.smithi188:mds.i> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.i.service 2023-11-16T18:22:15.220 INFO:journalctl@ceph.mds.f.smithi188.stdout:-- Logs begin at Thu 2023-11-16 17:59:32 UTC. -- 2023-11-16T18:22:15.221 DEBUG:teuthology.orchestra.run.smithi188:mds.l> sudo journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.l.service 2023-11-16T18:22:15.248 INFO:journalctl@ceph.mds.i.smithi188.stdout:-- Logs begin at Thu 2023-11-16 17:59:32 UTC. -- 2023-11-16T18:22:15.250 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-11-16T18:22:15.251 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:22:15.275 INFO:journalctl@ceph.mds.l.smithi188.stdout:-- Logs begin at Thu 2023-11-16 17:59:32 UTC. -- 2023-11-16T18:22:15.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.382+0000 7f339512c700 1 -- 172.21.15.2:0/2397914515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33901003b0 msgr2=0x7f3390100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:15.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.382+0000 7f339512c700 1 --2- 172.21.15.2:0/2397914515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33901003b0 0x7f3390100800 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3384009990 tx=0x7f3384009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.382+0000 7f339512c700 1 -- 172.21.15.2:0/2397914515 shutdown_connections 2023-11-16T18:22:15.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.382+0000 7f339512c700 1 --2- 172.21.15.2:0/2397914515 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3390100d40 0x7f3390107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.382+0000 7f339512c700 1 --2- 172.21.15.2:0/2397914515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33900ff1c0 0x7f33900ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.382+0000 7f339512c700 1 --2- 172.21.15.2:0/2397914515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33901003b0 0x7f3390100800 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.382+0000 7f339512c700 1 -- 172.21.15.2:0/2397914515 >> 172.21.15.2:0/2397914515 conn(0x7f33900fa7a0 msgr2=0x7f33900fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:15.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.382+0000 7f339512c700 1 -- 172.21.15.2:0/2397914515 shutdown_connections 2023-11-16T18:22:15.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.383+0000 7f339512c700 1 -- 172.21.15.2:0/2397914515 wait complete. 2023-11-16T18:22:15.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.383+0000 7f339512c700 1 Processor -- start 2023-11-16T18:22:15.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.383+0000 7f339512c700 1 -- start start 2023-11-16T18:22:15.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f339512c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33900ff1c0 0x7f3390109940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:15.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f339512c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33901003b0 0x7f3390107f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:15.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f339512c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3390100d40 0x7f33901084c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:15.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f339512c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3390108c60 con 0x7f33900ff1c0 2023-11-16T18:22:15.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f339512c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3390108dd0 con 0x7f3390100d40 2023-11-16T18:22:15.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f339512c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33901a0020 con 0x7f33901003b0 2023-11-16T18:22:15.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f338ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33900ff1c0 0x7f3390109940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:15.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f338ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33900ff1c0 0x7f3390109940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47468/0 (socket says 172.21.15.2:47468) 2023-11-16T18:22:15.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f338ed9d700 1 -- 172.21.15.2:0/936100728 learned_addr learned my addr 172.21.15.2:0/936100728 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:15.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f338f59e700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3390100d40 0x7f33901084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:15.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f338e59c700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33901003b0 0x7f3390107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:15.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f338ed9d700 1 -- 172.21.15.2:0/936100728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33901003b0 msgr2=0x7f3390107f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:15.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f338ed9d700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33901003b0 0x7f3390107f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f338ed9d700 1 -- 172.21.15.2:0/936100728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3390100d40 msgr2=0x7f33901084c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:15.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f338ed9d700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3390100d40 0x7f33901084c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f338ed9d700 1 -- 172.21.15.2:0/936100728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33840095f0 con 0x7f33900ff1c0 2023-11-16T18:22:15.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f338f59e700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3390100d40 0x7f33901084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:22:15.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f338ed9d700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33900ff1c0 0x7f3390109940 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f338000d980 tx=0x7f338000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:15.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f3377fff700 1 -- 172.21.15.2:0/936100728 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338000de60 con 0x7f33900ff1c0 2023-11-16T18:22:15.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f339512c700 1 -- 172.21.15.2:0/936100728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339010bc60 con 0x7f33900ff1c0 2023-11-16T18:22:15.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f3377fff700 1 -- 172.21.15.2:0/936100728 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33800044e0 con 0x7f33900ff1c0 2023-11-16T18:22:15.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f3377fff700 1 -- 172.21.15.2:0/936100728 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380003be0 con 0x7f33900ff1c0 2023-11-16T18:22:15.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.384+0000 7f339512c700 1 -- 172.21.15.2:0/936100728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339010bad0 con 0x7f33900ff1c0 2023-11-16T18:22:15.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.385+0000 7f339512c700 1 -- 172.21.15.2:0/936100728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3390106050 con 0x7f33900ff1c0 2023-11-16T18:22:15.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.387+0000 7f3377fff700 1 -- 172.21.15.2:0/936100728 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f3380009bb0 con 0x7f33900ff1c0 2023-11-16T18:22:15.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.388+0000 7f3377fff700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f33780a1150 0x7f33780a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:15.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.388+0000 7f338e59c700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f33780a1150 0x7f33780a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:15.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.388+0000 7f3377fff700 1 -- 172.21.15.2:0/936100728 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f33800cc650 con 0x7f33900ff1c0 2023-11-16T18:22:15.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.388+0000 7f338e59c700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f33780a1150 0x7f33780a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f338400b5c0 tx=0x7f3384005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:15.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.391+0000 7f3377fff700 1 -- 172.21.15.2:0/936100728 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f33800961a0 con 0x7f33900ff1c0 2023-11-16T18:22:15.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.592+0000 7f339512c700 1 -- 172.21.15.2:0/936100728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f339002cfe0 con 0x7f33900ff1c0 2023-11-16T18:22:15.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.592+0000 7f3377fff700 1 -- 172.21.15.2:0/936100728 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18315 (secure 0 0 0) 0x7f3380009940 con 0x7f33900ff1c0 2023-11-16T18:22:15.597 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:15.598 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":78,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:22:03.153518+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:22:15.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.595+0000 7f339512c700 1 -- 172.21.15.2:0/936100728 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f33780a1150 msgr2=0x7f33780a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:15.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.595+0000 7f339512c700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f33780a1150 0x7f33780a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f338400b5c0 tx=0x7f3384005fb0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.595+0000 7f339512c700 1 -- 172.21.15.2:0/936100728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33900ff1c0 msgr2=0x7f3390109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:15.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.595+0000 7f339512c700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33900ff1c0 0x7f3390109940 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f338000d980 tx=0x7f338000dd40 comp rx=0 tx=0).stop 2023-11-16T18:22:15.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.595+0000 7f339512c700 1 -- 172.21.15.2:0/936100728 shutdown_connections 2023-11-16T18:22:15.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.595+0000 7f339512c700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f33780a1150 0x7f33780a3600 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.595+0000 7f339512c700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3390100d40 0x7f33901084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.595+0000 7f339512c700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33900ff1c0 0x7f3390109940 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.595+0000 7f339512c700 1 --2- 172.21.15.2:0/936100728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33901003b0 0x7f3390107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.595+0000 7f339512c700 1 -- 172.21.15.2:0/936100728 >> 172.21.15.2:0/936100728 conn(0x7f33900fa7a0 msgr2=0x7f33900fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:15.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.595+0000 7f339512c700 1 -- 172.21.15.2:0/936100728 shutdown_connections 2023-11-16T18:22:15.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.595+0000 7f339512c700 1 -- 172.21.15.2:0/936100728 wait complete. 2023-11-16T18:22:15.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:22:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2023-11-16T18:22:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:22:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:22:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:22:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:22:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:15.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:15.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:15.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:15 smithi002 ceph-mon[98807]: Deploying daemon mds.l on smithi188 2023-11-16T18:22:15.610 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-16T18:22:15.610 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T18:22:15.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.736+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/581291650 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f28105e50 msgr2=0x7f8f28108300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:15.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.736+0000 7f8f2fcfd700 1 --2- 172.21.15.2:0/581291650 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f28105e50 0x7f8f28108300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8f2000b0a0 tx=0x7f8f2000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.736+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/581291650 shutdown_connections 2023-11-16T18:22:15.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.736+0000 7f8f2fcfd700 1 --2- 172.21.15.2:0/581291650 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f28105e50 0x7f8f28108300 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.736+0000 7f8f2fcfd700 1 --2- 172.21.15.2:0/581291650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f28068b00 0x7f8f28102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.736+0000 7f8f2fcfd700 1 --2- 172.21.15.2:0/581291650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f28103340 0x7f8f281057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.736+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/581291650 >> 172.21.15.2:0/581291650 conn(0x7f8f280fa750 msgr2=0x7f8f280fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:15.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.736+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/581291650 shutdown_connections 2023-11-16T18:22:15.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.736+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/581291650 wait complete. 2023-11-16T18:22:15.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.736+0000 7f8f2fcfd700 1 Processor -- start 2023-11-16T18:22:15.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.736+0000 7f8f2fcfd700 1 -- start start 2023-11-16T18:22:15.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2fcfd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f28068b00 0x7f8f281960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:15.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2fcfd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f28103340 0x7f8f281965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:15.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2fcfd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f28105e50 0x7f8f2819d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:15.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2fcfd700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2819dc20 con 0x7f8f28105e50 2023-11-16T18:22:15.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2fcfd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2819dd60 con 0x7f8f28103340 2023-11-16T18:22:15.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2fcfd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2819def0 con 0x7f8f28068b00 2023-11-16T18:22:15.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2e29a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f28105e50 0x7f8f2819d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2e29a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f28105e50 0x7f8f2819d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47486/0 (socket says 172.21.15.2:47486) 2023-11-16T18:22:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2e29a700 1 -- 172.21.15.2:0/4099670792 learned_addr learned my addr 172.21.15.2:0/4099670792 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2da99700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f28068b00 0x7f8f281960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2d298700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f28103340 0x7f8f281965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2e29a700 1 -- 172.21.15.2:0/4099670792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f28068b00 msgr2=0x7f8f281960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2e29a700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f28068b00 0x7f8f281960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2e29a700 1 -- 172.21.15.2:0/4099670792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f28103340 msgr2=0x7f8f281965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2e29a700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f28103340 0x7f8f281965e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2e29a700 1 -- 172.21.15.2:0/4099670792 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f20009d00 con 0x7f8f28105e50 2023-11-16T18:22:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2da99700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f28068b00 0x7f8f281960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:22:15.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2e29a700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f28105e50 0x7f8f2819d650 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8f2000b780 tx=0x7f8f2000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:15.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f1effd700 1 -- 172.21.15.2:0/4099670792 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2000e040 con 0x7f8f28105e50 2023-11-16T18:22:15.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f1effd700 1 -- 172.21.15.2:0/4099670792 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f20004e60 con 0x7f8f28105e50 2023-11-16T18:22:15.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/4099670792 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f2810bbf0 con 0x7f8f28105e50 2023-11-16T18:22:15.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f1effd700 1 -- 172.21.15.2:0/4099670792 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f200053b0 con 0x7f8f28105e50 2023-11-16T18:22:15.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.737+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/4099670792 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f2810ba60 con 0x7f8f28105e50 2023-11-16T18:22:15.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.738+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/4099670792 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f280fc2c0 con 0x7f8f28105e50 2023-11-16T18:22:15.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.739+0000 7f8f1effd700 1 -- 172.21.15.2:0/4099670792 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f8f20005510 con 0x7f8f28105e50 2023-11-16T18:22:15.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.740+0000 7f8f1effd700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8f140a1150 0x7f8f140a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:15.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.740+0000 7f8f2da99700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8f140a1150 0x7f8f140a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:15.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.740+0000 7f8f1effd700 1 -- 172.21.15.2:0/4099670792 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8f200ce1b0 con 0x7f8f28105e50 2023-11-16T18:22:15.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.740+0000 7f8f2da99700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8f140a1150 0x7f8f140a3600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8f28068820 tx=0x7f8f18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:15.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.743+0000 7f8f1effd700 1 -- 172.21.15.2:0/4099670792 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8f20097d90 con 0x7f8f28105e50 2023-11-16T18:22:15.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:22:15.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2023-11-16T18:22:15.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:15.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:15.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:15.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:15.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:22:15.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:15.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:22:15.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:22:15.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:22:15.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:15.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:15.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:15.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:15 smithi188 ceph-mon[100639]: Deploying daemon mds.l on smithi188 2023-11-16T18:22:15.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.942+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/4099670792 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8f28067a50 con 0x7f8f28105e50 2023-11-16T18:22:15.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.942+0000 7f8f1effd700 1 -- 172.21.15.2:0/4099670792 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8f2009b3b0 con 0x7f8f28105e50 2023-11-16T18:22:15.946 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:22:15.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.944+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/4099670792 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8f140a1150 msgr2=0x7f8f140a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:15.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.944+0000 7f8f2fcfd700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8f140a1150 0x7f8f140a3600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8f28068820 tx=0x7f8f18009260 comp rx=0 tx=0).stop 2023-11-16T18:22:15.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.944+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/4099670792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f28105e50 msgr2=0x7f8f2819d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:15.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.944+0000 7f8f2fcfd700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f28105e50 0x7f8f2819d650 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8f2000b780 tx=0x7f8f2000be10 comp rx=0 tx=0).stop 2023-11-16T18:22:15.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.944+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/4099670792 shutdown_connections 2023-11-16T18:22:15.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.944+0000 7f8f2fcfd700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8f140a1150 0x7f8f140a3600 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.944+0000 7f8f2fcfd700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f28068b00 0x7f8f281960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.944+0000 7f8f2fcfd700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f28105e50 0x7f8f2819d650 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.944+0000 7f8f2fcfd700 1 --2- 172.21.15.2:0/4099670792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f28103340 0x7f8f281965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:15.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.944+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/4099670792 >> 172.21.15.2:0/4099670792 conn(0x7f8f280fa750 msgr2=0x7f8f28101680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:15.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.944+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/4099670792 shutdown_connections 2023-11-16T18:22:15.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:15.944+0000 7f8f2fcfd700 1 -- 172.21.15.2:0/4099670792 wait complete. 2023-11-16T18:22:15.959 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:15.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:22:15.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2023-11-16T18:22:15.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:15.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:15.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:15.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:15.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:22:15.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:15.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:22:15.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:22:15.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:22:15.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:15.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:15.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:15.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:15 smithi040 ceph-mon[100251]: Deploying daemon mds.l on smithi188 2023-11-16T18:22:16.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.085+0000 7fbf5ce78700 1 -- 172.21.15.2:0/1288543991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf580ff1c0 msgr2=0x7fbf580ff5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:16.118 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.085+0000 7fbf5ce78700 1 --2- 172.21.15.2:0/1288543991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf580ff1c0 0x7fbf580ff5d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009990 tx=0x7fbf48009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:16.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.085+0000 7fbf5ce78700 1 -- 172.21.15.2:0/1288543991 shutdown_connections 2023-11-16T18:22:16.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.085+0000 7fbf5ce78700 1 --2- 172.21.15.2:0/1288543991 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf58100d40 0x7fbf58107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:16.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.085+0000 7fbf5ce78700 1 --2- 172.21.15.2:0/1288543991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf580ff1c0 0x7fbf580ff5d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:16.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.085+0000 7fbf5ce78700 1 --2- 172.21.15.2:0/1288543991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf581003b0 0x7fbf58100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:16.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.085+0000 7fbf5ce78700 1 -- 172.21.15.2:0/1288543991 >> 172.21.15.2:0/1288543991 conn(0x7fbf580fa7c0 msgr2=0x7fbf580fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:16.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.085+0000 7fbf5ce78700 1 -- 172.21.15.2:0/1288543991 shutdown_connections 2023-11-16T18:22:16.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.085+0000 7fbf5ce78700 1 -- 172.21.15.2:0/1288543991 wait complete. 2023-11-16T18:22:16.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.086+0000 7fbf5ce78700 1 Processor -- start 2023-11-16T18:22:16.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.086+0000 7fbf5ce78700 1 -- start start 2023-11-16T18:22:16.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.086+0000 7fbf5ce78700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf580ff1c0 0x7fbf58195160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:16.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.086+0000 7fbf5ce78700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf581003b0 0x7fbf581956a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:16.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.086+0000 7fbf5ce78700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf58100d40 0x7fbf5819c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:16.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.086+0000 7fbf5ce78700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5819cce0 con 0x7fbf580ff1c0 2023-11-16T18:22:16.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.086+0000 7fbf5ce78700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5819ce20 con 0x7fbf581003b0 2023-11-16T18:22:16.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.086+0000 7fbf5ce78700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5819cfb0 con 0x7fbf58100d40 2023-11-16T18:22:16.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.086+0000 7fbf5659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf580ff1c0 0x7fbf58195160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:16.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.086+0000 7fbf5659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf580ff1c0 0x7fbf58195160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47496/0 (socket says 172.21.15.2:47496) 2023-11-16T18:22:16.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.086+0000 7fbf5659c700 1 -- 172.21.15.2:0/414017118 learned_addr learned my addr 172.21.15.2:0/414017118 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:16.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.086+0000 7fbf55d9b700 1 --2- 172.21.15.2:0/414017118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf581003b0 0x7fbf581956a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:16.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.087+0000 7fbf5659c700 1 -- 172.21.15.2:0/414017118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf58100d40 msgr2=0x7fbf5819c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:16.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.087+0000 7fbf5659c700 1 --2- 172.21.15.2:0/414017118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf58100d40 0x7fbf5819c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:16.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.087+0000 7fbf5659c700 1 -- 172.21.15.2:0/414017118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf581003b0 msgr2=0x7fbf581956a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:16.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.087+0000 7fbf5659c700 1 --2- 172.21.15.2:0/414017118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf581003b0 0x7fbf581956a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:16.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.087+0000 7fbf5659c700 1 -- 172.21.15.2:0/414017118 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480095f0 con 0x7fbf580ff1c0 2023-11-16T18:22:16.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.087+0000 7fbf5659c700 1 --2- 172.21.15.2:0/414017118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf580ff1c0 0x7fbf58195160 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbf48004760 tx=0x7fbf48004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:16.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.087+0000 7fbf4f7fe700 1 -- 172.21.15.2:0/414017118 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4801d070 con 0x7fbf580ff1c0 2023-11-16T18:22:16.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.087+0000 7fbf4f7fe700 1 -- 172.21.15.2:0/414017118 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf480041a0 con 0x7fbf580ff1c0 2023-11-16T18:22:16.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.087+0000 7fbf5ce78700 1 -- 172.21.15.2:0/414017118 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5810ac70 con 0x7fbf580ff1c0 2023-11-16T18:22:16.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.087+0000 7fbf4f7fe700 1 -- 172.21.15.2:0/414017118 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48005310 con 0x7fbf580ff1c0 2023-11-16T18:22:16.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.087+0000 7fbf5ce78700 1 -- 172.21.15.2:0/414017118 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5810aae0 con 0x7fbf580ff1c0 2023-11-16T18:22:16.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.088+0000 7fbf5ce78700 1 -- 172.21.15.2:0/414017118 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5819d630 con 0x7fbf580ff1c0 2023-11-16T18:22:16.133 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.090+0000 7fbf4f7fe700 1 -- 172.21.15.2:0/414017118 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fbf480049c0 con 0x7fbf580ff1c0 2023-11-16T18:22:16.133 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.090+0000 7fbf4f7fe700 1 --2- 172.21.15.2:0/414017118 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbf440a11d0 0x7fbf440a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:16.133 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.091+0000 7fbf55d9b700 1 --2- 172.21.15.2:0/414017118 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbf440a11d0 0x7fbf440a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:16.133 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.091+0000 7fbf4f7fe700 1 -- 172.21.15.2:0/414017118 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbf480cdc40 con 0x7fbf580ff1c0 2023-11-16T18:22:16.133 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.091+0000 7fbf55d9b700 1 --2- 172.21.15.2:0/414017118 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbf440a11d0 0x7fbf440a3680 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009a10 tx=0x7fbf40009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:16.133 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.094+0000 7fbf4f7fe700 1 -- 172.21.15.2:0/414017118 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbf48097820 con 0x7fbf580ff1c0 2023-11-16T18:22:16.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.337+0000 7fbf5ce78700 1 -- 172.21.15.2:0/414017118 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf5819dd10 con 0x7fbf580ff1c0 2023-11-16T18:22:16.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.337+0000 7fbf4f7fe700 1 -- 172.21.15.2:0/414017118 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7fbf4809ae40 con 0x7fbf580ff1c0 2023-11-16T18:22:16.340 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:16.340 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-11-16T18:22:16.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.339+0000 7fbf5ce78700 1 -- 172.21.15.2:0/414017118 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbf440a11d0 msgr2=0x7fbf440a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:16.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.339+0000 7fbf5ce78700 1 --2- 172.21.15.2:0/414017118 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbf440a11d0 0x7fbf440a3680 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009a10 tx=0x7fbf40009190 comp rx=0 tx=0).stop 2023-11-16T18:22:16.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.339+0000 7fbf5ce78700 1 -- 172.21.15.2:0/414017118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf580ff1c0 msgr2=0x7fbf58195160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:16.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.339+0000 7fbf5ce78700 1 --2- 172.21.15.2:0/414017118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf580ff1c0 0x7fbf58195160 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbf48004760 tx=0x7fbf48004790 comp rx=0 tx=0).stop 2023-11-16T18:22:16.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.339+0000 7fbf5ce78700 1 -- 172.21.15.2:0/414017118 shutdown_connections 2023-11-16T18:22:16.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.339+0000 7fbf5ce78700 1 --2- 172.21.15.2:0/414017118 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbf440a11d0 0x7fbf440a3680 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:16.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.339+0000 7fbf5ce78700 1 --2- 172.21.15.2:0/414017118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf58100d40 0x7fbf5819c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:16.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.339+0000 7fbf5ce78700 1 --2- 172.21.15.2:0/414017118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf580ff1c0 0x7fbf58195160 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:16.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.339+0000 7fbf5ce78700 1 --2- 172.21.15.2:0/414017118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf581003b0 0x7fbf581956a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:16.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.339+0000 7fbf5ce78700 1 -- 172.21.15.2:0/414017118 >> 172.21.15.2:0/414017118 conn(0x7fbf580fa7c0 msgr2=0x7fbf580fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:16.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.340+0000 7fbf5ce78700 1 -- 172.21.15.2:0/414017118 shutdown_connections 2023-11-16T18:22:16.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:16.340+0000 7fbf5ce78700 1 -- 172.21.15.2:0/414017118 wait complete. 2023-11-16T18:22:16.343 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 1 2023-11-16T18:22:16.450 INFO:journalctl@ceph.mds.l.smithi188.stdout:Nov 16 18:22:16 smithi188 systemd[1]: Starting Ceph mds.l for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:22:16.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:16 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/936100728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:16.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:16 smithi002 ceph-mon[98807]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:16.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:16 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/4099670792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:22:16.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:16 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/414017118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:16.705 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:16 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/936100728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:16.705 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:16 smithi188 ceph-mon[100639]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:16.706 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:16 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/4099670792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:22:16.706 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:16 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/414017118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:16.706 INFO:journalctl@ceph.mds.l.smithi188.stdout:Nov 16 18:22:16 smithi188 podman[125769]: 2023-11-16T18:22:16.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:16 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/936100728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:16.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:16 smithi040 ceph-mon[100251]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:16.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:16 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/4099670792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:22:16.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:16 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/414017118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:17.046 INFO:journalctl@ceph.mds.l.smithi188.stdout:Nov 16 18:22:16 smithi188 ceph-mds[125809]: -- 172.21.15.188:0/1766998951 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d90a3b3520 con 0x55d90a388c00 2023-11-16T18:22:17.046 INFO:journalctl@ceph.mds.l.smithi188.stdout:Nov 16 18:22:16 smithi188 bash[125769]: c8f325dfda19d1782bbcd2a94eba617c34d2c5d86ec02488da66533fd778e4e4 2023-11-16T18:22:17.047 INFO:journalctl@ceph.mds.l.smithi188.stdout:Nov 16 18:22:16 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-l[125785]: starting mds.l at 2023-11-16T18:22:17.047 INFO:journalctl@ceph.mds.l.smithi188.stdout:Nov 16 18:22:16 smithi188 systemd[1]: Started Ceph mds.l for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:22:17.935 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:17 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.188:6832/381028209,v1:172.21.15.188:6833/381028209] up:boot 2023-11-16T18:22:17.935 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:17 smithi040 ceph-mon[100251]: fsmap 1 up:standby 2023-11-16T18:22:17.935 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T18:22:17.935 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:17.935 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:17.935 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:17.935 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:17.936 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:17.936 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:17 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:18.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:17 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.188:6832/381028209,v1:172.21.15.188:6833/381028209] up:boot 2023-11-16T18:22:18.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:17 smithi188 ceph-mon[100639]: fsmap 1 up:standby 2023-11-16T18:22:18.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T18:22:18.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:18.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:18.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:18.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:18.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:18.057 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:17 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:18.123 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:17 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.188:6832/381028209,v1:172.21.15.188:6833/381028209] up:boot 2023-11-16T18:22:18.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:17 smithi002 ceph-mon[98807]: fsmap 1 up:standby 2023-11-16T18:22:18.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T18:22:18.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:18.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:18.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:18.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:18.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:18.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:17 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:18.357 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:18.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.494+0000 7f549a2b5700 1 -- 172.21.15.2:0/185742583 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54940ff1c0 msgr2=0x7f54940ff5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:18.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.494+0000 7f549a2b5700 1 --2- 172.21.15.2:0/185742583 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54940ff1c0 0x7f54940ff5d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5484009990 tx=0x7f5484009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:18.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.495+0000 7f549a2b5700 1 -- 172.21.15.2:0/185742583 shutdown_connections 2023-11-16T18:22:18.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.495+0000 7f549a2b5700 1 --2- 172.21.15.2:0/185742583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5494100d40 0x7f5494107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:18.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.495+0000 7f549a2b5700 1 --2- 172.21.15.2:0/185742583 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54940ff1c0 0x7f54940ff5d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:18.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.495+0000 7f549a2b5700 1 --2- 172.21.15.2:0/185742583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54941003b0 0x7f5494100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:18.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.495+0000 7f549a2b5700 1 -- 172.21.15.2:0/185742583 >> 172.21.15.2:0/185742583 conn(0x7f54940fa7c0 msgr2=0x7f54940fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:18.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.495+0000 7f549a2b5700 1 -- 172.21.15.2:0/185742583 shutdown_connections 2023-11-16T18:22:18.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.495+0000 7f549a2b5700 1 -- 172.21.15.2:0/185742583 wait complete. 2023-11-16T18:22:18.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.495+0000 7f549a2b5700 1 Processor -- start 2023-11-16T18:22:18.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.495+0000 7f549a2b5700 1 -- start start 2023-11-16T18:22:18.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f549a2b5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54940ff1c0 0x7f5494198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:18.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f549a2b5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54941003b0 0x7f5494198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:18.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f549a2b5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5494100d40 0x7f549419f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:18.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f549a2b5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549419fe80 con 0x7f54940ff1c0 2023-11-16T18:22:18.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f549a2b5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549419ffc0 con 0x7f54941003b0 2023-11-16T18:22:18.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f549a2b5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54941a0150 con 0x7f5494100d40 2023-11-16T18:22:18.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f5493fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54940ff1c0 0x7f5494198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:18.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f5493fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54940ff1c0 0x7f5494198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47510/0 (socket says 172.21.15.2:47510) 2023-11-16T18:22:18.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f5493fff700 1 -- 172.21.15.2:0/3193390170 learned_addr learned my addr 172.21.15.2:0/3193390170 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:18.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f54937fe700 1 --2- 172.21.15.2:0/3193390170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54941003b0 0x7f5494198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:18.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f5493fff700 1 -- 172.21.15.2:0/3193390170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5494100d40 msgr2=0x7f549419f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:18.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f5493fff700 1 --2- 172.21.15.2:0/3193390170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5494100d40 0x7f549419f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:18.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f5493fff700 1 -- 172.21.15.2:0/3193390170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54941003b0 msgr2=0x7f5494198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:18.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f5493fff700 1 --2- 172.21.15.2:0/3193390170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54941003b0 0x7f5494198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:18.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f5493fff700 1 -- 172.21.15.2:0/3193390170 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54840095f0 con 0x7f54940ff1c0 2023-11-16T18:22:18.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f5493fff700 1 --2- 172.21.15.2:0/3193390170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54940ff1c0 0x7f5494198300 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5484006010 tx=0x7f5484004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:18.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f54917fa700 1 -- 172.21.15.2:0/3193390170 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548401d070 con 0x7f54940ff1c0 2023-11-16T18:22:18.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f54917fa700 1 -- 172.21.15.2:0/3193390170 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5484003fe0 con 0x7f54940ff1c0 2023-11-16T18:22:18.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f549a2b5700 1 -- 172.21.15.2:0/3193390170 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f549410aba0 con 0x7f54940ff1c0 2023-11-16T18:22:18.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f54917fa700 1 -- 172.21.15.2:0/3193390170 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5484022c10 con 0x7f54940ff1c0 2023-11-16T18:22:18.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.496+0000 7f549a2b5700 1 -- 172.21.15.2:0/3193390170 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549410aa10 con 0x7f54940ff1c0 2023-11-16T18:22:18.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.497+0000 7f549a2b5700 1 -- 172.21.15.2:0/3193390170 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54941a0770 con 0x7f54940ff1c0 2023-11-16T18:22:18.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.498+0000 7f54917fa700 1 -- 172.21.15.2:0/3193390170 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f5484022470 con 0x7f54940ff1c0 2023-11-16T18:22:18.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.499+0000 7f54917fa700 1 --2- 172.21.15.2:0/3193390170 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f547c0a1150 0x7f547c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:18.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.499+0000 7f54917fa700 1 -- 172.21.15.2:0/3193390170 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f54840ce1b0 con 0x7f54940ff1c0 2023-11-16T18:22:18.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.499+0000 7f54937fe700 1 --2- 172.21.15.2:0/3193390170 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f547c0a1150 0x7f547c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:18.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.500+0000 7f54937fe700 1 --2- 172.21.15.2:0/3193390170 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f547c0a1150 0x7f547c0a3600 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5488005950 tx=0x7f548800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:18.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.502+0000 7f54917fa700 1 -- 172.21.15.2:0/3193390170 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5484097d90 con 0x7f54940ff1c0 2023-11-16T18:22:18.646 INFO:journalctl@ceph.mds.k.smithi040.stdout:Nov 16 18:22:18 smithi040 systemd[1]: Starting Ceph mds.k for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:22:18.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:18 smithi002 ceph-mon[98807]: Deploying daemon mds.k on smithi040 2023-11-16T18:22:18.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:18 smithi002 ceph-mon[98807]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:18.755 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.750+0000 7f549a2b5700 1 -- 172.21.15.2:0/3193390170 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54941a0d20 con 0x7f54940ff1c0 2023-11-16T18:22:18.755 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.750+0000 7f54917fa700 1 -- 172.21.15.2:0/3193390170 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1257 (secure 0 0 0) 0x7f548409b3b0 con 0x7f54940ff1c0 2023-11-16T18:22:18.755 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:18.755 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-11-16T18:22:18.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.752+0000 7f549a2b5700 1 -- 172.21.15.2:0/3193390170 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f547c0a1150 msgr2=0x7f547c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:18.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.752+0000 7f549a2b5700 1 --2- 172.21.15.2:0/3193390170 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f547c0a1150 0x7f547c0a3600 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5488005950 tx=0x7f548800b410 comp rx=0 tx=0).stop 2023-11-16T18:22:18.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.752+0000 7f549a2b5700 1 -- 172.21.15.2:0/3193390170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54940ff1c0 msgr2=0x7f5494198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:18.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.752+0000 7f549a2b5700 1 --2- 172.21.15.2:0/3193390170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54940ff1c0 0x7f5494198300 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5484006010 tx=0x7f5484004730 comp rx=0 tx=0).stop 2023-11-16T18:22:18.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.753+0000 7f549a2b5700 1 -- 172.21.15.2:0/3193390170 shutdown_connections 2023-11-16T18:22:18.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.753+0000 7f549a2b5700 1 --2- 172.21.15.2:0/3193390170 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f547c0a1150 0x7f547c0a3600 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:18.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.753+0000 7f549a2b5700 1 --2- 172.21.15.2:0/3193390170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5494100d40 0x7f549419f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:18.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.753+0000 7f549a2b5700 1 --2- 172.21.15.2:0/3193390170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54940ff1c0 0x7f5494198300 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:18.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.753+0000 7f549a2b5700 1 --2- 172.21.15.2:0/3193390170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54941003b0 0x7f5494198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:18.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.753+0000 7f549a2b5700 1 -- 172.21.15.2:0/3193390170 >> 172.21.15.2:0/3193390170 conn(0x7f54940fa7c0 msgr2=0x7f5494067e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:18.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.753+0000 7f549a2b5700 1 -- 172.21.15.2:0/3193390170 shutdown_connections 2023-11-16T18:22:18.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:18.753+0000 7f549a2b5700 1 -- 172.21.15.2:0/3193390170 wait complete. 2023-11-16T18:22:18.757 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 2 2023-11-16T18:22:18.913 INFO:journalctl@ceph.mds.k.smithi040.stdout:Nov 16 18:22:18 smithi040 podman[124432]: 2023-11-16T18:22:18.914 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:18 smithi040 ceph-mon[100251]: Deploying daemon mds.k on smithi040 2023-11-16T18:22:18.914 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:18 smithi040 ceph-mon[100251]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:19.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:18 smithi188 ceph-mon[100639]: Deploying daemon mds.k on smithi040 2023-11-16T18:22:19.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:18 smithi188 ceph-mon[100639]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:19.233 INFO:journalctl@ceph.mds.k.smithi040.stdout:Nov 16 18:22:18 smithi040 ceph-mds[124472]: -- 172.21.15.40:0/1142478155 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555d178cf520 con 0x555d178a5000 2023-11-16T18:22:19.233 INFO:journalctl@ceph.mds.k.smithi040.stdout:Nov 16 18:22:18 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-k[124448]: starting mds.k at 2023-11-16T18:22:19.233 INFO:journalctl@ceph.mds.k.smithi040.stdout:Nov 16 18:22:18 smithi040 bash[124432]: 965126a20bd66956c3ee1a8b0082ad11297039f4e3836f12cb887aa22c3d5a69 2023-11-16T18:22:19.233 INFO:journalctl@ceph.mds.k.smithi040.stdout:Nov 16 18:22:18 smithi040 systemd[1]: Started Ceph mds.k for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:22:20.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:19 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3193390170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:20.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:19 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:20.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:19 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:20.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:19 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:20.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:19 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:20.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:19 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:20.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:19 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:20.048 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:19 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3193390170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:20.048 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:19 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:20.048 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:19 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:20.048 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:19 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:20.048 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:19 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:20.048 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:19 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:20.049 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:19 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:19 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3193390170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:19 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:19 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:19 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:20.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:19 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:20.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:19 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:20.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:19 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:20.558 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 18:22:20 smithi002 systemd[1]: Starting Ceph mds.j for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:22:20.773 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:20.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.912+0000 7fdb8e85b700 1 -- 172.21.15.2:0/1494642754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb88101e70 msgr2=0x7fdb88108730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:20.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.912+0000 7fdb8e85b700 1 --2- 172.21.15.2:0/1494642754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb88101e70 0x7fdb88108730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000b0a0 tx=0x7fdb8000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:20.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.912+0000 7fdb8e85b700 1 -- 172.21.15.2:0/1494642754 shutdown_connections 2023-11-16T18:22:20.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.912+0000 7fdb8e85b700 1 --2- 172.21.15.2:0/1494642754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb88101e70 0x7fdb88108730 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:20.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.912+0000 7fdb8e85b700 1 --2- 172.21.15.2:0/1494642754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb881002f0 0x7fdb88100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:20.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.912+0000 7fdb8e85b700 1 --2- 172.21.15.2:0/1494642754 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb881014e0 0x7fdb88101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:20.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.912+0000 7fdb8e85b700 1 -- 172.21.15.2:0/1494642754 >> 172.21.15.2:0/1494642754 conn(0x7fdb880fb8d0 msgr2=0x7fdb880fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:20.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.912+0000 7fdb8e85b700 1 -- 172.21.15.2:0/1494642754 shutdown_connections 2023-11-16T18:22:20.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.912+0000 7fdb8e85b700 1 -- 172.21.15.2:0/1494642754 wait complete. 2023-11-16T18:22:20.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.913+0000 7fdb8e85b700 1 Processor -- start 2023-11-16T18:22:20.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.913+0000 7fdb8e85b700 1 -- start start 2023-11-16T18:22:20.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.913+0000 7fdb8e85b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb881002f0 0x7fdb880748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:20.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.913+0000 7fdb8e85b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb881014e0 0x7fdb88072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:20.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.913+0000 7fdb8e85b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb88101e70 0x7fdb88073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:20.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.913+0000 7fdb8e85b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb88073bc0 con 0x7fdb88101e70 2023-11-16T18:22:20.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.913+0000 7fdb8e85b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb88073d30 con 0x7fdb881014e0 2023-11-16T18:22:20.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.913+0000 7fdb8e85b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb881a0140 con 0x7fdb881002f0 2023-11-16T18:22:20.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.913+0000 7fdb8cdf8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb88101e70 0x7fdb88073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:20.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.913+0000 7fdb8cdf8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb88101e70 0x7fdb88073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47526/0 (socket says 172.21.15.2:47526) 2023-11-16T18:22:20.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.913+0000 7fdb8cdf8700 1 -- 172.21.15.2:0/839630136 learned_addr learned my addr 172.21.15.2:0/839630136 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:20.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.913+0000 7fdb87fff700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb881002f0 0x7fdb880748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:20.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.913+0000 7fdb877fe700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb881014e0 0x7fdb88072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:20.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.914+0000 7fdb8cdf8700 1 -- 172.21.15.2:0/839630136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb881002f0 msgr2=0x7fdb880748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:20.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.914+0000 7fdb8cdf8700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb881002f0 0x7fdb880748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:20.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.914+0000 7fdb8cdf8700 1 -- 172.21.15.2:0/839630136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb881014e0 msgr2=0x7fdb88072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:20.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.914+0000 7fdb8cdf8700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb881014e0 0x7fdb88072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:20.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.914+0000 7fdb8cdf8700 1 -- 172.21.15.2:0/839630136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb80009d00 con 0x7fdb88101e70 2023-11-16T18:22:20.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.914+0000 7fdb8cdf8700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb88101e70 0x7fdb88073420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000b830 tx=0x7fdb80004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:20.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.914+0000 7fdb857fa700 1 -- 172.21.15.2:0/839630136 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8000e040 con 0x7fdb88101e70 2023-11-16T18:22:20.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.914+0000 7fdb857fa700 1 -- 172.21.15.2:0/839630136 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb8001db50 con 0x7fdb88101e70 2023-11-16T18:22:20.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.914+0000 7fdb8e85b700 1 -- 172.21.15.2:0/839630136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8810bc70 con 0x7fdb88101e70 2023-11-16T18:22:20.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.914+0000 7fdb857fa700 1 -- 172.21.15.2:0/839630136 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8001bb20 con 0x7fdb88101e70 2023-11-16T18:22:20.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.914+0000 7fdb8e85b700 1 -- 172.21.15.2:0/839630136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8810bae0 con 0x7fdb88101e70 2023-11-16T18:22:20.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.915+0000 7fdb8e85b700 1 -- 172.21.15.2:0/839630136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb88107740 con 0x7fdb88101e70 2023-11-16T18:22:20.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.916+0000 7fdb857fa700 1 -- 172.21.15.2:0/839630136 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fdb8001dcc0 con 0x7fdb88101e70 2023-11-16T18:22:20.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.917+0000 7fdb857fa700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdb700a1150 0x7fdb700a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:20.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.917+0000 7fdb87fff700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdb700a1150 0x7fdb700a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:20.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.917+0000 7fdb857fa700 1 -- 172.21.15.2:0/839630136 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdb800ce210 con 0x7fdb88101e70 2023-11-16T18:22:20.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.917+0000 7fdb87fff700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdb700a1150 0x7fdb700a3600 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdb88101340 tx=0x7fdb78007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:20.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:20.921+0000 7fdb857fa700 1 -- 172.21.15.2:0/839630136 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdb80097e90 con 0x7fdb88101e70 2023-11-16T18:22:21.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:20 smithi188 ceph-mon[100639]: Deploying daemon mds.j on smithi002 2023-11-16T18:22:21.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:20 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.40:6832/1110523894,v1:172.21.15.40:6833/1110523894] up:boot 2023-11-16T18:22:21.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:20 smithi188 ceph-mon[100639]: fsmap 2 up:standby 2023-11-16T18:22:21.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:20 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T18:22:21.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:20 smithi188 ceph-mon[100639]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:21.106 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 18:22:20 smithi002 podman[131941]: 2023-11-16T18:22:21.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:20 smithi002 ceph-mon[98807]: Deploying daemon mds.j on smithi002 2023-11-16T18:22:21.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:20 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.40:6832/1110523894,v1:172.21.15.40:6833/1110523894] up:boot 2023-11-16T18:22:21.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:20 smithi002 ceph-mon[98807]: fsmap 2 up:standby 2023-11-16T18:22:21.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:20 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T18:22:21.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:20 smithi002 ceph-mon[98807]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:21.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.195+0000 7fdb8e85b700 1 -- 172.21.15.2:0/839630136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb8802cfe0 con 0x7fdb88101e70 2023-11-16T18:22:21.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.195+0000 7fdb857fa700 1 -- 172.21.15.2:0/839630136 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2014 (secure 0 0 0) 0x7fdb8009b4b0 con 0x7fdb88101e70 2023-11-16T18:22:21.197 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:21.197 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-11-16T18:22:21.199 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.197+0000 7fdb8e85b700 1 -- 172.21.15.2:0/839630136 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdb700a1150 msgr2=0x7fdb700a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:21.199 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.197+0000 7fdb8e85b700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdb700a1150 0x7fdb700a3600 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdb88101340 tx=0x7fdb78007400 comp rx=0 tx=0).stop 2023-11-16T18:22:21.199 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.197+0000 7fdb8e85b700 1 -- 172.21.15.2:0/839630136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb88101e70 msgr2=0x7fdb88073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:21.199 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.197+0000 7fdb8e85b700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb88101e70 0x7fdb88073420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000b830 tx=0x7fdb80004c80 comp rx=0 tx=0).stop 2023-11-16T18:22:21.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.197+0000 7fdb8e85b700 1 -- 172.21.15.2:0/839630136 shutdown_connections 2023-11-16T18:22:21.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.197+0000 7fdb8e85b700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdb700a1150 0x7fdb700a3600 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:21.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.197+0000 7fdb8e85b700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb881002f0 0x7fdb880748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:21.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.197+0000 7fdb8e85b700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb88101e70 0x7fdb88073420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:21.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.197+0000 7fdb8e85b700 1 --2- 172.21.15.2:0/839630136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb881014e0 0x7fdb88072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:21.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.197+0000 7fdb8e85b700 1 -- 172.21.15.2:0/839630136 >> 172.21.15.2:0/839630136 conn(0x7fdb880fb8d0 msgr2=0x7fdb88104880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:21.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.197+0000 7fdb8e85b700 1 -- 172.21.15.2:0/839630136 shutdown_connections 2023-11-16T18:22:21.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:21.197+0000 7fdb8e85b700 1 -- 172.21.15.2:0/839630136 wait complete. 2023-11-16T18:22:21.201 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 3 2023-11-16T18:22:21.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:20 smithi040 ceph-mon[100251]: Deploying daemon mds.j on smithi002 2023-11-16T18:22:21.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:20 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.40:6832/1110523894,v1:172.21.15.40:6833/1110523894] up:boot 2023-11-16T18:22:21.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:20 smithi040 ceph-mon[100251]: fsmap 2 up:standby 2023-11-16T18:22:21.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:20 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T18:22:21.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:20 smithi040 ceph-mon[100251]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:21.605 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 18:22:21 smithi002 bash[131941]: 243d11b97b23ab68144928a498a25091ea2add1683b7a4638bc88b1fc85f021c 2023-11-16T18:22:21.606 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 18:22:21 smithi002 ceph-mds[132024]: -- 172.21.15.2:0/1546924130 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5631b0693520 con 0x5631b0668c00 2023-11-16T18:22:21.606 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 18:22:21 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-j[131991]: starting mds.j at 2023-11-16T18:22:21.606 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 18:22:21 smithi002 systemd[1]: Started Ceph mds.j for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:22:22.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:22 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:22.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:22 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:22.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:22 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/839630136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:22.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:22 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:22.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:22 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:22.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:22 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:22.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:22 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:22.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:22 smithi002 ceph-mon[98807]: Deploying daemon mds.d on smithi002 2023-11-16T18:22:22.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:22 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.2:6834/223055355,v1:172.21.15.2:6835/223055355] up:boot 2023-11-16T18:22:22.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:22 smithi002 ceph-mon[98807]: fsmap 3 up:standby 2023-11-16T18:22:22.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:22 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T18:22:22.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:22 smithi002 ceph-mon[98807]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:22 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:22 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:22 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/839630136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:22 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:22.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:22 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:22.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:22 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:22.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:22 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:22.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:22 smithi040 ceph-mon[100251]: Deploying daemon mds.d on smithi002 2023-11-16T18:22:22.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:22 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.2:6834/223055355,v1:172.21.15.2:6835/223055355] up:boot 2023-11-16T18:22:22.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:22 smithi040 ceph-mon[100251]: fsmap 3 up:standby 2023-11-16T18:22:22.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:22 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T18:22:22.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:22 smithi040 ceph-mon[100251]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:22.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:22 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:22 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:22 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/839630136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:22 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:22 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:22 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:22 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:22 smithi188 ceph-mon[100639]: Deploying daemon mds.d on smithi002 2023-11-16T18:22:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:22 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.2:6834/223055355,v1:172.21.15.2:6835/223055355] up:boot 2023-11-16T18:22:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:22 smithi188 ceph-mon[100639]: fsmap 3 up:standby 2023-11-16T18:22:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:22 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T18:22:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:22 smithi188 ceph-mon[100639]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:23.109 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 18:22:22 smithi002 systemd[1]: Starting Ceph mds.d for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:22:23.218 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:23.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.348+0000 7f1dc31a8700 1 -- 172.21.15.2:0/707887966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1dbc100d40 msgr2=0x7f1dbc107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:23.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.348+0000 7f1dc31a8700 1 --2- 172.21.15.2:0/707887966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1dbc100d40 0x7f1dbc107600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1db400b0a0 tx=0x7f1db400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:23.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.348+0000 7f1dc31a8700 1 -- 172.21.15.2:0/707887966 shutdown_connections 2023-11-16T18:22:23.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.348+0000 7f1dc31a8700 1 --2- 172.21.15.2:0/707887966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1dbc100d40 0x7f1dbc107600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:23.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.348+0000 7f1dc31a8700 1 --2- 172.21.15.2:0/707887966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc0ff1c0 0x7f1dbc0ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:23.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.348+0000 7f1dc31a8700 1 --2- 172.21.15.2:0/707887966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1dbc1003b0 0x7f1dbc100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:23.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.348+0000 7f1dc31a8700 1 -- 172.21.15.2:0/707887966 >> 172.21.15.2:0/707887966 conn(0x7f1dbc0fa7c0 msgr2=0x7f1dbc0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:23.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.348+0000 7f1dc31a8700 1 -- 172.21.15.2:0/707887966 shutdown_connections 2023-11-16T18:22:23.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.348+0000 7f1dc31a8700 1 -- 172.21.15.2:0/707887966 wait complete. 2023-11-16T18:22:23.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.348+0000 7f1dc31a8700 1 Processor -- start 2023-11-16T18:22:23.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.348+0000 7f1dc31a8700 1 -- start start 2023-11-16T18:22:23.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc31a8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1dbc0ff1c0 0x7f1dbc194fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:23.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc31a8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc1003b0 0x7f1dbc1954e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:23.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc31a8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1dbc100d40 0x7f1dbc19c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:23.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc31a8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc19cb20 con 0x7f1dbc100d40 2023-11-16T18:22:23.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc31a8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc19cc60 con 0x7f1dbc1003b0 2023-11-16T18:22:23.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc31a8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc19cdf0 con 0x7f1dbc0ff1c0 2023-11-16T18:22:23.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc1745700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1dbc100d40 0x7f1dbc19c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:23.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc1745700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1dbc100d40 0x7f1dbc19c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47554/0 (socket says 172.21.15.2:47554) 2023-11-16T18:22:23.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc1745700 1 -- 172.21.15.2:0/2825569194 learned_addr learned my addr 172.21.15.2:0/2825569194 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:23.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dbbfff700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc1003b0 0x7f1dbc1954e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:23.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc0f44700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1dbc0ff1c0 0x7f1dbc194fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:23.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc1745700 1 -- 172.21.15.2:0/2825569194 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1dbc0ff1c0 msgr2=0x7f1dbc194fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:23.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc1745700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1dbc0ff1c0 0x7f1dbc194fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:23.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc1745700 1 -- 172.21.15.2:0/2825569194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc1003b0 msgr2=0x7f1dbc1954e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:23.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc1745700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc1003b0 0x7f1dbc1954e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:23.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc1745700 1 -- 172.21.15.2:0/2825569194 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db4009d00 con 0x7f1dbc100d40 2023-11-16T18:22:23.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc0f44700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1dbc0ff1c0 0x7f1dbc194fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:22:23.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.349+0000 7f1dc1745700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1dbc100d40 0x7f1dbc19c550 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1db400b830 tx=0x7f1db4004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:23.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.350+0000 7f1db9ffb700 1 -- 172.21.15.2:0/2825569194 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db400e040 con 0x7f1dbc100d40 2023-11-16T18:22:23.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.350+0000 7f1db9ffb700 1 -- 172.21.15.2:0/2825569194 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1db40094c0 con 0x7f1dbc100d40 2023-11-16T18:22:23.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.350+0000 7f1dc31a8700 1 -- 172.21.15.2:0/2825569194 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dbc10ab40 con 0x7f1dbc100d40 2023-11-16T18:22:23.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.350+0000 7f1db9ffb700 1 -- 172.21.15.2:0/2825569194 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db401bd30 con 0x7f1dbc100d40 2023-11-16T18:22:23.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.350+0000 7f1dc31a8700 1 -- 172.21.15.2:0/2825569194 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dbc10a9b0 con 0x7f1dbc100d40 2023-11-16T18:22:23.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.351+0000 7f1dc31a8700 1 -- 172.21.15.2:0/2825569194 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dbc106000 con 0x7f1dbc100d40 2023-11-16T18:22:23.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.352+0000 7f1db9ffb700 1 -- 172.21.15.2:0/2825569194 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f1db401d890 con 0x7f1dbc100d40 2023-11-16T18:22:23.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.352+0000 7f1db9ffb700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1dac0a11d0 0x7f1dac0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:23.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.352+0000 7f1db9ffb700 1 -- 172.21.15.2:0/2825569194 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1db40cead0 con 0x7f1dbc100d40 2023-11-16T18:22:23.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.352+0000 7f1dc0f44700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1dac0a11d0 0x7f1dac0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:23.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.353+0000 7f1dc0f44700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1dac0a11d0 0x7f1dac0a3680 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1db0005fd0 tx=0x7f1db0005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:23.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.356+0000 7f1db9ffb700 1 -- 172.21.15.2:0/2825569194 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1db40987d0 con 0x7f1dbc100d40 2023-11-16T18:22:23.410 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 18:22:23 smithi002 podman[132327]: 2023-11-16T18:22:23.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.606+0000 7f1dc31a8700 1 -- 172.21.15.2:0/2825569194 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dbc02cfe0 con 0x7f1dbc100d40 2023-11-16T18:22:23.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.607+0000 7f1db9ffb700 1 -- 172.21.15.2:0/2825569194 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2765 (secure 0 0 0) 0x7f1db409bdf0 con 0x7f1dbc100d40 2023-11-16T18:22:23.609 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:23.609 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-11-16T18:22:23.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.609+0000 7f1dc31a8700 1 -- 172.21.15.2:0/2825569194 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1dac0a11d0 msgr2=0x7f1dac0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:23.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.609+0000 7f1dc31a8700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1dac0a11d0 0x7f1dac0a3680 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1db0005fd0 tx=0x7f1db0005dc0 comp rx=0 tx=0).stop 2023-11-16T18:22:23.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.609+0000 7f1dc31a8700 1 -- 172.21.15.2:0/2825569194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1dbc100d40 msgr2=0x7f1dbc19c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:23.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.609+0000 7f1dc31a8700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1dbc100d40 0x7f1dbc19c550 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1db400b830 tx=0x7f1db4004d30 comp rx=0 tx=0).stop 2023-11-16T18:22:23.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.609+0000 7f1dc31a8700 1 -- 172.21.15.2:0/2825569194 shutdown_connections 2023-11-16T18:22:23.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.609+0000 7f1dc31a8700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1dac0a11d0 0x7f1dac0a3680 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:23.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.609+0000 7f1dc31a8700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1dbc0ff1c0 0x7f1dbc194fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:23.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.609+0000 7f1dc31a8700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1dbc100d40 0x7f1dbc19c550 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:23.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.609+0000 7f1dc31a8700 1 --2- 172.21.15.2:0/2825569194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc1003b0 0x7f1dbc1954e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:23.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.609+0000 7f1dc31a8700 1 -- 172.21.15.2:0/2825569194 >> 172.21.15.2:0/2825569194 conn(0x7f1dbc0fa7c0 msgr2=0x7f1dbc0fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:23.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.609+0000 7f1dc31a8700 1 -- 172.21.15.2:0/2825569194 shutdown_connections 2023-11-16T18:22:23.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:23.609+0000 7f1dc31a8700 1 -- 172.21.15.2:0/2825569194 wait complete. 2023-11-16T18:22:23.612 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 4 2023-11-16T18:22:23.855 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 18:22:23 smithi002 ceph-mds[132410]: -- 172.21.15.2:0/1750140119 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5573837c3520 con 0x557383799000 2023-11-16T18:22:23.856 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 18:22:23 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-d[132343]: starting mds.d at 2023-11-16T18:22:23.856 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 18:22:23 smithi002 bash[132327]: 2fcf1e10ec4c093b5cbe01b5bae692b5fdcf9286ea6d2ac7d9e121b1acd7077d 2023-11-16T18:22:23.856 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 18:22:23 smithi002 systemd[1]: Started Ceph mds.d for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:22:24.691 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:24 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:24.692 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:24 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:24.692 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:24 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:24.692 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:24 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:24.692 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:24 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:24.692 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:24 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:24.692 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:24 smithi040 ceph-mon[100251]: Deploying daemon mds.e on smithi040 2023-11-16T18:22:24.692 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:24 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2825569194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:24.692 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:24 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.2:6836/508165139,v1:172.21.15.2:6837/508165139] up:boot 2023-11-16T18:22:24.692 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:24 smithi040 ceph-mon[100251]: fsmap 4 up:standby 2023-11-16T18:22:24.692 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:24 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T18:22:24.692 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:24 smithi040 ceph-mon[100251]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:24.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:24 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:24.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:24 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:24.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:24 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:24.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:24 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:24.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:24 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:24.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:24 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:24.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:24 smithi188 ceph-mon[100639]: Deploying daemon mds.e on smithi040 2023-11-16T18:22:24.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:24 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2825569194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:24.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:24 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.2:6836/508165139,v1:172.21.15.2:6837/508165139] up:boot 2023-11-16T18:22:24.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:24 smithi188 ceph-mon[100639]: fsmap 4 up:standby 2023-11-16T18:22:24.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:24 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T18:22:24.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:24 smithi188 ceph-mon[100639]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:24.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:24 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:24.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:24 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:24.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:24 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:24.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:24 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:24.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:24 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:24.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:24 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:24.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:24 smithi002 ceph-mon[98807]: Deploying daemon mds.e on smithi040 2023-11-16T18:22:24.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:24 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2825569194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:24.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:24 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.2:6836/508165139,v1:172.21.15.2:6837/508165139] up:boot 2023-11-16T18:22:24.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:24 smithi002 ceph-mon[98807]: fsmap 4 up:standby 2023-11-16T18:22:24.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:24 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T18:22:24.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:24 smithi002 ceph-mon[98807]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:25.483 INFO:journalctl@ceph.mds.e.smithi040.stdout:Nov 16 18:22:25 smithi040 systemd[1]: Starting Ceph mds.e for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:22:25.624 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:25.770 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.762+0000 7f6ad581a700 1 -- 172.21.15.2:0/2155345969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ad0100360 msgr2=0x7f6ad0100770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:25.771 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.762+0000 7f6ad581a700 1 --2- 172.21.15.2:0/2155345969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ad0100360 0x7f6ad0100770 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009990 tx=0x7f6ac0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:25.771 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.763+0000 7f6ad581a700 1 -- 172.21.15.2:0/2155345969 shutdown_connections 2023-11-16T18:22:25.771 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.763+0000 7f6ad581a700 1 --2- 172.21.15.2:0/2155345969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad0101ee0 0x7f6ad01087a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:25.771 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.763+0000 7f6ad581a700 1 --2- 172.21.15.2:0/2155345969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ad0100360 0x7f6ad0100770 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:25.771 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.763+0000 7f6ad581a700 1 --2- 172.21.15.2:0/2155345969 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ad0101550 0x7f6ad01019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:25.771 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.763+0000 7f6ad581a700 1 -- 172.21.15.2:0/2155345969 >> 172.21.15.2:0/2155345969 conn(0x7f6ad00fb960 msgr2=0x7f6ad00fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:25.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.763+0000 7f6ad581a700 1 -- 172.21.15.2:0/2155345969 shutdown_connections 2023-11-16T18:22:25.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.763+0000 7f6ad581a700 1 -- 172.21.15.2:0/2155345969 wait complete. 2023-11-16T18:22:25.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.763+0000 7f6ad581a700 1 Processor -- start 2023-11-16T18:22:25.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.763+0000 7f6ad581a700 1 -- start start 2023-11-16T18:22:25.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6ad581a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad0100360 0x7f6ad0195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:25.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6ad581a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ad0101550 0x7f6ad01964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:25.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6ad581a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ad0101ee0 0x7f6ad019d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:25.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6ad581a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad019db30 con 0x7f6ad0101ee0 2023-11-16T18:22:25.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6ad581a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad019dc70 con 0x7f6ad0100360 2023-11-16T18:22:25.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6ad581a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad019de00 con 0x7f6ad0101550 2023-11-16T18:22:25.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6acf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ad0101ee0 0x7f6ad019d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:25.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6acf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ad0101ee0 0x7f6ad019d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47580/0 (socket says 172.21.15.2:47580) 2023-11-16T18:22:25.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6acf7fe700 1 -- 172.21.15.2:0/1748865949 learned_addr learned my addr 172.21.15.2:0/1748865949 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:25.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6aceffd700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad0100360 0x7f6ad0195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:25.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6ace7fc700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ad0101550 0x7f6ad01964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:25.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6acf7fe700 1 -- 172.21.15.2:0/1748865949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ad0101550 msgr2=0x7f6ad01964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:25.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6acf7fe700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ad0101550 0x7f6ad01964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:25.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6acf7fe700 1 -- 172.21.15.2:0/1748865949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad0100360 msgr2=0x7f6ad0195fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:25.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6acf7fe700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad0100360 0x7f6ad0195fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:25.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6acf7fe700 1 -- 172.21.15.2:0/1748865949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac00095f0 con 0x7f6ad0101ee0 2023-11-16T18:22:25.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6ace7fc700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ad0101550 0x7f6ad01964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:22:25.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6aceffd700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad0100360 0x7f6ad0195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:22:25.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6acf7fe700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ad0101ee0 0x7f6ad019d560 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800ef30 tx=0x7f6ac800c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:25.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6ad4818700 1 -- 172.21.15.2:0/1748865949 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8018070 con 0x7f6ad0101ee0 2023-11-16T18:22:25.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6ad581a700 1 -- 172.21.15.2:0/1748865949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad010bc70 con 0x7f6ad0101ee0 2023-11-16T18:22:25.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6ad4818700 1 -- 172.21.15.2:0/1748865949 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ac800f040 con 0x7f6ad0101ee0 2023-11-16T18:22:25.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.764+0000 7f6ad4818700 1 -- 172.21.15.2:0/1748865949 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8003dc0 con 0x7f6ad0101ee0 2023-11-16T18:22:25.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.765+0000 7f6ad581a700 1 -- 172.21.15.2:0/1748865949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad010bae0 con 0x7f6ad0101ee0 2023-11-16T18:22:25.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.766+0000 7f6ad581a700 1 -- 172.21.15.2:0/1748865949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad0067f40 con 0x7f6ad0101ee0 2023-11-16T18:22:25.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.767+0000 7f6ad4818700 1 -- 172.21.15.2:0/1748865949 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f6ac800caa0 con 0x7f6ad0101ee0 2023-11-16T18:22:25.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.767+0000 7f6ad4818700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6ab80a1150 0x7f6ab80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:25.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.767+0000 7f6aceffd700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6ab80a1150 0x7f6ab80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:25.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.768+0000 7f6ad4818700 1 -- 172.21.15.2:0/1748865949 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6ac80cc140 con 0x7f6ad0101ee0 2023-11-16T18:22:25.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.768+0000 7f6aceffd700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6ab80a1150 0x7f6ab80a3600 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6ac000b2e0 tx=0x7f6ac0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:25.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:25.772+0000 7f6ad4818700 1 -- 172.21.15.2:0/1748865949 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6ac8095dc0 con 0x7f6ad0101ee0 2023-11-16T18:22:25.798 INFO:journalctl@ceph.mds.e.smithi040.stdout:Nov 16 18:22:25 smithi040 podman[124772]: 2023-11-16T18:22:26.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.027+0000 7f6ad581a700 1 -- 172.21.15.2:0/1748865949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ad01071a0 con 0x7f6ad0101ee0 2023-11-16T18:22:26.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.028+0000 7f6ad4818700 1 -- 172.21.15.2:0/1748865949 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4267 (secure 0 0 0) 0x7f6ac80993e0 con 0x7f6ad0101ee0 2023-11-16T18:22:26.031 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:26.031 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-11-16T18:22:26.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.030+0000 7f6ad581a700 1 -- 172.21.15.2:0/1748865949 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6ab80a1150 msgr2=0x7f6ab80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:26.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.030+0000 7f6ad581a700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6ab80a1150 0x7f6ab80a3600 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6ac000b2e0 tx=0x7f6ac0005fd0 comp rx=0 tx=0).stop 2023-11-16T18:22:26.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.030+0000 7f6ad581a700 1 -- 172.21.15.2:0/1748865949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ad0101ee0 msgr2=0x7f6ad019d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:26.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.030+0000 7f6ad581a700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ad0101ee0 0x7f6ad019d560 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800ef30 tx=0x7f6ac800c450 comp rx=0 tx=0).stop 2023-11-16T18:22:26.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.030+0000 7f6ad581a700 1 -- 172.21.15.2:0/1748865949 shutdown_connections 2023-11-16T18:22:26.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.030+0000 7f6ad581a700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6ab80a1150 0x7f6ab80a3600 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:26.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.030+0000 7f6ad581a700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ad0101ee0 0x7f6ad019d560 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:26.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.030+0000 7f6ad581a700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad0100360 0x7f6ad0195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:26.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.030+0000 7f6ad581a700 1 --2- 172.21.15.2:0/1748865949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ad0101550 0x7f6ad01964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:26.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.030+0000 7f6ad581a700 1 -- 172.21.15.2:0/1748865949 >> 172.21.15.2:0/1748865949 conn(0x7f6ad00fb960 msgr2=0x7f6ad00fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:26.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.030+0000 7f6ad581a700 1 -- 172.21.15.2:0/1748865949 shutdown_connections 2023-11-16T18:22:26.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:26.030+0000 7f6ad581a700 1 -- 172.21.15.2:0/1748865949 wait complete. 2023-11-16T18:22:26.033 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 6 2023-11-16T18:22:26.233 INFO:journalctl@ceph.mds.e.smithi040.stdout:Nov 16 18:22:25 smithi040 bash[124772]: cfd9deb2856d772a237fc34e455e0c03e7f1ea1581acc4edb44700b5bb88a424 2023-11-16T18:22:26.233 INFO:journalctl@ceph.mds.e.smithi040.stdout:Nov 16 18:22:25 smithi040 ceph-mds[124813]: -- 172.21.15.40:0/1257604340 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f1437cf520 con 0x55f1437a4c00 2023-11-16T18:22:26.233 INFO:journalctl@ceph.mds.e.smithi040.stdout:Nov 16 18:22:25 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-e[124788]: starting mds.e at 2023-11-16T18:22:26.233 INFO:journalctl@ceph.mds.e.smithi040.stdout:Nov 16 18:22:25 smithi040 systemd[1]: Started Ceph mds.e for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:22:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:26 smithi188 ceph-mon[100639]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:26 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:26 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.40:6834/17956118,v1:172.21.15.40:6835/17956118] up:boot 2023-11-16T18:22:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:26 smithi188 ceph-mon[100639]: fsmap 5 up:standby 2023-11-16T18:22:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:26 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T18:22:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:26 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:26.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:26 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:26.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:26 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:26.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:26 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:26.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:26 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:26.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:26 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1748865949' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:27.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:26 smithi002 ceph-mon[98807]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:27.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:26 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:27.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:26 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.40:6834/17956118,v1:172.21.15.40:6835/17956118] up:boot 2023-11-16T18:22:27.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:26 smithi002 ceph-mon[98807]: fsmap 5 up:standby 2023-11-16T18:22:27.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:26 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T18:22:27.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:26 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:27.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:26 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:27.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:26 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:27.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:26 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:27.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:26 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:27.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:26 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1748865949' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:27.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:26 smithi040 ceph-mon[100251]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:27.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:26 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:27.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:26 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.40:6834/17956118,v1:172.21.15.40:6835/17956118] up:boot 2023-11-16T18:22:27.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:26 smithi040 ceph-mon[100251]: fsmap 5 up:standby 2023-11-16T18:22:27.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:26 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T18:22:27.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:26 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:27.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:26 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:27.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:26 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:27.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:26 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:27.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:26 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:27.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:26 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1748865949' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:27.547 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:22:27 smithi188 systemd[1]: Starting Ceph mds.f for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:22:27.899 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:27 smithi188 ceph-mon[100639]: Deploying daemon mds.f on smithi188 2023-11-16T18:22:27.899 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:22:27 smithi188 podman[126109]: 2023-11-16T18:22:28.045 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:28.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:27 smithi002 ceph-mon[98807]: Deploying daemon mds.f on smithi188 2023-11-16T18:22:28.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.184+0000 7f414b079700 1 -- 172.21.15.2:0/3944819921 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41441016d0 msgr2=0x7f4144101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:28.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.184+0000 7f414b079700 1 --2- 172.21.15.2:0/3944819921 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41441016d0 0x7f4144101b20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4130009990 tx=0x7f4130009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:28.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.184+0000 7f414b079700 1 -- 172.21.15.2:0/3944819921 shutdown_connections 2023-11-16T18:22:28.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.184+0000 7f414b079700 1 --2- 172.21.15.2:0/3944819921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41441004e0 0x7f41441008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:28.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.184+0000 7f414b079700 1 --2- 172.21.15.2:0/3944819921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4144102060 0x7f4144108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:28.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.184+0000 7f414b079700 1 --2- 172.21.15.2:0/3944819921 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41441016d0 0x7f4144101b20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:28.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.184+0000 7f414b079700 1 -- 172.21.15.2:0/3944819921 >> 172.21.15.2:0/3944819921 conn(0x7f41440fbac0 msgr2=0x7f41440fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:28.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.184+0000 7f414b079700 1 -- 172.21.15.2:0/3944819921 shutdown_connections 2023-11-16T18:22:28.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.184+0000 7f414b079700 1 -- 172.21.15.2:0/3944819921 wait complete. 2023-11-16T18:22:28.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f414b079700 1 Processor -- start 2023-11-16T18:22:28.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f414b079700 1 -- start start 2023-11-16T18:22:28.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f414b079700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41441004e0 0x7f41440748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:28.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f414b079700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41441016d0 0x7f4144072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:28.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f414b079700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4144102060 0x7f4144073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:28.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f414b079700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4144073bc0 con 0x7f4144102060 2023-11-16T18:22:28.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f414b079700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4144073d30 con 0x7f41441004e0 2023-11-16T18:22:28.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f414b079700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41441a0250 con 0x7f41441016d0 2023-11-16T18:22:28.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f4149616700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4144102060 0x7f4144073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:28.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f4143fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41441016d0 0x7f4144072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:28.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f4149616700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4144102060 0x7f4144073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47594/0 (socket says 172.21.15.2:47594) 2023-11-16T18:22:28.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f4143fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41441016d0 0x7f4144072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:39122/0 (socket says 172.21.15.2:39122) 2023-11-16T18:22:28.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f4149616700 1 -- 172.21.15.2:0/2880477066 learned_addr learned my addr 172.21.15.2:0/2880477066 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:28.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.185+0000 7f4148e15700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41441004e0 0x7f41440748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:28.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.187+0000 7f4149616700 1 -- 172.21.15.2:0/2880477066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41441016d0 msgr2=0x7f4144072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:28.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.187+0000 7f4149616700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41441016d0 0x7f4144072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:28.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.187+0000 7f4149616700 1 -- 172.21.15.2:0/2880477066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41441004e0 msgr2=0x7f41440748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:28.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.187+0000 7f4149616700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41441004e0 0x7f41440748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:28.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.187+0000 7f4149616700 1 -- 172.21.15.2:0/2880477066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41300095f0 con 0x7f4144102060 2023-11-16T18:22:28.192 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.187+0000 7f4148e15700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41441004e0 0x7f41440748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:22:28.192 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.187+0000 7f4143fff700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41441016d0 0x7f4144072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:22:28.192 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.187+0000 7f4149616700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4144102060 0x7f4144073420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f413c00ef30 tx=0x7f413c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:28.192 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.187+0000 7f4141ffb700 1 -- 172.21.15.2:0/2880477066 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c018070 con 0x7f4144102060 2023-11-16T18:22:28.192 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.187+0000 7f4141ffb700 1 -- 172.21.15.2:0/2880477066 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f413c00f040 con 0x7f4144102060 2023-11-16T18:22:28.192 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.187+0000 7f414b079700 1 -- 172.21.15.2:0/2880477066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f414410be50 con 0x7f4144102060 2023-11-16T18:22:28.192 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.187+0000 7f4141ffb700 1 -- 172.21.15.2:0/2880477066 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c00cdb0 con 0x7f4144102060 2023-11-16T18:22:28.192 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.187+0000 7f414b079700 1 -- 172.21.15.2:0/2880477066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f414410bcc0 con 0x7f4144102060 2023-11-16T18:22:28.193 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.188+0000 7f414b079700 1 -- 172.21.15.2:0/2880477066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41441073e0 con 0x7f4144102060 2023-11-16T18:22:28.194 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.189+0000 7f4141ffb700 1 -- 172.21.15.2:0/2880477066 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f413c003a00 con 0x7f4144102060 2023-11-16T18:22:28.194 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.189+0000 7f4141ffb700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41340a1150 0x7f41340a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:28.194 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.189+0000 7f4148e15700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41340a1150 0x7f41340a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:28.194 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.189+0000 7f4141ffb700 1 -- 172.21.15.2:0/2880477066 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f413c0cc0a0 con 0x7f4144102060 2023-11-16T18:22:28.194 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.189+0000 7f4148e15700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41340a1150 0x7f41340a3600 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4144101530 tx=0x7f4138009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:28.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.194+0000 7f4141ffb700 1 -- 172.21.15.2:0/2880477066 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f413c095d20 con 0x7f4144102060 2023-11-16T18:22:28.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:27 smithi040 ceph-mon[100251]: Deploying daemon mds.f on smithi188 2023-11-16T18:22:28.296 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:22:27 smithi188 ceph-mds[126149]: -- 172.21.15.188:0/2478003635 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5571dee53520 con 0x5571dee28c00 2023-11-16T18:22:28.296 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:22:27 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-f[126125]: starting mds.f at 2023-11-16T18:22:28.297 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:22:27 smithi188 bash[126109]: 076cd0c592f718fe0f4490341dfff8c13afa6ae663bcd61e7f254ba3947467fc 2023-11-16T18:22:28.297 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:22:27 smithi188 systemd[1]: Started Ceph mds.f for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:22:28.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.441+0000 7f414b079700 1 -- 172.21.15.2:0/2880477066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f414402cfe0 con 0x7f4144102060 2023-11-16T18:22:28.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.442+0000 7f4141ffb700 1 -- 172.21.15.2:0/2880477066 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4267 (secure 0 0 0) 0x7f413c099340 con 0x7f4144102060 2023-11-16T18:22:28.443 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:28.443 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-11-16T18:22:28.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.444+0000 7f414b079700 1 -- 172.21.15.2:0/2880477066 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41340a1150 msgr2=0x7f41340a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:28.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.444+0000 7f414b079700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41340a1150 0x7f41340a3600 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4144101530 tx=0x7f4138009260 comp rx=0 tx=0).stop 2023-11-16T18:22:28.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.444+0000 7f414b079700 1 -- 172.21.15.2:0/2880477066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4144102060 msgr2=0x7f4144073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:28.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.444+0000 7f414b079700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4144102060 0x7f4144073420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f413c00ef30 tx=0x7f413c00c3a0 comp rx=0 tx=0).stop 2023-11-16T18:22:28.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.444+0000 7f414b079700 1 -- 172.21.15.2:0/2880477066 shutdown_connections 2023-11-16T18:22:28.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.444+0000 7f414b079700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41340a1150 0x7f41340a3600 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:28.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.444+0000 7f414b079700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4144102060 0x7f4144073420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:28.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.444+0000 7f414b079700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41441004e0 0x7f41440748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:28.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.444+0000 7f414b079700 1 --2- 172.21.15.2:0/2880477066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41441016d0 0x7f4144072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:28.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.444+0000 7f414b079700 1 -- 172.21.15.2:0/2880477066 >> 172.21.15.2:0/2880477066 conn(0x7f41440fbac0 msgr2=0x7f41440fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:28.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.444+0000 7f414b079700 1 -- 172.21.15.2:0/2880477066 shutdown_connections 2023-11-16T18:22:28.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:28.444+0000 7f414b079700 1 -- 172.21.15.2:0/2880477066 wait complete. 2023-11-16T18:22:28.447 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 6 2023-11-16T18:22:29.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:28 smithi188 ceph-mon[100639]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:28 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:28 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:28 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:28 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:28 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:28 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:28 smithi188 ceph-mon[100639]: Deploying daemon mds.c on smithi188 2023-11-16T18:22:29.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:28 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2880477066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:29.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:28 smithi002 ceph-mon[98807]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:28 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:28 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:28 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:28 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:28 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:28 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:28 smithi002 ceph-mon[98807]: Deploying daemon mds.c on smithi188 2023-11-16T18:22:29.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:28 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2880477066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:29.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:28 smithi040 ceph-mon[100251]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:29.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:28 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:29.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:28 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:29.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:28 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:29.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:28 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:29.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:28 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:29.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:28 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:29.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:28 smithi040 ceph-mon[100251]: Deploying daemon mds.c on smithi188 2023-11-16T18:22:29.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:28 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2880477066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:29.797 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:22:29 smithi188 systemd[1]: Starting Ceph mds.c for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:22:30.085 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:22:29 smithi188 podman[126451]: 2023-11-16T18:22:30.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:29 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.188:6834/3630447129,v1:172.21.15.188:6835/3630447129] up:boot 2023-11-16T18:22:30.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:29 smithi188 ceph-mon[100639]: fsmap 6 up:standby 2023-11-16T18:22:30.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:29 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T18:22:30.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:29 smithi188 ceph-mon[100639]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:30.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:29 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.188:6834/3630447129,v1:172.21.15.188:6835/3630447129] up:boot 2023-11-16T18:22:30.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:29 smithi002 ceph-mon[98807]: fsmap 6 up:standby 2023-11-16T18:22:30.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:29 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T18:22:30.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:29 smithi002 ceph-mon[98807]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:30.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:29 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.188:6834/3630447129,v1:172.21.15.188:6835/3630447129] up:boot 2023-11-16T18:22:30.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:29 smithi040 ceph-mon[100251]: fsmap 6 up:standby 2023-11-16T18:22:30.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:29 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T18:22:30.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:29 smithi040 ceph-mon[100251]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:30.459 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:30.546 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:22:30 smithi188 ceph-mds[126491]: -- 172.21.15.188:0/1015744941 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f81b571520 con 0x55f81b546c00 2023-11-16T18:22:30.546 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:22:30 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-c[126467]: starting mds.c at 2023-11-16T18:22:30.547 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:22:30 smithi188 bash[126451]: dd8f0aa2fa9661aafc438fa8a6808c9f8abf1626c7805ea012f0191a97cfc0a7 2023-11-16T18:22:30.547 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:22:30 smithi188 systemd[1]: Started Ceph mds.c for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:22:30.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.598+0000 7f77d9278700 1 -- 172.21.15.2:0/2329303051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77d4100cd0 msgr2=0x7f77d4107590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:30.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.598+0000 7f77d9278700 1 --2- 172.21.15.2:0/2329303051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77d4100cd0 0x7f77d4107590 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00b0a0 tx=0x7f77cc00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:30.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.598+0000 7f77d9278700 1 -- 172.21.15.2:0/2329303051 shutdown_connections 2023-11-16T18:22:30.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.598+0000 7f77d9278700 1 --2- 172.21.15.2:0/2329303051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77d4100cd0 0x7f77d4107590 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:30.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.598+0000 7f77d9278700 1 --2- 172.21.15.2:0/2329303051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77d40ff150 0x7f77d40ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:30.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.598+0000 7f77d9278700 1 --2- 172.21.15.2:0/2329303051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d4100340 0x7f77d4100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:30.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.598+0000 7f77d9278700 1 -- 172.21.15.2:0/2329303051 >> 172.21.15.2:0/2329303051 conn(0x7f77d40fa750 msgr2=0x7f77d40fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:30.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.598+0000 7f77d9278700 1 -- 172.21.15.2:0/2329303051 shutdown_connections 2023-11-16T18:22:30.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.598+0000 7f77d9278700 1 -- 172.21.15.2:0/2329303051 wait complete. 2023-11-16T18:22:30.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.599+0000 7f77d9278700 1 Processor -- start 2023-11-16T18:22:30.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.599+0000 7f77d9278700 1 -- start start 2023-11-16T18:22:30.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.599+0000 7f77d9278700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d40ff150 0x7f77d4198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:30.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.599+0000 7f77d9278700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77d4100340 0x7f77d4198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:30.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.599+0000 7f77d9278700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77d4100cd0 0x7f77d419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:30.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.599+0000 7f77d9278700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d419fdc0 con 0x7f77d4100cd0 2023-11-16T18:22:30.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.599+0000 7f77d9278700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d419ff00 con 0x7f77d4100340 2023-11-16T18:22:30.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.599+0000 7f77d9278700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d41a0090 con 0x7f77d40ff150 2023-11-16T18:22:30.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.599+0000 7f77d37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77d4100cd0 0x7f77d419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:30.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.599+0000 7f77d37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77d4100cd0 0x7f77d419f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47608/0 (socket says 172.21.15.2:47608) 2023-11-16T18:22:30.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.599+0000 7f77d2ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d40ff150 0x7f77d4198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:30.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.599+0000 7f77d37fe700 1 -- 172.21.15.2:0/1685078226 learned_addr learned my addr 172.21.15.2:0/1685078226 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:30.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.599+0000 7f77d27fc700 1 --2- 172.21.15.2:0/1685078226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77d4100340 0x7f77d4198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:30.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.600+0000 7f77d37fe700 1 -- 172.21.15.2:0/1685078226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d40ff150 msgr2=0x7f77d4198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:30.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.600+0000 7f77d37fe700 1 --2- 172.21.15.2:0/1685078226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d40ff150 0x7f77d4198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:30.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.600+0000 7f77d37fe700 1 -- 172.21.15.2:0/1685078226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77d4100340 msgr2=0x7f77d4198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:30.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.600+0000 7f77d37fe700 1 --2- 172.21.15.2:0/1685078226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77d4100340 0x7f77d4198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:30.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.600+0000 7f77d37fe700 1 -- 172.21.15.2:0/1685078226 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c4009520 con 0x7f77d4100cd0 2023-11-16T18:22:30.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.600+0000 7f77d37fe700 1 --2- 172.21.15.2:0/1685078226 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77d4100cd0 0x7f77d419f7f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f77cc007950 tx=0x7f77cc007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:30.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.600+0000 7f77cbfff700 1 -- 172.21.15.2:0/1685078226 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77cc00e040 con 0x7f77d4100cd0 2023-11-16T18:22:30.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.600+0000 7f77cbfff700 1 -- 172.21.15.2:0/1685078226 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77cc01d480 con 0x7f77d4100cd0 2023-11-16T18:22:30.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.600+0000 7f77d9278700 1 -- 172.21.15.2:0/1685078226 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77cc009d00 con 0x7f77d4100cd0 2023-11-16T18:22:30.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.600+0000 7f77cbfff700 1 -- 172.21.15.2:0/1685078226 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77cc01b6b0 con 0x7f77d4100cd0 2023-11-16T18:22:30.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.600+0000 7f77d9278700 1 -- 172.21.15.2:0/1685078226 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d410aac0 con 0x7f77d4100cd0 2023-11-16T18:22:30.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.601+0000 7f77d9278700 1 -- 172.21.15.2:0/1685078226 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d4067ea0 con 0x7f77d4100cd0 2023-11-16T18:22:30.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.602+0000 7f77cbfff700 1 -- 172.21.15.2:0/1685078226 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f77cc01d5f0 con 0x7f77d4100cd0 2023-11-16T18:22:30.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.603+0000 7f77cbfff700 1 --2- 172.21.15.2:0/1685078226 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f77c00a11d0 0x7f77c00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:30.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.603+0000 7f77d2ffd700 1 --2- 172.21.15.2:0/1685078226 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f77c00a11d0 0x7f77c00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:30.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.603+0000 7f77cbfff700 1 -- 172.21.15.2:0/1685078226 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f77cc0cd8e0 con 0x7f77d4100cd0 2023-11-16T18:22:30.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.604+0000 7f77d2ffd700 1 --2- 172.21.15.2:0/1685078226 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f77c00a11d0 0x7f77c00a3680 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f77d41001a0 tx=0x7f77c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:30.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.607+0000 7f77cbfff700 1 -- 172.21.15.2:0/1685078226 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f77cc097540 con 0x7f77d4100cd0 2023-11-16T18:22:30.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.853+0000 7f77d9278700 1 -- 172.21.15.2:0/1685078226 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77d41a0b00 con 0x7f77d4100cd0 2023-11-16T18:22:30.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.853+0000 7f77cbfff700 1 -- 172.21.15.2:0/1685078226 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5784 (secure 0 0 0) 0x7f77cc00be60 con 0x7f77d4100cd0 2023-11-16T18:22:30.855 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:30.855 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-11-16T18:22:30.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.856+0000 7f77d9278700 1 -- 172.21.15.2:0/1685078226 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f77c00a11d0 msgr2=0x7f77c00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:30.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.856+0000 7f77d9278700 1 --2- 172.21.15.2:0/1685078226 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f77c00a11d0 0x7f77c00a3680 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f77d41001a0 tx=0x7f77c4009260 comp rx=0 tx=0).stop 2023-11-16T18:22:30.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.856+0000 7f77d9278700 1 -- 172.21.15.2:0/1685078226 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77d4100cd0 msgr2=0x7f77d419f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:30.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.856+0000 7f77d9278700 1 --2- 172.21.15.2:0/1685078226 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77d4100cd0 0x7f77d419f7f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f77cc007950 tx=0x7f77cc007c40 comp rx=0 tx=0).stop 2023-11-16T18:22:30.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.856+0000 7f77d9278700 1 -- 172.21.15.2:0/1685078226 shutdown_connections 2023-11-16T18:22:30.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.856+0000 7f77d9278700 1 --2- 172.21.15.2:0/1685078226 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f77c00a11d0 0x7f77c00a3680 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:30.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.856+0000 7f77d9278700 1 --2- 172.21.15.2:0/1685078226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d40ff150 0x7f77d4198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:30.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.856+0000 7f77d9278700 1 --2- 172.21.15.2:0/1685078226 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77d4100cd0 0x7f77d419f7f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:30.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.856+0000 7f77d9278700 1 --2- 172.21.15.2:0/1685078226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77d4100340 0x7f77d4198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:30.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.856+0000 7f77d9278700 1 -- 172.21.15.2:0/1685078226 >> 172.21.15.2:0/1685078226 conn(0x7f77d40fa750 msgr2=0x7f77d40690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:30.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.856+0000 7f77d9278700 1 -- 172.21.15.2:0/1685078226 shutdown_connections 2023-11-16T18:22:30.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:30.856+0000 7f77d9278700 1 -- 172.21.15.2:0/1685078226 wait complete. 2023-11-16T18:22:30.858 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 8 2023-11-16T18:22:31.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:31.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:31.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:31.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:31.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:31.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:31.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:31 smithi040 ceph-mon[100251]: Deploying daemon mds.h on smithi040 2023-11-16T18:22:31.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:31 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.188:6836/264232322,v1:172.21.15.188:6837/264232322] up:boot 2023-11-16T18:22:31.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:31 smithi040 ceph-mon[100251]: fsmap 7 up:standby 2023-11-16T18:22:31.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:31 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T18:22:31.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:31 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1685078226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:31.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:31.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:31.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:31.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:31.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:31.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:31.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:31 smithi188 ceph-mon[100639]: Deploying daemon mds.h on smithi040 2023-11-16T18:22:31.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:31 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.188:6836/264232322,v1:172.21.15.188:6837/264232322] up:boot 2023-11-16T18:22:31.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:31 smithi188 ceph-mon[100639]: fsmap 7 up:standby 2023-11-16T18:22:31.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:31 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T18:22:31.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:31 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1685078226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:31.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:31.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:31.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:31.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:31.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:31.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:31.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:31 smithi002 ceph-mon[98807]: Deploying daemon mds.h on smithi040 2023-11-16T18:22:31.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:31 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.188:6836/264232322,v1:172.21.15.188:6837/264232322] up:boot 2023-11-16T18:22:31.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:31 smithi002 ceph-mon[98807]: fsmap 7 up:standby 2023-11-16T18:22:31.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:31 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T18:22:31.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:31 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1685078226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:31.775 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:22:31 smithi040 systemd[1]: Starting Ceph mds.h for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:22:32.354 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:22:32 smithi040 podman[125114]: 2023-11-16T18:22:32.355 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:32 smithi040 ceph-mon[100251]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:32.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:32 smithi188 ceph-mon[100639]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:32.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:32 smithi002 ceph-mon[98807]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:32.733 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:22:32 smithi040 ceph-mds[125155]: -- 172.21.15.40:0/2860931338 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d213765520 con 0x55d21373a800 2023-11-16T18:22:32.733 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:22:32 smithi040 bash[125114]: cbae0c4cdcb0545d9f947cda1659980956333b5860e6c86be496623dadfbf5a3 2023-11-16T18:22:32.734 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:22:32 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: starting mds.h at 2023-11-16T18:22:32.734 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:22:32 smithi040 systemd[1]: Started Ceph mds.h for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:22:32.873 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:33.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.011+0000 7f431f359700 1 -- 172.21.15.2:0/243737103 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4318101550 msgr2=0x7f43181039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:33.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.011+0000 7f431f359700 1 --2- 172.21.15.2:0/243737103 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4318101550 0x7f43181039d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4308009990 tx=0x7f4308009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:33.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.012+0000 7f431f359700 1 -- 172.21.15.2:0/243737103 shutdown_connections 2023-11-16T18:22:33.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.012+0000 7f431f359700 1 --2- 172.21.15.2:0/243737103 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43180febf0 0x7f4318101010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:33.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.012+0000 7f431f359700 1 --2- 172.21.15.2:0/243737103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4318103f10 0x7f4318104390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:33.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.012+0000 7f431f359700 1 --2- 172.21.15.2:0/243737103 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4318101550 0x7f43181039d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:33.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.012+0000 7f431f359700 1 -- 172.21.15.2:0/243737103 >> 172.21.15.2:0/243737103 conn(0x7f43180fa800 msgr2=0x7f43180fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:33.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.012+0000 7f431f359700 1 -- 172.21.15.2:0/243737103 shutdown_connections 2023-11-16T18:22:33.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.012+0000 7f431f359700 1 -- 172.21.15.2:0/243737103 wait complete. 2023-11-16T18:22:33.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.012+0000 7f431f359700 1 Processor -- start 2023-11-16T18:22:33.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.012+0000 7f431f359700 1 -- start start 2023-11-16T18:22:33.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431f359700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43180febf0 0x7f431819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:33.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431f359700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4318101550 0x7f431819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:33.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431f359700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4318103f10 0x7f43181a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:33.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431f359700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43181a4220 con 0x7f4318101550 2023-11-16T18:22:33.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431f359700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43181a4360 con 0x7f4318103f10 2023-11-16T18:22:33.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431f359700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43181a44f0 con 0x7f43180febf0 2023-11-16T18:22:33.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431c8f4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4318101550 0x7f431819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431c8f4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4318101550 0x7f431819cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47620/0 (socket says 172.21.15.2:47620) 2023-11-16T18:22:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431c8f4700 1 -- 172.21.15.2:0/1523783675 learned_addr learned my addr 172.21.15.2:0/1523783675 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431d0f5700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43180febf0 0x7f431819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431d8f6700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4318103f10 0x7f43181a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431c8f4700 1 -- 172.21.15.2:0/1523783675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43180febf0 msgr2=0x7f431819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431c8f4700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43180febf0 0x7f431819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431c8f4700 1 -- 172.21.15.2:0/1523783675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4318103f10 msgr2=0x7f43181a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431c8f4700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4318103f10 0x7f43181a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431c8f4700 1 -- 172.21.15.2:0/1523783675 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43080095f0 con 0x7f4318101550 2023-11-16T18:22:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431c8f4700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4318101550 0x7f431819cbe0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4308000c00 tx=0x7f4308009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f430e7fc700 1 -- 172.21.15.2:0/1523783675 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430801d070 con 0x7f4318101550 2023-11-16T18:22:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f430e7fc700 1 -- 172.21.15.2:0/1523783675 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4308022470 con 0x7f4318101550 2023-11-16T18:22:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431f359700 1 -- 172.21.15.2:0/1523783675 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4318112250 con 0x7f4318101550 2023-11-16T18:22:33.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f430e7fc700 1 -- 172.21.15.2:0/1523783675 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308017650 con 0x7f4318101550 2023-11-16T18:22:33.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.013+0000 7f431f359700 1 -- 172.21.15.2:0/1523783675 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43181120c0 con 0x7f4318101550 2023-11-16T18:22:33.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.014+0000 7f431f359700 1 -- 172.21.15.2:0/1523783675 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43180fc370 con 0x7f4318101550 2023-11-16T18:22:33.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.016+0000 7f430e7fc700 1 -- 172.21.15.2:0/1523783675 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f4308022850 con 0x7f4318101550 2023-11-16T18:22:33.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.017+0000 7f430e7fc700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f43040a11d0 0x7f43040a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:33.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.017+0000 7f430e7fc700 1 -- 172.21.15.2:0/1523783675 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f43080cdd50 con 0x7f4318101550 2023-11-16T18:22:33.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.017+0000 7f431d0f5700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f43040a11d0 0x7f43040a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:33.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.017+0000 7f431d0f5700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f43040a11d0 0x7f43040a3680 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4318069be0 tx=0x7f4314009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:33.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.020+0000 7f430e7fc700 1 -- 172.21.15.2:0/1523783675 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f43080979d0 con 0x7f4318101550 2023-11-16T18:22:33.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.271+0000 7f431f359700 1 -- 172.21.15.2:0/1523783675 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f431802cfe0 con 0x7f4318101550 2023-11-16T18:22:33.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.272+0000 7f430e7fc700 1 -- 172.21.15.2:0/1523783675 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6538 (secure 0 0 0) 0x7f430809aff0 con 0x7f4318101550 2023-11-16T18:22:33.273 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:33.273 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-11-16T18:22:33.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.274+0000 7f431f359700 1 -- 172.21.15.2:0/1523783675 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f43040a11d0 msgr2=0x7f43040a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:33.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.274+0000 7f431f359700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f43040a11d0 0x7f43040a3680 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4318069be0 tx=0x7f4314009260 comp rx=0 tx=0).stop 2023-11-16T18:22:33.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.274+0000 7f431f359700 1 -- 172.21.15.2:0/1523783675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4318101550 msgr2=0x7f431819cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:33.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.274+0000 7f431f359700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4318101550 0x7f431819cbe0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4308000c00 tx=0x7f4308009580 comp rx=0 tx=0).stop 2023-11-16T18:22:33.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.274+0000 7f431f359700 1 -- 172.21.15.2:0/1523783675 shutdown_connections 2023-11-16T18:22:33.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.274+0000 7f431f359700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f43040a11d0 0x7f43040a3680 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:33.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.274+0000 7f431f359700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43180febf0 0x7f431819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:33.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.274+0000 7f431f359700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4318103f10 0x7f43181a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:33.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.274+0000 7f431f359700 1 --2- 172.21.15.2:0/1523783675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4318101550 0x7f431819cbe0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:33.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.274+0000 7f431f359700 1 -- 172.21.15.2:0/1523783675 >> 172.21.15.2:0/1523783675 conn(0x7f43180fa800 msgr2=0x7f43180ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:33.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.274+0000 7f431f359700 1 -- 172.21.15.2:0/1523783675 shutdown_connections 2023-11-16T18:22:33.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:33.274+0000 7f431f359700 1 -- 172.21.15.2:0/1523783675 wait complete. 2023-11-16T18:22:33.276 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 9 2023-11-16T18:22:33.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:33.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:33.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:33.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:33.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:33.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:33.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:33 smithi188 ceph-mon[100639]: Deploying daemon mds.i on smithi188 2023-11-16T18:22:33.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:33 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] up:boot 2023-11-16T18:22:33.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:33 smithi188 ceph-mon[100639]: fsmap 8 up:standby 2023-11-16T18:22:33.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:33 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T18:22:33.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:33 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1523783675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:33.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:33.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:33.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:33.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:33.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:33.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:33.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:33 smithi002 ceph-mon[98807]: Deploying daemon mds.i on smithi188 2023-11-16T18:22:33.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:33 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] up:boot 2023-11-16T18:22:33.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:33 smithi002 ceph-mon[98807]: fsmap 8 up:standby 2023-11-16T18:22:33.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:33 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T18:22:33.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:33 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1523783675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:33.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:33.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:33.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:33.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:33.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:33.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:33.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:33 smithi040 ceph-mon[100251]: Deploying daemon mds.i on smithi188 2023-11-16T18:22:33.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:33 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] up:boot 2023-11-16T18:22:33.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:33 smithi040 ceph-mon[100251]: fsmap 8 up:standby 2023-11-16T18:22:33.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:33 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T18:22:33.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:33 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1523783675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:34.297 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:22:34 smithi188 systemd[1]: Starting Ceph mds.i for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:22:34.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:34 smithi188 ceph-mon[100639]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:34.623 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:22:34 smithi188 podman[126797]: 2023-11-16T18:22:34.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:34 smithi040 ceph-mon[100251]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:34.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:34 smithi002 ceph-mon[98807]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:35.046 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:22:34 smithi188 ceph-mds[126837]: -- 172.21.15.188:0/2342455642 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ab45ecd520 con 0x55ab45ea2c00 2023-11-16T18:22:35.047 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:22:34 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-i[126813]: starting mds.i at 2023-11-16T18:22:35.047 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:22:34 smithi188 bash[126797]: 3741ae60c0b5d7862c241c22f8057f792b551319ed8645ef71f8b63c3287e9c4 2023-11-16T18:22:35.047 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:22:34 smithi188 systemd[1]: Started Ceph mds.i for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:22:35.290 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:35.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.420+0000 7f8b57feb700 1 -- 172.21.15.2:0/3574882708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b501003b0 msgr2=0x7f8b50100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:35.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.420+0000 7f8b57feb700 1 --2- 172.21.15.2:0/3574882708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b501003b0 0x7f8b50100800 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000d960 tx=0x7f8b4000dc70 comp rx=0 tx=0).stop 2023-11-16T18:22:35.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.421+0000 7f8b57feb700 1 -- 172.21.15.2:0/3574882708 shutdown_connections 2023-11-16T18:22:35.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.421+0000 7f8b57feb700 1 --2- 172.21.15.2:0/3574882708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b50100d40 0x7f8b50107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:35.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.421+0000 7f8b57feb700 1 --2- 172.21.15.2:0/3574882708 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b500ff1c0 0x7f8b500ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:35.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.421+0000 7f8b57feb700 1 --2- 172.21.15.2:0/3574882708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b501003b0 0x7f8b50100800 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.421+0000 7f8b57feb700 1 -- 172.21.15.2:0/3574882708 >> 172.21.15.2:0/3574882708 conn(0x7f8b500fa7a0 msgr2=0x7f8b500fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.421+0000 7f8b57feb700 1 -- 172.21.15.2:0/3574882708 shutdown_connections 2023-11-16T18:22:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.421+0000 7f8b57feb700 1 -- 172.21.15.2:0/3574882708 wait complete. 2023-11-16T18:22:35.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.422+0000 7f8b57feb700 1 Processor -- start 2023-11-16T18:22:35.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.422+0000 7f8b57feb700 1 -- start start 2023-11-16T18:22:35.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.422+0000 7f8b57feb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b500ff1c0 0x7f8b50198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:35.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.422+0000 7f8b57feb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b501003b0 0x7f8b50198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:35.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.422+0000 7f8b57feb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b50100d40 0x7f8b5019f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:35.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.422+0000 7f8b57feb700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5019fec0 con 0x7f8b50100d40 2023-11-16T18:22:35.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.422+0000 7f8b57feb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b501a0000 con 0x7f8b500ff1c0 2023-11-16T18:22:35.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.422+0000 7f8b57feb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b501a0190 con 0x7f8b501003b0 2023-11-16T18:22:35.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.422+0000 7f8b56588700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b50100d40 0x7f8b5019f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:35.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.422+0000 7f8b56588700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b50100d40 0x7f8b5019f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47636/0 (socket says 172.21.15.2:47636) 2023-11-16T18:22:35.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.422+0000 7f8b56588700 1 -- 172.21.15.2:0/747667780 learned_addr learned my addr 172.21.15.2:0/747667780 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:35.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.422+0000 7f8b55586700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b501003b0 0x7f8b50198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:35.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.422+0000 7f8b55d87700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b500ff1c0 0x7f8b50198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:35.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.423+0000 7f8b56588700 1 -- 172.21.15.2:0/747667780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b501003b0 msgr2=0x7f8b50198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:35.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.423+0000 7f8b56588700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b501003b0 0x7f8b50198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:35.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.423+0000 7f8b56588700 1 -- 172.21.15.2:0/747667780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b500ff1c0 msgr2=0x7f8b50198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:35.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.423+0000 7f8b56588700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b500ff1c0 0x7f8b50198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:35.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.423+0000 7f8b56588700 1 -- 172.21.15.2:0/747667780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4000d590 con 0x7f8b50100d40 2023-11-16T18:22:35.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.423+0000 7f8b56588700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b50100d40 0x7f8b5019f8f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800b970 tx=0x7f8b4800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:35.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.423+0000 7f8b46ffd700 1 -- 172.21.15.2:0/747667780 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4800c870 con 0x7f8b50100d40 2023-11-16T18:22:35.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.423+0000 7f8b57feb700 1 -- 172.21.15.2:0/747667780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b50109ba0 con 0x7f8b50100d40 2023-11-16T18:22:35.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.423+0000 7f8b46ffd700 1 -- 172.21.15.2:0/747667780 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b4800ce90 con 0x7f8b50100d40 2023-11-16T18:22:35.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.423+0000 7f8b46ffd700 1 -- 172.21.15.2:0/747667780 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48007370 con 0x7f8b50100d40 2023-11-16T18:22:35.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.423+0000 7f8b57feb700 1 -- 172.21.15.2:0/747667780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b50109a10 con 0x7f8b50100d40 2023-11-16T18:22:35.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.424+0000 7f8b57feb700 1 -- 172.21.15.2:0/747667780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b501a0960 con 0x7f8b50100d40 2023-11-16T18:22:35.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.425+0000 7f8b46ffd700 1 -- 172.21.15.2:0/747667780 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f8b4800c9d0 con 0x7f8b50100d40 2023-11-16T18:22:35.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.426+0000 7f8b46ffd700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8b3c0a1150 0x7f8b3c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:35.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.426+0000 7f8b55d87700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8b3c0a1150 0x7f8b3c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:35.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.426+0000 7f8b46ffd700 1 -- 172.21.15.2:0/747667780 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8b48006e40 con 0x7f8b50100d40 2023-11-16T18:22:35.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.426+0000 7f8b55d87700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8b3c0a1150 0x7f8b3c0a3600 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8b50100210 tx=0x7f8b4c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:35.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.429+0000 7f8b46ffd700 1 -- 172.21.15.2:0/747667780 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8b48095ad0 con 0x7f8b50100d40 2023-11-16T18:22:35.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.680+0000 7f8b57feb700 1 -- 172.21.15.2:0/747667780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b501a0f10 con 0x7f8b50100d40 2023-11-16T18:22:35.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.681+0000 7f8b46ffd700 1 -- 172.21.15.2:0/747667780 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7300 (secure 0 0 0) 0x7f8b480990f0 con 0x7f8b50100d40 2023-11-16T18:22:35.683 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:35.683 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-11-16T18:22:35.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.683+0000 7f8b57feb700 1 -- 172.21.15.2:0/747667780 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8b3c0a1150 msgr2=0x7f8b3c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:35.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.683+0000 7f8b57feb700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8b3c0a1150 0x7f8b3c0a3600 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8b50100210 tx=0x7f8b4c009310 comp rx=0 tx=0).stop 2023-11-16T18:22:35.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.683+0000 7f8b57feb700 1 -- 172.21.15.2:0/747667780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b50100d40 msgr2=0x7f8b5019f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:35.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.683+0000 7f8b57feb700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b50100d40 0x7f8b5019f8f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800b970 tx=0x7f8b4800bc80 comp rx=0 tx=0).stop 2023-11-16T18:22:35.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.683+0000 7f8b57feb700 1 -- 172.21.15.2:0/747667780 shutdown_connections 2023-11-16T18:22:35.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.683+0000 7f8b57feb700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8b3c0a1150 0x7f8b3c0a3600 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:35.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.683+0000 7f8b57feb700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b50100d40 0x7f8b5019f8f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:35.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.683+0000 7f8b57feb700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b500ff1c0 0x7f8b50198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:35.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.683+0000 7f8b57feb700 1 --2- 172.21.15.2:0/747667780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b501003b0 0x7f8b50198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:35.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.683+0000 7f8b57feb700 1 -- 172.21.15.2:0/747667780 >> 172.21.15.2:0/747667780 conn(0x7f8b500fa7a0 msgr2=0x7f8b500fc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:35.686 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.684+0000 7f8b57feb700 1 -- 172.21.15.2:0/747667780 shutdown_connections 2023-11-16T18:22:35.686 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:35.684+0000 7f8b57feb700 1 -- 172.21.15.2:0/747667780 wait complete. 2023-11-16T18:22:35.686 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 10 2023-11-16T18:22:35.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:35.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:35.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:35.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:35.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:35.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:35.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:35 smithi002 ceph-mon[98807]: Deploying daemon mds.g on smithi002 2023-11-16T18:22:35.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:35 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] up:boot 2023-11-16T18:22:35.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:35 smithi002 ceph-mon[98807]: fsmap 9 up:standby 2023-11-16T18:22:35.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:35 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T18:22:35.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:35 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/747667780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:35.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:35.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:35.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:35.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:35.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:35.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:35.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:35 smithi040 ceph-mon[100251]: Deploying daemon mds.g on smithi002 2023-11-16T18:22:35.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:35 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] up:boot 2023-11-16T18:22:35.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:35 smithi040 ceph-mon[100251]: fsmap 9 up:standby 2023-11-16T18:22:35.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:35 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T18:22:35.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:35 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/747667780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:36.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:36.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:36.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:36.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:36.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:36.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:36.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:35 smithi188 ceph-mon[100639]: Deploying daemon mds.g on smithi002 2023-11-16T18:22:36.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:35 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] up:boot 2023-11-16T18:22:36.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:35 smithi188 ceph-mon[100639]: fsmap 9 up:standby 2023-11-16T18:22:36.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:35 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T18:22:36.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:35 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/747667780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:36.556 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 18:22:36 smithi002 systemd[1]: Starting Ceph mds.g for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:22:36.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:36 smithi002 ceph-mon[98807]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:36.856 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 18:22:36 smithi002 podman[132937]: 2023-11-16T18:22:36.857 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 18:22:36 smithi002 bash[132937]: 73ab18e3bb5b05b5ad9d356650336be16c0f27f115c7fdd683bd44f07939ce28 2023-11-16T18:22:36.857 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 18:22:36 smithi002 ceph-mds[132977]: -- 172.21.15.2:0/2261723848 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558845887520 con 0x55884585cc00 2023-11-16T18:22:36.857 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 18:22:36 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-g[132953]: starting mds.g at 2023-11-16T18:22:36.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:36 smithi040 ceph-mon[100251]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:37.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:36 smithi188 ceph-mon[100639]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:37.355 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 18:22:36 smithi002 systemd[1]: Started Ceph mds.g for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:22:37.698 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:37.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.833+0000 7f1856c0e700 1 -- 172.21.15.2:0/3076384440 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1850101fb0 msgr2=0x7f1850108870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:37.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.833+0000 7f1856c0e700 1 --2- 172.21.15.2:0/3076384440 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1850101fb0 0x7f1850108870 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f184800b0a0 tx=0x7f184800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:37.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.833+0000 7f1856c0e700 1 -- 172.21.15.2:0/3076384440 shutdown_connections 2023-11-16T18:22:37.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.833+0000 7f1856c0e700 1 --2- 172.21.15.2:0/3076384440 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1850101fb0 0x7f1850108870 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:37.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.833+0000 7f1856c0e700 1 --2- 172.21.15.2:0/3076384440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18500fe5b0 0x7f18500fe9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:37.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.833+0000 7f1856c0e700 1 --2- 172.21.15.2:0/3076384440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1850101620 0x7f1850101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:37.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.833+0000 7f1856c0e700 1 -- 172.21.15.2:0/3076384440 >> 172.21.15.2:0/3076384440 conn(0x7f18500f9f50 msgr2=0x7f18500fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:37.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.833+0000 7f1856c0e700 1 -- 172.21.15.2:0/3076384440 shutdown_connections 2023-11-16T18:22:37.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.833+0000 7f1856c0e700 1 -- 172.21.15.2:0/3076384440 wait complete. 2023-11-16T18:22:37.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.834+0000 7f1856c0e700 1 Processor -- start 2023-11-16T18:22:37.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.834+0000 7f1856c0e700 1 -- start start 2023-11-16T18:22:37.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f1856c0e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18500fe5b0 0x7f1850196160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:37.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f1856c0e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1850101620 0x7f18501966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:37.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f1856c0e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1850101fb0 0x7f185019d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:37.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f1856c0e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185019dce0 con 0x7f18500fe5b0 2023-11-16T18:22:37.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f1856c0e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185019de20 con 0x7f1850101fb0 2023-11-16T18:22:37.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f1856c0e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185019dfb0 con 0x7f1850101620 2023-11-16T18:22:37.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f18549aa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18500fe5b0 0x7f1850196160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:37.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f18549aa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18500fe5b0 0x7f1850196160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47660/0 (socket says 172.21.15.2:47660) 2023-11-16T18:22:37.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f18549aa700 1 -- 172.21.15.2:0/3683301467 learned_addr learned my addr 172.21.15.2:0/3683301467 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:37.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f184ffff700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1850101620 0x7f18501966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:37.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f18551ab700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1850101fb0 0x7f185019d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:37.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f18549aa700 1 -- 172.21.15.2:0/3683301467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1850101620 msgr2=0x7f18501966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:37.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f18549aa700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1850101620 0x7f18501966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:37.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f18549aa700 1 -- 172.21.15.2:0/3683301467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1850101fb0 msgr2=0x7f185019d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:37.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f18549aa700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1850101fb0 0x7f185019d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:37.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f18549aa700 1 -- 172.21.15.2:0/3683301467 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1848009d00 con 0x7f18500fe5b0 2023-11-16T18:22:37.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f18549aa700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18500fe5b0 0x7f1850196160 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f184400c970 tx=0x7f184400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:37.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f184dffb700 1 -- 172.21.15.2:0/3683301467 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844004220 con 0x7f18500fe5b0 2023-11-16T18:22:37.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f184dffb700 1 -- 172.21.15.2:0/3683301467 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1844007d60 con 0x7f18500fe5b0 2023-11-16T18:22:37.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f1856c0e700 1 -- 172.21.15.2:0/3683301467 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185010bc90 con 0x7f18500fe5b0 2023-11-16T18:22:37.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.835+0000 7f184dffb700 1 -- 172.21.15.2:0/3683301467 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844003d00 con 0x7f18500fe5b0 2023-11-16T18:22:37.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.836+0000 7f1856c0e700 1 -- 172.21.15.2:0/3683301467 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185010bb00 con 0x7f18500fe5b0 2023-11-16T18:22:37.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.837+0000 7f1856c0e700 1 -- 172.21.15.2:0/3683301467 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185019e700 con 0x7f18500fe5b0 2023-11-16T18:22:37.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.838+0000 7f184dffb700 1 -- 172.21.15.2:0/3683301467 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f1844004380 con 0x7f18500fe5b0 2023-11-16T18:22:37.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.838+0000 7f184dffb700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f18400a1150 0x7f18400a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:37.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.838+0000 7f184dffb700 1 -- 172.21.15.2:0/3683301467 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f18440cc410 con 0x7f18500fe5b0 2023-11-16T18:22:37.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.838+0000 7f184ffff700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f18400a1150 0x7f18400a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:37.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.839+0000 7f184ffff700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f18400a1150 0x7f18400a3600 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f183c005950 tx=0x7f183c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:37.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:37.842+0000 7f184dffb700 1 -- 172.21.15.2:0/3683301467 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1844096000 con 0x7f18500fe5b0 2023-11-16T18:22:37.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:37.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:37.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:37.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:37.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:37.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:37.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:37 smithi002 ceph-mon[98807]: Deploying daemon mds.a on smithi002 2023-11-16T18:22:37.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:37 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.2:6838/1181439264,v1:172.21.15.2:6839/1181439264] up:boot 2023-11-16T18:22:37.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:37 smithi002 ceph-mon[98807]: fsmap 10 up:standby 2023-11-16T18:22:37.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:37 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T18:22:37.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:37 smithi002 ceph-mon[98807]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:38.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.092+0000 7f1856c0e700 1 -- 172.21.15.2:0/3683301467 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18501072e0 con 0x7f18500fe5b0 2023-11-16T18:22:38.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.093+0000 7f184dffb700 1 -- 172.21.15.2:0/3683301467 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8055 (secure 0 0 0) 0x7f1844099620 con 0x7f18500fe5b0 2023-11-16T18:22:38.094 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:38.095 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-11-16T18:22:38.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.095+0000 7f1856c0e700 1 -- 172.21.15.2:0/3683301467 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f18400a1150 msgr2=0x7f18400a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:38.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.095+0000 7f1856c0e700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f18400a1150 0x7f18400a3600 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f183c005950 tx=0x7f183c0058e0 comp rx=0 tx=0).stop 2023-11-16T18:22:38.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.095+0000 7f1856c0e700 1 -- 172.21.15.2:0/3683301467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18500fe5b0 msgr2=0x7f1850196160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:38.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.095+0000 7f1856c0e700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18500fe5b0 0x7f1850196160 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f184400c970 tx=0x7f184400cd30 comp rx=0 tx=0).stop 2023-11-16T18:22:38.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.096+0000 7f1856c0e700 1 -- 172.21.15.2:0/3683301467 shutdown_connections 2023-11-16T18:22:38.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.096+0000 7f1856c0e700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f18400a1150 0x7f18400a3600 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:38.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.096+0000 7f1856c0e700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1850101fb0 0x7f185019d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:38.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.096+0000 7f1856c0e700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18500fe5b0 0x7f1850196160 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:38.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.096+0000 7f1856c0e700 1 --2- 172.21.15.2:0/3683301467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1850101620 0x7f18501966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:38.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.096+0000 7f1856c0e700 1 -- 172.21.15.2:0/3683301467 >> 172.21.15.2:0/3683301467 conn(0x7f18500f9f50 msgr2=0x7f18500fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:38.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.096+0000 7f1856c0e700 1 -- 172.21.15.2:0/3683301467 shutdown_connections 2023-11-16T18:22:38.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:38.096+0000 7f1856c0e700 1 -- 172.21.15.2:0/3683301467 wait complete. 2023-11-16T18:22:38.101 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 11 2023-11-16T18:22:38.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:38.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:38.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:38.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:38.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:38.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:38.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:37 smithi040 ceph-mon[100251]: Deploying daemon mds.a on smithi002 2023-11-16T18:22:38.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:37 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.2:6838/1181439264,v1:172.21.15.2:6839/1181439264] up:boot 2023-11-16T18:22:38.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:37 smithi040 ceph-mon[100251]: fsmap 10 up:standby 2023-11-16T18:22:38.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:37 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T18:22:38.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:37 smithi040 ceph-mon[100251]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:38.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:38.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:38.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:38.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:38.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:38.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:38.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:37 smithi188 ceph-mon[100639]: Deploying daemon mds.a on smithi002 2023-11-16T18:22:38.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:37 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.2:6838/1181439264,v1:172.21.15.2:6839/1181439264] up:boot 2023-11-16T18:22:38.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:37 smithi188 ceph-mon[100639]: fsmap 10 up:standby 2023-11-16T18:22:38.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:37 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T18:22:38.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:37 smithi188 ceph-mon[100639]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:38.746 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 18:22:38 smithi002 systemd[1]: Starting Ceph mds.a for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:22:39.036 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 18:22:38 smithi002 podman[133320]: 2023-11-16T18:22:39.036 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:38 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3683301467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:39.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:38 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3683301467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:39.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:38 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3683301467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:39.356 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 18:22:39 smithi002 bash[133320]: f106bfdbe3e148832004322c045e10f236a10b5790999554468c2c304bb0a06a 2023-11-16T18:22:39.356 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 18:22:39 smithi002 ceph-mds[133360]: -- 172.21.15.2:0/2689313073 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55697a785520 con 0x55697a75b000 2023-11-16T18:22:39.356 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 18:22:39 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-a[133336]: starting mds.a at 2023-11-16T18:22:39.356 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 18:22:39 smithi002 systemd[1]: Started Ceph mds.a for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:22:40.111 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:40.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:40.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:40.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:40.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:40.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:40.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:40.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:40 smithi040 ceph-mon[100251]: Deploying daemon mds.b on smithi040 2023-11-16T18:22:40.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:40 smithi040 ceph-mon[100251]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:40.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:40 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.2:6840/3213213799,v1:172.21.15.2:6841/3213213799] up:boot 2023-11-16T18:22:40.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:40 smithi040 ceph-mon[100251]: fsmap 11 up:standby 2023-11-16T18:22:40.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:40 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T18:22:40.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.255+0000 7f9c77f98700 1 -- 172.21.15.2:0/1359455764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c70100d40 msgr2=0x7f9c70107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:40.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.255+0000 7f9c77f98700 1 --2- 172.21.15.2:0/1359455764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c70100d40 0x7f9c70107600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9c6800b0a0 tx=0x7f9c6800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:40.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.255+0000 7f9c77f98700 1 -- 172.21.15.2:0/1359455764 shutdown_connections 2023-11-16T18:22:40.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.255+0000 7f9c77f98700 1 --2- 172.21.15.2:0/1359455764 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c700ff1c0 0x7f9c700ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.255+0000 7f9c77f98700 1 --2- 172.21.15.2:0/1359455764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c70100d40 0x7f9c70107600 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.255+0000 7f9c77f98700 1 --2- 172.21.15.2:0/1359455764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c701003b0 0x7f9c70100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.255+0000 7f9c77f98700 1 -- 172.21.15.2:0/1359455764 >> 172.21.15.2:0/1359455764 conn(0x7f9c700fa7c0 msgr2=0x7f9c700fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.255+0000 7f9c77f98700 1 -- 172.21.15.2:0/1359455764 shutdown_connections 2023-11-16T18:22:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.255+0000 7f9c77f98700 1 -- 172.21.15.2:0/1359455764 wait complete. 2023-11-16T18:22:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.256+0000 7f9c77f98700 1 Processor -- start 2023-11-16T18:22:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.256+0000 7f9c77f98700 1 -- start start 2023-11-16T18:22:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.256+0000 7f9c77f98700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c700ff1c0 0x7f9c701961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.256+0000 7f9c77f98700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c701003b0 0x7f9c70196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.256+0000 7f9c77f98700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70100d40 0x7f9c7019d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.256+0000 7f9c77f98700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7019dd40 con 0x7f9c700ff1c0 2023-11-16T18:22:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.256+0000 7f9c77f98700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7019de80 con 0x7f9c701003b0 2023-11-16T18:22:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.256+0000 7f9c77f98700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7019e010 con 0x7f9c70100d40 2023-11-16T18:22:40.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.256+0000 7f9c75533700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c701003b0 0x7f9c70196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:40.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.256+0000 7f9c75533700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c701003b0 0x7f9c70196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:38126/0 (socket says 172.21.15.2:38126) 2023-11-16T18:22:40.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.256+0000 7f9c75533700 1 -- 172.21.15.2:0/2792371035 learned_addr learned my addr 172.21.15.2:0/2792371035 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:40.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.256+0000 7f9c76535700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70100d40 0x7f9c7019d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:40.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.257+0000 7f9c75d34700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c700ff1c0 0x7f9c701961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:40.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.257+0000 7f9c75533700 1 -- 172.21.15.2:0/2792371035 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70100d40 msgr2=0x7f9c7019d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:40.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.257+0000 7f9c75533700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70100d40 0x7f9c7019d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:40.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.257+0000 7f9c75533700 1 -- 172.21.15.2:0/2792371035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c700ff1c0 msgr2=0x7f9c701961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:40.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.257+0000 7f9c75533700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c700ff1c0 0x7f9c701961c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:40.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.257+0000 7f9c75533700 1 -- 172.21.15.2:0/2792371035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c68009d00 con 0x7f9c701003b0 2023-11-16T18:22:40.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.257+0000 7f9c75d34700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c700ff1c0 0x7f9c701961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:22:40.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.257+0000 7f9c75533700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c701003b0 0x7f9c70196700 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400e9a0 tx=0x7f9c6400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:40.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.257+0000 7f9c62ffd700 1 -- 172.21.15.2:0/2792371035 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c64009790 con 0x7f9c701003b0 2023-11-16T18:22:40.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.257+0000 7f9c77f98700 1 -- 172.21.15.2:0/2792371035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7010aba0 con 0x7f9c701003b0 2023-11-16T18:22:40.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.257+0000 7f9c62ffd700 1 -- 172.21.15.2:0/2792371035 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c6400cbd0 con 0x7f9c701003b0 2023-11-16T18:22:40.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.257+0000 7f9c62ffd700 1 -- 172.21.15.2:0/2792371035 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c64018680 con 0x7f9c701003b0 2023-11-16T18:22:40.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.257+0000 7f9c77f98700 1 -- 172.21.15.2:0/2792371035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7010aa10 con 0x7f9c701003b0 2023-11-16T18:22:40.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.258+0000 7f9c77f98700 1 -- 172.21.15.2:0/2792371035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c70067af0 con 0x7f9c701003b0 2023-11-16T18:22:40.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.259+0000 7f9c62ffd700 1 -- 172.21.15.2:0/2792371035 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f9c6400cd40 con 0x7f9c701003b0 2023-11-16T18:22:40.264 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.260+0000 7f9c62ffd700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9c5c0a1150 0x7f9c5c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:40.264 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.260+0000 7f9c75d34700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9c5c0a1150 0x7f9c5c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:40.264 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.260+0000 7f9c62ffd700 1 -- 172.21.15.2:0/2792371035 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9c64014070 con 0x7f9c701003b0 2023-11-16T18:22:40.264 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.260+0000 7f9c75d34700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9c5c0a1150 0x7f9c5c0a3600 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c005fd0 tx=0x7f9c6c005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:40.264 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.263+0000 7f9c62ffd700 1 -- 172.21.15.2:0/2792371035 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9c640963b0 con 0x7f9c701003b0 2023-11-16T18:22:40.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:40 smithi002 ceph-mon[98807]: Deploying daemon mds.b on smithi040 2023-11-16T18:22:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:40 smithi002 ceph-mon[98807]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:40 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.2:6840/3213213799,v1:172.21.15.2:6841/3213213799] up:boot 2023-11-16T18:22:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:40 smithi002 ceph-mon[98807]: fsmap 11 up:standby 2023-11-16T18:22:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:40 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T18:22:40.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.510+0000 7f9c77f98700 1 -- 172.21.15.2:0/2792371035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c7019eb20 con 0x7f9c701003b0 2023-11-16T18:22:40.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.511+0000 7f9c62ffd700 1 -- 172.21.15.2:0/2792371035 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8810 (secure 0 0 0) 0x7f9c640999d0 con 0x7f9c701003b0 2023-11-16T18:22:40.512 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:40.512 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-11-16T18:22:40.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.513+0000 7f9c77f98700 1 -- 172.21.15.2:0/2792371035 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9c5c0a1150 msgr2=0x7f9c5c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:40.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.513+0000 7f9c77f98700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9c5c0a1150 0x7f9c5c0a3600 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c005fd0 tx=0x7f9c6c005f00 comp rx=0 tx=0).stop 2023-11-16T18:22:40.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.513+0000 7f9c77f98700 1 -- 172.21.15.2:0/2792371035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c701003b0 msgr2=0x7f9c70196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:40.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.513+0000 7f9c77f98700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c701003b0 0x7f9c70196700 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400e9a0 tx=0x7f9c6400ecb0 comp rx=0 tx=0).stop 2023-11-16T18:22:40.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.513+0000 7f9c77f98700 1 -- 172.21.15.2:0/2792371035 shutdown_connections 2023-11-16T18:22:40.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.513+0000 7f9c77f98700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9c5c0a1150 0x7f9c5c0a3600 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:40.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.513+0000 7f9c77f98700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c70100d40 0x7f9c7019d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:40.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.513+0000 7f9c77f98700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c700ff1c0 0x7f9c701961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:40.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.513+0000 7f9c77f98700 1 --2- 172.21.15.2:0/2792371035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c701003b0 0x7f9c70196700 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:40.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.513+0000 7f9c77f98700 1 -- 172.21.15.2:0/2792371035 >> 172.21.15.2:0/2792371035 conn(0x7f9c700fa7c0 msgr2=0x7f9c70067dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:40.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.513+0000 7f9c77f98700 1 -- 172.21.15.2:0/2792371035 shutdown_connections 2023-11-16T18:22:40.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:40.513+0000 7f9c77f98700 1 -- 172.21.15.2:0/2792371035 wait complete. 2023-11-16T18:22:40.517 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 12 2023-11-16T18:22:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T18:22:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T18:22:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:40 smithi188 ceph-mon[100639]: Deploying daemon mds.b on smithi040 2023-11-16T18:22:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:40 smithi188 ceph-mon[100639]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:40 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.2:6840/3213213799,v1:172.21.15.2:6841/3213213799] up:boot 2023-11-16T18:22:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:40 smithi188 ceph-mon[100639]: fsmap 11 up:standby 2023-11-16T18:22:40.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:40 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T18:22:40.940 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:22:40 smithi040 systemd[1]: Starting Ceph mds.b for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:22:41.204 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:41 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2792371035' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:41.204 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:22:40 smithi040 podman[125461]: 2023-11-16T18:22:41.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:41 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2792371035' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:41.484 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:22:41 smithi040 bash[125461]: 5cc16666f3a06de11f91abbf8f763407ca4033340fdf81c40a56c2028c7e2fe9 2023-11-16T18:22:41.484 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:22:41 smithi040 ceph-mds[125503]: -- 172.21.15.40:0/2386226478 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556fc04c7520 con 0x556fc049cc00 2023-11-16T18:22:41.484 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:22:41 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-b[125477]: starting mds.b at 2023-11-16T18:22:41.484 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:22:41 smithi040 systemd[1]: Started Ceph mds.b for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:22:41.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:41 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2792371035' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:42.528 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:42.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:42 smithi188 ceph-mon[100639]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:42 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] up:boot 2023-11-16T18:22:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:42 smithi188 ceph-mon[100639]: fsmap 12 up:standby 2023-11-16T18:22:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:42 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T18:22:42.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:42.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:42 smithi002 ceph-mon[98807]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:42.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:42 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] up:boot 2023-11-16T18:22:42.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:42 smithi002 ceph-mon[98807]: fsmap 12 up:standby 2023-11-16T18:22:42.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:42 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T18:22:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.660+0000 7fe8522fd700 1 -- 172.21.15.2:0/1168325709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe84c0ff1c0 msgr2=0x7fe84c0ff5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.660+0000 7fe8522fd700 1 --2- 172.21.15.2:0/1168325709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe84c0ff1c0 0x7fe84c0ff5d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe83c009990 tx=0x7fe83c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.660+0000 7fe8522fd700 1 -- 172.21.15.2:0/1168325709 shutdown_connections 2023-11-16T18:22:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.660+0000 7fe8522fd700 1 --2- 172.21.15.2:0/1168325709 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe84c100d40 0x7fe84c107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.660+0000 7fe8522fd700 1 --2- 172.21.15.2:0/1168325709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe84c0ff1c0 0x7fe84c0ff5d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.661+0000 7fe8522fd700 1 --2- 172.21.15.2:0/1168325709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe84c1003b0 0x7fe84c100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.661+0000 7fe8522fd700 1 -- 172.21.15.2:0/1168325709 >> 172.21.15.2:0/1168325709 conn(0x7fe84c0fa7c0 msgr2=0x7fe84c0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:42.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.661+0000 7fe8522fd700 1 -- 172.21.15.2:0/1168325709 shutdown_connections 2023-11-16T18:22:42.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.661+0000 7fe8522fd700 1 -- 172.21.15.2:0/1168325709 wait complete. 2023-11-16T18:22:42.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.661+0000 7fe8522fd700 1 Processor -- start 2023-11-16T18:22:42.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.661+0000 7fe8522fd700 1 -- start start 2023-11-16T18:22:42.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe8522fd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe84c0ff1c0 0x7fe84c194fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:42.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe8522fd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe84c1003b0 0x7fe84c1954f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:42.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe8522fd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe84c100d40 0x7fe84c19c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:42.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe8522fd700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c19cb30 con 0x7fe84c0ff1c0 2023-11-16T18:22:42.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe8522fd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c19cc70 con 0x7fe84c1003b0 2023-11-16T18:22:42.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe8522fd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c19ce00 con 0x7fe84c100d40 2023-11-16T18:22:42.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe84bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe84c0ff1c0 0x7fe84c194fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:42.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe84bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe84c0ff1c0 0x7fe84c194fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47700/0 (socket says 172.21.15.2:47700) 2023-11-16T18:22:42.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe84bfff700 1 -- 172.21.15.2:0/397156721 learned_addr learned my addr 172.21.15.2:0/397156721 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:42.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe84b7fe700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe84c1003b0 0x7fe84c1954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:42.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe85089a700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe84c100d40 0x7fe84c19c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:42.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe84bfff700 1 -- 172.21.15.2:0/397156721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe84c100d40 msgr2=0x7fe84c19c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:42.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe84bfff700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe84c100d40 0x7fe84c19c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:42.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe84bfff700 1 -- 172.21.15.2:0/397156721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe84c1003b0 msgr2=0x7fe84c1954f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:42.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe84bfff700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe84c1003b0 0x7fe84c1954f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:42.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe84bfff700 1 -- 172.21.15.2:0/397156721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c0095f0 con 0x7fe84c0ff1c0 2023-11-16T18:22:42.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.662+0000 7fe84bfff700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe84c0ff1c0 0x7fe84c194fb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe83c006010 tx=0x7fe83c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:42.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.663+0000 7fe8497fa700 1 -- 172.21.15.2:0/397156721 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c01d070 con 0x7fe84c0ff1c0 2023-11-16T18:22:42.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.663+0000 7fe8522fd700 1 -- 172.21.15.2:0/397156721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe84c10ab20 con 0x7fe84c0ff1c0 2023-11-16T18:22:42.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.663+0000 7fe8497fa700 1 -- 172.21.15.2:0/397156721 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe83c003fe0 con 0x7fe84c0ff1c0 2023-11-16T18:22:42.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.663+0000 7fe8497fa700 1 -- 172.21.15.2:0/397156721 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c022c10 con 0x7fe84c0ff1c0 2023-11-16T18:22:42.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.663+0000 7fe8522fd700 1 -- 172.21.15.2:0/397156721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe84c10a990 con 0x7fe84c0ff1c0 2023-11-16T18:22:42.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.664+0000 7fe8522fd700 1 -- 172.21.15.2:0/397156721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe84c19d3a0 con 0x7fe84c0ff1c0 2023-11-16T18:22:42.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.665+0000 7fe8497fa700 1 -- 172.21.15.2:0/397156721 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fe83c022470 con 0x7fe84c0ff1c0 2023-11-16T18:22:42.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.666+0000 7fe8497fa700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe8340a1150 0x7fe8340a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:42.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.666+0000 7fe8497fa700 1 -- 172.21.15.2:0/397156721 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe83c0cda20 con 0x7fe84c0ff1c0 2023-11-16T18:22:42.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.666+0000 7fe84b7fe700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe8340a1150 0x7fe8340a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:42.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.667+0000 7fe84b7fe700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe8340a1150 0x7fe8340a3600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe840005950 tx=0x7fe84000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:42.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.671+0000 7fe8497fa700 1 -- 172.21.15.2:0/397156721 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe83c0976a0 con 0x7fe84c0ff1c0 2023-11-16T18:22:42.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:42.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:42.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:42 smithi040 ceph-mon[100251]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:42.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:42 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] up:boot 2023-11-16T18:22:42.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:42 smithi040 ceph-mon[100251]: fsmap 12 up:standby 2023-11-16T18:22:42.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:42 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T18:22:42.931 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.930+0000 7fe8522fd700 1 -- 172.21.15.2:0/397156721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe84c106130 con 0x7fe84c0ff1c0 2023-11-16T18:22:42.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.931+0000 7fe8497fa700 1 -- 172.21.15.2:0/397156721 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9568 (secure 0 0 0) 0x7fe83c09acc0 con 0x7fe84c0ff1c0 2023-11-16T18:22:42.932 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:42.932 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24473,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-11-16T18:22:42.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.933+0000 7fe8522fd700 1 -- 172.21.15.2:0/397156721 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe8340a1150 msgr2=0x7fe8340a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:42.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.933+0000 7fe8522fd700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe8340a1150 0x7fe8340a3600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe840005950 tx=0x7fe84000b410 comp rx=0 tx=0).stop 2023-11-16T18:22:42.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.933+0000 7fe8522fd700 1 -- 172.21.15.2:0/397156721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe84c0ff1c0 msgr2=0x7fe84c194fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:42.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.933+0000 7fe8522fd700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe84c0ff1c0 0x7fe84c194fb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe83c006010 tx=0x7fe83c004730 comp rx=0 tx=0).stop 2023-11-16T18:22:42.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.934+0000 7fe8522fd700 1 -- 172.21.15.2:0/397156721 shutdown_connections 2023-11-16T18:22:42.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.934+0000 7fe8522fd700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe8340a1150 0x7fe8340a3600 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:42.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.934+0000 7fe8522fd700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe84c100d40 0x7fe84c19c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:42.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.934+0000 7fe8522fd700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe84c0ff1c0 0x7fe84c194fb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:42.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.934+0000 7fe8522fd700 1 --2- 172.21.15.2:0/397156721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe84c1003b0 0x7fe84c1954f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:42.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.934+0000 7fe8522fd700 1 -- 172.21.15.2:0/397156721 >> 172.21.15.2:0/397156721 conn(0x7fe84c0fa7c0 msgr2=0x7fe84c0fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:42.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.934+0000 7fe8522fd700 1 -- 172.21.15.2:0/397156721 shutdown_connections 2023-11-16T18:22:42.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:42.934+0000 7fe8522fd700 1 -- 172.21.15.2:0/397156721 wait complete. 2023-11-16T18:22:42.936 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 13 2023-11-16T18:22:42.947 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:22:43.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.122+0000 7eff257d5700 1 -- 172.21.15.2:0/2218450065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff20105f80 msgr2=0x7eff201083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.122+0000 7eff257d5700 1 --2- 172.21.15.2:0/2218450065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff20105f80 0x7eff201083e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7eff1800b0a0 tx=0x7eff1800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.122+0000 7eff257d5700 1 -- 172.21.15.2:0/2218450065 shutdown_connections 2023-11-16T18:22:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.122+0000 7eff257d5700 1 --2- 172.21.15.2:0/2218450065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff20105f80 0x7eff201083e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.122+0000 7eff257d5700 1 --2- 172.21.15.2:0/2218450065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff20100d40 0x7eff20103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.122+0000 7eff257d5700 1 --2- 172.21.15.2:0/2218450065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff20103660 0x7eff20105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.122+0000 7eff257d5700 1 -- 172.21.15.2:0/2218450065 >> 172.21.15.2:0/2218450065 conn(0x7eff200fa7a0 msgr2=0x7eff200fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.122+0000 7eff257d5700 1 -- 172.21.15.2:0/2218450065 shutdown_connections 2023-11-16T18:22:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.123+0000 7eff257d5700 1 -- 172.21.15.2:0/2218450065 wait complete. 2023-11-16T18:22:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.123+0000 7eff257d5700 1 Processor -- start 2023-11-16T18:22:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.123+0000 7eff257d5700 1 -- start start 2023-11-16T18:22:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.123+0000 7eff257d5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff20100d40 0x7eff201960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.123+0000 7eff257d5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff20103660 0x7eff201965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.123+0000 7eff257d5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff20105f80 0x7eff2019d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.123+0000 7eff257d5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2019dc30 con 0x7eff20100d40 2023-11-16T18:22:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.123+0000 7eff257d5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2019dd70 con 0x7eff20105f80 2023-11-16T18:22:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.123+0000 7eff257d5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2019df00 con 0x7eff20103660 2023-11-16T18:22:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.123+0000 7eff1effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff20100d40 0x7eff201960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff1effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff20100d40 0x7eff201960b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47714/0 (socket says 172.21.15.2:47714) 2023-11-16T18:22:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff1effd700 1 -- 172.21.15.2:0/3397027477 learned_addr learned my addr 172.21.15.2:0/3397027477 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff1f7fe700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff20105f80 0x7eff2019d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff1e7fc700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff20103660 0x7eff201965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff1effd700 1 -- 172.21.15.2:0/3397027477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff20103660 msgr2=0x7eff201965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff1effd700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff20103660 0x7eff201965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff1effd700 1 -- 172.21.15.2:0/3397027477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff20105f80 msgr2=0x7eff2019d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:43.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff1effd700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff20105f80 0x7eff2019d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff1effd700 1 -- 172.21.15.2:0/3397027477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff18009d00 con 0x7eff20100d40 2023-11-16T18:22:43.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff1f7fe700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff20105f80 0x7eff2019d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:22:43.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff1effd700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff20100d40 0x7eff201960b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7eff1400e9a0 tx=0x7eff1400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:43.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff07fff700 1 -- 172.21.15.2:0/3397027477 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14009790 con 0x7eff20100d40 2023-11-16T18:22:43.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff257d5700 1 -- 172.21.15.2:0/3397027477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff2010bc20 con 0x7eff20100d40 2023-11-16T18:22:43.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff07fff700 1 -- 172.21.15.2:0/3397027477 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff14010430 con 0x7eff20100d40 2023-11-16T18:22:43.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff07fff700 1 -- 172.21.15.2:0/3397027477 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14018680 con 0x7eff20100d40 2023-11-16T18:22:43.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.124+0000 7eff257d5700 1 -- 172.21.15.2:0/3397027477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff2010ba90 con 0x7eff20100d40 2023-11-16T18:22:43.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.125+0000 7eff257d5700 1 -- 172.21.15.2:0/3397027477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff200fc310 con 0x7eff20100d40 2023-11-16T18:22:43.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.126+0000 7eff07fff700 1 -- 172.21.15.2:0/3397027477 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7eff140105a0 con 0x7eff20100d40 2023-11-16T18:22:43.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.127+0000 7eff07fff700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7eff080a1150 0x7eff080a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:43.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.127+0000 7eff1e7fc700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7eff080a1150 0x7eff080a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:43.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.127+0000 7eff07fff700 1 -- 172.21.15.2:0/3397027477 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7eff14014070 con 0x7eff20100d40 2023-11-16T18:22:43.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.127+0000 7eff1e7fc700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7eff080a1150 0x7eff080a3600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7eff10005950 tx=0x7eff1000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:43.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.130+0000 7eff07fff700 1 -- 172.21.15.2:0/3397027477 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7eff14096400 con 0x7eff20100d40 2023-11-16T18:22:43.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.347+0000 7eff257d5700 1 -- 172.21.15.2:0/3397027477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff2002cfe0 con 0x7eff20100d40 2023-11-16T18:22:43.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.348+0000 7eff07fff700 1 -- 172.21.15.2:0/3397027477 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18315 (secure 0 0 0) 0x7eff14099a20 con 0x7eff20100d40 2023-11-16T18:22:43.352 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:43.353 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":78,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:22:03.153518+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:22:43.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.350+0000 7eff257d5700 1 -- 172.21.15.2:0/3397027477 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7eff080a1150 msgr2=0x7eff080a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:43.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.350+0000 7eff257d5700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7eff080a1150 0x7eff080a3600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7eff10005950 tx=0x7eff1000b410 comp rx=0 tx=0).stop 2023-11-16T18:22:43.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.350+0000 7eff257d5700 1 -- 172.21.15.2:0/3397027477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff20100d40 msgr2=0x7eff201960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:43.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.350+0000 7eff257d5700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff20100d40 0x7eff201960b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7eff1400e9a0 tx=0x7eff1400ecb0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.350+0000 7eff257d5700 1 -- 172.21.15.2:0/3397027477 shutdown_connections 2023-11-16T18:22:43.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.350+0000 7eff257d5700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7eff080a1150 0x7eff080a3600 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.350+0000 7eff257d5700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff20105f80 0x7eff2019d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.350+0000 7eff257d5700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff20100d40 0x7eff201960b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.350+0000 7eff257d5700 1 --2- 172.21.15.2:0/3397027477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff20103660 0x7eff201965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.350+0000 7eff257d5700 1 -- 172.21.15.2:0/3397027477 >> 172.21.15.2:0/3397027477 conn(0x7eff200fa7a0 msgr2=0x7eff201019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:43.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.351+0000 7eff257d5700 1 -- 172.21.15.2:0/3397027477 shutdown_connections 2023-11-16T18:22:43.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.351+0000 7eff257d5700 1 -- 172.21.15.2:0/3397027477 wait complete. 2023-11-16T18:22:43.370 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-16T18:22:43.370 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T18:22:43.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.502+0000 7fc04262a700 1 -- 172.21.15.2:0/3803300365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc03c1036a0 msgr2=0x7fc03c105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:43.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.502+0000 7fc04262a700 1 --2- 172.21.15.2:0/3803300365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc03c1036a0 0x7fc03c105a80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc030009990 tx=0x7fc030009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.502+0000 7fc04262a700 1 -- 172.21.15.2:0/3803300365 shutdown_connections 2023-11-16T18:22:43.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.502+0000 7fc04262a700 1 --2- 172.21.15.2:0/3803300365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc03c100d80 0x7fc03c103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.502+0000 7fc04262a700 1 --2- 172.21.15.2:0/3803300365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc03c105fc0 0x7fc03c1083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.502+0000 7fc04262a700 1 --2- 172.21.15.2:0/3803300365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc03c1036a0 0x7fc03c105a80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.502+0000 7fc04262a700 1 -- 172.21.15.2:0/3803300365 >> 172.21.15.2:0/3803300365 conn(0x7fc03c0fa7a0 msgr2=0x7fc03c0fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:43.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.502+0000 7fc04262a700 1 -- 172.21.15.2:0/3803300365 shutdown_connections 2023-11-16T18:22:43.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.503+0000 7fc04262a700 1 -- 172.21.15.2:0/3803300365 wait complete. 2023-11-16T18:22:43.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.503+0000 7fc04262a700 1 Processor -- start 2023-11-16T18:22:43.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.503+0000 7fc04262a700 1 -- start start 2023-11-16T18:22:43.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc04262a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc03c100d80 0x7fc03c195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:43.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc04262a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc03c1036a0 0x7fc03c1964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:43.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc04262a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc03c105fc0 0x7fc03c19d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:43.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc04262a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c19db00 con 0x7fc03c105fc0 2023-11-16T18:22:43.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc04262a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c19dc40 con 0x7fc03c100d80 2023-11-16T18:22:43.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc04262a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c19ddd0 con 0x7fc03c1036a0 2023-11-16T18:22:43.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc040bc7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc03c105fc0 0x7fc03c19d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:43.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc040bc7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc03c105fc0 0x7fc03c19d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47732/0 (socket says 172.21.15.2:47732) 2023-11-16T18:22:43.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc03bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc03c100d80 0x7fc03c195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:43.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc040bc7700 1 -- 172.21.15.2:0/2982111617 learned_addr learned my addr 172.21.15.2:0/2982111617 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:43.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc03b7fe700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc03c1036a0 0x7fc03c1964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:43.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc040bc7700 1 -- 172.21.15.2:0/2982111617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc03c1036a0 msgr2=0x7fc03c1964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:43.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc040bc7700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc03c1036a0 0x7fc03c1964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc040bc7700 1 -- 172.21.15.2:0/2982111617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc03c100d80 msgr2=0x7fc03c195f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:43.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.504+0000 7fc040bc7700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc03c100d80 0x7fc03c195f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.505+0000 7fc040bc7700 1 -- 172.21.15.2:0/2982111617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0300095f0 con 0x7fc03c105fc0 2023-11-16T18:22:43.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.505+0000 7fc03bfff700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc03c100d80 0x7fc03c195f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:22:43.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.505+0000 7fc040bc7700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc03c105fc0 0x7fc03c19d530 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc034007ee0 tx=0x7fc03400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:43.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.505+0000 7fc0397fa700 1 -- 172.21.15.2:0/2982111617 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03400dd20 con 0x7fc03c105fc0 2023-11-16T18:22:43.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.505+0000 7fc0397fa700 1 -- 172.21.15.2:0/2982111617 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc03400f040 con 0x7fc03c105fc0 2023-11-16T18:22:43.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.505+0000 7fc04262a700 1 -- 172.21.15.2:0/2982111617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03c10bc60 con 0x7fc03c105fc0 2023-11-16T18:22:43.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.505+0000 7fc0397fa700 1 -- 172.21.15.2:0/2982111617 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0340048a0 con 0x7fc03c105fc0 2023-11-16T18:22:43.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.505+0000 7fc04262a700 1 -- 172.21.15.2:0/2982111617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03c10bad0 con 0x7fc03c105fc0 2023-11-16T18:22:43.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.506+0000 7fc04262a700 1 -- 172.21.15.2:0/2982111617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03c0fc310 con 0x7fc03c105fc0 2023-11-16T18:22:43.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.507+0000 7fc0397fa700 1 -- 172.21.15.2:0/2982111617 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc034003a00 con 0x7fc03c105fc0 2023-11-16T18:22:43.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.508+0000 7fc0397fa700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc0240a1150 0x7fc0240a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:43.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.508+0000 7fc0397fa700 1 -- 172.21.15.2:0/2982111617 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc0340cc500 con 0x7fc03c105fc0 2023-11-16T18:22:43.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.508+0000 7fc03bfff700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc0240a1150 0x7fc0240a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:43.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.508+0000 7fc03bfff700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc0240a1150 0x7fc0240a3600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc02c005950 tx=0x7fc02c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:43.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.512+0000 7fc0397fa700 1 -- 172.21.15.2:0/2982111617 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc0340960f0 con 0x7fc03c105fc0 2023-11-16T18:22:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.722+0000 7fc04262a700 1 -- 172.21.15.2:0/2982111617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc03c067ea0 con 0x7fc03c105fc0 2023-11-16T18:22:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.722+0000 7fc0397fa700 1 -- 172.21.15.2:0/2982111617 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc034099710 con 0x7fc03c105fc0 2023-11-16T18:22:43.726 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:22:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.724+0000 7fc04262a700 1 -- 172.21.15.2:0/2982111617 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc0240a1150 msgr2=0x7fc0240a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.724+0000 7fc04262a700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc0240a1150 0x7fc0240a3600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc02c005950 tx=0x7fc02c00b410 comp rx=0 tx=0).stop 2023-11-16T18:22:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.724+0000 7fc04262a700 1 -- 172.21.15.2:0/2982111617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc03c105fc0 msgr2=0x7fc03c19d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.724+0000 7fc04262a700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc03c105fc0 0x7fc03c19d530 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc034007ee0 tx=0x7fc03400d3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.725+0000 7fc04262a700 1 -- 172.21.15.2:0/2982111617 shutdown_connections 2023-11-16T18:22:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.725+0000 7fc04262a700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc0240a1150 0x7fc0240a3600 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.725+0000 7fc04262a700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc03c105fc0 0x7fc03c19d530 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.725+0000 7fc04262a700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc03c100d80 0x7fc03c195f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.725+0000 7fc04262a700 1 --2- 172.21.15.2:0/2982111617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc03c1036a0 0x7fc03c1964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.725+0000 7fc04262a700 1 -- 172.21.15.2:0/2982111617 >> 172.21.15.2:0/2982111617 conn(0x7fc03c0fa7a0 msgr2=0x7fc03c101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.725+0000 7fc04262a700 1 -- 172.21.15.2:0/2982111617 shutdown_connections 2023-11-16T18:22:43.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.725+0000 7fc04262a700 1 -- 172.21.15.2:0/2982111617 wait complete. 2023-11-16T18:22:43.738 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-16T18:22:43.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.871+0000 7fb7062fa700 1 -- 172.21.15.2:0/1263043161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7001003e0 msgr2=0x7fb700100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:43.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.871+0000 7fb7062fa700 1 --2- 172.21.15.2:0/1263043161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7001003e0 0x7fb700100830 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009990 tx=0x7fb6f4009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.871+0000 7fb7062fa700 1 -- 172.21.15.2:0/1263043161 shutdown_connections 2023-11-16T18:22:43.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.871+0000 7fb7062fa700 1 --2- 172.21.15.2:0/1263043161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7000ff1f0 0x7fb7000ff600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.871+0000 7fb7062fa700 1 --2- 172.21.15.2:0/1263043161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb700100d70 0x7fb700107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.871+0000 7fb7062fa700 1 --2- 172.21.15.2:0/1263043161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7001003e0 0x7fb700100830 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.871+0000 7fb7062fa700 1 -- 172.21.15.2:0/1263043161 >> 172.21.15.2:0/1263043161 conn(0x7fb7000fa7d0 msgr2=0x7fb7000fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:43.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.871+0000 7fb7062fa700 1 -- 172.21.15.2:0/1263043161 shutdown_connections 2023-11-16T18:22:43.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.871+0000 7fb7062fa700 1 -- 172.21.15.2:0/1263043161 wait complete. 2023-11-16T18:22:43.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.872+0000 7fb7062fa700 1 Processor -- start 2023-11-16T18:22:43.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.872+0000 7fb7062fa700 1 -- start start 2023-11-16T18:22:43.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.872+0000 7fb7062fa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7000ff1f0 0x7fb700196000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:43.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.872+0000 7fb7062fa700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7001003e0 0x7fb700196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:43.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.872+0000 7fb7062fa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb700100d70 0x7fb70019d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:43.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.872+0000 7fb7062fa700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70019db80 con 0x7fb700100d70 2023-11-16T18:22:43.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.872+0000 7fb7062fa700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70019dcc0 con 0x7fb7000ff1f0 2023-11-16T18:22:43.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.872+0000 7fb7062fa700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70019de50 con 0x7fb7001003e0 2023-11-16T18:22:43.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.872+0000 7fb704897700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb700100d70 0x7fb70019d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:43.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb704897700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb700100d70 0x7fb70019d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47746/0 (socket says 172.21.15.2:47746) 2023-11-16T18:22:43.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb704897700 1 -- 172.21.15.2:0/2747830740 learned_addr learned my addr 172.21.15.2:0/2747830740 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:43.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb6fffff700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7000ff1f0 0x7fb700196000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb6ff7fe700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7001003e0 0x7fb700196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb704897700 1 -- 172.21.15.2:0/2747830740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7001003e0 msgr2=0x7fb700196540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb704897700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7001003e0 0x7fb700196540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb704897700 1 -- 172.21.15.2:0/2747830740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7000ff1f0 msgr2=0x7fb700196000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb704897700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7000ff1f0 0x7fb700196000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb704897700 1 -- 172.21.15.2:0/2747830740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f40095f0 con 0x7fb700100d70 2023-11-16T18:22:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb6ff7fe700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7001003e0 0x7fb700196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:22:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb704897700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb700100d70 0x7fb70019d5b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb6f800ef30 tx=0x7fb6f800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb6fd7fa700 1 -- 172.21.15.2:0/2747830740 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8018070 con 0x7fb700100d70 2023-11-16T18:22:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb6fd7fa700 1 -- 172.21.15.2:0/2747830740 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6f800f040 con 0x7fb700100d70 2023-11-16T18:22:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb6fd7fa700 1 -- 172.21.15.2:0/2747830740 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f800cdb0 con 0x7fb700100d70 2023-11-16T18:22:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb7062fa700 1 -- 172.21.15.2:0/2747830740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70010ab50 con 0x7fb700100d70 2023-11-16T18:22:43.878 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.873+0000 7fb7062fa700 1 -- 172.21.15.2:0/2747830740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70010a9c0 con 0x7fb700100d70 2023-11-16T18:22:43.878 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.874+0000 7fb7062fa700 1 -- 172.21.15.2:0/2747830740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70019e490 con 0x7fb700100d70 2023-11-16T18:22:43.879 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.875+0000 7fb6fd7fa700 1 -- 172.21.15.2:0/2747830740 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fb6f8003a00 con 0x7fb700100d70 2023-11-16T18:22:43.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.876+0000 7fb6fd7fa700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb6e80a11d0 0x7fb6e80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:43.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.876+0000 7fb6fd7fa700 1 -- 172.21.15.2:0/2747830740 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb6f80cc940 con 0x7fb700100d70 2023-11-16T18:22:43.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.876+0000 7fb6fffff700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb6e80a11d0 0x7fb6e80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:43.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.876+0000 7fb6fffff700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb6e80a11d0 0x7fb6e80a3680 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb700100240 tx=0x7fb6f0007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:43.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:43.879+0000 7fb6fd7fa700 1 -- 172.21.15.2:0/2747830740 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb6f80965c0 con 0x7fb700100d70 2023-11-16T18:22:44.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.089+0000 7fb7062fa700 1 -- 172.21.15.2:0/2747830740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb70019ea40 con 0x7fb700100d70 2023-11-16T18:22:44.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.089+0000 7fb6fd7fa700 1 -- 172.21.15.2:0/2747830740 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb6f8099be0 con 0x7fb700100d70 2023-11-16T18:22:44.090 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:44.090 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2023-11-16T18:22:44.091 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:22:44.091 INFO:teuthology.orchestra.run.smithi002.stdout: "poolnum": 1, 2023-11-16T18:22:44.091 INFO:teuthology.orchestra.run.smithi002.stdout: "poolname": ".mgr" 2023-11-16T18:22:44.091 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:22:44.091 INFO:teuthology.orchestra.run.smithi002.stdout:] 2023-11-16T18:22:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.091+0000 7fb7062fa700 1 -- 172.21.15.2:0/2747830740 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb6e80a11d0 msgr2=0x7fb6e80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.091+0000 7fb7062fa700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb6e80a11d0 0x7fb6e80a3680 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb700100240 tx=0x7fb6f0007400 comp rx=0 tx=0).stop 2023-11-16T18:22:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.091+0000 7fb7062fa700 1 -- 172.21.15.2:0/2747830740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb700100d70 msgr2=0x7fb70019d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.091+0000 7fb7062fa700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb700100d70 0x7fb70019d5b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb6f800ef30 tx=0x7fb6f800c3a0 comp rx=0 tx=0).stop 2023-11-16T18:22:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.092+0000 7fb7062fa700 1 -- 172.21.15.2:0/2747830740 shutdown_connections 2023-11-16T18:22:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.092+0000 7fb7062fa700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb6e80a11d0 0x7fb6e80a3680 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.092+0000 7fb7062fa700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb700100d70 0x7fb70019d5b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.092+0000 7fb7062fa700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7000ff1f0 0x7fb700196000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.092+0000 7fb7062fa700 1 --2- 172.21.15.2:0/2747830740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7001003e0 0x7fb700196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.092+0000 7fb7062fa700 1 -- 172.21.15.2:0/2747830740 >> 172.21.15.2:0/2747830740 conn(0x7fb7000fa7d0 msgr2=0x7fb700068e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.092+0000 7fb7062fa700 1 -- 172.21.15.2:0/2747830740 shutdown_connections 2023-11-16T18:22:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.092+0000 7fb7062fa700 1 -- 172.21.15.2:0/2747830740 wait complete. 2023-11-16T18:22:44.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:43 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:44.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:43 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/397156721' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:44.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:43 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3397027477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:44.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:43 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2982111617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:22:44.106 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-16T18:22:44.106 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-16T18:22:44.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:43 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:44.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:43 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/397156721' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:44.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:43 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3397027477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:44.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:43 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2982111617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:22:44.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.243+0000 7fec027f0700 1 -- 172.21.15.2:0/3449355481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febfc103640 msgr2=0x7febfc105a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:44.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.243+0000 7fec027f0700 1 --2- 172.21.15.2:0/3449355481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febfc103640 0x7febfc105a20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7febf0009990 tx=0x7febf0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:44.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.243+0000 7fec027f0700 1 -- 172.21.15.2:0/3449355481 shutdown_connections 2023-11-16T18:22:44.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.243+0000 7fec027f0700 1 --2- 172.21.15.2:0/3449355481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febfc105f60 0x7febfc1083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:44.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.243+0000 7fec027f0700 1 --2- 172.21.15.2:0/3449355481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febfc100d20 0x7febfc103100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:44.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.243+0000 7fec027f0700 1 --2- 172.21.15.2:0/3449355481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febfc103640 0x7febfc105a20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:44.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.243+0000 7fec027f0700 1 -- 172.21.15.2:0/3449355481 >> 172.21.15.2:0/3449355481 conn(0x7febfc0fa7a0 msgr2=0x7febfc0fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:44.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.243+0000 7fec027f0700 1 -- 172.21.15.2:0/3449355481 shutdown_connections 2023-11-16T18:22:44.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.243+0000 7fec027f0700 1 -- 172.21.15.2:0/3449355481 wait complete. 2023-11-16T18:22:44.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.244+0000 7fec027f0700 1 Processor -- start 2023-11-16T18:22:44.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.244+0000 7fec027f0700 1 -- start start 2023-11-16T18:22:44.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.244+0000 7fec027f0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febfc100d20 0x7febfc195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:44.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.244+0000 7fec027f0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febfc103640 0x7febfc1964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:44.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.244+0000 7fec027f0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febfc105f60 0x7febfc19d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:44.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.244+0000 7fec027f0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc19dae0 con 0x7febfc100d20 2023-11-16T18:22:44.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.244+0000 7fec027f0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc19dc20 con 0x7febfc103640 2023-11-16T18:22:44.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.244+0000 7fec027f0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc19ddb0 con 0x7febfc105f60 2023-11-16T18:22:44.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.244+0000 7febfbfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febfc100d20 0x7febfc195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:44.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.244+0000 7febfbfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febfc100d20 0x7febfc195f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47756/0 (socket says 172.21.15.2:47756) 2023-11-16T18:22:44.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.244+0000 7febfbfff700 1 -- 172.21.15.2:0/16609298 learned_addr learned my addr 172.21.15.2:0/16609298 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:44.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.244+0000 7febfb7fe700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febfc103640 0x7febfc1964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:44.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7fec00d8d700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febfc105f60 0x7febfc19d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:44.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7febfbfff700 1 -- 172.21.15.2:0/16609298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febfc105f60 msgr2=0x7febfc19d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:44.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7febfbfff700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febfc105f60 0x7febfc19d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:44.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7febfbfff700 1 -- 172.21.15.2:0/16609298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febfc103640 msgr2=0x7febfc1964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:44.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7febfbfff700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febfc103640 0x7febfc1964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:44.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7febfbfff700 1 -- 172.21.15.2:0/16609298 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf00095f0 con 0x7febfc100d20 2023-11-16T18:22:44.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7fec00d8d700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febfc105f60 0x7febfc19d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:22:44.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7febfb7fe700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febfc103640 0x7febfc1964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:22:44.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7febfbfff700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febfc100d20 0x7febfc195f60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7febec00e9a0 tx=0x7febec00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:44.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7febf97fa700 1 -- 172.21.15.2:0/16609298 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec009790 con 0x7febfc100d20 2023-11-16T18:22:44.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7febf97fa700 1 -- 172.21.15.2:0/16609298 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7febec00cc80 con 0x7febfc100d20 2023-11-16T18:22:44.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7febf97fa700 1 -- 172.21.15.2:0/16609298 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec01b9e0 con 0x7febfc100d20 2023-11-16T18:22:44.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7fec027f0700 1 -- 172.21.15.2:0/16609298 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febfc10bc60 con 0x7febfc100d20 2023-11-16T18:22:44.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.245+0000 7fec027f0700 1 -- 172.21.15.2:0/16609298 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febfc10bad0 con 0x7febfc100d20 2023-11-16T18:22:44.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.247+0000 7fec027f0700 1 -- 172.21.15.2:0/16609298 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febfc0fc310 con 0x7febfc100d20 2023-11-16T18:22:44.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.248+0000 7febf97fa700 1 -- 172.21.15.2:0/16609298 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7febec00c7c0 con 0x7febfc100d20 2023-11-16T18:22:44.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.249+0000 7febf97fa700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7febe40a11d0 0x7febe40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:44.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.249+0000 7febfb7fe700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7febe40a11d0 0x7febe40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:44.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.249+0000 7febf97fa700 1 -- 172.21.15.2:0/16609298 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7febec0d06e0 con 0x7febfc100d20 2023-11-16T18:22:44.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.249+0000 7febfb7fe700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7febe40a11d0 0x7febe40a3680 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7febf000b2e0 tx=0x7febf0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:44.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.257+0000 7febf97fa700 1 -- 172.21.15.2:0/16609298 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7febec09a2d0 con 0x7febfc100d20 2023-11-16T18:22:44.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:43 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:44.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:43 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/397156721' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:44.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:43 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3397027477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:44.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:43 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2982111617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:22:44.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:44.471+0000 7fec027f0700 1 -- 172.21.15.2:0/16609298 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7febfc02cfe0 con 0x7febfc100d20 2023-11-16T18:22:45.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:44 smithi040 ceph-mon[100251]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:45.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:44 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2747830740' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-11-16T18:22:45.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:44 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:44 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:44 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:44 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:44 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:44 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:44 smithi188 ceph-mon[100639]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:45.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:44 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2747830740' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-11-16T18:22:45.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:44 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:44 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:44 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:44 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:44 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:44 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:44 smithi002 ceph-mon[98807]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:45.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:44 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2747830740' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-11-16T18:22:45.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:44 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:44 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:44 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:44 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:44 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:44 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:45.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.359+0000 7febf97fa700 1 -- 172.21.15.2:0/16609298 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7febec09d8f0 con 0x7febfc100d20 2023-11-16T18:22:45.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.363+0000 7fec027f0700 1 -- 172.21.15.2:0/16609298 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7febe40a11d0 msgr2=0x7febe40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:45.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.363+0000 7fec027f0700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7febe40a11d0 0x7febe40a3680 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7febf000b2e0 tx=0x7febf0005fb0 comp rx=0 tx=0).stop 2023-11-16T18:22:45.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.363+0000 7fec027f0700 1 -- 172.21.15.2:0/16609298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febfc100d20 msgr2=0x7febfc195f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:45.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.363+0000 7fec027f0700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febfc100d20 0x7febfc195f60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7febec00e9a0 tx=0x7febec00ecb0 comp rx=0 tx=0).stop 2023-11-16T18:22:45.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.364+0000 7fec027f0700 1 -- 172.21.15.2:0/16609298 shutdown_connections 2023-11-16T18:22:45.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.364+0000 7fec027f0700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7febe40a11d0 0x7febe40a3680 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:45.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.364+0000 7fec027f0700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febfc105f60 0x7febfc19d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:45.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.364+0000 7fec027f0700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febfc100d20 0x7febfc195f60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:45.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.364+0000 7fec027f0700 1 --2- 172.21.15.2:0/16609298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febfc103640 0x7febfc1964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:45.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.364+0000 7fec027f0700 1 -- 172.21.15.2:0/16609298 >> 172.21.15.2:0/16609298 conn(0x7febfc0fa7a0 msgr2=0x7febfc101980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:45.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.364+0000 7fec027f0700 1 -- 172.21.15.2:0/16609298 shutdown_connections 2023-11-16T18:22:45.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.364+0000 7fec027f0700 1 -- 172.21.15.2:0/16609298 wait complete. 2023-11-16T18:22:45.367 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' created 2023-11-16T18:22:45.384 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-11-16T18:22:45.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.523+0000 7fc290c5e700 1 -- 172.21.15.2:0/1555225943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc28c103580 msgr2=0x7fc28c105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:45.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.523+0000 7fc290c5e700 1 --2- 172.21.15.2:0/1555225943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc28c103580 0x7fc28c105960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc27c009990 tx=0x7fc27c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:45.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.523+0000 7fc290c5e700 1 -- 172.21.15.2:0/1555225943 shutdown_connections 2023-11-16T18:22:45.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.523+0000 7fc290c5e700 1 --2- 172.21.15.2:0/1555225943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc28c105ea0 0x7fc28c108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:45.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.523+0000 7fc290c5e700 1 --2- 172.21.15.2:0/1555225943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc28c100c60 0x7fc28c103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:45.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.523+0000 7fc290c5e700 1 --2- 172.21.15.2:0/1555225943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc28c103580 0x7fc28c105960 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:45.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.523+0000 7fc290c5e700 1 -- 172.21.15.2:0/1555225943 >> 172.21.15.2:0/1555225943 conn(0x7fc28c0fa720 msgr2=0x7fc28c0fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:45.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.523+0000 7fc290c5e700 1 -- 172.21.15.2:0/1555225943 shutdown_connections 2023-11-16T18:22:45.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.523+0000 7fc290c5e700 1 -- 172.21.15.2:0/1555225943 wait complete. 2023-11-16T18:22:45.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.524+0000 7fc290c5e700 1 Processor -- start 2023-11-16T18:22:45.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.524+0000 7fc290c5e700 1 -- start start 2023-11-16T18:22:45.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.524+0000 7fc290c5e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc28c100c60 0x7fc28c195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:45.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.524+0000 7fc290c5e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc28c103580 0x7fc28c196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:45.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.524+0000 7fc290c5e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc28c105ea0 0x7fc28c19d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:45.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.524+0000 7fc290c5e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c19dab0 con 0x7fc28c100c60 2023-11-16T18:22:45.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc290c5e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c19dbf0 con 0x7fc28c103580 2023-11-16T18:22:45.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc290c5e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c19dd80 con 0x7fc28c105ea0 2023-11-16T18:22:45.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc28a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc28c100c60 0x7fc28c195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:45.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc28a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc28c100c60 0x7fc28c195f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47770/0 (socket says 172.21.15.2:47770) 2023-11-16T18:22:45.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc28a59c700 1 -- 172.21.15.2:0/1600099593 learned_addr learned my addr 172.21.15.2:0/1600099593 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:45.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc289d9b700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc28c103580 0x7fc28c196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:45.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc28ad9d700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc28c105ea0 0x7fc28c19d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:45.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc28a59c700 1 -- 172.21.15.2:0/1600099593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc28c105ea0 msgr2=0x7fc28c19d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:45.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc28a59c700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc28c105ea0 0x7fc28c19d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:45.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc28a59c700 1 -- 172.21.15.2:0/1600099593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc28c103580 msgr2=0x7fc28c196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:45.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc28a59c700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc28c103580 0x7fc28c196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:45.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc28a59c700 1 -- 172.21.15.2:0/1600099593 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc27c0095f0 con 0x7fc28c100c60 2023-11-16T18:22:45.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc28a59c700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc28c100c60 0x7fc28c195f30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc28000e9a0 tx=0x7fc28000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:45.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc27b7fe700 1 -- 172.21.15.2:0/1600099593 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc280009790 con 0x7fc28c100c60 2023-11-16T18:22:45.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc27b7fe700 1 -- 172.21.15.2:0/1600099593 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc28000cbd0 con 0x7fc28c100c60 2023-11-16T18:22:45.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc27b7fe700 1 -- 172.21.15.2:0/1600099593 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc280018680 con 0x7fc28c100c60 2023-11-16T18:22:45.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc290c5e700 1 -- 172.21.15.2:0/1600099593 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c10bb60 con 0x7fc28c100c60 2023-11-16T18:22:45.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.525+0000 7fc290c5e700 1 -- 172.21.15.2:0/1600099593 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c10b9d0 con 0x7fc28c100c60 2023-11-16T18:22:45.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.526+0000 7fc290c5e700 1 -- 172.21.15.2:0/1600099593 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c0fc290 con 0x7fc28c100c60 2023-11-16T18:22:45.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.528+0000 7fc27b7fe700 1 -- 172.21.15.2:0/1600099593 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc28000cd40 con 0x7fc28c100c60 2023-11-16T18:22:45.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.529+0000 7fc27b7fe700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc2740a11d0 0x7fc2740a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:45.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.529+0000 7fc27b7fe700 1 -- 172.21.15.2:0/1600099593 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fc280014070 con 0x7fc28c100c60 2023-11-16T18:22:45.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.529+0000 7fc289d9b700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc2740a11d0 0x7fc2740a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:45.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.529+0000 7fc289d9b700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc2740a11d0 0x7fc2740a3680 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00b2e0 tx=0x7fc27c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:45.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.533+0000 7fc27b7fe700 1 -- 172.21.15.2:0/1600099593 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc2800964f0 con 0x7fc28c100c60 2023-11-16T18:22:45.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:45.746+0000 7fc290c5e700 1 -- 172.21.15.2:0/1600099593 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7fc28c02cfe0 con 0x7fc28c100c60 2023-11-16T18:22:46.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.363+0000 7fc27b7fe700 1 -- 172.21.15.2:0/1600099593 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v80) v1 ==== 171+0+0 (secure 0 0 0) 0x7fc280099b10 con 0x7fc28c100c60 2023-11-16T18:22:46.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.372+0000 7fc290c5e700 1 -- 172.21.15.2:0/1600099593 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc2740a11d0 msgr2=0x7fc2740a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:46.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.372+0000 7fc290c5e700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc2740a11d0 0x7fc2740a3680 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00b2e0 tx=0x7fc27c005fb0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.372+0000 7fc290c5e700 1 -- 172.21.15.2:0/1600099593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc28c100c60 msgr2=0x7fc28c195f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:46.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.372+0000 7fc290c5e700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc28c100c60 0x7fc28c195f30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc28000e9a0 tx=0x7fc28000ecb0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.372+0000 7fc290c5e700 1 -- 172.21.15.2:0/1600099593 shutdown_connections 2023-11-16T18:22:46.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.372+0000 7fc290c5e700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc2740a11d0 0x7fc2740a3680 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.372+0000 7fc290c5e700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc28c105ea0 0x7fc28c19d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.372+0000 7fc290c5e700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc28c100c60 0x7fc28c195f30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.372+0000 7fc290c5e700 1 --2- 172.21.15.2:0/1600099593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc28c103580 0x7fc28c196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.372+0000 7fc290c5e700 1 -- 172.21.15.2:0/1600099593 >> 172.21.15.2:0/1600099593 conn(0x7fc28c0fa720 msgr2=0x7fc28c1018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:46.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.373+0000 7fc290c5e700 1 -- 172.21.15.2:0/1600099593 shutdown_connections 2023-11-16T18:22:46.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.373+0000 7fc290c5e700 1 -- 172.21.15.2:0/1600099593 wait complete. 2023-11-16T18:22:46.379 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' created 2023-11-16T18:22:46.395 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-16T18:22:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:46 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/16609298' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T18:22:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:46 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:46 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:46 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:46 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:46 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/16609298' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-11-16T18:22:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:46 smithi040 ceph-mon[100251]: osdmap e79: 12 total, 12 up, 12 in 2023-11-16T18:22:46.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:46 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1600099593' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T18:22:46.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.533+0000 7f3e7dbab700 1 -- 172.21.15.2:0/1217498998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e78103650 msgr2=0x7f3e78105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:46.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.533+0000 7f3e7dbab700 1 --2- 172.21.15.2:0/1217498998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e78103650 0x7f3e78105a30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c009990 tx=0x7f3e6c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.533+0000 7f3e7dbab700 1 -- 172.21.15.2:0/1217498998 shutdown_connections 2023-11-16T18:22:46.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.533+0000 7f3e7dbab700 1 --2- 172.21.15.2:0/1217498998 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e78105f70 0x7f3e781083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.533+0000 7f3e7dbab700 1 --2- 172.21.15.2:0/1217498998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e78100d30 0x7f3e78103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.536 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.533+0000 7f3e7dbab700 1 --2- 172.21.15.2:0/1217498998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e78103650 0x7f3e78105a30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.536 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.533+0000 7f3e7dbab700 1 -- 172.21.15.2:0/1217498998 >> 172.21.15.2:0/1217498998 conn(0x7f3e780fa7b0 msgr2=0x7f3e780fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:46.536 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.533+0000 7f3e7dbab700 1 -- 172.21.15.2:0/1217498998 shutdown_connections 2023-11-16T18:22:46.536 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.533+0000 7f3e7dbab700 1 -- 172.21.15.2:0/1217498998 wait complete. 2023-11-16T18:22:46.536 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.534+0000 7f3e7dbab700 1 Processor -- start 2023-11-16T18:22:46.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.534+0000 7f3e7dbab700 1 -- start start 2023-11-16T18:22:46.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e7dbab700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e78100d30 0x7f3e78195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:46.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e7dbab700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e78103650 0x7f3e78196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:46.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e7dbab700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e78105f70 0x7f3e7819d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:46.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e7dbab700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7819dad0 con 0x7f3e78100d30 2023-11-16T18:22:46.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e7dbab700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7819dc10 con 0x7f3e78103650 2023-11-16T18:22:46.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e7dbab700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7819dda0 con 0x7f3e78105f70 2023-11-16T18:22:46.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e777fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e78100d30 0x7f3e78195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:46.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e777fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e78100d30 0x7f3e78195f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47788/0 (socket says 172.21.15.2:47788) 2023-11-16T18:22:46.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e777fe700 1 -- 172.21.15.2:0/3936446899 learned_addr learned my addr 172.21.15.2:0/3936446899 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:46.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e76ffd700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e78103650 0x7f3e78196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:46.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e77fff700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e78105f70 0x7f3e7819d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:46.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e777fe700 1 -- 172.21.15.2:0/3936446899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e78105f70 msgr2=0x7f3e7819d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:46.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e777fe700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e78105f70 0x7f3e7819d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e777fe700 1 -- 172.21.15.2:0/3936446899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e78103650 msgr2=0x7f3e78196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:46.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e777fe700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e78103650 0x7f3e78196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e777fe700 1 -- 172.21.15.2:0/3936446899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c0095f0 con 0x7f3e78100d30 2023-11-16T18:22:46.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e76ffd700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e78103650 0x7f3e78196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:22:46.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.535+0000 7f3e777fe700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e78100d30 0x7f3e78195f50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3e6800e9a0 tx=0x7f3e6800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:46.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.536+0000 7f3e74ff9700 1 -- 172.21.15.2:0/3936446899 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e68009790 con 0x7f3e78100d30 2023-11-16T18:22:46.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.536+0000 7f3e74ff9700 1 -- 172.21.15.2:0/3936446899 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e68010430 con 0x7f3e78100d30 2023-11-16T18:22:46.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.536+0000 7f3e7dbab700 1 -- 172.21.15.2:0/3936446899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7810bc70 con 0x7f3e78100d30 2023-11-16T18:22:46.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.536+0000 7f3e74ff9700 1 -- 172.21.15.2:0/3936446899 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e68018680 con 0x7f3e78100d30 2023-11-16T18:22:46.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.536+0000 7f3e7dbab700 1 -- 172.21.15.2:0/3936446899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e7810bae0 con 0x7f3e78100d30 2023-11-16T18:22:46.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.537+0000 7f3e7dbab700 1 -- 172.21.15.2:0/3936446899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e780fc320 con 0x7f3e78100d30 2023-11-16T18:22:46.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.539+0000 7f3e74ff9700 1 -- 172.21.15.2:0/3936446899 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f3e680105a0 con 0x7f3e78100d30 2023-11-16T18:22:46.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.539+0000 7f3e74ff9700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3e600a1150 0x7f3e600a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:46.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.540+0000 7f3e76ffd700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3e600a1150 0x7f3e600a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:46.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.540+0000 7f3e74ff9700 1 -- 172.21.15.2:0/3936446899 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f3e68014070 con 0x7f3e78100d30 2023-11-16T18:22:46.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.540+0000 7f3e76ffd700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3e600a1150 0x7f3e600a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00b2e0 tx=0x7f3e6c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:46.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.544+0000 7f3e74ff9700 1 -- 172.21.15.2:0/3936446899 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3e68096410 con 0x7f3e78100d30 2023-11-16T18:22:46.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:46 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/16609298' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T18:22:46.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:46 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:46.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:46 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:46.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:46 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:46.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:46 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:46.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:46 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/16609298' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-11-16T18:22:46.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:46 smithi188 ceph-mon[100639]: osdmap e79: 12 total, 12 up, 12 in 2023-11-16T18:22:46.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:46 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1600099593' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T18:22:46.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:46 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/16609298' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T18:22:46.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:46 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:46.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:46 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:46.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:46 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:46.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:46 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:22:46.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:46 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/16609298' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-11-16T18:22:46.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:46 smithi002 ceph-mon[98807]: osdmap e79: 12 total, 12 up, 12 in 2023-11-16T18:22:46.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:46 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1600099593' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T18:22:46.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.804+0000 7f3e7dbab700 1 -- 172.21.15.2:0/3936446899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3e7802cfe0 con 0x7f3e78100d30 2023-11-16T18:22:46.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.811+0000 7f3e74ff9700 1 -- 172.21.15.2:0/3936446899 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f3e680260c0 con 0x7f3e78100d30 2023-11-16T18:22:46.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.815+0000 7f3e7dbab700 1 -- 172.21.15.2:0/3936446899 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3e600a1150 msgr2=0x7f3e600a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:46.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.815+0000 7f3e7dbab700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3e600a1150 0x7f3e600a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00b2e0 tx=0x7f3e6c005fb0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.815+0000 7f3e7dbab700 1 -- 172.21.15.2:0/3936446899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e78100d30 msgr2=0x7f3e78195f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:46.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.815+0000 7f3e7dbab700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e78100d30 0x7f3e78195f50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3e6800e9a0 tx=0x7f3e6800ecb0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.816+0000 7f3e7dbab700 1 -- 172.21.15.2:0/3936446899 shutdown_connections 2023-11-16T18:22:46.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.816+0000 7f3e7dbab700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3e600a1150 0x7f3e600a3600 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.816+0000 7f3e7dbab700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e78105f70 0x7f3e7819d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.816+0000 7f3e7dbab700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e78100d30 0x7f3e78195f50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.816+0000 7f3e7dbab700 1 --2- 172.21.15.2:0/3936446899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e78103650 0x7f3e78196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.816+0000 7f3e7dbab700 1 -- 172.21.15.2:0/3936446899 >> 172.21.15.2:0/3936446899 conn(0x7f3e780fa7b0 msgr2=0x7f3e78101990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:46.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.816+0000 7f3e7dbab700 1 -- 172.21.15.2:0/3936446899 shutdown_connections 2023-11-16T18:22:46.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.816+0000 7f3e7dbab700 1 -- 172.21.15.2:0/3936446899 wait complete. 2023-11-16T18:22:46.830 INFO:teuthology.orchestra.run.smithi002.stderr:new fs with metadata pool 2 and data pool 3 2023-11-16T18:22:46.846 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-11-16T18:22:46.846 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-11-16T18:22:46.991 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.988+0000 7f99b2906700 1 -- 172.21.15.2:0/3396386437 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99ac0734d0 msgr2=0x7f99ac073940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:46.991 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.988+0000 7f99b2906700 1 --2- 172.21.15.2:0/3396386437 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99ac0734d0 0x7f99ac073940 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f99a0009990 tx=0x7f99a0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.991 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.988+0000 7f99b2906700 1 -- 172.21.15.2:0/3396386437 shutdown_connections 2023-11-16T18:22:46.992 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.988+0000 7f99b2906700 1 --2- 172.21.15.2:0/3396386437 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99ac1081b0 0x7f99ac10a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.992 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.988+0000 7f99b2906700 1 --2- 172.21.15.2:0/3396386437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ac074ab0 0x7f99ac072f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.992 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.988+0000 7f99b2906700 1 --2- 172.21.15.2:0/3396386437 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99ac0734d0 0x7f99ac073940 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:46.992 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.988+0000 7f99b2906700 1 -- 172.21.15.2:0/3396386437 >> 172.21.15.2:0/3396386437 conn(0x7f99ac0fb8e0 msgr2=0x7f99ac0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:46.992 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.988+0000 7f99b2906700 1 -- 172.21.15.2:0/3396386437 shutdown_connections 2023-11-16T18:22:46.992 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.988+0000 7f99b2906700 1 -- 172.21.15.2:0/3396386437 wait complete. 2023-11-16T18:22:46.992 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.988+0000 7f99b2906700 1 Processor -- start 2023-11-16T18:22:46.992 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.988+0000 7f99b2906700 1 -- start start 2023-11-16T18:22:46.992 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99b2906700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99ac0734d0 0x7f99ac198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:46.992 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99b2906700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ac074ab0 0x7f99ac198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:46.992 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99b2906700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99ac1081b0 0x7f99ac19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:46.992 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99b2906700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac19fcc0 con 0x7f99ac0734d0 2023-11-16T18:22:46.993 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99b2906700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac19fe00 con 0x7f99ac074ab0 2023-11-16T18:22:46.993 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99b2906700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac19ff90 con 0x7f99ac1081b0 2023-11-16T18:22:47.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99abfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99ac0734d0 0x7f99ac198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:47.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99abfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99ac0734d0 0x7f99ac198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47806/0 (socket says 172.21.15.2:47806) 2023-11-16T18:22:47.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99abfff700 1 -- 172.21.15.2:0/1635115441 learned_addr learned my addr 172.21.15.2:0/1635115441 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:47.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99ab7fe700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ac074ab0 0x7f99ac198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:47.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99b0ea3700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99ac1081b0 0x7f99ac19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:47.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99abfff700 1 -- 172.21.15.2:0/1635115441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99ac1081b0 msgr2=0x7f99ac19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:47.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99abfff700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99ac1081b0 0x7f99ac19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:47.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99abfff700 1 -- 172.21.15.2:0/1635115441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ac074ab0 msgr2=0x7f99ac198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:47.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99abfff700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ac074ab0 0x7f99ac198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:47.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99abfff700 1 -- 172.21.15.2:0/1635115441 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a00095f0 con 0x7f99ac0734d0 2023-11-16T18:22:47.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99ab7fe700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ac074ab0 0x7f99ac198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:22:47.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99b0ea3700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99ac1081b0 0x7f99ac19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:22:47.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.989+0000 7f99abfff700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99ac0734d0 0x7f99ac198140 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f999c00d980 tx=0x7f999c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:47.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.990+0000 7f99a97fa700 1 -- 172.21.15.2:0/1635115441 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c004220 con 0x7f99ac0734d0 2023-11-16T18:22:47.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.990+0000 7f99a97fa700 1 -- 172.21.15.2:0/1635115441 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f999c007640 con 0x7f99ac0734d0 2023-11-16T18:22:47.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.990+0000 7f99a97fa700 1 -- 172.21.15.2:0/1635115441 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c005610 con 0x7f99ac0734d0 2023-11-16T18:22:47.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.990+0000 7f99b2906700 1 -- 172.21.15.2:0/1635115441 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ac10de30 con 0x7f99ac0734d0 2023-11-16T18:22:47.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.990+0000 7f99b2906700 1 -- 172.21.15.2:0/1635115441 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ac10dca0 con 0x7f99ac0734d0 2023-11-16T18:22:47.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.991+0000 7f99b2906700 1 -- 172.21.15.2:0/1635115441 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ac1068f0 con 0x7f99ac0734d0 2023-11-16T18:22:47.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.992+0000 7f99a97fa700 1 -- 172.21.15.2:0/1635115441 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f999c004750 con 0x7f99ac0734d0 2023-11-16T18:22:47.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.993+0000 7f99a97fa700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f99940a11d0 0x7f99940a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:47.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.993+0000 7f99ab7fe700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f99940a11d0 0x7f99940a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:47.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.993+0000 7f99a97fa700 1 -- 172.21.15.2:0/1635115441 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f999c0ccae0 con 0x7f99ac0734d0 2023-11-16T18:22:47.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.993+0000 7f99ab7fe700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f99940a11d0 0x7f99940a3680 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f99a000b5c0 tx=0x7f99a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:47.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:46.997+0000 7f99a97fa700 1 -- 172.21.15.2:0/1635115441 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f999c096380 con 0x7f99ac0734d0 2023-11-16T18:22:47.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.215+0000 7f99b2906700 1 -- 172.21.15.2:0/1635115441 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f99ac02cfe0 con 0x7f99ac0734d0 2023-11-16T18:22:47.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:47 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:22:47.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:47 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:22:47.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:47 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:22:47.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:47 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:22:47.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:47 smithi040 ceph-mon[100251]: pgmap v253: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:47.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:47 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1600099593' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-11-16T18:22:47.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:47 smithi040 ceph-mon[100251]: osdmap e80: 12 total, 12 up, 12 in 2023-11-16T18:22:47.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:47 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3936446899' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T18:22:47.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:47 smithi040 ceph-mon[100251]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-11-16T18:22:47.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:47 smithi040 ceph-mon[100251]: osdmap e81: 12 total, 12 up, 12 in 2023-11-16T18:22:47.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:47 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3936446899' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-11-16T18:22:47.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:47 smithi040 ceph-mon[100251]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-11-16T18:22:47.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:47 smithi040 ceph-mon[100251]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-11-16T18:22:47.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:47 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:22:47.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:47 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:22:47.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:47 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:22:47.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:47 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:22:47.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:47 smithi188 ceph-mon[100639]: pgmap v253: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:47.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:47 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1600099593' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-11-16T18:22:47.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:47 smithi188 ceph-mon[100639]: osdmap e80: 12 total, 12 up, 12 in 2023-11-16T18:22:47.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:47 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3936446899' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T18:22:47.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:47 smithi188 ceph-mon[100639]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-11-16T18:22:47.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:47 smithi188 ceph-mon[100639]: osdmap e81: 12 total, 12 up, 12 in 2023-11-16T18:22:47.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:47 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3936446899' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-11-16T18:22:47.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:47 smithi188 ceph-mon[100639]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-11-16T18:22:47.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:47 smithi188 ceph-mon[100639]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-11-16T18:22:47.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:47 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:22:47.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:47 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:22:47.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:47 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:22:47.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:47 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:22:47.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:47 smithi002 ceph-mon[98807]: pgmap v253: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:47.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:47 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1600099593' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-11-16T18:22:47.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:47 smithi002 ceph-mon[98807]: osdmap e80: 12 total, 12 up, 12 in 2023-11-16T18:22:47.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:47 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3936446899' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T18:22:47.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:47 smithi002 ceph-mon[98807]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-11-16T18:22:47.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:47 smithi002 ceph-mon[98807]: osdmap e81: 12 total, 12 up, 12 in 2023-11-16T18:22:47.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:47 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3936446899' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-11-16T18:22:47.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:47 smithi002 ceph-mon[98807]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-11-16T18:22:47.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:47 smithi002 ceph-mon[98807]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-11-16T18:22:47.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.814+0000 7f99a97fa700 1 -- 172.21.15.2:0/1635115441 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v82) v1 ==== 157+0+0 (secure 0 0 0) 0x7f999c016020 con 0x7f99ac0734d0 2023-11-16T18:22:47.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.816+0000 7f99b2906700 1 -- 172.21.15.2:0/1635115441 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f99940a11d0 msgr2=0x7f99940a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:47.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.816+0000 7f99b2906700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f99940a11d0 0x7f99940a3680 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f99a000b5c0 tx=0x7f99a0005fd0 comp rx=0 tx=0).stop 2023-11-16T18:22:47.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.816+0000 7f99b2906700 1 -- 172.21.15.2:0/1635115441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99ac0734d0 msgr2=0x7f99ac198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:47.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.816+0000 7f99b2906700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99ac0734d0 0x7f99ac198140 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f999c00d980 tx=0x7f999c00dc90 comp rx=0 tx=0).stop 2023-11-16T18:22:47.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.816+0000 7f99b2906700 1 -- 172.21.15.2:0/1635115441 shutdown_connections 2023-11-16T18:22:47.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.816+0000 7f99b2906700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f99940a11d0 0x7f99940a3680 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:47.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.816+0000 7f99b2906700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99ac1081b0 0x7f99ac19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:47.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.816+0000 7f99b2906700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99ac0734d0 0x7f99ac198140 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:47.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.816+0000 7f99b2906700 1 --2- 172.21.15.2:0/1635115441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ac074ab0 0x7f99ac198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:47.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.816+0000 7f99b2906700 1 -- 172.21.15.2:0/1635115441 >> 172.21.15.2:0/1635115441 conn(0x7f99ac0fb8e0 msgr2=0x7f99ac0fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:47.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.817+0000 7f99b2906700 1 -- 172.21.15.2:0/1635115441 shutdown_connections 2023-11-16T18:22:47.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.817+0000 7f99b2906700 1 -- 172.21.15.2:0/1635115441 wait complete. 2023-11-16T18:22:47.830 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-11-16T18:22:47.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.966+0000 7f2867071700 1 -- 172.21.15.2:0/1046867526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2860100d40 msgr2=0x7f2860107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:47.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.966+0000 7f2867071700 1 --2- 172.21.15.2:0/1046867526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2860100d40 0x7f2860107600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f285800b0a0 tx=0x7f285800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:47.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.966+0000 7f2867071700 1 -- 172.21.15.2:0/1046867526 shutdown_connections 2023-11-16T18:22:47.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.966+0000 7f2867071700 1 --2- 172.21.15.2:0/1046867526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2860100d40 0x7f2860107600 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:47.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.966+0000 7f2867071700 1 --2- 172.21.15.2:0/1046867526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28600ff1c0 0x7f28600ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:47.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.966+0000 7f2867071700 1 --2- 172.21.15.2:0/1046867526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28601003b0 0x7f2860100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:47.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.966+0000 7f2867071700 1 -- 172.21.15.2:0/1046867526 >> 172.21.15.2:0/1046867526 conn(0x7f28600fa7a0 msgr2=0x7f28600fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:47.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.966+0000 7f2867071700 1 -- 172.21.15.2:0/1046867526 shutdown_connections 2023-11-16T18:22:47.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.966+0000 7f2867071700 1 -- 172.21.15.2:0/1046867526 wait complete. 2023-11-16T18:22:47.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.966+0000 7f2867071700 1 Processor -- start 2023-11-16T18:22:47.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.966+0000 7f2867071700 1 -- start start 2023-11-16T18:22:47.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.966+0000 7f2867071700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28600ff1c0 0x7f2860103100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:47.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.966+0000 7f2867071700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28601003b0 0x7f2860101790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2867071700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2860100d40 0x7f2860101cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2867071700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2860102470 con 0x7f28600ff1c0 2023-11-16T18:22:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2867071700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28601025e0 con 0x7f2860100d40 2023-11-16T18:22:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2867071700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28601a0150 con 0x7f28601003b0 2023-11-16T18:22:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2864e0d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28600ff1c0 0x7f2860103100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2864e0d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28600ff1c0 0x7f2860103100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47820/0 (socket says 172.21.15.2:47820) 2023-11-16T18:22:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2864e0d700 1 -- 172.21.15.2:0/1123438900 learned_addr learned my addr 172.21.15.2:0/1123438900 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f285ffff700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28601003b0 0x7f2860101790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f286560e700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2860100d40 0x7f2860101cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2864e0d700 1 -- 172.21.15.2:0/1123438900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28601003b0 msgr2=0x7f2860101790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2864e0d700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28601003b0 0x7f2860101790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2864e0d700 1 -- 172.21.15.2:0/1123438900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2860100d40 msgr2=0x7f2860101cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:47.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2864e0d700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2860100d40 0x7f2860101cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:47.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2864e0d700 1 -- 172.21.15.2:0/1123438900 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2858009d00 con 0x7f28600ff1c0 2023-11-16T18:22:47.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2864e0d700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28600ff1c0 0x7f2860103100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f284800e970 tx=0x7f284800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:47.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f285dffb700 1 -- 172.21.15.2:0/1123438900 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848009790 con 0x7f28600ff1c0 2023-11-16T18:22:47.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f2867071700 1 -- 172.21.15.2:0/1123438900 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286010aba0 con 0x7f28600ff1c0 2023-11-16T18:22:47.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f285dffb700 1 -- 172.21.15.2:0/1123438900 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f284800eec0 con 0x7f28600ff1c0 2023-11-16T18:22:47.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.967+0000 7f285dffb700 1 -- 172.21.15.2:0/1123438900 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848003c90 con 0x7f28600ff1c0 2023-11-16T18:22:47.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.968+0000 7f2867071700 1 -- 172.21.15.2:0/1123438900 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286010aa10 con 0x7f28600ff1c0 2023-11-16T18:22:47.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.968+0000 7f2867071700 1 -- 172.21.15.2:0/1123438900 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28600671c0 con 0x7f28600ff1c0 2023-11-16T18:22:47.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.969+0000 7f285dffb700 1 -- 172.21.15.2:0/1123438900 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f284800cc70 con 0x7f28600ff1c0 2023-11-16T18:22:47.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.970+0000 7f285dffb700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f284c0a1150 0x7f284c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:47.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.970+0000 7f285dffb700 1 -- 172.21.15.2:0/1123438900 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8205+0+0 (secure 0 0 0) 0x7f2848014070 con 0x7f28600ff1c0 2023-11-16T18:22:47.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.970+0000 7f285ffff700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f284c0a1150 0x7f284c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:47.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.970+0000 7f285ffff700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f284c0a1150 0x7f284c0a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2850005950 tx=0x7f285000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:47.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:47.974+0000 7f285dffb700 1 -- 172.21.15.2:0/1123438900 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2848096280 con 0x7f28600ff1c0 2023-11-16T18:22:48.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:48.185+0000 7f2867071700 1 -- 172.21.15.2:0/1123438900 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f28601a0cc0 con 0x7f28600ff1c0 2023-11-16T18:22:48.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1635115441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T18:22:48.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: pgmap v256: 97 pgs: 14 active+clean, 14 creating+peering, 69 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:48.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] up:active 2023-11-16T18:22:48.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:22:48.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1635115441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-11-16T18:22:48.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: osdmap e82: 12 total, 12 up, 12 in 2023-11-16T18:22:48.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:22:48.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:48.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:48.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:48 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1123438900' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T18:22:48.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1635115441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T18:22:48.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: pgmap v256: 97 pgs: 14 active+clean, 14 creating+peering, 69 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:48.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] up:active 2023-11-16T18:22:48.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:22:48.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1635115441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-11-16T18:22:48.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: osdmap e82: 12 total, 12 up, 12 in 2023-11-16T18:22:48.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:22:48.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:48.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:48.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:48 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1123438900' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T18:22:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1635115441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T18:22:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: pgmap v256: 97 pgs: 14 active+clean, 14 creating+peering, 69 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:22:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] up:active 2023-11-16T18:22:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:22:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1635115441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-11-16T18:22:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: osdmap e82: 12 total, 12 up, 12 in 2023-11-16T18:22:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:22:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:22:48.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:22:48.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:22:48.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:48 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1123438900' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T18:22:49.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:49 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:22:49.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:49 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:22:49.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:49 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:22:49.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:49 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:22:49.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:49 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:22:49.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:49 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:22:49.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:49 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:22:49.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:49 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:22:49.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:49 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:22:49.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:49 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:22:49.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:49 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:22:49.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:49 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:22:50.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.102+0000 7f285dffb700 1 -- 172.21.15.2:0/1123438900 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v84) v1 ==== 228+0+0 (secure 0 0 0) 0x7f28480998a0 con 0x7f28600ff1c0 2023-11-16T18:22:50.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.110+0000 7f2867071700 1 -- 172.21.15.2:0/1123438900 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f284c0a1150 msgr2=0x7f284c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:50.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.110+0000 7f2867071700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f284c0a1150 0x7f284c0a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2850005950 tx=0x7f285000a300 comp rx=0 tx=0).stop 2023-11-16T18:22:50.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.110+0000 7f2867071700 1 -- 172.21.15.2:0/1123438900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28600ff1c0 msgr2=0x7f2860103100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:50.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.110+0000 7f2867071700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28600ff1c0 0x7f2860103100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f284800e970 tx=0x7f284800ec80 comp rx=0 tx=0).stop 2023-11-16T18:22:50.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.110+0000 7f2867071700 1 -- 172.21.15.2:0/1123438900 shutdown_connections 2023-11-16T18:22:50.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.110+0000 7f2867071700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f284c0a1150 0x7f284c0a3600 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:50.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.110+0000 7f2867071700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2860100d40 0x7f2860101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:50.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.110+0000 7f2867071700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28600ff1c0 0x7f2860103100 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f284800e970 tx=0x7f284800ec80 comp rx=0 tx=0).stop 2023-11-16T18:22:50.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.110+0000 7f2867071700 1 --2- 172.21.15.2:0/1123438900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28601003b0 0x7f2860101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:50.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.110+0000 7f2867071700 1 -- 172.21.15.2:0/1123438900 >> 172.21.15.2:0/1123438900 conn(0x7f28600fa7a0 msgr2=0x7f2860068e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:50.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.112+0000 7f2867071700 1 -- 172.21.15.2:0/1123438900 shutdown_connections 2023-11-16T18:22:50.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.112+0000 7f2867071700 1 -- 172.21.15.2:0/1123438900 wait complete. 2023-11-16T18:22:50.115 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data_ec' created 2023-11-16T18:22:50.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-11-16T18:22:50.264 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.263+0000 7f95b90d4700 1 -- 172.21.15.2:0/1815310377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95b41002c0 msgr2=0x7f95b41006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:50.264 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.263+0000 7f95b90d4700 1 --2- 172.21.15.2:0/1815310377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95b41002c0 0x7f95b41006d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f95a4009990 tx=0x7f95a4009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:50.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.263+0000 7f95b90d4700 1 -- 172.21.15.2:0/1815310377 shutdown_connections 2023-11-16T18:22:50.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.263+0000 7f95b90d4700 1 --2- 172.21.15.2:0/1815310377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b4101e40 0x7f95b4108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:50.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.263+0000 7f95b90d4700 1 --2- 172.21.15.2:0/1815310377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95b41002c0 0x7f95b41006d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:50.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.263+0000 7f95b90d4700 1 --2- 172.21.15.2:0/1815310377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95b41014b0 0x7f95b4101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:50.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.263+0000 7f95b90d4700 1 -- 172.21.15.2:0/1815310377 >> 172.21.15.2:0/1815310377 conn(0x7f95b40fb8c0 msgr2=0x7f95b40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:50.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.263+0000 7f95b90d4700 1 -- 172.21.15.2:0/1815310377 shutdown_connections 2023-11-16T18:22:50.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.263+0000 7f95b90d4700 1 -- 172.21.15.2:0/1815310377 wait complete. 2023-11-16T18:22:50.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.263+0000 7f95b90d4700 1 Processor -- start 2023-11-16T18:22:50.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b90d4700 1 -- start start 2023-11-16T18:22:50.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b90d4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95b41002c0 0x7f95b4195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:50.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b90d4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95b41014b0 0x7f95b41964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:50.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b90d4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b4101e40 0x7f95b419d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:50.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b90d4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b419db20 con 0x7f95b41014b0 2023-11-16T18:22:50.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b90d4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b419dc60 con 0x7f95b4101e40 2023-11-16T18:22:50.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b90d4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b419ddf0 con 0x7f95b41002c0 2023-11-16T18:22:50.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95b41014b0 0x7f95b41964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:50.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95b41014b0 0x7f95b41964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47842/0 (socket says 172.21.15.2:47842) 2023-11-16T18:22:50.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b259c700 1 -- 172.21.15.2:0/1306582429 learned_addr learned my addr 172.21.15.2:0/1306582429 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:50.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b2d9d700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95b41002c0 0x7f95b4195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:50.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b359e700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b4101e40 0x7f95b419d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:50.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b259c700 1 -- 172.21.15.2:0/1306582429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95b41002c0 msgr2=0x7f95b4195fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:50.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b259c700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95b41002c0 0x7f95b4195fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:50.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b259c700 1 -- 172.21.15.2:0/1306582429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b4101e40 msgr2=0x7f95b419d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:50.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b259c700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b4101e40 0x7f95b419d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:50.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b259c700 1 -- 172.21.15.2:0/1306582429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a40095f0 con 0x7f95b41014b0 2023-11-16T18:22:50.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b359e700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b4101e40 0x7f95b419d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:22:50.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.264+0000 7f95b259c700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95b41014b0 0x7f95b41964e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f95a800e9a0 tx=0x7f95a800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:50.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.265+0000 7f959bfff700 1 -- 172.21.15.2:0/1306582429 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a8009790 con 0x7f95b41014b0 2023-11-16T18:22:50.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.265+0000 7f959bfff700 1 -- 172.21.15.2:0/1306582429 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95a800cbd0 con 0x7f95b41014b0 2023-11-16T18:22:50.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.265+0000 7f95b90d4700 1 -- 172.21.15.2:0/1306582429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b410bc20 con 0x7f95b41014b0 2023-11-16T18:22:50.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.265+0000 7f959bfff700 1 -- 172.21.15.2:0/1306582429 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a8018680 con 0x7f95b41014b0 2023-11-16T18:22:50.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.265+0000 7f95b90d4700 1 -- 172.21.15.2:0/1306582429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b410ba90 con 0x7f95b41014b0 2023-11-16T18:22:50.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.266+0000 7f95b90d4700 1 -- 172.21.15.2:0/1306582429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b4107240 con 0x7f95b41014b0 2023-11-16T18:22:50.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.267+0000 7f959bfff700 1 -- 172.21.15.2:0/1306582429 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f95a800cd40 con 0x7f95b41014b0 2023-11-16T18:22:50.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.267+0000 7f959bfff700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f959c0a11d0 0x7f959c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:50.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.268+0000 7f959bfff700 1 -- 172.21.15.2:0/1306582429 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f95a8014070 con 0x7f95b41014b0 2023-11-16T18:22:50.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.268+0000 7f95b2d9d700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f959c0a11d0 0x7f959c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:50.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.268+0000 7f95b2d9d700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f959c0a11d0 0x7f959c0a3680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f95a4006010 tx=0x7f95a400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:50.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.271+0000 7f959bfff700 1 -- 172.21.15.2:0/1306582429 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f95a80967e0 con 0x7f95b41014b0 2023-11-16T18:22:50.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:50.472+0000 7f95b90d4700 1 -- 172.21.15.2:0/1306582429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f95b402cfe0 con 0x7f95b41014b0 2023-11-16T18:22:50.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:50 smithi188 ceph-mon[100639]: osdmap e83: 12 total, 12 up, 12 in 2023-11-16T18:22:50.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:50 smithi188 ceph-mon[100639]: pgmap v259: 97 pgs: 50 active+clean, 14 creating+peering, 33 unknown; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2023-11-16T18:22:50.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:50 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1123438900' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-11-16T18:22:50.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:50 smithi188 ceph-mon[100639]: osdmap e84: 12 total, 12 up, 12 in 2023-11-16T18:22:50.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:50 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1306582429' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T18:22:50.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:50 smithi002 ceph-mon[98807]: osdmap e83: 12 total, 12 up, 12 in 2023-11-16T18:22:50.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:50 smithi002 ceph-mon[98807]: pgmap v259: 97 pgs: 50 active+clean, 14 creating+peering, 33 unknown; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2023-11-16T18:22:50.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:50 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1123438900' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-11-16T18:22:50.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:50 smithi002 ceph-mon[98807]: osdmap e84: 12 total, 12 up, 12 in 2023-11-16T18:22:50.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:50 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1306582429' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T18:22:50.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:50 smithi040 ceph-mon[100251]: osdmap e83: 12 total, 12 up, 12 in 2023-11-16T18:22:50.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:50 smithi040 ceph-mon[100251]: pgmap v259: 97 pgs: 50 active+clean, 14 creating+peering, 33 unknown; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2023-11-16T18:22:50.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:50 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1123438900' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-11-16T18:22:50.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:50 smithi040 ceph-mon[100251]: osdmap e84: 12 total, 12 up, 12 in 2023-11-16T18:22:50.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:50 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1306582429' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T18:22:51.106 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.105+0000 7f959bfff700 1 -- 172.21.15.2:0/1306582429 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v85) v1 ==== 169+0+0 (secure 0 0 0) 0x7f95a8099e00 con 0x7f95b41014b0 2023-11-16T18:22:51.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.117+0000 7f95b90d4700 1 -- 172.21.15.2:0/1306582429 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f959c0a11d0 msgr2=0x7f959c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:51.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.117+0000 7f95b90d4700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f959c0a11d0 0x7f959c0a3680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f95a4006010 tx=0x7f95a400b260 comp rx=0 tx=0).stop 2023-11-16T18:22:51.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.117+0000 7f95b90d4700 1 -- 172.21.15.2:0/1306582429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95b41014b0 msgr2=0x7f95b41964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:51.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.117+0000 7f95b90d4700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95b41014b0 0x7f95b41964e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f95a800e9a0 tx=0x7f95a800ed60 comp rx=0 tx=0).stop 2023-11-16T18:22:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.122+0000 7f95b90d4700 1 -- 172.21.15.2:0/1306582429 shutdown_connections 2023-11-16T18:22:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.122+0000 7f95b90d4700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f959c0a11d0 0x7f959c0a3680 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.122+0000 7f95b90d4700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95b41002c0 0x7f95b4195fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.122+0000 7f95b90d4700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b4101e40 0x7f95b419d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.122+0000 7f95b90d4700 1 --2- 172.21.15.2:0/1306582429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95b41014b0 0x7f95b41964e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.122+0000 7f95b90d4700 1 -- 172.21.15.2:0/1306582429 >> 172.21.15.2:0/1306582429 conn(0x7f95b40fb8c0 msgr2=0x7f95b40fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:51.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.128+0000 7f95b90d4700 1 -- 172.21.15.2:0/1306582429 shutdown_connections 2023-11-16T18:22:51.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.129+0000 7f95b90d4700 1 -- 172.21.15.2:0/1306582429 wait complete. 2023-11-16T18:22:51.131 INFO:teuthology.orchestra.run.smithi002.stderr:set pool 4 allow_ec_overwrites to true 2023-11-16T18:22:51.146 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-11-16T18:22:51.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.278+0000 7f58af0bf700 1 -- 172.21.15.2:0/2897334498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a8100790 msgr2=0x7f58a8100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:51.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.278+0000 7f58af0bf700 1 --2- 172.21.15.2:0/2897334498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a8100790 0x7f58a8100ba0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f589c009990 tx=0x7f589c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.278+0000 7f58af0bf700 1 -- 172.21.15.2:0/2897334498 shutdown_connections 2023-11-16T18:22:51.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.278+0000 7f58af0bf700 1 --2- 172.21.15.2:0/2897334498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80ffdc0 0x7f58a8107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.278+0000 7f58af0bf700 1 --2- 172.21.15.2:0/2897334498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a8100790 0x7f58a8100ba0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.278+0000 7f58af0bf700 1 --2- 172.21.15.2:0/2897334498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58a80ff430 0x7f58a80ff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.278+0000 7f58af0bf700 1 -- 172.21.15.2:0/2897334498 >> 172.21.15.2:0/2897334498 conn(0x7f58a80f9f70 msgr2=0x7f58a80fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:51.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.278+0000 7f58af0bf700 1 -- 172.21.15.2:0/2897334498 shutdown_connections 2023-11-16T18:22:51.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.278+0000 7f58af0bf700 1 -- 172.21.15.2:0/2897334498 wait complete. 2023-11-16T18:22:51.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58af0bf700 1 Processor -- start 2023-11-16T18:22:51.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58af0bf700 1 -- start start 2023-11-16T18:22:51.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58af0bf700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a80ff430 0x7f58a80fee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:51.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58af0bf700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80ffdc0 0x7f58a819ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:51.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58af0bf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58a8100790 0x7f58a819d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:51.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58af0bf700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a819db60 con 0x7f58a80ff430 2023-11-16T18:22:51.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58af0bf700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a819dcd0 con 0x7f58a8100790 2023-11-16T18:22:51.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58af0bf700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a819de90 con 0x7f58a80ffdc0 2023-11-16T18:22:51.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58ace5b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a80ff430 0x7f58a80fee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:51.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58ace5b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a80ff430 0x7f58a80fee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47856/0 (socket says 172.21.15.2:47856) 2023-11-16T18:22:51.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58ace5b700 1 -- 172.21.15.2:0/3938825022 learned_addr learned my addr 172.21.15.2:0/3938825022 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:51.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58a7fff700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80ffdc0 0x7f58a819ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:51.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58ad65c700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58a8100790 0x7f58a819d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:51.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58ace5b700 1 -- 172.21.15.2:0/3938825022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80ffdc0 msgr2=0x7f58a819ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:51.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58ace5b700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80ffdc0 0x7f58a819ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58ace5b700 1 -- 172.21.15.2:0/3938825022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58a8100790 msgr2=0x7f58a819d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:51.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58ace5b700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58a8100790 0x7f58a819d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58ace5b700 1 -- 172.21.15.2:0/3938825022 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589c0095f0 con 0x7f58a80ff430 2023-11-16T18:22:51.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.279+0000 7f58ad65c700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58a8100790 0x7f58a819d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:22:51.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.280+0000 7f58ace5b700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a80ff430 0x7f58a80fee90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f589c0095c0 tx=0x7f589c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:51.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.280+0000 7f58a5ffb700 1 -- 172.21.15.2:0/3938825022 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c01d070 con 0x7f58a80ff430 2023-11-16T18:22:51.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.280+0000 7f58af0bf700 1 -- 172.21.15.2:0/3938825022 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a810ab10 con 0x7f58a80ff430 2023-11-16T18:22:51.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.280+0000 7f58a5ffb700 1 -- 172.21.15.2:0/3938825022 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f589c022470 con 0x7f58a80ff430 2023-11-16T18:22:51.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.280+0000 7f58a5ffb700 1 -- 172.21.15.2:0/3938825022 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c017650 con 0x7f58a80ff430 2023-11-16T18:22:51.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.280+0000 7f58af0bf700 1 -- 172.21.15.2:0/3938825022 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a810a980 con 0x7f58a80ff430 2023-11-16T18:22:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.281+0000 7f58af0bf700 1 -- 172.21.15.2:0/3938825022 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a81068e0 con 0x7f58a80ff430 2023-11-16T18:22:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.282+0000 7f58a5ffb700 1 -- 172.21.15.2:0/3938825022 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f589c00fad0 con 0x7f58a80ff430 2023-11-16T18:22:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.283+0000 7f58a5ffb700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58980a1150 0x7f58980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.283+0000 7f58a7fff700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58980a1150 0x7f58980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.283+0000 7f58a5ffb700 1 -- 172.21.15.2:0/3938825022 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f589c0ce4e0 con 0x7f58a80ff430 2023-11-16T18:22:51.286 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.283+0000 7f58a7fff700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58980a1150 0x7f58980a3600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5894005950 tx=0x7f589400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:51.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.287+0000 7f58a5ffb700 1 -- 172.21.15.2:0/3938825022 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f589c097bb0 con 0x7f58a80ff430 2023-11-16T18:22:51.534 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.533+0000 7f58af0bf700 1 -- 172.21.15.2:0/3938825022 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f58a8061700 con 0x7f58a80ff430 2023-11-16T18:22:51.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.539+0000 7f58a5ffb700 1 -- 172.21.15.2:0/3938825022 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f589c0270b0 con 0x7f58a80ff430 2023-11-16T18:22:51.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.541+0000 7f58af0bf700 1 -- 172.21.15.2:0/3938825022 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58980a1150 msgr2=0x7f58980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:51.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.541+0000 7f58af0bf700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58980a1150 0x7f58980a3600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5894005950 tx=0x7f589400b410 comp rx=0 tx=0).stop 2023-11-16T18:22:51.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.541+0000 7f58af0bf700 1 -- 172.21.15.2:0/3938825022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a80ff430 msgr2=0x7f58a80fee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:51.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.541+0000 7f58af0bf700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a80ff430 0x7f58a80fee90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f589c0095c0 tx=0x7f589c00f690 comp rx=0 tx=0).stop 2023-11-16T18:22:51.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.541+0000 7f58af0bf700 1 -- 172.21.15.2:0/3938825022 shutdown_connections 2023-11-16T18:22:51.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.541+0000 7f58af0bf700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f58980a1150 0x7f58980a3600 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.541+0000 7f58af0bf700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58a8100790 0x7f58a819d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.541+0000 7f58af0bf700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a80ff430 0x7f58a80fee90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.541+0000 7f58af0bf700 1 --2- 172.21.15.2:0/3938825022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80ffdc0 0x7f58a819ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.541+0000 7f58af0bf700 1 -- 172.21.15.2:0/3938825022 >> 172.21.15.2:0/3938825022 conn(0x7f58a80f9f70 msgr2=0x7f58a8103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:51.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.541+0000 7f58af0bf700 1 -- 172.21.15.2:0/3938825022 shutdown_connections 2023-11-16T18:22:51.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.541+0000 7f58af0bf700 1 -- 172.21.15.2:0/3938825022 wait complete. 2023-11-16T18:22:51.544 INFO:teuthology.orchestra.run.smithi002.stderr:added data pool 4 to fsmap 2023-11-16T18:22:51.555 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:51.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.683+0000 7f13f6da0700 1 -- 172.21.15.2:0/3830153856 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13f0105de0 msgr2=0x7f13f0108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:51.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.683+0000 7f13f6da0700 1 --2- 172.21.15.2:0/3830153856 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13f0105de0 0x7f13f0108200 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f13e800b0a0 tx=0x7f13e800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.683+0000 7f13f6da0700 1 -- 172.21.15.2:0/3830153856 shutdown_connections 2023-11-16T18:22:51.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.683+0000 7f13f6da0700 1 --2- 172.21.15.2:0/3830153856 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13f0100ba0 0x7f13f0102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.683+0000 7f13f6da0700 1 --2- 172.21.15.2:0/3830153856 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13f0105de0 0x7f13f0108200 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.683+0000 7f13f6da0700 1 --2- 172.21.15.2:0/3830153856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f01034c0 0x7f13f01058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.683+0000 7f13f6da0700 1 -- 172.21.15.2:0/3830153856 >> 172.21.15.2:0/3830153856 conn(0x7f13f00fa5e0 msgr2=0x7f13f00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:51.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.683+0000 7f13f6da0700 1 -- 172.21.15.2:0/3830153856 shutdown_connections 2023-11-16T18:22:51.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.683+0000 7f13f6da0700 1 -- 172.21.15.2:0/3830153856 wait complete. 2023-11-16T18:22:51.687 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.683+0000 7f13f6da0700 1 Processor -- start 2023-11-16T18:22:51.687 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f6da0700 1 -- start start 2023-11-16T18:22:51.687 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f6da0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13f0100ba0 0x7f13f0195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:51.687 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f6da0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f01034c0 0x7f13f0196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:51.687 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f6da0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13f0105de0 0x7f13f019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:51.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f6da0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f019da60 con 0x7f13f0100ba0 2023-11-16T18:22:51.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f6da0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f019dba0 con 0x7f13f01034c0 2023-11-16T18:22:51.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f6da0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f019dd30 con 0x7f13f0105de0 2023-11-16T18:22:51.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f5d9e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13f0100ba0 0x7f13f0195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:51.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f5d9e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13f0100ba0 0x7f13f0195f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47870/0 (socket says 172.21.15.2:47870) 2023-11-16T18:22:51.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f5d9e700 1 -- 172.21.15.2:0/2427068084 learned_addr learned my addr 172.21.15.2:0/2427068084 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:51.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f559d700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f01034c0 0x7f13f0196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:51.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f5d9e700 1 -- 172.21.15.2:0/2427068084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13f0105de0 msgr2=0x7f13f019d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:51.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f659f700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13f0105de0 0x7f13f019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:51.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f5d9e700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13f0105de0 0x7f13f019d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f5d9e700 1 -- 172.21.15.2:0/2427068084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f01034c0 msgr2=0x7f13f0196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:51.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f5d9e700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f01034c0 0x7f13f0196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f5d9e700 1 -- 172.21.15.2:0/2427068084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e8009d00 con 0x7f13f0100ba0 2023-11-16T18:22:51.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f659f700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13f0105de0 0x7f13f019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:22:51.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.684+0000 7f13f5d9e700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13f0100ba0 0x7f13f0195f30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f13ec00b530 tx=0x7f13ec00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:51.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.685+0000 7f13e6ffd700 1 -- 172.21.15.2:0/2427068084 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec011830 con 0x7f13f0100ba0 2023-11-16T18:22:51.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.685+0000 7f13e6ffd700 1 -- 172.21.15.2:0/2427068084 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13ec011e50 con 0x7f13f0100ba0 2023-11-16T18:22:51.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.685+0000 7f13f6da0700 1 -- 172.21.15.2:0/2427068084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f010baa0 con 0x7f13f0100ba0 2023-11-16T18:22:51.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.685+0000 7f13e6ffd700 1 -- 172.21.15.2:0/2427068084 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec00f2e0 con 0x7f13f0100ba0 2023-11-16T18:22:51.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.685+0000 7f13f6da0700 1 -- 172.21.15.2:0/2427068084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f010b910 con 0x7f13f0100ba0 2023-11-16T18:22:51.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.686+0000 7f13f6da0700 1 -- 172.21.15.2:0/2427068084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f00fc150 con 0x7f13f0100ba0 2023-11-16T18:22:51.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.687+0000 7f13e6ffd700 1 -- 172.21.15.2:0/2427068084 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f13ec010580 con 0x7f13f0100ba0 2023-11-16T18:22:51.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.687+0000 7f13e6ffd700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f13dc0a1150 0x7f13dc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:51.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.687+0000 7f13e6ffd700 1 -- 172.21.15.2:0/2427068084 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f13ec0cc600 con 0x7f13f0100ba0 2023-11-16T18:22:51.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.687+0000 7f13f559d700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f13dc0a1150 0x7f13dc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:51.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.688+0000 7f13f559d700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f13dc0a1150 0x7f13dc0a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f13e0005950 tx=0x7f13e00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:51.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.692+0000 7f13e6ffd700 1 -- 172.21.15.2:0/2427068084 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f13ec095b60 con 0x7f13f0100ba0 2023-11-16T18:22:51.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.936+0000 7f13f6da0700 1 -- 172.21.15.2:0/2427068084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13f0061930 con 0x7f13f0100ba0 2023-11-16T18:22:51.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.936+0000 7f13e6ffd700 1 -- 172.21.15.2:0/2427068084 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10714 (secure 0 0 0) 0x7f13ec099180 con 0x7f13f0100ba0 2023-11-16T18:22:51.938 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:51.938 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:22:51.534922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24473},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-11-16T18:22:51.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.939+0000 7f13f6da0700 1 -- 172.21.15.2:0/2427068084 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f13dc0a1150 msgr2=0x7f13dc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:51.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.939+0000 7f13f6da0700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f13dc0a1150 0x7f13dc0a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f13e0005950 tx=0x7f13e00058e0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.939+0000 7f13f6da0700 1 -- 172.21.15.2:0/2427068084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13f0100ba0 msgr2=0x7f13f0195f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:51.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.939+0000 7f13f6da0700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13f0100ba0 0x7f13f0195f30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f13ec00b530 tx=0x7f13ec00b8f0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.939+0000 7f13f6da0700 1 -- 172.21.15.2:0/2427068084 shutdown_connections 2023-11-16T18:22:51.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.939+0000 7f13f6da0700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f13dc0a1150 0x7f13dc0a3600 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.939+0000 7f13f6da0700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13f0105de0 0x7f13f019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.939+0000 7f13f6da0700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13f0100ba0 0x7f13f0195f30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.939+0000 7f13f6da0700 1 --2- 172.21.15.2:0/2427068084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f01034c0 0x7f13f0196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:51.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.939+0000 7f13f6da0700 1 -- 172.21.15.2:0/2427068084 >> 172.21.15.2:0/2427068084 conn(0x7f13f00fa5e0 msgr2=0x7f13f0101820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:51.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.939+0000 7f13f6da0700 1 -- 172.21.15.2:0/2427068084 shutdown_connections 2023-11-16T18:22:51.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:51.939+0000 7f13f6da0700 1 -- 172.21.15.2:0/2427068084 wait complete. 2023-11-16T18:22:51.941 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 17 2023-11-16T18:22:51.952 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:22:52.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.077+0000 7fdbfec67700 1 -- 172.21.15.2:0/2911336429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf81036c0 msgr2=0x7fdbf8105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:52.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.077+0000 7fdbfec67700 1 --2- 172.21.15.2:0/2911336429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf81036c0 0x7fdbf8105aa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4009990 tx=0x7fdbe4009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.077+0000 7fdbfec67700 1 -- 172.21.15.2:0/2911336429 shutdown_connections 2023-11-16T18:22:52.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.077+0000 7fdbfec67700 1 --2- 172.21.15.2:0/2911336429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbf8100da0 0x7fdbf8103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.077+0000 7fdbfec67700 1 --2- 172.21.15.2:0/2911336429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbf8105fe0 0x7fdbf81083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.077+0000 7fdbfec67700 1 --2- 172.21.15.2:0/2911336429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf81036c0 0x7fdbf8105aa0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.077+0000 7fdbfec67700 1 -- 172.21.15.2:0/2911336429 >> 172.21.15.2:0/2911336429 conn(0x7fdbf80fa7c0 msgr2=0x7fdbf80fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:52.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.077+0000 7fdbfec67700 1 -- 172.21.15.2:0/2911336429 shutdown_connections 2023-11-16T18:22:52.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.077+0000 7fdbfec67700 1 -- 172.21.15.2:0/2911336429 wait complete. 2023-11-16T18:22:52.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.078+0000 7fdbfec67700 1 Processor -- start 2023-11-16T18:22:52.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.078+0000 7fdbfec67700 1 -- start start 2023-11-16T18:22:52.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.078+0000 7fdbfec67700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbf8100da0 0x7fdbf81960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:52.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.078+0000 7fdbfec67700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf81036c0 0x7fdbf81965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:52.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.078+0000 7fdbfec67700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbf8105fe0 0x7fdbf819d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:52.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.078+0000 7fdbfec67700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf819dc20 con 0x7fdbf81036c0 2023-11-16T18:22:52.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.078+0000 7fdbfec67700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf819dd60 con 0x7fdbf8100da0 2023-11-16T18:22:52.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.078+0000 7fdbfec67700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf819def0 con 0x7fdbf8105fe0 2023-11-16T18:22:52.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.078+0000 7fdbf7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf81036c0 0x7fdbf81965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:52.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.078+0000 7fdbf7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf81036c0 0x7fdbf81965e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47884/0 (socket says 172.21.15.2:47884) 2023-11-16T18:22:52.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.078+0000 7fdbf7fff700 1 -- 172.21.15.2:0/399145301 learned_addr learned my addr 172.21.15.2:0/399145301 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:52.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.078+0000 7fdbfd204700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbf8105fe0 0x7fdbf819d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:52.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.079+0000 7fdbfca03700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbf8100da0 0x7fdbf81960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:52.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.079+0000 7fdbf7fff700 1 -- 172.21.15.2:0/399145301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbf8105fe0 msgr2=0x7fdbf819d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:52.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.079+0000 7fdbf7fff700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbf8105fe0 0x7fdbf819d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.079+0000 7fdbf7fff700 1 -- 172.21.15.2:0/399145301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbf8100da0 msgr2=0x7fdbf81960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:52.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.079+0000 7fdbf7fff700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbf8100da0 0x7fdbf81960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.079+0000 7fdbf7fff700 1 -- 172.21.15.2:0/399145301 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe40095f0 con 0x7fdbf81036c0 2023-11-16T18:22:52.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.079+0000 7fdbfca03700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbf8100da0 0x7fdbf81960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:22:52.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.079+0000 7fdbf7fff700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf81036c0 0x7fdbf81965e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40095c0 tx=0x7fdbe400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:52.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.079+0000 7fdbf5ffb700 1 -- 172.21.15.2:0/399145301 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe401d070 con 0x7fdbf81036c0 2023-11-16T18:22:52.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.079+0000 7fdbf5ffb700 1 -- 172.21.15.2:0/399145301 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdbe4022470 con 0x7fdbf81036c0 2023-11-16T18:22:52.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.079+0000 7fdbfec67700 1 -- 172.21.15.2:0/399145301 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf810bcd0 con 0x7fdbf81036c0 2023-11-16T18:22:52.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.079+0000 7fdbf5ffb700 1 -- 172.21.15.2:0/399145301 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4017650 con 0x7fdbf81036c0 2023-11-16T18:22:52.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.079+0000 7fdbfec67700 1 -- 172.21.15.2:0/399145301 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf810bb40 con 0x7fdbf81036c0 2023-11-16T18:22:52.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.080+0000 7fdbfec67700 1 -- 172.21.15.2:0/399145301 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf819e490 con 0x7fdbf81036c0 2023-11-16T18:22:52.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.081+0000 7fdbf5ffb700 1 -- 172.21.15.2:0/399145301 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fdbe4022890 con 0x7fdbf81036c0 2023-11-16T18:22:52.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.082+0000 7fdbf5ffb700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdbe80a1150 0x7fdbe80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:52.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.082+0000 7fdbf5ffb700 1 -- 172.21.15.2:0/399145301 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdbe40cdef0 con 0x7fdbf81036c0 2023-11-16T18:22:52.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.082+0000 7fdbfca03700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdbe80a1150 0x7fdbe80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:52.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.082+0000 7fdbfca03700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdbe80a1150 0x7fdbe80a3600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdbec005fd0 tx=0x7fdbec005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:52.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.086+0000 7fdbf5ffb700 1 -- 172.21.15.2:0/399145301 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdbe40975a0 con 0x7fdbf81036c0 2023-11-16T18:22:52.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.289+0000 7fdbfec67700 1 -- 172.21.15.2:0/399145301 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbf819ea40 con 0x7fdbf81036c0 2023-11-16T18:22:52.291 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.291+0000 7fdbf5ffb700 1 -- 172.21.15.2:0/399145301 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23853 (secure 0 0 0) 0x7fdbe4027090 con 0x7fdbf81036c0 2023-11-16T18:22:52.292 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:52.292 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":86,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:22:51.534683+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:22:52.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.293+0000 7fdbfec67700 1 -- 172.21.15.2:0/399145301 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdbe80a1150 msgr2=0x7fdbe80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:52.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.293+0000 7fdbfec67700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdbe80a1150 0x7fdbe80a3600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdbec005fd0 tx=0x7fdbec005ee0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.293+0000 7fdbfec67700 1 -- 172.21.15.2:0/399145301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf81036c0 msgr2=0x7fdbf81965e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:52.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.293+0000 7fdbfec67700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf81036c0 0x7fdbf81965e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40095c0 tx=0x7fdbe400f690 comp rx=0 tx=0).stop 2023-11-16T18:22:52.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.293+0000 7fdbfec67700 1 -- 172.21.15.2:0/399145301 shutdown_connections 2023-11-16T18:22:52.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.293+0000 7fdbfec67700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdbe80a1150 0x7fdbe80a3600 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.293+0000 7fdbfec67700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdbf8105fe0 0x7fdbf819d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.293+0000 7fdbfec67700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbf8100da0 0x7fdbf81960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.293+0000 7fdbfec67700 1 --2- 172.21.15.2:0/399145301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf81036c0 0x7fdbf81965e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.293+0000 7fdbfec67700 1 -- 172.21.15.2:0/399145301 >> 172.21.15.2:0/399145301 conn(0x7fdbf80fa7c0 msgr2=0x7fdbf8102260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:52.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.293+0000 7fdbfec67700 1 -- 172.21.15.2:0/399145301 shutdown_connections 2023-11-16T18:22:52.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.293+0000 7fdbfec67700 1 -- 172.21.15.2:0/399145301 wait complete. 2023-11-16T18:22:52.306 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:52 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1306582429' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-11-16T18:22:52.306 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:52 smithi002 ceph-mon[98807]: osdmap e85: 12 total, 12 up, 12 in 2023-11-16T18:22:52.306 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:52 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3938825022' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T18:22:52.306 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:52 smithi002 ceph-mon[98807]: osdmap e86: 12 total, 12 up, 12 in 2023-11-16T18:22:52.306 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:52 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3938825022' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-11-16T18:22:52.306 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:52 smithi002 ceph-mon[98807]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:22:52.306 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:52 smithi002 ceph-mon[98807]: pgmap v263: 129 pgs: 101 active+clean, 12 creating+peering, 16 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2023-11-16T18:22:52.306 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:52 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2427068084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:52.308 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:22:52.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.432+0000 7fbf3094e700 1 -- 172.21.15.2:0/3468564129 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf2c0ff1f0 msgr2=0x7fbf2c0ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:52.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.432+0000 7fbf3094e700 1 --2- 172.21.15.2:0/3468564129 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf2c0ff1f0 0x7fbf2c0ff600 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009990 tx=0x7fbf1c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.432+0000 7fbf3094e700 1 -- 172.21.15.2:0/3468564129 shutdown_connections 2023-11-16T18:22:52.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.432+0000 7fbf3094e700 1 --2- 172.21.15.2:0/3468564129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf2c100d70 0x7fbf2c107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.432+0000 7fbf3094e700 1 --2- 172.21.15.2:0/3468564129 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf2c0ff1f0 0x7fbf2c0ff600 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.432+0000 7fbf3094e700 1 --2- 172.21.15.2:0/3468564129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c1003e0 0x7fbf2c100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.432+0000 7fbf3094e700 1 -- 172.21.15.2:0/3468564129 >> 172.21.15.2:0/3468564129 conn(0x7fbf2c0fa7d0 msgr2=0x7fbf2c0fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:52.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.432+0000 7fbf3094e700 1 -- 172.21.15.2:0/3468564129 shutdown_connections 2023-11-16T18:22:52.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.432+0000 7fbf3094e700 1 -- 172.21.15.2:0/3468564129 wait complete. 2023-11-16T18:22:52.436 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.433+0000 7fbf3094e700 1 Processor -- start 2023-11-16T18:22:52.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.433+0000 7fbf3094e700 1 -- start start 2023-11-16T18:22:52.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.433+0000 7fbf3094e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c0ff1f0 0x7fbf2c196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:52.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.433+0000 7fbf3094e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf2c1003e0 0x7fbf2c1965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:52.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.433+0000 7fbf3094e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf2c100d70 0x7fbf2c19d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:52.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.433+0000 7fbf3094e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c19dc10 con 0x7fbf2c100d70 2023-11-16T18:22:52.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.433+0000 7fbf3094e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c19dd50 con 0x7fbf2c0ff1f0 2023-11-16T18:22:52.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.433+0000 7fbf3094e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c19dee0 con 0x7fbf2c1003e0 2023-11-16T18:22:52.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.433+0000 7fbf2ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf2c100d70 0x7fbf2c19d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:52.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.433+0000 7fbf2ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf2c100d70 0x7fbf2c19d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47900/0 (socket says 172.21.15.2:47900) 2023-11-16T18:22:52.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.433+0000 7fbf2ad9d700 1 -- 172.21.15.2:0/2868040917 learned_addr learned my addr 172.21.15.2:0/2868040917 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:52.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.433+0000 7fbf2a59c700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c0ff1f0 0x7fbf2c196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:52.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.434+0000 7fbf29d9b700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf2c1003e0 0x7fbf2c1965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:52.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.434+0000 7fbf2ad9d700 1 -- 172.21.15.2:0/2868040917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf2c1003e0 msgr2=0x7fbf2c1965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:52.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.434+0000 7fbf2ad9d700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf2c1003e0 0x7fbf2c1965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.434+0000 7fbf2ad9d700 1 -- 172.21.15.2:0/2868040917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c0ff1f0 msgr2=0x7fbf2c196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:52.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.434+0000 7fbf2ad9d700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c0ff1f0 0x7fbf2c196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.434+0000 7fbf2ad9d700 1 -- 172.21.15.2:0/2868040917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1c0095f0 con 0x7fbf2c100d70 2023-11-16T18:22:52.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.434+0000 7fbf2ad9d700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf2c100d70 0x7fbf2c19d640 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbf24007ee0 tx=0x7fbf2400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:52.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.434+0000 7fbf1b7fe700 1 -- 172.21.15.2:0/2868040917 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24017070 con 0x7fbf2c100d70 2023-11-16T18:22:52.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.434+0000 7fbf3094e700 1 -- 172.21.15.2:0/2868040917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2c10ab60 con 0x7fbf2c100d70 2023-11-16T18:22:52.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.434+0000 7fbf1b7fe700 1 -- 172.21.15.2:0/2868040917 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf2400f040 con 0x7fbf2c100d70 2023-11-16T18:22:52.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.434+0000 7fbf1b7fe700 1 -- 172.21.15.2:0/2868040917 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24005550 con 0x7fbf2c100d70 2023-11-16T18:22:52.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.434+0000 7fbf3094e700 1 -- 172.21.15.2:0/2868040917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2c10a9d0 con 0x7fbf2c100d70 2023-11-16T18:22:52.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.435+0000 7fbf3094e700 1 -- 172.21.15.2:0/2868040917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2c107320 con 0x7fbf2c100d70 2023-11-16T18:22:52.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.436+0000 7fbf1b7fe700 1 -- 172.21.15.2:0/2868040917 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fbf2400db20 con 0x7fbf2c100d70 2023-11-16T18:22:52.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.437+0000 7fbf1b7fe700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbf140a1150 0x7fbf140a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:52.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.437+0000 7fbf1b7fe700 1 -- 172.21.15.2:0/2868040917 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbf240cc880 con 0x7fbf2c100d70 2023-11-16T18:22:52.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.437+0000 7fbf2a59c700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbf140a1150 0x7fbf140a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:52.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.437+0000 7fbf2a59c700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbf140a1150 0x7fbf140a3600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c00b5c0 tx=0x7fbf1c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:52.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.440+0000 7fbf1b7fe700 1 -- 172.21.15.2:0/2868040917 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbf24095de0 con 0x7fbf2c100d70 2023-11-16T18:22:52.483 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:22:52 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-b[125477]: 2023-11-16T18:22:52.169+0000 7f68bf57f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-16T18:22:52.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:52 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1306582429' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-11-16T18:22:52.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:52 smithi040 ceph-mon[100251]: osdmap e85: 12 total, 12 up, 12 in 2023-11-16T18:22:52.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:52 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3938825022' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T18:22:52.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:52 smithi040 ceph-mon[100251]: osdmap e86: 12 total, 12 up, 12 in 2023-11-16T18:22:52.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:52 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3938825022' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-11-16T18:22:52.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:52 smithi040 ceph-mon[100251]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:22:52.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:52 smithi040 ceph-mon[100251]: pgmap v263: 129 pgs: 101 active+clean, 12 creating+peering, 16 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2023-11-16T18:22:52.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:52 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2427068084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:52.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:52 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1306582429' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-11-16T18:22:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:52 smithi188 ceph-mon[100639]: osdmap e85: 12 total, 12 up, 12 in 2023-11-16T18:22:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:52 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3938825022' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T18:22:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:52 smithi188 ceph-mon[100639]: osdmap e86: 12 total, 12 up, 12 in 2023-11-16T18:22:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:52 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3938825022' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-11-16T18:22:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:52 smithi188 ceph-mon[100639]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:22:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:52 smithi188 ceph-mon[100639]: pgmap v263: 129 pgs: 101 active+clean, 12 creating+peering, 16 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2023-11-16T18:22:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:52 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2427068084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:52.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.643+0000 7fbf3094e700 1 -- 172.21.15.2:0/2868040917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf2c02cfe0 con 0x7fbf2c100d70 2023-11-16T18:22:52.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.644+0000 7fbf1b7fe700 1 -- 172.21.15.2:0/2868040917 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23853 (secure 0 0 0) 0x7fbf24099400 con 0x7fbf2c100d70 2023-11-16T18:22:52.645 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:52.646 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":86,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:22:51.534683+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:22:52.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.646+0000 7fbf3094e700 1 -- 172.21.15.2:0/2868040917 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbf140a1150 msgr2=0x7fbf140a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:52.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.646+0000 7fbf3094e700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbf140a1150 0x7fbf140a3600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c00b5c0 tx=0x7fbf1c005fb0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.646+0000 7fbf3094e700 1 -- 172.21.15.2:0/2868040917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf2c100d70 msgr2=0x7fbf2c19d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:52.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.646+0000 7fbf3094e700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf2c100d70 0x7fbf2c19d640 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbf24007ee0 tx=0x7fbf2400d3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.646+0000 7fbf3094e700 1 -- 172.21.15.2:0/2868040917 shutdown_connections 2023-11-16T18:22:52.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.646+0000 7fbf3094e700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbf140a1150 0x7fbf140a3600 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.646+0000 7fbf3094e700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf2c100d70 0x7fbf2c19d640 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.646+0000 7fbf3094e700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c0ff1f0 0x7fbf2c196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.646+0000 7fbf3094e700 1 --2- 172.21.15.2:0/2868040917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf2c1003e0 0x7fbf2c1965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.646+0000 7fbf3094e700 1 -- 172.21.15.2:0/2868040917 >> 172.21.15.2:0/2868040917 conn(0x7fbf2c0fa7d0 msgr2=0x7fbf2c0fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:52.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.646+0000 7fbf3094e700 1 -- 172.21.15.2:0/2868040917 shutdown_connections 2023-11-16T18:22:52.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.646+0000 7fbf3094e700 1 -- 172.21.15.2:0/2868040917 wait complete. 2023-11-16T18:22:52.689 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:22:52.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.814+0000 7fa6011e7700 1 -- 172.21.15.2:0/475258317 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5fc0ff1c0 msgr2=0x7fa5fc0ff5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:52.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.814+0000 7fa6011e7700 1 --2- 172.21.15.2:0/475258317 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5fc0ff1c0 0x7fa5fc0ff5d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec009990 tx=0x7fa5ec009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.814+0000 7fa6011e7700 1 -- 172.21.15.2:0/475258317 shutdown_connections 2023-11-16T18:22:52.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.814+0000 7fa6011e7700 1 --2- 172.21.15.2:0/475258317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5fc100d40 0x7fa5fc107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.814+0000 7fa6011e7700 1 --2- 172.21.15.2:0/475258317 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5fc0ff1c0 0x7fa5fc0ff5d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.814+0000 7fa6011e7700 1 --2- 172.21.15.2:0/475258317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5fc1003b0 0x7fa5fc100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.814+0000 7fa6011e7700 1 -- 172.21.15.2:0/475258317 >> 172.21.15.2:0/475258317 conn(0x7fa5fc0fa7c0 msgr2=0x7fa5fc0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:52.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.814+0000 7fa6011e7700 1 -- 172.21.15.2:0/475258317 shutdown_connections 2023-11-16T18:22:52.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.814+0000 7fa6011e7700 1 -- 172.21.15.2:0/475258317 wait complete. 2023-11-16T18:22:52.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.815+0000 7fa6011e7700 1 Processor -- start 2023-11-16T18:22:52.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa6011e7700 1 -- start start 2023-11-16T18:22:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa6011e7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5fc0ff1c0 0x7fa5fc194f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa6011e7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5fc1003b0 0x7fa5fc195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa6011e7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5fc100d40 0x7fa5fc19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa6011e7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc19cad0 con 0x7fa5fc0ff1c0 2023-11-16T18:22:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa6011e7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc19cc10 con 0x7fa5fc1003b0 2023-11-16T18:22:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa6011e7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc19cda0 con 0x7fa5fc100d40 2023-11-16T18:22:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa5fad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5fc0ff1c0 0x7fa5fc194f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa5fad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5fc0ff1c0 0x7fa5fc194f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47910/0 (socket says 172.21.15.2:47910) 2023-11-16T18:22:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa5fad9d700 1 -- 172.21.15.2:0/448871990 learned_addr learned my addr 172.21.15.2:0/448871990 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa5fa59c700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5fc1003b0 0x7fa5fc195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:52.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa5fad9d700 1 -- 172.21.15.2:0/448871990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5fc100d40 msgr2=0x7fa5fc19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:52.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa5fb59e700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5fc100d40 0x7fa5fc19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:52.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa5fad9d700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5fc100d40 0x7fa5fc19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa5fad9d700 1 -- 172.21.15.2:0/448871990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5fc1003b0 msgr2=0x7fa5fc195490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:52.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa5fad9d700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5fc1003b0 0x7fa5fc195490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:52.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa5fad9d700 1 -- 172.21.15.2:0/448871990 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec0095f0 con 0x7fa5fc0ff1c0 2023-11-16T18:22:52.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa5fa59c700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5fc1003b0 0x7fa5fc195490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:22:52.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.816+0000 7fa5fad9d700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5fc0ff1c0 0x7fa5fc194f50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec005740 tx=0x7fa5ec009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:52.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.817+0000 7fa5e3fff700 1 -- 172.21.15.2:0/448871990 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec01d070 con 0x7fa5fc0ff1c0 2023-11-16T18:22:52.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.817+0000 7fa6011e7700 1 -- 172.21.15.2:0/448871990 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5fc10ab80 con 0x7fa5fc0ff1c0 2023-11-16T18:22:52.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.817+0000 7fa5e3fff700 1 -- 172.21.15.2:0/448871990 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5ec0040a0 con 0x7fa5fc0ff1c0 2023-11-16T18:22:52.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.817+0000 7fa5e3fff700 1 -- 172.21.15.2:0/448871990 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec022d50 con 0x7fa5fc0ff1c0 2023-11-16T18:22:52.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.817+0000 7fa6011e7700 1 -- 172.21.15.2:0/448871990 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5fc10a9f0 con 0x7fa5fc0ff1c0 2023-11-16T18:22:52.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.818+0000 7fa6011e7700 1 -- 172.21.15.2:0/448871990 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5fc19d3a0 con 0x7fa5fc0ff1c0 2023-11-16T18:22:52.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.819+0000 7fa5e3fff700 1 -- 172.21.15.2:0/448871990 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fa5ec022470 con 0x7fa5fc0ff1c0 2023-11-16T18:22:52.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.820+0000 7fa5e3fff700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa5e40a1150 0x7fa5e40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:52.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.820+0000 7fa5e3fff700 1 -- 172.21.15.2:0/448871990 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa5ec0ce720 con 0x7fa5fc0ff1c0 2023-11-16T18:22:52.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.820+0000 7fa5fa59c700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa5e40a1150 0x7fa5e40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:52.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.820+0000 7fa5fa59c700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa5e40a1150 0x7fa5e40a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0005fd0 tx=0x7fa5f0005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:52.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:52.823+0000 7fa5e3fff700 1 -- 172.21.15.2:0/448871990 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa5ec097d10 con 0x7fa5fc0ff1c0 2023-11-16T18:22:53.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.068+0000 7fa6011e7700 1 -- 172.21.15.2:0/448871990 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5fc106070 con 0x7fa5fc0ff1c0 2023-11-16T18:22:53.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.069+0000 7fa5e3fff700 1 -- 172.21.15.2:0/448871990 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10714 (secure 0 0 0) 0x7fa5ec09b330 con 0x7fa5fc0ff1c0 2023-11-16T18:22:53.070 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:53.070 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:22:51.534922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24473},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-11-16T18:22:53.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.071+0000 7fa6011e7700 1 -- 172.21.15.2:0/448871990 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa5e40a1150 msgr2=0x7fa5e40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.071+0000 7fa6011e7700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa5e40a1150 0x7fa5e40a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0005fd0 tx=0x7fa5f0005e20 comp rx=0 tx=0).stop 2023-11-16T18:22:53.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.071+0000 7fa6011e7700 1 -- 172.21.15.2:0/448871990 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5fc0ff1c0 msgr2=0x7fa5fc194f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.071+0000 7fa6011e7700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5fc0ff1c0 0x7fa5fc194f50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec005740 tx=0x7fa5ec009580 comp rx=0 tx=0).stop 2023-11-16T18:22:53.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.071+0000 7fa6011e7700 1 -- 172.21.15.2:0/448871990 shutdown_connections 2023-11-16T18:22:53.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.071+0000 7fa6011e7700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa5e40a1150 0x7fa5e40a3600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.071+0000 7fa6011e7700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5fc100d40 0x7fa5fc19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.071+0000 7fa6011e7700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5fc0ff1c0 0x7fa5fc194f50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.071+0000 7fa6011e7700 1 --2- 172.21.15.2:0/448871990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5fc1003b0 0x7fa5fc195490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.071+0000 7fa6011e7700 1 -- 172.21.15.2:0/448871990 >> 172.21.15.2:0/448871990 conn(0x7fa5fc0fa7c0 msgr2=0x7fa5fc0fc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:53.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.071+0000 7fa6011e7700 1 -- 172.21.15.2:0/448871990 shutdown_connections 2023-11-16T18:22:53.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.071+0000 7fa6011e7700 1 -- 172.21.15.2:0/448871990 wait complete. 2023-11-16T18:22:53.074 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 17 2023-11-16T18:22:53.085 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:22:53.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.212+0000 7f45e9d51700 1 -- 172.21.15.2:0/295589059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e4100d40 msgr2=0x7f45e4107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.212+0000 7f45e9d51700 1 --2- 172.21.15.2:0/295589059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e4100d40 0x7f45e4107600 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00b0a0 tx=0x7f45dc00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.212+0000 7f45e9d51700 1 -- 172.21.15.2:0/295589059 shutdown_connections 2023-11-16T18:22:53.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.212+0000 7f45e9d51700 1 --2- 172.21.15.2:0/295589059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45e40ff1c0 0x7f45e40ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.212+0000 7f45e9d51700 1 --2- 172.21.15.2:0/295589059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e4100d40 0x7f45e4107600 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.212+0000 7f45e9d51700 1 --2- 172.21.15.2:0/295589059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45e41003b0 0x7f45e4100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.212+0000 7f45e9d51700 1 -- 172.21.15.2:0/295589059 >> 172.21.15.2:0/295589059 conn(0x7f45e40fa7a0 msgr2=0x7f45e40fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:53.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.212+0000 7f45e9d51700 1 -- 172.21.15.2:0/295589059 shutdown_connections 2023-11-16T18:22:53.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.212+0000 7f45e9d51700 1 -- 172.21.15.2:0/295589059 wait complete. 2023-11-16T18:22:53.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.212+0000 7f45e9d51700 1 Processor -- start 2023-11-16T18:22:53.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.212+0000 7f45e9d51700 1 -- start start 2023-11-16T18:22:53.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e9d51700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45e40ff1c0 0x7f45e40699c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e9d51700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45e41003b0 0x7f45e419adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e9d51700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e4100d40 0x7f45e419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e9d51700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e419dc20 con 0x7f45e4100d40 2023-11-16T18:22:53.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e9d51700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e419dd90 con 0x7f45e40ff1c0 2023-11-16T18:22:53.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e9d51700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e419df50 con 0x7f45e41003b0 2023-11-16T18:22:53.217 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e4100d40 0x7f45e419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.217 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e4100d40 0x7f45e419d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47928/0 (socket says 172.21.15.2:47928) 2023-11-16T18:22:53.217 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e3fff700 1 -- 172.21.15.2:0/730471940 learned_addr learned my addr 172.21.15.2:0/730471940 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:53.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e2ffd700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45e41003b0 0x7f45e419adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e37fe700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45e40ff1c0 0x7f45e40699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e3fff700 1 -- 172.21.15.2:0/730471940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45e41003b0 msgr2=0x7f45e419adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e3fff700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45e41003b0 0x7f45e419adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e3fff700 1 -- 172.21.15.2:0/730471940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45e40ff1c0 msgr2=0x7f45e40699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e3fff700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45e40ff1c0 0x7f45e40699c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e3fff700 1 -- 172.21.15.2:0/730471940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc009d00 con 0x7f45e4100d40 2023-11-16T18:22:53.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e3fff700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e4100d40 0x7f45e419d480 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00b780 tx=0x7f45dc00bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:53.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e0ff9700 1 -- 172.21.15.2:0/730471940 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc00e040 con 0x7f45e4100d40 2023-11-16T18:22:53.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.213+0000 7f45e0ff9700 1 -- 172.21.15.2:0/730471940 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45dc004c60 con 0x7f45e4100d40 2023-11-16T18:22:53.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.214+0000 7f45e9d51700 1 -- 172.21.15.2:0/730471940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e410ab40 con 0x7f45e4100d40 2023-11-16T18:22:53.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.214+0000 7f45e0ff9700 1 -- 172.21.15.2:0/730471940 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc01b6b0 con 0x7f45e4100d40 2023-11-16T18:22:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.214+0000 7f45e9d51700 1 -- 172.21.15.2:0/730471940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e410a9b0 con 0x7f45e4100d40 2023-11-16T18:22:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.214+0000 7f45e9d51700 1 -- 172.21.15.2:0/730471940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e410acd0 con 0x7f45e4100d40 2023-11-16T18:22:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.216+0000 7f45e0ff9700 1 -- 172.21.15.2:0/730471940 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f45dc01b810 con 0x7f45e4100d40 2023-11-16T18:22:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.217+0000 7f45e0ff9700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45cc0a1150 0x7f45cc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.217+0000 7f45e37fe700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45cc0a1150 0x7f45cc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.220 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.217+0000 7f45e0ff9700 1 -- 172.21.15.2:0/730471940 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f45dc0cec50 con 0x7f45e4100d40 2023-11-16T18:22:53.220 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.217+0000 7f45e37fe700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45cc0a1150 0x7f45cc0a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f45e4100210 tx=0x7f45d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:53.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.220+0000 7f45e0ff9700 1 -- 172.21.15.2:0/730471940 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f45dc098240 con 0x7f45e4100d40 2023-11-16T18:22:53.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:53 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/399145301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:53.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:53 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2868040917' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:53.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:53 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/448871990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:53.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.422+0000 7f45e9d51700 1 -- 172.21.15.2:0/730471940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45e4105ff0 con 0x7f45e4100d40 2023-11-16T18:22:53.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.424+0000 7f45e0ff9700 1 -- 172.21.15.2:0/730471940 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23853 (secure 0 0 0) 0x7f45dc09b860 con 0x7f45e4100d40 2023-11-16T18:22:53.425 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:53.425 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":86,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:22:51.534683+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:22:53.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.426+0000 7f45e9d51700 1 -- 172.21.15.2:0/730471940 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45cc0a1150 msgr2=0x7f45cc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.426+0000 7f45e9d51700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45cc0a1150 0x7f45cc0a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f45e4100210 tx=0x7f45d8009260 comp rx=0 tx=0).stop 2023-11-16T18:22:53.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.426+0000 7f45e9d51700 1 -- 172.21.15.2:0/730471940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e4100d40 msgr2=0x7f45e419d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.426+0000 7f45e9d51700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e4100d40 0x7f45e419d480 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00b780 tx=0x7f45dc00bd00 comp rx=0 tx=0).stop 2023-11-16T18:22:53.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.426+0000 7f45e9d51700 1 -- 172.21.15.2:0/730471940 shutdown_connections 2023-11-16T18:22:53.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.426+0000 7f45e9d51700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45cc0a1150 0x7f45cc0a3600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.426+0000 7f45e9d51700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e4100d40 0x7f45e419d480 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.426+0000 7f45e9d51700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45e40ff1c0 0x7f45e40699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.426+0000 7f45e9d51700 1 --2- 172.21.15.2:0/730471940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45e41003b0 0x7f45e419adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.426+0000 7f45e9d51700 1 -- 172.21.15.2:0/730471940 >> 172.21.15.2:0/730471940 conn(0x7f45e40fa7a0 msgr2=0x7f45e40fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:53.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.426+0000 7f45e9d51700 1 -- 172.21.15.2:0/730471940 shutdown_connections 2023-11-16T18:22:53.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.426+0000 7f45e9d51700 1 -- 172.21.15.2:0/730471940 wait complete. 2023-11-16T18:22:53.440 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-16T18:22:46.806019+0000', 'modified': '2023-11-16T18:22:51.534922+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24473}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24473': {'gid': 24473, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6839/3162585010', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 3162585010}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 3162585010}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-11-16T18:22:53.440 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-16T18:22:53.440 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-16T18:22:53.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:53 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/399145301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:53.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:53 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2868040917' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:53.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:53 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/448871990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:53.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:53 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/399145301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:53.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:53 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2868040917' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:53.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:53 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/448871990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:22:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.566+0000 7ffa6734d700 1 -- 172.21.15.2:0/1324248989 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa60103610 msgr2=0x7ffa601059f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.566+0000 7ffa6734d700 1 --2- 172.21.15.2:0/1324248989 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa60103610 0x7ffa601059f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009990 tx=0x7ffa50009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.566+0000 7ffa6734d700 1 -- 172.21.15.2:0/1324248989 shutdown_connections 2023-11-16T18:22:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.566+0000 7ffa6734d700 1 --2- 172.21.15.2:0/1324248989 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa60100cf0 0x7ffa601030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.566+0000 7ffa6734d700 1 --2- 172.21.15.2:0/1324248989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa60105f30 0x7ffa60108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.566+0000 7ffa6734d700 1 --2- 172.21.15.2:0/1324248989 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa60103610 0x7ffa601059f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.566+0000 7ffa6734d700 1 -- 172.21.15.2:0/1324248989 >> 172.21.15.2:0/1324248989 conn(0x7ffa600fa6f0 msgr2=0x7ffa600fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.566+0000 7ffa6734d700 1 -- 172.21.15.2:0/1324248989 shutdown_connections 2023-11-16T18:22:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.566+0000 7ffa6734d700 1 -- 172.21.15.2:0/1324248989 wait complete. 2023-11-16T18:22:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.566+0000 7ffa6734d700 1 Processor -- start 2023-11-16T18:22:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.566+0000 7ffa6734d700 1 -- start start 2023-11-16T18:22:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa6734d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa60100cf0 0x7ffa60100660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa6734d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa60103610 0x7ffa600feca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa6734d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa60105f30 0x7ffa600ff1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa6734d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600ff980 con 0x7ffa60105f30 2023-11-16T18:22:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa6734d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600ffaf0 con 0x7ffa60103610 2023-11-16T18:22:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa6734d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6019ffc0 con 0x7ffa60100cf0 2023-11-16T18:22:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa658ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa60105f30 0x7ffa600ff1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa658ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa60105f30 0x7ffa600ff1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47942/0 (socket says 172.21.15.2:47942) 2023-11-16T18:22:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa658ea700 1 -- 172.21.15.2:0/1429606032 learned_addr learned my addr 172.21.15.2:0/1429606032 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:53.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa648e8700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa60103610 0x7ffa600feca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa650e9700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa60100cf0 0x7ffa60100660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa658ea700 1 -- 172.21.15.2:0/1429606032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa60100cf0 msgr2=0x7ffa60100660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa658ea700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa60100cf0 0x7ffa60100660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa658ea700 1 -- 172.21.15.2:0/1429606032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa60103610 msgr2=0x7ffa600feca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa658ea700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa60103610 0x7ffa600feca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa658ea700 1 -- 172.21.15.2:0/1429606032 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa500095f0 con 0x7ffa60105f30 2023-11-16T18:22:53.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa658ea700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa60105f30 0x7ffa600ff1e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffa58007ee0 tx=0x7ffa5800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:53.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa567fc700 1 -- 172.21.15.2:0/1429606032 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58017070 con 0x7ffa60105f30 2023-11-16T18:22:53.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa567fc700 1 -- 172.21.15.2:0/1429606032 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa5800f040 con 0x7ffa60105f30 2023-11-16T18:22:53.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.567+0000 7ffa6734d700 1 -- 172.21.15.2:0/1429606032 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6010bbd0 con 0x7ffa60105f30 2023-11-16T18:22:53.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.568+0000 7ffa567fc700 1 -- 172.21.15.2:0/1429606032 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58005740 con 0x7ffa60105f30 2023-11-16T18:22:53.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.568+0000 7ffa6734d700 1 -- 172.21.15.2:0/1429606032 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6010ba40 con 0x7ffa60105f30 2023-11-16T18:22:53.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.571+0000 7ffa734d9b80 1 Processor -- start 2023-11-16T18:22:53.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.572+0000 7ffa567fc700 1 -- 172.21.15.2:0/1429606032 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7ffa5800dc60 con 0x7ffa60105f30 2023-11-16T18:22:53.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.572+0000 7ffa734d9b80 1 -- start start 2023-11-16T18:22:53.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.572+0000 7ffa734d9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f21213bb0 0x555f21213fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.572+0000 7ffa734d9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21214500 0x555f21214ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.572+0000 7ffa734d9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f212150c0 0x555f21215530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.572+0000 7ffa734d9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f21119010 con 0x555f21213bb0 2023-11-16T18:22:53.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.572+0000 7ffa734d9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f2116b850 con 0x555f21214500 2023-11-16T18:22:53.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.572+0000 7ffa734d9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f21164ce0 con 0x555f212150c0 2023-11-16T18:22:53.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa648e8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f21213bb0 0x555f21213fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa648e8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f21213bb0 0x555f21213fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47944/0 (socket says 172.21.15.2:47944) 2023-11-16T18:22:53.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa648e8700 1 -- 172.21.15.2:0/2663960929 learned_addr learned my addr 172.21.15.2:0/2663960929 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:53.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa650e9700 1 --2- 172.21.15.2:0/2663960929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21214500 0x555f21214ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa648e8700 1 --2- 172.21.15.2:0/2663960929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f212150c0 0x555f21215530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa648e8700 1 -- 172.21.15.2:0/2663960929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f212150c0 msgr2=0x555f21215530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:22:53.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa648e8700 1 --2- 172.21.15.2:0/2663960929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f212150c0 0x555f21215530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa648e8700 1 -- 172.21.15.2:0/2663960929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21214500 msgr2=0x555f21214ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:22:53.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa648e8700 1 --2- 172.21.15.2:0/2663960929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21214500 0x555f21214ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa648e8700 1 -- 172.21.15.2:0/2663960929 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f20fe65e0 con 0x555f21213bb0 2023-11-16T18:22:53.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa567fc700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ffa4c0a1150 0x7ffa4c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa648e8700 1 --2- 172.21.15.2:0/2663960929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f21213bb0 0x555f21213fc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffa50015000 tx=0x7ffa50005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb215774279dd4c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:53.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa650e9700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ffa4c0a1150 0x7ffa4c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa567fc700 1 -- 172.21.15.2:0/1429606032 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ffa580cc340 con 0x7ffa60105f30 2023-11-16T18:22:53.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa4b7fe700 1 -- 172.21.15.2:0/2663960929 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5001d070 con 0x555f21213bb0 2023-11-16T18:22:53.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa4b7fe700 1 -- 172.21.15.2:0/2663960929 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa500059c0 con 0x555f21213bb0 2023-11-16T18:22:53.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.573+0000 7ffa4b7fe700 1 -- 172.21.15.2:0/2663960929 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa50003bd0 con 0x555f21213bb0 2023-11-16T18:22:53.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.574+0000 7ffa734d9b80 1 -- 172.21.15.2:0/2663960929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f21213bb0 msgr2=0x555f21213fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.574+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/2663960929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f21213bb0 0x555f21213fc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffa50015000 tx=0x7ffa50005e80 comp rx=0 tx=0).stop 2023-11-16T18:22:53.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.574+0000 7ffa650e9700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ffa4c0a1150 0x7ffa4c0a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x555f2120f460 tx=0x7ffa5c00c4d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:53.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.574+0000 7ffa734d9b80 1 -- 172.21.15.2:0/2663960929 shutdown_connections 2023-11-16T18:22:53.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.574+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/2663960929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f212150c0 0x555f21215530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.574+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/2663960929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f21213bb0 0x555f21213fc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.574+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/2663960929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21214500 0x555f21214ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.574+0000 7ffa734d9b80 1 -- 172.21.15.2:0/2663960929 >> 172.21.15.2:0/2663960929 conn(0x555f21209310 msgr2=0x555f2120e620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:53.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.574+0000 7ffa734d9b80 1 -- 172.21.15.2:0/2663960929 shutdown_connections 2023-11-16T18:22:53.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.574+0000 7ffa734d9b80 1 -- 172.21.15.2:0/2663960929 wait complete. 2023-11-16T18:22:53.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.574+0000 7ffa734d9b80 1 -- start start 2023-11-16T18:22:53.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa734d9b80 1 Processor -- start 2023-11-16T18:22:53.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa734d9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21213bb0 0x555f2121d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa734d9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21214500 0x555f2121da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa734d9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f21224ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa734d9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f21106890 con 0x555f212150c0 2023-11-16T18:22:53.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa734d9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f21106530 con 0x555f21214500 2023-11-16T18:22:53.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa734d9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f20fe88d0 con 0x555f21213bb0 2023-11-16T18:22:53.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa734d9b80 10 client.0 _ll_register_callbacks cb 0x555f21124730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:22:53.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa658ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f21224ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa658ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f21224ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47956/0 (socket says 172.21.15.2:47956) 2023-11-16T18:22:53.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa658ea700 1 -- 172.21.15.2:0/58373889 learned_addr learned my addr 172.21.15.2:0/58373889 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:53.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa648e8700 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21214500 0x555f2121da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa650e9700 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21213bb0 0x555f2121d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa658ea700 1 -- 172.21.15.2:0/58373889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21213bb0 msgr2=0x555f2121d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:22:53.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa658ea700 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21213bb0 0x555f2121d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa658ea700 1 -- 172.21.15.2:0/58373889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21214500 msgr2=0x555f2121da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:22:53.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa658ea700 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21214500 0x555f2121da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa658ea700 1 -- 172.21.15.2:0/58373889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa50020040 con 0x555f212150c0 2023-11-16T18:22:53.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.575+0000 7ffa648e8700 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21214500 0x555f2121da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:22:53.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa658ea700 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f21224ad0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffa58099260 tx=0x7ffa58098f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed6f9100b97e1f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:53.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa48ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:22:53.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa48ff9700 1 -- 172.21.15.2:0/58373889 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa580a19b0 con 0x555f212150c0 2023-11-16T18:22:53.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa48ff9700 1 -- 172.21.15.2:0/58373889 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ffa580058a0 con 0x555f212150c0 2023-11-16T18:22:53.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa734d9b80 1 -- 172.21.15.2:0/58373889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555f21206ad0 con 0x555f212150c0 2023-11-16T18:22:53.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa48ff9700 1 -- 172.21.15.2:0/58373889 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa58005a60 con 0x555f212150c0 2023-11-16T18:22:53.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa734d9b80 10 client.14727 fetch_fsmap learned FSMap version 17 2023-11-16T18:22:53.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa734d9b80 1 -- 172.21.15.2:0/58373889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x555f2112b140 con 0x555f212150c0 2023-11-16T18:22:53.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa48ff9700 1 -- 172.21.15.2:0/58373889 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa580b5020 con 0x555f212150c0 2023-11-16T18:22:53.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa48ff9700 1 -- 172.21.15.2:0/58373889 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7ffa580998c0 con 0x555f212150c0 2023-11-16T18:22:53.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa734d9b80 10 client.14727 fetch_fsmap finished waiting for FSMap version 17 2023-11-16T18:22:53.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa734d9b80 10 client.14727 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T18:22:53.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa734d9b80 20 client.14727 populate_metadata read hostname 'smithi002' 2023-11-16T18:22:53.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555f212286b0 0x555f21227b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa734d9b80 4 client.14727 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:22:53.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.576+0000 7ffa734d9b80 1 -- 172.21.15.2:0/58373889 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555f21127050 con 0x555f212286b0 2023-11-16T18:22:53.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.577+0000 7ffa650e9700 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555f212286b0 0x555f21227b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.577+0000 7ffa650e9700 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555f212286b0 0x555f21227b70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73f1f492d419aaab server_cookie=b5cc418092927850 in_seq=0 out_seq=0 2023-11-16T18:22:53.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.577+0000 7ffa48ff9700 10 client.14727 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:22:53.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa48ff9700 1 -- 172.21.15.2:0/58373889 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555f21127050 con 0x555f212286b0 2023-11-16T18:22:53.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa48ff9700 10 client.14727 handle_command_reply: tid=0 2023-11-16T18:22:53.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa734d9b80 1 client.14727 shutdown 2023-11-16T18:22:53.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa734d9b80 1 -- 172.21.15.2:0/58373889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 msgr2=0x555f21224ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f21224ad0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffa58099260 tx=0x7ffa58098f70 comp rx=0 tx=0).stop 2023-11-16T18:22:53.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa734d9b80 1 -- 172.21.15.2:0/58373889 shutdown_connections 2023-11-16T18:22:53.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555f212286b0 0x555f21227b70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21213bb0 0x555f2121d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa48ff9700 0 client.14727 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:22:53.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f21224ad0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/58373889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21214500 0x555f2121da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa734d9b80 1 -- 172.21.15.2:0/58373889 >> 172.21.15.2:0/58373889 conn(0x555f21209310 msgr2=0x555f2120dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:53.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa734d9b80 1 -- 172.21.15.2:0/58373889 shutdown_connections 2023-11-16T18:22:53.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.578+0000 7ffa734d9b80 1 -- 172.21.15.2:0/58373889 wait complete. 2023-11-16T18:22:53.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.579+0000 7ffa734d9b80 20 client.14727 trim_cache size 0 max 16384 2023-11-16T18:22:53.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa734d9b80 1 Processor -- start 2023-11-16T18:22:53.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa734d9b80 1 -- start start 2023-11-16T18:22:53.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa734d9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21213bb0 0x555f2121b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa734d9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21214500 0x555f21224a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa734d9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f21217fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa734d9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f212148d0 con 0x555f212150c0 2023-11-16T18:22:53.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa734d9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f2121e450 con 0x555f21213bb0 2023-11-16T18:22:53.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa734d9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f21127050 con 0x555f21214500 2023-11-16T18:22:53.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa658ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f21217fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa658ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f21217fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47964/0 (socket says 172.21.15.2:47964) 2023-11-16T18:22:53.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa658ea700 1 -- 172.21.15.2:0/3590571562 learned_addr learned my addr 172.21.15.2:0/3590571562 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:53.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa648e8700 1 --2- 172.21.15.2:0/3590571562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21214500 0x555f21224a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa650e9700 1 --2- 172.21.15.2:0/3590571562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21213bb0 0x555f2121b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa658ea700 1 -- 172.21.15.2:0/3590571562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21214500 msgr2=0x555f21224a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:22:53.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa658ea700 1 --2- 172.21.15.2:0/3590571562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21214500 0x555f21224a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa658ea700 1 -- 172.21.15.2:0/3590571562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21213bb0 msgr2=0x555f2121b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:22:53.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa658ea700 1 --2- 172.21.15.2:0/3590571562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21213bb0 0x555f2121b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa658ea700 1 -- 172.21.15.2:0/3590571562 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f2112ae90 con 0x555f212150c0 2023-11-16T18:22:53.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa658ea700 1 --2- 172.21.15.2:0/3590571562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f21217fb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800d7b0 tx=0x7ffa58022530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0df7764dab988b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:53.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa4bfff700 1 -- 172.21.15.2:0/3590571562 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa580842b0 con 0x555f212150c0 2023-11-16T18:22:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa4bfff700 1 -- 172.21.15.2:0/3590571562 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa580b5030 con 0x555f212150c0 2023-11-16T18:22:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.597+0000 7ffa4bfff700 1 -- 172.21.15.2:0/3590571562 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa580058a0 con 0x555f212150c0 2023-11-16T18:22:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.598+0000 7ffa734d9b80 1 -- 172.21.15.2:0/3590571562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 msgr2=0x555f21217fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.598+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/3590571562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f21217fb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800d7b0 tx=0x7ffa58022530 comp rx=0 tx=0).stop 2023-11-16T18:22:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.598+0000 7ffa734d9b80 1 -- 172.21.15.2:0/3590571562 shutdown_connections 2023-11-16T18:22:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.598+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/3590571562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f21217fb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.598+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/3590571562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21213bb0 0x555f2121b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.598+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/3590571562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21214500 0x555f21224a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.598+0000 7ffa734d9b80 1 -- 172.21.15.2:0/3590571562 >> 172.21.15.2:0/3590571562 conn(0x555f21209310 msgr2=0x555f2121c240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.598+0000 7ffa734d9b80 1 -- 172.21.15.2:0/3590571562 shutdown_connections 2023-11-16T18:22:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.598+0000 7ffa734d9b80 1 -- 172.21.15.2:0/3590571562 wait complete. 2023-11-16T18:22:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.598+0000 7ffa734d9b80 1 -- start start 2023-11-16T18:22:53.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.598+0000 7ffa734d9b80 1 Processor -- start 2023-11-16T18:22:53.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa734d9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21213bb0 0x555f2121f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa734d9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21214500 0x555f2121fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa734d9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f2127c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa734d9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f21224b60 con 0x555f212150c0 2023-11-16T18:22:53.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa734d9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f21224cd0 con 0x555f21213bb0 2023-11-16T18:22:53.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa734d9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f21224e90 con 0x555f21214500 2023-11-16T18:22:53.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa734d9b80 10 client.0 _ll_register_callbacks cb 0x7ffa58099330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:22:53.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa658ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f2127c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa650e9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21213bb0 0x555f2121f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa658ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f2127c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47970/0 (socket says 172.21.15.2:47970) 2023-11-16T18:22:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa658ea700 1 -- 172.21.15.2:0/4132813272 learned_addr learned my addr 172.21.15.2:0/4132813272 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa648e8700 1 --2- 172.21.15.2:0/4132813272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21214500 0x555f2121fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa658ea700 1 -- 172.21.15.2:0/4132813272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21214500 msgr2=0x555f2121fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:22:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa658ea700 1 --2- 172.21.15.2:0/4132813272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21214500 0x555f2121fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa658ea700 1 -- 172.21.15.2:0/4132813272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21213bb0 msgr2=0x555f2121f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:22:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa658ea700 1 --2- 172.21.15.2:0/4132813272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21213bb0 0x555f2121f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa658ea700 1 -- 172.21.15.2:0/4132813272 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f2112b140 con 0x555f212150c0 2023-11-16T18:22:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.599+0000 7ffa650e9700 1 --2- 172.21.15.2:0/4132813272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21213bb0 0x555f2121f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:22:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa658ea700 1 --2- 172.21.15.2:0/4132813272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f2127c550 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffa580225c0 tx=0x7ffa580b7ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a3f99c5224ac79 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa48ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:22:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa48ff9700 1 -- 172.21.15.2:0/4132813272 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58005aa0 con 0x555f212150c0 2023-11-16T18:22:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa48ff9700 1 -- 172.21.15.2:0/4132813272 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ffa580b6080 con 0x555f212150c0 2023-11-16T18:22:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa734d9b80 1 -- 172.21.15.2:0/4132813272 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffa500059c0 con 0x555f212150c0 2023-11-16T18:22:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa48ff9700 1 -- 172.21.15.2:0/4132813272 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa58098c40 con 0x555f212150c0 2023-11-16T18:22:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa734d9b80 10 client.14733 fetch_fsmap learned FSMap version 17 2023-11-16T18:22:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa734d9b80 1 -- 172.21.15.2:0/4132813272 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x555f20f7bc10 con 0x555f212150c0 2023-11-16T18:22:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa48ff9700 1 -- 172.21.15.2:0/4132813272 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa500059c0 con 0x555f212150c0 2023-11-16T18:22:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa48ff9700 1 -- 172.21.15.2:0/4132813272 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7ffa580a1620 con 0x555f212150c0 2023-11-16T18:22:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa734d9b80 10 client.14733 fetch_fsmap finished waiting for FSMap version 17 2023-11-16T18:22:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa734d9b80 10 client.14733 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T18:22:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa734d9b80 20 client.14733 populate_metadata read hostname 'smithi002' 2023-11-16T18:22:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/4132813272 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555f21280e80 0x555f2127db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa734d9b80 4 client.14733 mds_command: new command op to 24473 tid=0[{"prefix": "status"}] 2023-11-16T18:22:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.600+0000 7ffa734d9b80 1 -- 172.21.15.2:0/4132813272 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555f21133610 con 0x555f21280e80 2023-11-16T18:22:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa650e9700 1 --2- 172.21.15.2:0/4132813272 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555f21280e80 0x555f2127db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa650e9700 1 --2- 172.21.15.2:0/4132813272 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555f21280e80 0x555f2127db20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e3286ccf5983ba9 server_cookie=653e1ff60e51cf34 in_seq=0 out_seq=0 2023-11-16T18:22:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa48ff9700 10 client.14733 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:22:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa48ff9700 1 -- 172.21.15.2:0/4132813272 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x555f21133610 con 0x555f21280e80 2023-11-16T18:22:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa48ff9700 10 client.14733 handle_command_reply: tid=0 2023-11-16T18:22:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa734d9b80 1 client.14733 shutdown 2023-11-16T18:22:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa734d9b80 1 -- 172.21.15.2:0/4132813272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 msgr2=0x555f2127c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/4132813272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f2127c550 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffa580225c0 tx=0x7ffa580b7ec0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa734d9b80 1 -- 172.21.15.2:0/4132813272 shutdown_connections 2023-11-16T18:22:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/4132813272 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555f21280e80 0x555f2127db20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/4132813272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555f212150c0 0x555f2127c550 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/4132813272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f21213bb0 0x555f2121f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa48ff9700 0 client.14733 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:22:53.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.601+0000 7ffa734d9b80 1 --2- 172.21.15.2:0/4132813272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f21214500 0x555f2121fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa734d9b80 1 -- 172.21.15.2:0/4132813272 >> 172.21.15.2:0/4132813272 conn(0x555f21209310 msgr2=0x555f2121b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:53.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa734d9b80 1 -- 172.21.15.2:0/4132813272 shutdown_connections 2023-11-16T18:22:53.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa734d9b80 1 -- 172.21.15.2:0/4132813272 wait complete. 2023-11-16T18:22:53.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa734d9b80 20 client.14733 trim_cache size 0 max 16384 2023-11-16T18:22:53.611 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:22:53.612 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "08160fac-84ac-11ee-8dbe-212e2dc638e7", 2023-11-16T18:22:53.612 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2023-11-16T18:22:53.612 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24473, 2023-11-16T18:22:53.612 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T18:22:53.612 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T18:22:53.612 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T18:22:53.612 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 6.7933263210000003, 2023-11-16T18:22:53.612 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 17, 2023-11-16T18:22:53.612 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 86, 2023-11-16T18:22:53.612 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 85, 2023-11-16T18:22:53.612 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 12.393508766 2023-11-16T18:22:53.612 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:22:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa6734d700 1 -- 172.21.15.2:0/1429606032 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ffa4c0a1150 msgr2=0x7ffa4c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa6734d700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ffa4c0a1150 0x7ffa4c0a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x555f2120f460 tx=0x7ffa5c00c4d0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa6734d700 1 -- 172.21.15.2:0/1429606032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa60105f30 msgr2=0x7ffa600ff1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa6734d700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa60105f30 0x7ffa600ff1e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffa58007ee0 tx=0x7ffa5800d3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa6734d700 1 -- 172.21.15.2:0/1429606032 shutdown_connections 2023-11-16T18:22:53.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa6734d700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ffa4c0a1150 0x7ffa4c0a3600 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa6734d700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa60100cf0 0x7ffa60100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa6734d700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa60105f30 0x7ffa600ff1e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa6734d700 1 --2- 172.21.15.2:0/1429606032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa60103610 0x7ffa600feca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:53.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa6734d700 1 -- 172.21.15.2:0/1429606032 >> 172.21.15.2:0/1429606032 conn(0x7ffa600fa6f0 msgr2=0x7ffa60101950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:53.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa6734d700 1 -- 172.21.15.2:0/1429606032 shutdown_connections 2023-11-16T18:22:53.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:53.602+0000 7ffa6734d700 1 -- 172.21.15.2:0/1429606032 wait complete. 2023-11-16T18:22:53.615 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-16T18:22:53.616 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-11-16T18:22:53.616 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph fs ls 2023-11-16T18:22:54.261 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:22:54.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:54 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/730471940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:54.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:54 smithi002 ceph-mon[98807]: pgmap v264: 129 pgs: 112 active+clean, 12 creating+peering, 5 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 4 op/s 2023-11-16T18:22:54.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:54 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/730471940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:54.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:54 smithi040 ceph-mon[100251]: pgmap v264: 129 pgs: 112 active+clean, 12 creating+peering, 5 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 4 op/s 2023-11-16T18:22:54.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:54 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/730471940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:54.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:54 smithi188 ceph-mon[100639]: pgmap v264: 129 pgs: 112 active+clean, 12 creating+peering, 5 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 4 op/s 2023-11-16T18:22:55.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.483+0000 7fe329b12700 1 -- 172.21.15.2:0/211873241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3240ffe00 msgr2=0x7fe324103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:55.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.483+0000 7fe329b12700 1 --2- 172.21.15.2:0/211873241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3240ffe00 0x7fe324103dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe3140099c0 tx=0x7fe314009cd0 comp rx=0 tx=0).stop 2023-11-16T18:22:55.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.483+0000 7fe329b12700 1 -- 172.21.15.2:0/211873241 shutdown_connections 2023-11-16T18:22:55.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.483+0000 7fe329b12700 1 --2- 172.21.15.2:0/211873241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe324104500 0x7fe32410c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:55.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.483+0000 7fe329b12700 1 --2- 172.21.15.2:0/211873241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3240ff460 0x7fe3240ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:55.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.483+0000 7fe329b12700 1 --2- 172.21.15.2:0/211873241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3240ffe00 0x7fe324103dd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:55.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.483+0000 7fe329b12700 1 -- 172.21.15.2:0/211873241 >> 172.21.15.2:0/211873241 conn(0x7fe3240770a0 msgr2=0x7fe3240774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:55.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.483+0000 7fe329b12700 1 -- 172.21.15.2:0/211873241 shutdown_connections 2023-11-16T18:22:55.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.483+0000 7fe329b12700 1 -- 172.21.15.2:0/211873241 wait complete. 2023-11-16T18:22:55.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.484+0000 7fe329b12700 1 Processor -- start 2023-11-16T18:22:55.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.484+0000 7fe329b12700 1 -- start start 2023-11-16T18:22:55.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.484+0000 7fe329b12700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3240ff460 0x7fe324196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:55.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.484+0000 7fe329b12700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3240ffe00 0x7fe3241969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:55.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.484+0000 7fe329b12700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe324104500 0x7fe32419ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:55.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.484+0000 7fe329b12700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32419b450 con 0x7fe3240ffe00 2023-11-16T18:22:55.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.484+0000 7fe329b12700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32419b5c0 con 0x7fe3240ff460 2023-11-16T18:22:55.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.484+0000 7fe329b12700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32419b780 con 0x7fe324104500 2023-11-16T18:22:55.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.484+0000 7fe322ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3240ffe00 0x7fe3241969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:55.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.484+0000 7fe322ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3240ffe00 0x7fe3241969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47980/0 (socket says 172.21.15.2:47980) 2023-11-16T18:22:55.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.484+0000 7fe322ffd700 1 -- 172.21.15.2:0/4239686133 learned_addr learned my addr 172.21.15.2:0/4239686133 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:55.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.484+0000 7fe3237fe700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3240ff460 0x7fe324196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:55.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.485+0000 7fe323fff700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe324104500 0x7fe32419ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:55.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.485+0000 7fe322ffd700 1 -- 172.21.15.2:0/4239686133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe324104500 msgr2=0x7fe32419ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:55.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.485+0000 7fe322ffd700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe324104500 0x7fe32419ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:55.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.485+0000 7fe322ffd700 1 -- 172.21.15.2:0/4239686133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3240ff460 msgr2=0x7fe324196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:55.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.485+0000 7fe322ffd700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3240ff460 0x7fe324196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:55.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.485+0000 7fe322ffd700 1 -- 172.21.15.2:0/4239686133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3140095f0 con 0x7fe3240ffe00 2023-11-16T18:22:55.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.485+0000 7fe323fff700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe324104500 0x7fe32419ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:22:55.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.485+0000 7fe322ffd700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3240ffe00 0x7fe3241969d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe314009990 tx=0x7fe314004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:55.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.485+0000 7fe320ff9700 1 -- 172.21.15.2:0/4239686133 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31401d070 con 0x7fe3240ffe00 2023-11-16T18:22:55.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.485+0000 7fe320ff9700 1 -- 172.21.15.2:0/4239686133 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3140054f0 con 0x7fe3240ffe00 2023-11-16T18:22:55.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.485+0000 7fe329b12700 1 -- 172.21.15.2:0/4239686133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3241103e0 con 0x7fe3240ffe00 2023-11-16T18:22:55.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.485+0000 7fe320ff9700 1 -- 172.21.15.2:0/4239686133 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314003ad0 con 0x7fe3240ffe00 2023-11-16T18:22:55.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.485+0000 7fe329b12700 1 -- 172.21.15.2:0/4239686133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe324110250 con 0x7fe3240ffe00 2023-11-16T18:22:55.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.486+0000 7fe329b12700 1 -- 172.21.15.2:0/4239686133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe324110570 con 0x7fe3240ffe00 2023-11-16T18:22:55.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.487+0000 7fe320ff9700 1 -- 172.21.15.2:0/4239686133 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fe3140047a0 con 0x7fe3240ffe00 2023-11-16T18:22:55.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.488+0000 7fe320ff9700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe3100a1150 0x7fe3100a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:55.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.488+0000 7fe3237fe700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe3100a1150 0x7fe3100a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:55.497 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.488+0000 7fe320ff9700 1 -- 172.21.15.2:0/4239686133 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe3140cdcf0 con 0x7fe3240ffe00 2023-11-16T18:22:55.497 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.488+0000 7fe3237fe700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe3100a1150 0x7fe3100a3600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe30c007be0 tx=0x7fe30c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:55.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.491+0000 7fe320ff9700 1 -- 172.21.15.2:0/4239686133 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe3140972e0 con 0x7fe3240ffe00 2023-11-16T18:22:55.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.734+0000 7fe329b12700 1 -- 172.21.15.2:0/4239686133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe324066ea0 con 0x7fe3240ffe00 2023-11-16T18:22:55.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.734+0000 7fe320ff9700 1 -- 172.21.15.2:0/4239686133 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7fe31409a900 con 0x7fe3240ffe00 2023-11-16T18:22:55.737 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-16T18:22:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.737+0000 7fe329b12700 1 -- 172.21.15.2:0/4239686133 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe3100a1150 msgr2=0x7fe3100a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.737+0000 7fe329b12700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe3100a1150 0x7fe3100a3600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe30c007be0 tx=0x7fe30c0073d0 comp rx=0 tx=0).stop 2023-11-16T18:22:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.737+0000 7fe329b12700 1 -- 172.21.15.2:0/4239686133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3240ffe00 msgr2=0x7fe3241969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.737+0000 7fe329b12700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3240ffe00 0x7fe3241969d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe314009990 tx=0x7fe314004730 comp rx=0 tx=0).stop 2023-11-16T18:22:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.737+0000 7fe329b12700 1 -- 172.21.15.2:0/4239686133 shutdown_connections 2023-11-16T18:22:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.737+0000 7fe329b12700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe3100a1150 0x7fe3100a3600 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.737+0000 7fe329b12700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe324104500 0x7fe32419ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.737+0000 7fe329b12700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3240ff460 0x7fe324196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.737+0000 7fe329b12700 1 --2- 172.21.15.2:0/4239686133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3240ffe00 0x7fe3241969d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.737+0000 7fe329b12700 1 -- 172.21.15.2:0/4239686133 >> 172.21.15.2:0/4239686133 conn(0x7fe3240770a0 msgr2=0x7fe3240fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.737+0000 7fe329b12700 1 -- 172.21.15.2:0/4239686133 shutdown_connections 2023-11-16T18:22:55.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:55.737+0000 7fe329b12700 1 -- 172.21.15.2:0/4239686133 wait complete. 2023-11-16T18:22:56.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:55 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/4239686133' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T18:22:56.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:55 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/4239686133' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T18:22:56.222 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-16T18:22:56.223 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-16T18:22:56.223 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2023-11-16T18:22:56.223 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-11-16T18:22:56.223 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-11-16T18:22:56.223 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-16T18:22:56.223 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-16T18:22:56.223 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-16T18:22:56.223 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-11-16T18:22:56.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:56.224 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2023-11-16T18:22:56.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:55 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/4239686133' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T18:22:56.242 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-16T18:22:56.242 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-16T18:22:56.242 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2023-11-16T18:22:56.242 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2023-11-16T18:22:56.243 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2023-11-16T18:22:56.243 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2023-11-16T18:22:56.243 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T18:22:56.243 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2023-11-16T18:22:56.243 INFO:teuthology.orchestra.run.smithi002.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-16T18:22:56.243 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2023-11-16T18:22:56.243 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-16T18:22:56.243 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 41785sec preferred_lft 41785sec 2023-11-16T18:22:56.243 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link 2023-11-16T18:22:56.243 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2023-11-16T18:22:56.243 INFO:teuthology.orchestra.run.smithi002.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T18:22:56.243 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2023-11-16T18:22:56.244 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T18:22:56.244 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2023-11-16T18:22:56.245 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-16T18:22:56.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:56.245 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T18:22:56.245 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T18:22:56.245 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add name ceph-brx type bridge 2023-11-16T18:22:56.245 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr flush dev ceph-brx 2023-11-16T18:22:56.246 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set ceph-brx up 2023-11-16T18:22:56.246 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-16T18:22:56.246 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T18:22:56.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:56.420 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-16T18:22:56.449 INFO:teuthology.orchestra.run.smithi002.stdout:1 2023-11-16T18:22:56.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:56.450 DEBUG:teuthology.orchestra.run.smithi002:> route 2023-11-16T18:22:56.526 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2023-11-16T18:22:56.526 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-16T18:22:56.526 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-16T18:22:56.526 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-16T18:22:56.526 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-16T18:22:56.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:56.527 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T18:22:56.527 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T18:22:56.527 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-16T18:22:56.527 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-16T18:22:56.527 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-16T18:22:56.527 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T18:22:56.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:56.800 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2023-11-16T18:22:56.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:56.818 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list-id 2023-11-16T18:22:56.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:56.875 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T18:22:56.875 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T18:22:56.876 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-16T18:22:56.876 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-11-16T18:22:56.876 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T18:22:56.973 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-11-16T18:22:56.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:56.973 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T18:22:56.973 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T18:22:56.974 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-11-16T18:22:56.974 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-16T18:22:56.974 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-11-16T18:22:56.974 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-11-16T18:22:56.974 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-11-16T18:22:56.974 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T18:22:57.014 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:56 smithi002 ceph-mon[98807]: pgmap v265: 129 pgs: 128 active+clean, 1 creating+peering; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2023-11-16T18:22:57.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:56 smithi188 ceph-mon[100639]: pgmap v265: 129 pgs: 128 active+clean, 1 creating+peering; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2023-11-16T18:22:57.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:57.182 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T18:22:57.182 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T18:22:57.182 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2023-11-16T18:22:57.182 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set dev brx.0 master ceph-brx 2023-11-16T18:22:57.183 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T18:22:57.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:56 smithi040 ceph-mon[100251]: pgmap v265: 129 pgs: 128 active+clean, 1 creating+peering; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2023-11-16T18:22:57.322 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-11-16T18:22:57.322 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T18:22:57.322 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-11-16T18:22:57.339 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-11-16T18:22:57.340 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T18:22:57.340 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-11-16T18:22:57.397 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2023-11-16T18:22:57.465 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-16T18:22:57.465 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-16T18:22:57.536 INFO:teuthology.orchestra.run.smithi002.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-16T18:22:57.537 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-16T18:22:57.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:57.537 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2023-11-16T18:22:57.595 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-16T18:22:57.595 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-11-16T18:22:57.638 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2023-11-16T18:22:57.665 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-16T18:22:57.665 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-16T18:22:57.732 INFO:teuthology.orchestra.run.smithi002.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-16T18:22:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-16T18:22:57.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:57.733 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2023-11-16T18:22:57.745 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi002.stderr:2023-11-16T18:22:57.743+0000 7f63f7c095c0 -1 init, newargv = 0x555a5ef63340 newargc=15 2023-11-16T18:22:57.745 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi002.stderr:ceph-fuse[135801]: starting ceph client 2023-11-16T18:22:57.751 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi002.stderr:ceph-fuse[135801]: starting fuse 2023-11-16T18:22:57.763 INFO:teuthology.orchestra.run.smithi002.stdout:124 2023-11-16T18:22:57.763 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2023-11-16T18:22:57.763 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-11-16T18:22:57.763 DEBUG:teuthology.orchestra.run.smithi002:> import glob 2023-11-16T18:22:57.764 DEBUG:teuthology.orchestra.run.smithi002:> import re 2023-11-16T18:22:57.764 DEBUG:teuthology.orchestra.run.smithi002:> import os 2023-11-16T18:22:57.764 DEBUG:teuthology.orchestra.run.smithi002:> import subprocess 2023-11-16T18:22:57.764 DEBUG:teuthology.orchestra.run.smithi002:> 2023-11-16T18:22:57.764 DEBUG:teuthology.orchestra.run.smithi002:> def _find_admin_socket(client_name): 2023-11-16T18:22:57.764 DEBUG:teuthology.orchestra.run.smithi002:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-11-16T18:22:57.764 DEBUG:teuthology.orchestra.run.smithi002:> files = glob.glob(asok_path) 2023-11-16T18:22:57.764 DEBUG:teuthology.orchestra.run.smithi002:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-11-16T18:22:57.765 DEBUG:teuthology.orchestra.run.smithi002:> 2023-11-16T18:22:57.765 DEBUG:teuthology.orchestra.run.smithi002:> # Given a non-glob path, it better be there 2023-11-16T18:22:57.765 DEBUG:teuthology.orchestra.run.smithi002:> if "*" not in asok_path: 2023-11-16T18:22:57.765 DEBUG:teuthology.orchestra.run.smithi002:> assert(len(files) == 1) 2023-11-16T18:22:57.765 DEBUG:teuthology.orchestra.run.smithi002:> return files[0] 2023-11-16T18:22:57.766 DEBUG:teuthology.orchestra.run.smithi002:> 2023-11-16T18:22:57.766 DEBUG:teuthology.orchestra.run.smithi002:> for f in files: 2023-11-16T18:22:57.766 DEBUG:teuthology.orchestra.run.smithi002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-16T18:22:57.766 DEBUG:teuthology.orchestra.run.smithi002:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-16T18:22:57.766 DEBUG:teuthology.orchestra.run.smithi002:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-16T18:22:57.766 DEBUG:teuthology.orchestra.run.smithi002:> contents = proc_f.read() 2023-11-16T18:22:57.766 DEBUG:teuthology.orchestra.run.smithi002:> if mountpoint in contents: 2023-11-16T18:22:57.766 DEBUG:teuthology.orchestra.run.smithi002:> return f 2023-11-16T18:22:57.766 DEBUG:teuthology.orchestra.run.smithi002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-16T18:22:57.766 DEBUG:teuthology.orchestra.run.smithi002:> 2023-11-16T18:22:57.766 DEBUG:teuthology.orchestra.run.smithi002:> print(_find_admin_socket("client.admin")) 2023-11-16T18:22:57.766 DEBUG:teuthology.orchestra.run.smithi002:> ' 2023-11-16T18:22:57.890 INFO:teuthology.orchestra.run.smithi002.stdout:/var/run/ceph/ceph-client.admin.135801.asok 2023-11-16T18:22:58.072 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.135801.asok 2023-11-16T18:22:58.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:58.072 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.135801.asok status 2023-11-16T18:22:58.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:57 smithi002 ceph-mon[98807]: pgmap v266: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 2 op/s 2023-11-16T18:22:58.225 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:22:58.225 INFO:teuthology.orchestra.run.smithi002.stdout: "metadata": { 2023-11-16T18:22:58.225 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_sha1": "5e4a743ad7fd1422ebd6c4dc950a21a303279207", 2023-11-16T18:22:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev)", 2023-11-16T18:22:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "entity_id": "admin", 2023-11-16T18:22:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": "smithi002", 2023-11-16T18:22:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-11-16T18:22:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "pid": "135801", 2023-11-16T18:22:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "root": "/" 2023-11-16T18:22:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:22:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_count": 0, 2023-11-16T18:22:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_pinned_count": 0, 2023-11-16T18:22:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 14745, 2023-11-16T18:22:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "inst": { 2023-11-16T18:22:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "name": { 2023-11-16T18:22:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "client", 2023-11-16T18:22:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "num": 14745 2023-11-16T18:22:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:22:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2023-11-16T18:22:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-16T18:22:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2023-11-16T18:22:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1702669555 2023-11-16T18:22:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:22:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:22:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2023-11-16T18:22:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-16T18:22:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2023-11-16T18:22:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1702669555 2023-11-16T18:22:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:22:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "inst_str": "client.14745 192.168.0.1:0/1702669555", 2023-11-16T18:22:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "addr_str": "192.168.0.1:0/1702669555", 2023-11-16T18:22:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "inode_count": 1, 2023-11-16T18:22:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_epoch": 17, 2023-11-16T18:22:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch": 86, 2023-11-16T18:22:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch_barrier": 0, 2023-11-16T18:22:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "blocklisted": false, 2023-11-16T18:22:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs" 2023-11-16T18:22:58.228 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:22:58.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:57 smithi040 ceph-mon[100251]: pgmap v266: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 2 op/s 2023-11-16T18:22:58.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:58.236 DEBUG:teuthology.orchestra.run.smithi002:> stat --file-system '--printf=%T 2023-11-16T18:22:58.236 DEBUG:teuthology.orchestra.run.smithi002:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-11-16T18:22:58.254 INFO:teuthology.orchestra.run.smithi002.stdout:fuseblk 2023-11-16T18:22:58.254 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-11-16T18:22:58.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:58.255 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-11-16T18:22:58.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:57 smithi188 ceph-mon[100639]: pgmap v266: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 2 op/s 2023-11-16T18:22:58.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:22:58.323 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-11-16T18:22:58.430 INFO:teuthology.orchestra.run.smithi002.stdout:# file: . 2023-11-16T18:22:58.430 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-11-16T18:22:58.430 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:58.430 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:22:58.430 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T18:22:58.449 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:22:58.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.613+0000 7f06963ee700 1 -- 172.21.15.2:0/3445240782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06900ffdc0 msgr2=0x7f0690107670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:58.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.613+0000 7f06963ee700 1 --2- 172.21.15.2:0/3445240782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06900ffdc0 0x7f0690107670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f068800b0a0 tx=0x7f068800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.613+0000 7f06963ee700 1 -- 172.21.15.2:0/3445240782 shutdown_connections 2023-11-16T18:22:58.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.613+0000 7f06963ee700 1 --2- 172.21.15.2:0/3445240782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0690100850 0x7f0690100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.613+0000 7f06963ee700 1 --2- 172.21.15.2:0/3445240782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06900ffdc0 0x7f0690107670 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.613+0000 7f06963ee700 1 --2- 172.21.15.2:0/3445240782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06900ff410 0x7f06900ff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.613+0000 7f06963ee700 1 -- 172.21.15.2:0/3445240782 >> 172.21.15.2:0/3445240782 conn(0x7f06900f9f50 msgr2=0x7f06900fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:58.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.613+0000 7f06963ee700 1 -- 172.21.15.2:0/3445240782 shutdown_connections 2023-11-16T18:22:58.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.613+0000 7f06963ee700 1 -- 172.21.15.2:0/3445240782 wait complete. 2023-11-16T18:22:58.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.614+0000 7f06963ee700 1 Processor -- start 2023-11-16T18:22:58.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.614+0000 7f06963ee700 1 -- start start 2023-11-16T18:22:58.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.614+0000 7f06963ee700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06900ff410 0x7f0690198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:58.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.614+0000 7f06963ee700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06900ffdc0 0x7f0690198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:58.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.614+0000 7f06963ee700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0690100850 0x7f069019f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:58.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.614+0000 7f06963ee700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069019fe80 con 0x7f06900ffdc0 2023-11-16T18:22:58.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.614+0000 7f06963ee700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069019ffc0 con 0x7f0690100850 2023-11-16T18:22:58.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.614+0000 7f06963ee700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06901a0150 con 0x7f06900ff410 2023-11-16T18:22:58.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.614+0000 7f068f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06900ffdc0 0x7f0690198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:58.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.614+0000 7f068f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06900ffdc0 0x7f0690198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48006/0 (socket says 172.21.15.2:48006) 2023-11-16T18:22:58.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.614+0000 7f068f7fe700 1 -- 172.21.15.2:0/2218121007 learned_addr learned my addr 172.21.15.2:0/2218121007 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:58.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.614+0000 7f068ffff700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06900ff410 0x7f0690198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:58.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.615+0000 7f069498b700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0690100850 0x7f069019f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:58.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.615+0000 7f068f7fe700 1 -- 172.21.15.2:0/2218121007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06900ff410 msgr2=0x7f0690198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:58.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.615+0000 7f068f7fe700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06900ff410 0x7f0690198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.615+0000 7f068f7fe700 1 -- 172.21.15.2:0/2218121007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0690100850 msgr2=0x7f069019f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:58.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.615+0000 7f068f7fe700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0690100850 0x7f069019f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.615+0000 7f068f7fe700 1 -- 172.21.15.2:0/2218121007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0688009d00 con 0x7f06900ffdc0 2023-11-16T18:22:58.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.615+0000 7f069498b700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0690100850 0x7f069019f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:22:58.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.615+0000 7f068f7fe700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06900ffdc0 0x7f0690198840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f068400d950 tx=0x7f068400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:58.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.615+0000 7f068d7fa700 1 -- 172.21.15.2:0/2218121007 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684009940 con 0x7f06900ffdc0 2023-11-16T18:22:58.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.615+0000 7f068d7fa700 1 -- 172.21.15.2:0/2218121007 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0684010460 con 0x7f06900ffdc0 2023-11-16T18:22:58.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.615+0000 7f06963ee700 1 -- 172.21.15.2:0/2218121007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069010abd0 con 0x7f06900ffdc0 2023-11-16T18:22:58.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.615+0000 7f068d7fa700 1 -- 172.21.15.2:0/2218121007 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068400f620 con 0x7f06900ffdc0 2023-11-16T18:22:58.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.615+0000 7f06963ee700 1 -- 172.21.15.2:0/2218121007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069010aa40 con 0x7f06900ffdc0 2023-11-16T18:22:58.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.616+0000 7f06963ee700 1 -- 172.21.15.2:0/2218121007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06901a0890 con 0x7f06900ffdc0 2023-11-16T18:22:58.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.617+0000 7f068d7fa700 1 -- 172.21.15.2:0/2218121007 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f06840108d0 con 0x7f06900ffdc0 2023-11-16T18:22:58.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.618+0000 7f068d7fa700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f06780a11d0 0x7f06780a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:58.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.618+0000 7f068d7fa700 1 -- 172.21.15.2:0/2218121007 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f06840cc900 con 0x7f06900ffdc0 2023-11-16T18:22:58.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.618+0000 7f068ffff700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f06780a11d0 0x7f06780a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:58.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.618+0000 7f068ffff700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f06780a11d0 0x7f06780a3680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0680005950 tx=0x7f068000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:58.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.621+0000 7f068d7fa700 1 -- 172.21.15.2:0/2218121007 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0684095e60 con 0x7f06900ffdc0 2023-11-16T18:22:58.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.821+0000 7f06963ee700 1 -- 172.21.15.2:0/2218121007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06901a0e40 con 0x7f06900ffdc0 2023-11-16T18:22:58.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.823+0000 7f068d7fa700 1 -- 172.21.15.2:0/2218121007 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23853 (secure 0 0 0) 0x7f0684020090 con 0x7f06900ffdc0 2023-11-16T18:22:58.825 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:22:58.825 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":86,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:22:51.534683+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:22:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.825+0000 7f06963ee700 1 -- 172.21.15.2:0/2218121007 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f06780a11d0 msgr2=0x7f06780a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.825+0000 7f06963ee700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f06780a11d0 0x7f06780a3680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0680005950 tx=0x7f068000b410 comp rx=0 tx=0).stop 2023-11-16T18:22:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.825+0000 7f06963ee700 1 -- 172.21.15.2:0/2218121007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06900ffdc0 msgr2=0x7f0690198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.825+0000 7f06963ee700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06900ffdc0 0x7f0690198840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f068400d950 tx=0x7f068400dc60 comp rx=0 tx=0).stop 2023-11-16T18:22:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.825+0000 7f06963ee700 1 -- 172.21.15.2:0/2218121007 shutdown_connections 2023-11-16T18:22:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.825+0000 7f06963ee700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f06780a11d0 0x7f06780a3680 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.825+0000 7f06963ee700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06900ff410 0x7f0690198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.825+0000 7f06963ee700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0690100850 0x7f069019f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.825+0000 7f06963ee700 1 --2- 172.21.15.2:0/2218121007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06900ffdc0 0x7f0690198840 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.825+0000 7f06963ee700 1 -- 172.21.15.2:0/2218121007 >> 172.21.15.2:0/2218121007 conn(0x7f06900f9f50 msgr2=0x7f06900fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.825+0000 7f06963ee700 1 -- 172.21.15.2:0/2218121007 shutdown_connections 2023-11-16T18:22:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.825+0000 7f06963ee700 1 -- 172.21.15.2:0/2218121007 wait complete. 2023-11-16T18:22:58.840 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T18:22:44.902889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T18:22:46.187275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T18:22:49.539148+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T18:22:58.840 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T18:22:58.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.966+0000 7f4cb2be2700 1 -- 172.21.15.2:0/69083675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4cac0ff480 msgr2=0x7f4cac0ff8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:58.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.966+0000 7f4cb2be2700 1 --2- 172.21.15.2:0/69083675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4cac0ff480 0x7f4cac0ff8f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4c98009990 tx=0x7f4c98009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.966+0000 7f4cb2be2700 1 -- 172.21.15.2:0/69083675 shutdown_connections 2023-11-16T18:22:58.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 --2- 172.21.15.2:0/69083675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4cac0ffe30 0x7f4cac1076e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 --2- 172.21.15.2:0/69083675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cac100790 0x7f4cac100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 --2- 172.21.15.2:0/69083675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4cac0ff480 0x7f4cac0ff8f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 -- 172.21.15.2:0/69083675 >> 172.21.15.2:0/69083675 conn(0x7f4cac0f9f00 msgr2=0x7f4cac0fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:58.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 -- 172.21.15.2:0/69083675 shutdown_connections 2023-11-16T18:22:58.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 -- 172.21.15.2:0/69083675 wait complete. 2023-11-16T18:22:58.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 Processor -- start 2023-11-16T18:22:58.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 -- start start 2023-11-16T18:22:58.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cac0ff480 0x7f4cac0feee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4cac0ffe30 0x7f4cac19ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4cac100790 0x7f4cac19d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac19da40 con 0x7f4cac0ffe30 2023-11-16T18:22:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac19dbb0 con 0x7f4cac0ff480 2023-11-16T18:22:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cb2be2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac19dd70 con 0x7f4cac100790 2023-11-16T18:22:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cabfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4cac0ffe30 0x7f4cac19ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cabfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4cac0ffe30 0x7f4cac19ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48020/0 (socket says 172.21.15.2:48020) 2023-11-16T18:22:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.967+0000 7f4cabfff700 1 -- 172.21.15.2:0/3766806701 learned_addr learned my addr 172.21.15.2:0/3766806701 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.968+0000 7f4cb117f700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4cac100790 0x7f4cac19d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.968+0000 7f4cb097e700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cac0ff480 0x7f4cac0feee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.968+0000 7f4cabfff700 1 -- 172.21.15.2:0/3766806701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4cac100790 msgr2=0x7f4cac19d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.968+0000 7f4cabfff700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4cac100790 0x7f4cac19d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.968+0000 7f4cabfff700 1 -- 172.21.15.2:0/3766806701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cac0ff480 msgr2=0x7f4cac0feee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:58.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.968+0000 7f4cabfff700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cac0ff480 0x7f4cac0feee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:58.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.968+0000 7f4cabfff700 1 -- 172.21.15.2:0/3766806701 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c980095f0 con 0x7f4cac0ffe30 2023-11-16T18:22:58.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.968+0000 7f4cabfff700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4cac0ffe30 0x7f4cac19ac60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4c98005f50 tx=0x7f4c980057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:58.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.968+0000 7f4ca9ffb700 1 -- 172.21.15.2:0/3766806701 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9801d070 con 0x7f4cac0ffe30 2023-11-16T18:22:58.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.968+0000 7f4cb2be2700 1 -- 172.21.15.2:0/3766806701 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cac10ab10 con 0x7f4cac0ffe30 2023-11-16T18:22:58.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.968+0000 7f4ca9ffb700 1 -- 172.21.15.2:0/3766806701 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c9800fbf0 con 0x7f4cac0ffe30 2023-11-16T18:22:58.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.968+0000 7f4ca9ffb700 1 -- 172.21.15.2:0/3766806701 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c98017650 con 0x7f4cac0ffe30 2023-11-16T18:22:58.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.968+0000 7f4cb2be2700 1 -- 172.21.15.2:0/3766806701 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cac10a980 con 0x7f4cac0ffe30 2023-11-16T18:22:58.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.969+0000 7f4cb2be2700 1 -- 172.21.15.2:0/3766806701 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cac19e2f0 con 0x7f4cac0ffe30 2023-11-16T18:22:58.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.970+0000 7f4ca9ffb700 1 -- 172.21.15.2:0/3766806701 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f4c98021410 con 0x7f4cac0ffe30 2023-11-16T18:22:58.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.971+0000 7f4ca9ffb700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4c9c0a1150 0x7f4c9c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:58.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.971+0000 7f4cb097e700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4c9c0a1150 0x7f4c9c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:58.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.971+0000 7f4ca9ffb700 1 -- 172.21.15.2:0/3766806701 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4c980ce020 con 0x7f4cac0ffe30 2023-11-16T18:22:58.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.972+0000 7f4cb097e700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4c9c0a1150 0x7f4c9c0a3600 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4cac067d90 tx=0x7f4ca0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:58.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:58.975+0000 7f4ca9ffb700 1 -- 172.21.15.2:0/3766806701 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4c98097610 con 0x7f4cac0ffe30 2023-11-16T18:22:59.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:58 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2218121007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:59.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.176+0000 7f4cb2be2700 1 -- 172.21.15.2:0/3766806701 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4cac1061a0 con 0x7f4cac0ffe30 2023-11-16T18:22:59.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.176+0000 7f4ca9ffb700 1 -- 172.21.15.2:0/3766806701 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4c9809ac30 con 0x7f4cac0ffe30 2023-11-16T18:22:59.179 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:22:59.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.179+0000 7f4cb2be2700 1 -- 172.21.15.2:0/3766806701 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4c9c0a1150 msgr2=0x7f4c9c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:59.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.179+0000 7f4cb2be2700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4c9c0a1150 0x7f4c9c0a3600 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4cac067d90 tx=0x7f4ca0009260 comp rx=0 tx=0).stop 2023-11-16T18:22:59.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.179+0000 7f4cb2be2700 1 -- 172.21.15.2:0/3766806701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4cac0ffe30 msgr2=0x7f4cac19ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:59.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.179+0000 7f4cb2be2700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4cac0ffe30 0x7f4cac19ac60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4c98005f50 tx=0x7f4c980057b0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.179+0000 7f4cb2be2700 1 -- 172.21.15.2:0/3766806701 shutdown_connections 2023-11-16T18:22:59.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.179+0000 7f4cb2be2700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4c9c0a1150 0x7f4c9c0a3600 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.179+0000 7f4cb2be2700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4cac100790 0x7f4cac19d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.179+0000 7f4cb2be2700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cac0ff480 0x7f4cac0feee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.179+0000 7f4cb2be2700 1 --2- 172.21.15.2:0/3766806701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4cac0ffe30 0x7f4cac19ac60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.179+0000 7f4cb2be2700 1 -- 172.21.15.2:0/3766806701 >> 172.21.15.2:0/3766806701 conn(0x7f4cac0f9f00 msgr2=0x7f4cac0fc170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:59.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.179+0000 7f4cb2be2700 1 -- 172.21.15.2:0/3766806701 shutdown_connections 2023-11-16T18:22:59.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.179+0000 7f4cb2be2700 1 -- 172.21.15.2:0/3766806701 wait complete. 2023-11-16T18:22:59.193 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T18:22:59.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:58 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2218121007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:59.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:58 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2218121007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:22:59.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.319+0000 7fdc0f575700 1 -- 172.21.15.2:0/3704170827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc08100d20 msgr2=0x7fdc081075e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:59.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 --2- 172.21.15.2:0/3704170827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc08100d20 0x7fdc081075e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdc0000b0a0 tx=0x7fdc0000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 -- 172.21.15.2:0/3704170827 shutdown_connections 2023-11-16T18:22:59.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 --2- 172.21.15.2:0/3704170827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc080ff1a0 0x7fdc080ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 --2- 172.21.15.2:0/3704170827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc08100d20 0x7fdc081075e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 --2- 172.21.15.2:0/3704170827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc08100390 0x7fdc081007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 -- 172.21.15.2:0/3704170827 >> 172.21.15.2:0/3704170827 conn(0x7fdc080fa7a0 msgr2=0x7fdc080fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:59.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 -- 172.21.15.2:0/3704170827 shutdown_connections 2023-11-16T18:22:59.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 -- 172.21.15.2:0/3704170827 wait complete. 2023-11-16T18:22:59.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 Processor -- start 2023-11-16T18:22:59.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 -- start start 2023-11-16T18:22:59.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc080ff1a0 0x7fdc08196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:59.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc08100390 0x7fdc08196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:59.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc08100d20 0x7fdc0819d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:59.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0819dba0 con 0x7fdc080ff1a0 2023-11-16T18:22:59.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0819dce0 con 0x7fdc08100390 2023-11-16T18:22:59.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.320+0000 7fdc0f575700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0819de70 con 0x7fdc08100d20 2023-11-16T18:22:59.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdc0d311700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc080ff1a0 0x7fdc08196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:59.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdc0d311700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc080ff1a0 0x7fdc08196020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48044/0 (socket says 172.21.15.2:48044) 2023-11-16T18:22:59.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdc0d311700 1 -- 172.21.15.2:0/334868099 learned_addr learned my addr 172.21.15.2:0/334868099 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:59.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdc0cb10700 1 --2- 172.21.15.2:0/334868099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc08100390 0x7fdc08196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:59.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdc0d311700 1 -- 172.21.15.2:0/334868099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc08100d20 msgr2=0x7fdc0819d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:59.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdc0d311700 1 --2- 172.21.15.2:0/334868099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc08100d20 0x7fdc0819d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdc0d311700 1 -- 172.21.15.2:0/334868099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc08100390 msgr2=0x7fdc08196560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:59.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdc0d311700 1 --2- 172.21.15.2:0/334868099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc08100390 0x7fdc08196560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdc0d311700 1 -- 172.21.15.2:0/334868099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc00009d00 con 0x7fdc080ff1a0 2023-11-16T18:22:59.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdc0d311700 1 --2- 172.21.15.2:0/334868099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc080ff1a0 0x7fdc08196020 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400c970 tx=0x7fdc0400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:59.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdbfe7fc700 1 -- 172.21.15.2:0/334868099 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04004220 con 0x7fdc080ff1a0 2023-11-16T18:22:59.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdbfe7fc700 1 -- 172.21.15.2:0/334868099 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc04007d60 con 0x7fdc080ff1a0 2023-11-16T18:22:59.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdc0f575700 1 -- 172.21.15.2:0/334868099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0810ab20 con 0x7fdc080ff1a0 2023-11-16T18:22:59.327 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdbfe7fc700 1 -- 172.21.15.2:0/334868099 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04003c90 con 0x7fdc080ff1a0 2023-11-16T18:22:59.327 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.321+0000 7fdc0f575700 1 -- 172.21.15.2:0/334868099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0810a990 con 0x7fdc080ff1a0 2023-11-16T18:22:59.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.322+0000 7fdc0f575700 1 -- 172.21.15.2:0/334868099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0819e5c0 con 0x7fdc080ff1a0 2023-11-16T18:22:59.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.323+0000 7fdbfe7fc700 1 -- 172.21.15.2:0/334868099 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fdc04004380 con 0x7fdc080ff1a0 2023-11-16T18:22:59.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.324+0000 7fdbfe7fc700 1 --2- 172.21.15.2:0/334868099 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdbf40a1150 0x7fdbf40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:59.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.324+0000 7fdbfe7fc700 1 -- 172.21.15.2:0/334868099 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdc040cc6e0 con 0x7fdc080ff1a0 2023-11-16T18:22:59.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.324+0000 7fdc0cb10700 1 --2- 172.21.15.2:0/334868099 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdbf40a1150 0x7fdbf40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:59.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.324+0000 7fdc0cb10700 1 --2- 172.21.15.2:0/334868099 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdbf40a1150 0x7fdbf40a3600 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8005950 tx=0x7fdbf800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:59.330 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.328+0000 7fdbfe7fc700 1 -- 172.21.15.2:0/334868099 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdc04095be0 con 0x7fdc080ff1a0 2023-11-16T18:22:59.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.525+0000 7fdc0f575700 1 -- 172.21.15.2:0/334868099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdc0819eb70 con 0x7fdc080ff1a0 2023-11-16T18:22:59.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.526+0000 7fdbfe7fc700 1 -- 172.21.15.2:0/334868099 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdc04099200 con 0x7fdc080ff1a0 2023-11-16T18:22:59.528 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:22:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.528+0000 7fdc0f575700 1 -- 172.21.15.2:0/334868099 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdbf40a1150 msgr2=0x7fdbf40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.528+0000 7fdc0f575700 1 --2- 172.21.15.2:0/334868099 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdbf40a1150 0x7fdbf40a3600 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8005950 tx=0x7fdbf800a300 comp rx=0 tx=0).stop 2023-11-16T18:22:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.528+0000 7fdc0f575700 1 -- 172.21.15.2:0/334868099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc080ff1a0 msgr2=0x7fdc08196020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.528+0000 7fdc0f575700 1 --2- 172.21.15.2:0/334868099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc080ff1a0 0x7fdc08196020 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400c970 tx=0x7fdc0400cc80 comp rx=0 tx=0).stop 2023-11-16T18:22:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.528+0000 7fdc0f575700 1 -- 172.21.15.2:0/334868099 shutdown_connections 2023-11-16T18:22:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.528+0000 7fdc0f575700 1 --2- 172.21.15.2:0/334868099 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdbf40a1150 0x7fdbf40a3600 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.528+0000 7fdc0f575700 1 --2- 172.21.15.2:0/334868099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc08100d20 0x7fdc0819d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.528+0000 7fdc0f575700 1 --2- 172.21.15.2:0/334868099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc080ff1a0 0x7fdc08196020 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.528+0000 7fdc0f575700 1 --2- 172.21.15.2:0/334868099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc08100390 0x7fdc08196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.528+0000 7fdc0f575700 1 -- 172.21.15.2:0/334868099 >> 172.21.15.2:0/334868099 conn(0x7fdc080fa7a0 msgr2=0x7fdc08068e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:59.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.528+0000 7fdc0f575700 1 -- 172.21.15.2:0/334868099 shutdown_connections 2023-11-16T18:22:59.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.528+0000 7fdc0f575700 1 -- 172.21.15.2:0/334868099 wait complete. 2023-11-16T18:22:59.542 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T18:22:59.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.668+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2195409930 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c28068ae0 msgr2=0x7f0c28102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:59.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.668+0000 7f0c2cfbd700 1 --2- 172.21.15.2:0/2195409930 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c28068ae0 0x7f0c28102e20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009990 tx=0x7f0c18009ca0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.668+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2195409930 shutdown_connections 2023-11-16T18:22:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.668+0000 7f0c2cfbd700 1 --2- 172.21.15.2:0/2195409930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c28105e50 0x7f0c28108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.668+0000 7f0c2cfbd700 1 --2- 172.21.15.2:0/2195409930 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c28068ae0 0x7f0c28102e20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.668+0000 7f0c2cfbd700 1 --2- 172.21.15.2:0/2195409930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c28103360 0x7f0c281057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.669+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2195409930 >> 172.21.15.2:0/2195409930 conn(0x7f0c280fa750 msgr2=0x7f0c280fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.669+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2195409930 shutdown_connections 2023-11-16T18:22:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.669+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2195409930 wait complete. 2023-11-16T18:22:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.669+0000 7f0c2cfbd700 1 Processor -- start 2023-11-16T18:22:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.669+0000 7f0c2cfbd700 1 -- start start 2023-11-16T18:22:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.669+0000 7f0c2cfbd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c28068ae0 0x7f0c28195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:59.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c2cfbd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c28103360 0x7f0c281964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:59.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c2cfbd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c28105e50 0x7f0c2819d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:59.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c2cfbd700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2819db30 con 0x7f0c28105e50 2023-11-16T18:22:59.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c2cfbd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2819dc70 con 0x7f0c28068ae0 2023-11-16T18:22:59.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c2cfbd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2819de00 con 0x7f0c28103360 2023-11-16T18:22:59.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c26d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c28105e50 0x7f0c2819d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:59.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c2659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c28068ae0 0x7f0c28195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:59.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c26d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c28105e50 0x7f0c2819d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48062/0 (socket says 172.21.15.2:48062) 2023-11-16T18:22:59.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c26d9d700 1 -- 172.21.15.2:0/2140213771 learned_addr learned my addr 172.21.15.2:0/2140213771 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:22:59.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c25d9b700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c28103360 0x7f0c281964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:59.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c26d9d700 1 -- 172.21.15.2:0/2140213771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c28103360 msgr2=0x7f0c281964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:59.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c26d9d700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c28103360 0x7f0c281964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c26d9d700 1 -- 172.21.15.2:0/2140213771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c28068ae0 msgr2=0x7f0c28195fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:59.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c26d9d700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c28068ae0 0x7f0c28195fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c26d9d700 1 -- 172.21.15.2:0/2140213771 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c180095f0 con 0x7f0c28105e50 2023-11-16T18:22:59.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c2659c700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c28068ae0 0x7f0c28195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:22:59.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c26d9d700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c28105e50 0x7f0c2819d560 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0c20007ee0 tx=0x7f0c2000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:59.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c1f7fe700 1 -- 172.21.15.2:0/2140213771 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20017070 con 0x7f0c28105e50 2023-11-16T18:22:59.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c1f7fe700 1 -- 172.21.15.2:0/2140213771 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c2000f040 con 0x7f0c28105e50 2023-11-16T18:22:59.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2140213771 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2810bbe0 con 0x7f0c28105e50 2023-11-16T18:22:59.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c1f7fe700 1 -- 172.21.15.2:0/2140213771 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20005580 con 0x7f0c28105e50 2023-11-16T18:22:59.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.670+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2140213771 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c2810ba50 con 0x7f0c28105e50 2023-11-16T18:22:59.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.671+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2140213771 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2819e540 con 0x7f0c28105e50 2023-11-16T18:22:59.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.672+0000 7f0c1f7fe700 1 -- 172.21.15.2:0/2140213771 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f0c2000db20 con 0x7f0c28105e50 2023-11-16T18:22:59.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.673+0000 7f0c1f7fe700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0c140a11d0 0x7f0c140a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:22:59.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.673+0000 7f0c2659c700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0c140a11d0 0x7f0c140a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:22:59.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.673+0000 7f0c1f7fe700 1 -- 172.21.15.2:0/2140213771 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0c200cc7d0 con 0x7f0c28105e50 2023-11-16T18:22:59.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.674+0000 7f0c2659c700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0c140a11d0 0x7f0c140a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0c1800b5c0 tx=0x7f0c18005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:22:59.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.676+0000 7f0c1f7fe700 1 -- 172.21.15.2:0/2140213771 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0c20095d30 con 0x7f0c28105e50 2023-11-16T18:22:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.880+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2140213771 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0c2819eaf0 con 0x7f0c28105e50 2023-11-16T18:22:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.880+0000 7f0c1f7fe700 1 -- 172.21.15.2:0/2140213771 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0c20099350 con 0x7f0c28105e50 2023-11-16T18:22:59.882 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T18:22:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.882+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2140213771 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0c140a11d0 msgr2=0x7f0c140a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.882+0000 7f0c2cfbd700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0c140a11d0 0x7f0c140a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0c1800b5c0 tx=0x7f0c18005fd0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.882+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2140213771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c28105e50 msgr2=0x7f0c2819d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:22:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.882+0000 7f0c2cfbd700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c28105e50 0x7f0c2819d560 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0c20007ee0 tx=0x7f0c2000d3b0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.882+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2140213771 shutdown_connections 2023-11-16T18:22:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.882+0000 7f0c2cfbd700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0c140a11d0 0x7f0c140a3680 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.882+0000 7f0c2cfbd700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c28105e50 0x7f0c2819d560 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.882+0000 7f0c2cfbd700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c28068ae0 0x7f0c28195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.882+0000 7f0c2cfbd700 1 --2- 172.21.15.2:0/2140213771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c28103360 0x7f0c281964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:22:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.882+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2140213771 >> 172.21.15.2:0/2140213771 conn(0x7f0c280fa750 msgr2=0x7f0c28101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:22:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.882+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2140213771 shutdown_connections 2023-11-16T18:22:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:22:59.883+0000 7f0c2cfbd700 1 -- 172.21.15.2:0/2140213771 wait complete. 2023-11-16T18:22:59.897 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T18:23:00.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.025+0000 7f5bfefa3700 1 -- 172.21.15.2:0/2976200981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bf81003b0 msgr2=0x7f5bf8100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:00.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.025+0000 7f5bfefa3700 1 --2- 172.21.15.2:0/2976200981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bf81003b0 0x7f5bf8100800 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5be4009990 tx=0x7f5be4009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.025+0000 7f5bfefa3700 1 -- 172.21.15.2:0/2976200981 shutdown_connections 2023-11-16T18:23:00.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.025+0000 7f5bfefa3700 1 --2- 172.21.15.2:0/2976200981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf80ff1c0 0x7f5bf80ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.025+0000 7f5bfefa3700 1 --2- 172.21.15.2:0/2976200981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bf8100d40 0x7f5bf8107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.025+0000 7f5bfefa3700 1 --2- 172.21.15.2:0/2976200981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bf81003b0 0x7f5bf8100800 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.025+0000 7f5bfefa3700 1 -- 172.21.15.2:0/2976200981 >> 172.21.15.2:0/2976200981 conn(0x7f5bf80fa800 msgr2=0x7f5bf80fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:00.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.025+0000 7f5bfefa3700 1 -- 172.21.15.2:0/2976200981 shutdown_connections 2023-11-16T18:23:00.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.025+0000 7f5bfefa3700 1 -- 172.21.15.2:0/2976200981 wait complete. 2023-11-16T18:23:00.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.025+0000 7f5bfefa3700 1 Processor -- start 2023-11-16T18:23:00.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.025+0000 7f5bfefa3700 1 -- start start 2023-11-16T18:23:00.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfefa3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bf80ff1c0 0x7f5bf8069950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:00.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfefa3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf81003b0 0x7f5bf819acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:00.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfefa3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bf8100d40 0x7f5bf819d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:00.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfefa3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf819daa0 con 0x7f5bf80ff1c0 2023-11-16T18:23:00.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfefa3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf819dc10 con 0x7f5bf8100d40 2023-11-16T18:23:00.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfefa3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf819ddd0 con 0x7f5bf81003b0 2023-11-16T18:23:00.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfcd3f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bf80ff1c0 0x7f5bf8069950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:00.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfcd3f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bf80ff1c0 0x7f5bf8069950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48072/0 (socket says 172.21.15.2:48072) 2023-11-16T18:23:00.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfcd3f700 1 -- 172.21.15.2:0/4029627887 learned_addr learned my addr 172.21.15.2:0/4029627887 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:00.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfd540700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bf8100d40 0x7f5bf819d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:00.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bf7fff700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf81003b0 0x7f5bf819acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:00.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfcd3f700 1 -- 172.21.15.2:0/4029627887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf81003b0 msgr2=0x7f5bf819acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:00.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfcd3f700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf81003b0 0x7f5bf819acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfcd3f700 1 -- 172.21.15.2:0/4029627887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bf8100d40 msgr2=0x7f5bf819d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:00.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfcd3f700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bf8100d40 0x7f5bf819d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfcd3f700 1 -- 172.21.15.2:0/4029627887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be40095f0 con 0x7f5bf80ff1c0 2023-11-16T18:23:00.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bf7fff700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf81003b0 0x7f5bf819acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:00.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfcd3f700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bf80ff1c0 0x7f5bf8069950 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5bec00d980 tx=0x7f5bec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:00.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bf5ffb700 1 -- 172.21.15.2:0/4029627887 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec004020 con 0x7f5bf80ff1c0 2023-11-16T18:23:00.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bfefa3700 1 -- 172.21.15.2:0/4029627887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf810ab30 con 0x7f5bf80ff1c0 2023-11-16T18:23:00.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bf5ffb700 1 -- 172.21.15.2:0/4029627887 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bec009730 con 0x7f5bf80ff1c0 2023-11-16T18:23:00.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.026+0000 7f5bf5ffb700 1 -- 172.21.15.2:0/4029627887 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec003a30 con 0x7f5bf80ff1c0 2023-11-16T18:23:00.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.027+0000 7f5bfefa3700 1 -- 172.21.15.2:0/4029627887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf810a9a0 con 0x7f5bf80ff1c0 2023-11-16T18:23:00.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.028+0000 7f5bfefa3700 1 -- 172.21.15.2:0/4029627887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf819e520 con 0x7f5bf80ff1c0 2023-11-16T18:23:00.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.028+0000 7f5bf5ffb700 1 -- 172.21.15.2:0/4029627887 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f5bec0098a0 con 0x7f5bf80ff1c0 2023-11-16T18:23:00.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.029+0000 7f5bf5ffb700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5be80a1150 0x7f5be80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:00.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.029+0000 7f5bf7fff700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5be80a1150 0x7f5be80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:00.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.029+0000 7f5bf5ffb700 1 -- 172.21.15.2:0/4029627887 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5bec0cc800 con 0x7f5bf80ff1c0 2023-11-16T18:23:00.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.030+0000 7f5bf7fff700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5be80a1150 0x7f5be80a3600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5be4006010 tx=0x7f5be400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:00.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.034+0000 7f5bf5ffb700 1 -- 172.21.15.2:0/4029627887 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5bec095d60 con 0x7f5bf80ff1c0 2023-11-16T18:23:00.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:59 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3766806701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:23:00.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:59 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/334868099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:23:00.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:22:59 smithi040 ceph-mon[100251]: pgmap v267: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 235 B/s wr, 0 op/s 2023-11-16T18:23:00.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.233+0000 7f5bfefa3700 1 -- 172.21.15.2:0/4029627887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5bf8106100 con 0x7f5bf80ff1c0 2023-11-16T18:23:00.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.234+0000 7f5bf5ffb700 1 -- 172.21.15.2:0/4029627887 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v86) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5bec099380 con 0x7f5bf80ff1c0 2023-11-16T18:23:00.236 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:23:00.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.236+0000 7f5bfefa3700 1 -- 172.21.15.2:0/4029627887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5be80a1150 msgr2=0x7f5be80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:00.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.236+0000 7f5bfefa3700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5be80a1150 0x7f5be80a3600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5be4006010 tx=0x7f5be400b540 comp rx=0 tx=0).stop 2023-11-16T18:23:00.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.236+0000 7f5bfefa3700 1 -- 172.21.15.2:0/4029627887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bf80ff1c0 msgr2=0x7f5bf8069950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:00.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.236+0000 7f5bfefa3700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bf80ff1c0 0x7f5bf8069950 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5bec00d980 tx=0x7f5bec00dc90 comp rx=0 tx=0).stop 2023-11-16T18:23:00.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.236+0000 7f5bfefa3700 1 -- 172.21.15.2:0/4029627887 shutdown_connections 2023-11-16T18:23:00.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.236+0000 7f5bfefa3700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5be80a1150 0x7f5be80a3600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.236+0000 7f5bfefa3700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bf8100d40 0x7f5bf819d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.236+0000 7f5bfefa3700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bf80ff1c0 0x7f5bf8069950 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.236+0000 7f5bfefa3700 1 --2- 172.21.15.2:0/4029627887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf81003b0 0x7f5bf819acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.236+0000 7f5bfefa3700 1 -- 172.21.15.2:0/4029627887 >> 172.21.15.2:0/4029627887 conn(0x7f5bf80fa800 msgr2=0x7f5bf80fca90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:00.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.236+0000 7f5bfefa3700 1 -- 172.21.15.2:0/4029627887 shutdown_connections 2023-11-16T18:23:00.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.236+0000 7f5bfefa3700 1 -- 172.21.15.2:0/4029627887 wait complete. 2023-11-16T18:23:00.250 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:59 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3766806701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:23:00.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:59 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/334868099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:23:00.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:22:59 smithi002 ceph-mon[98807]: pgmap v267: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 235 B/s wr, 0 op/s 2023-11-16T18:23:00.252 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-16T18:23:00.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:59 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3766806701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:23:00.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:59 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/334868099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:23:00.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:22:59 smithi188 ceph-mon[100639]: pgmap v267: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 235 B/s wr, 0 op/s 2023-11-16T18:23:00.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.377+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/595619488 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bec100c90 msgr2=0x7f7bec107550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:00.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.377+0000 7f7bf1e3d700 1 --2- 172.21.15.2:0/595619488 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bec100c90 0x7f7bec107550 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00b0a0 tx=0x7f7bdc00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.377+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/595619488 shutdown_connections 2023-11-16T18:23:00.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.377+0000 7f7bf1e3d700 1 --2- 172.21.15.2:0/595619488 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bec100c90 0x7f7bec107550 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.377+0000 7f7bf1e3d700 1 --2- 172.21.15.2:0/595619488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec0ff110 0x7f7bec0ff520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.377+0000 7f7bf1e3d700 1 --2- 172.21.15.2:0/595619488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bec100300 0x7f7bec100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.377+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/595619488 >> 172.21.15.2:0/595619488 conn(0x7f7bec0fa6f0 msgr2=0x7f7bec0fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:00.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.377+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/595619488 shutdown_connections 2023-11-16T18:23:00.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.377+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/595619488 wait complete. 2023-11-16T18:23:00.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.377+0000 7f7bf1e3d700 1 Processor -- start 2023-11-16T18:23:00.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7bf1e3d700 1 -- start start 2023-11-16T18:23:00.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7bf1e3d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bec0ff110 0x7f7bec1960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:00.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7bf1e3d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec100300 0x7f7bec196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:00.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7bf1e3d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bec100c90 0x7f7bec19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:00.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7bf1e3d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec19dc40 con 0x7f7bec0ff110 2023-11-16T18:23:00.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7bf1e3d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec19dd80 con 0x7f7bec100300 2023-11-16T18:23:00.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7bf1e3d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec19df10 con 0x7f7bec100c90 2023-11-16T18:23:00.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7beb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bec0ff110 0x7f7bec1960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:00.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7beb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bec0ff110 0x7f7bec1960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48086/0 (socket says 172.21.15.2:48086) 2023-11-16T18:23:00.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7beb7fe700 1 -- 172.21.15.2:0/1341468800 learned_addr learned my addr 172.21.15.2:0/1341468800 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:00.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7bebfff700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bec100c90 0x7f7bec19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:00.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7beb7fe700 1 -- 172.21.15.2:0/1341468800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bec100c90 msgr2=0x7f7bec19d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:00.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7beaffd700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec100300 0x7f7bec196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:00.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7beb7fe700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bec100c90 0x7f7bec19d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7beb7fe700 1 -- 172.21.15.2:0/1341468800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec100300 msgr2=0x7f7bec196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:00.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7beb7fe700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec100300 0x7f7bec196600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7beb7fe700 1 -- 172.21.15.2:0/1341468800 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc009d00 con 0x7f7bec0ff110 2023-11-16T18:23:00.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7bebfff700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bec100c90 0x7f7bec19d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:00.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.378+0000 7f7beb7fe700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bec0ff110 0x7f7bec1960c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7be400d980 tx=0x7f7be400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:00.384 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.379+0000 7f7be8ff9700 1 -- 172.21.15.2:0/1341468800 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be400de60 con 0x7f7bec0ff110 2023-11-16T18:23:00.384 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.379+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/1341468800 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bec10aad0 con 0x7f7bec0ff110 2023-11-16T18:23:00.384 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.379+0000 7f7be8ff9700 1 -- 172.21.15.2:0/1341468800 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7be40044e0 con 0x7f7bec0ff110 2023-11-16T18:23:00.384 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.379+0000 7f7be8ff9700 1 -- 172.21.15.2:0/1341468800 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4003be0 con 0x7f7bec0ff110 2023-11-16T18:23:00.384 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.379+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/1341468800 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bec10a940 con 0x7f7bec0ff110 2023-11-16T18:23:00.384 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.380+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/1341468800 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bec1071a0 con 0x7f7bec0ff110 2023-11-16T18:23:00.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.381+0000 7f7be8ff9700 1 -- 172.21.15.2:0/1341468800 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f7be4009bb0 con 0x7f7bec0ff110 2023-11-16T18:23:00.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.381+0000 7f7be8ff9700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7bd40a1150 0x7f7bd40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:00.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.382+0000 7f7beaffd700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7bd40a1150 0x7f7bd40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:00.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.382+0000 7f7be8ff9700 1 -- 172.21.15.2:0/1341468800 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7be40cccb0 con 0x7f7bec0ff110 2023-11-16T18:23:00.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.382+0000 7f7beaffd700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7bd40a1150 0x7f7bd40a3600 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009af0 tx=0x7f7be00094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:00.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.385+0000 7f7be8ff9700 1 -- 172.21.15.2:0/1341468800 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7be40961b0 con 0x7f7bec0ff110 2023-11-16T18:23:00.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.600+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/1341468800 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7bec02cfe0 con 0x7f7bec0ff110 2023-11-16T18:23:00.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.600+0000 7f7be8ff9700 1 -- 172.21.15.2:0/1341468800 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v86) v1 ==== 81+0+579 (secure 0 0 0) 0x7f7be4009940 con 0x7f7bec0ff110 2023-11-16T18:23:00.602 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/857143886 2023-11-17T18:16:37.723675+0000 2023-11-16T18:23:00.602 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/251514255 2023-11-17T18:16:37.723675+0000 2023-11-16T18:23:00.602 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/1605535303 2023-11-17T18:16:37.723675+0000 2023-11-16T18:23:00.603 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/1605535303 2023-11-17T18:16:37.723675+0000 2023-11-16T18:23:00.603 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/1609662822 2023-11-17T18:16:13.217812+0000 2023-11-16T18:23:00.603 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/1609662822 2023-11-17T18:16:13.217812+0000 2023-11-16T18:23:00.603 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/708195783 2023-11-17T18:16:13.217812+0000 2023-11-16T18:23:00.603 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/1158002645 2023-11-17T18:16:13.217812+0000 2023-11-16T18:23:00.603 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/1986390775 2023-11-17T18:16:13.217812+0000 2023-11-16T18:23:00.603 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/2955607810 2023-11-17T18:16:37.723675+0000 2023-11-16T18:23:00.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.602+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/1341468800 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7bd40a1150 msgr2=0x7f7bd40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:00.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.602+0000 7f7bf1e3d700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7bd40a1150 0x7f7bd40a3600 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009af0 tx=0x7f7be00094f0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.602+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/1341468800 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bec0ff110 msgr2=0x7f7bec1960c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:00.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.602+0000 7f7bf1e3d700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bec0ff110 0x7f7bec1960c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7be400d980 tx=0x7f7be400dd40 comp rx=0 tx=0).stop 2023-11-16T18:23:00.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.602+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/1341468800 shutdown_connections 2023-11-16T18:23:00.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.602+0000 7f7bf1e3d700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7bd40a1150 0x7f7bd40a3600 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.602+0000 7f7bf1e3d700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bec100c90 0x7f7bec19d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.602+0000 7f7bf1e3d700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bec0ff110 0x7f7bec1960c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.602+0000 7f7bf1e3d700 1 --2- 172.21.15.2:0/1341468800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec100300 0x7f7bec196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.602+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/1341468800 >> 172.21.15.2:0/1341468800 conn(0x7f7bec0fa6f0 msgr2=0x7f7bec0fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:00.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.602+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/1341468800 shutdown_connections 2023-11-16T18:23:00.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.602+0000 7f7bf1e3d700 1 -- 172.21.15.2:0/1341468800 wait complete. 2023-11-16T18:23:00.606 INFO:teuthology.orchestra.run.smithi002.stderr:listed 10 entries 2023-11-16T18:23:00.617 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:23:00.617 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T18:23:00.636 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:23:00.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.798+0000 7febbc0dd700 1 -- 172.21.15.2:0/1741932808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb40ff1c0 msgr2=0x7febb40ff5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:00.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.798+0000 7febbc0dd700 1 --2- 172.21.15.2:0/1741932808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb40ff1c0 0x7febb40ff5d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7febb0009990 tx=0x7febb0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 -- 172.21.15.2:0/1741932808 shutdown_connections 2023-11-16T18:23:00.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 --2- 172.21.15.2:0/1741932808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febb4100d40 0x7febb4107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 --2- 172.21.15.2:0/1741932808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb40ff1c0 0x7febb40ff5d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 --2- 172.21.15.2:0/1741932808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febb41003b0 0x7febb4100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 -- 172.21.15.2:0/1741932808 >> 172.21.15.2:0/1741932808 conn(0x7febb40fa7c0 msgr2=0x7febb40fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:00.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 -- 172.21.15.2:0/1741932808 shutdown_connections 2023-11-16T18:23:00.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 -- 172.21.15.2:0/1741932808 wait complete. 2023-11-16T18:23:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 Processor -- start 2023-11-16T18:23:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 -- start start 2023-11-16T18:23:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febb40ff1c0 0x7febb4103150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febb41003b0 0x7febb4101790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb4100d40 0x7febb4101cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb4102470 con 0x7febb4100d40 2023-11-16T18:23:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb41025e0 con 0x7febb41003b0 2023-11-16T18:23:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.799+0000 7febbc0dd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb41a0090 con 0x7febb40ff1c0 2023-11-16T18:23:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febba67a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb4100d40 0x7febb4101cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febba67a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb4100d40 0x7febb4101cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48104/0 (socket says 172.21.15.2:48104) 2023-11-16T18:23:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febba67a700 1 -- 172.21.15.2:0/2884103510 learned_addr learned my addr 172.21.15.2:0/2884103510 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febb9e79700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febb40ff1c0 0x7febb4103150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febb9678700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febb41003b0 0x7febb4101790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febba67a700 1 -- 172.21.15.2:0/2884103510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febb40ff1c0 msgr2=0x7febb4103150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febba67a700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febb40ff1c0 0x7febb4103150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febba67a700 1 -- 172.21.15.2:0/2884103510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febb41003b0 msgr2=0x7febb4101790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febba67a700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febb41003b0 0x7febb4101790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febba67a700 1 -- 172.21.15.2:0/2884103510 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb00095f0 con 0x7febb4100d40 2023-11-16T18:23:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febb9678700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febb41003b0 0x7febb4101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febba67a700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb4100d40 0x7febb4101cd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7febac007d70 tx=0x7febac00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febaaffd700 1 -- 172.21.15.2:0/2884103510 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febac00ad40 con 0x7febb4100d40 2023-11-16T18:23:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febaaffd700 1 -- 172.21.15.2:0/2884103510 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7febac0128e0 con 0x7febb4100d40 2023-11-16T18:23:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febbc0dd700 1 -- 172.21.15.2:0/2884103510 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb410ab20 con 0x7febb4100d40 2023-11-16T18:23:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febaaffd700 1 -- 172.21.15.2:0/2884103510 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febac0116b0 con 0x7febb4100d40 2023-11-16T18:23:00.806 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.800+0000 7febbc0dd700 1 -- 172.21.15.2:0/2884103510 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb410a990 con 0x7febb4100d40 2023-11-16T18:23:00.806 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.801+0000 7febbc0dd700 1 -- 172.21.15.2:0/2884103510 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb41a07e0 con 0x7febb4100d40 2023-11-16T18:23:00.806 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.802+0000 7febaaffd700 1 -- 172.21.15.2:0/2884103510 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7febac012420 con 0x7febb4100d40 2023-11-16T18:23:00.807 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.803+0000 7febaaffd700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feba00a1150 0x7feba00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:00.807 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.803+0000 7febb9e79700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feba00a1150 0x7feba00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:00.807 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.803+0000 7febaaffd700 1 -- 172.21.15.2:0/2884103510 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7febac0cd140 con 0x7febb4100d40 2023-11-16T18:23:00.807 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.803+0000 7febb9e79700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feba00a1150 0x7feba00a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7febb000b5c0 tx=0x7febb0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:00.808 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:00.806+0000 7febaaffd700 1 -- 172.21.15.2:0/2884103510 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7febac0966a0 con 0x7febb4100d40 2023-11-16T18:23:01.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.010+0000 7febbc0dd700 1 -- 172.21.15.2:0/2884103510 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febb41a0d90 con 0x7febb4100d40 2023-11-16T18:23:01.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.011+0000 7febaaffd700 1 -- 172.21.15.2:0/2884103510 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23853 (secure 0 0 0) 0x7febac099cc0 con 0x7febb4100d40 2023-11-16T18:23:01.014 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:01.014 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":86,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:22:51.534683+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:23:01.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.013+0000 7febbc0dd700 1 -- 172.21.15.2:0/2884103510 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feba00a1150 msgr2=0x7feba00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.013+0000 7febbc0dd700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feba00a1150 0x7feba00a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7febb000b5c0 tx=0x7febb0005fb0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.013+0000 7febbc0dd700 1 -- 172.21.15.2:0/2884103510 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb4100d40 msgr2=0x7febb4101cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.013+0000 7febbc0dd700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb4100d40 0x7febb4101cd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7febac007d70 tx=0x7febac00a340 comp rx=0 tx=0).stop 2023-11-16T18:23:01.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.013+0000 7febbc0dd700 1 -- 172.21.15.2:0/2884103510 shutdown_connections 2023-11-16T18:23:01.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.013+0000 7febbc0dd700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feba00a1150 0x7feba00a3600 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.013+0000 7febbc0dd700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febb40ff1c0 0x7febb4103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.013+0000 7febbc0dd700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb4100d40 0x7febb4101cd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.013+0000 7febbc0dd700 1 --2- 172.21.15.2:0/2884103510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febb41003b0 0x7febb4101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.013+0000 7febbc0dd700 1 -- 172.21.15.2:0/2884103510 >> 172.21.15.2:0/2884103510 conn(0x7febb40fa7c0 msgr2=0x7febb4067dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:01.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.013+0000 7febbc0dd700 1 -- 172.21.15.2:0/2884103510 shutdown_connections 2023-11-16T18:23:01.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.014+0000 7febbc0dd700 1 -- 172.21.15.2:0/2884103510 wait complete. 2023-11-16T18:23:01.028 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T18:22:44.902889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T18:22:46.187275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T18:22:49.539148+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T18:23:01.029 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T18:23:01.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.155+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2208088438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0d0100450 msgr2=0x7fd0d01008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.155+0000 7fd0d4e84700 1 --2- 172.21.15.2:0/2208088438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0d0100450 0x7fd0d01008c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0009990 tx=0x7fd0c0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.155+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2208088438 shutdown_connections 2023-11-16T18:23:01.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.155+0000 7fd0d4e84700 1 --2- 172.21.15.2:0/2208088438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0d0100e00 0x7fd0d01076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.155+0000 7fd0d4e84700 1 --2- 172.21.15.2:0/2208088438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0d00ff6a0 0x7fd0d00ffab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.155+0000 7fd0d4e84700 1 --2- 172.21.15.2:0/2208088438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0d0100450 0x7fd0d01008c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.155+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2208088438 >> 172.21.15.2:0/2208088438 conn(0x7fd0d00f9f00 msgr2=0x7fd0d00fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:01.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.155+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2208088438 shutdown_connections 2023-11-16T18:23:01.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.155+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2208088438 wait complete. 2023-11-16T18:23:01.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0d4e84700 1 Processor -- start 2023-11-16T18:23:01.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0d4e84700 1 -- start start 2023-11-16T18:23:01.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0d4e84700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0d00ff6a0 0x7fd0d00fee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:01.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0d4e84700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0d0100450 0x7fd0d019ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:01.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0d4e84700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0d0100e00 0x7fd0d019d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:01.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0d4e84700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d019db10 con 0x7fd0d0100450 2023-11-16T18:23:01.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0d4e84700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d019dc80 con 0x7fd0d00ff6a0 2023-11-16T18:23:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0d4e84700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d019de40 con 0x7fd0d0100e00 2023-11-16T18:23:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0cdd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0d0100450 0x7fd0d019ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0cdd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0d0100450 0x7fd0d019ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48116/0 (socket says 172.21.15.2:48116) 2023-11-16T18:23:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0cdd9b700 1 -- 172.21.15.2:0/2783220026 learned_addr learned my addr 172.21.15.2:0/2783220026 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0ce59c700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0d00ff6a0 0x7fd0d00fee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0ced9d700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0d0100e00 0x7fd0d019d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0cdd9b700 1 -- 172.21.15.2:0/2783220026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0d0100e00 msgr2=0x7fd0d019d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0cdd9b700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0d0100e00 0x7fd0d019d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0cdd9b700 1 -- 172.21.15.2:0/2783220026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0d00ff6a0 msgr2=0x7fd0d00fee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0cdd9b700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0d00ff6a0 0x7fd0d00fee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.156+0000 7fd0cdd9b700 1 -- 172.21.15.2:0/2783220026 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c00095f0 con 0x7fd0d0100450 2023-11-16T18:23:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.157+0000 7fd0cdd9b700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0d0100450 0x7fd0d019ad30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00095c0 tx=0x7fd0c000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.157+0000 7fd0bf7fe700 1 -- 172.21.15.2:0/2783220026 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c001d070 con 0x7fd0d0100450 2023-11-16T18:23:01.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.157+0000 7fd0bf7fe700 1 -- 172.21.15.2:0/2783220026 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0c0004830 con 0x7fd0d0100450 2023-11-16T18:23:01.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.157+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2783220026 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d010ab50 con 0x7fd0d0100450 2023-11-16T18:23:01.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.157+0000 7fd0bf7fe700 1 -- 172.21.15.2:0/2783220026 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0005420 con 0x7fd0d0100450 2023-11-16T18:23:01.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.157+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2783220026 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d010a9c0 con 0x7fd0d0100450 2023-11-16T18:23:01.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.158+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2783220026 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d0106210 con 0x7fd0d0100450 2023-11-16T18:23:01.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.159+0000 7fd0bf7fe700 1 -- 172.21.15.2:0/2783220026 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fd0c00049a0 con 0x7fd0d0100450 2023-11-16T18:23:01.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.159+0000 7fd0bf7fe700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd0b80a1150 0x7fd0b80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:01.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.160+0000 7fd0ce59c700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd0b80a1150 0x7fd0b80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:01.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.160+0000 7fd0bf7fe700 1 -- 172.21.15.2:0/2783220026 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd0c00ce220 con 0x7fd0d0100450 2023-11-16T18:23:01.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.160+0000 7fd0ce59c700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd0b80a1150 0x7fd0b80a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0067930 tx=0x7fd0c400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:01.165 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.163+0000 7fd0bf7fe700 1 -- 172.21.15.2:0/2783220026 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd0c00977b0 con 0x7fd0d0100450 2023-11-16T18:23:01.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:00 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2140213771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:23:01.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:00 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/4029627887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:23:01.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:00 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1341468800' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T18:23:01.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:00 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2140213771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:23:01.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:00 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/4029627887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:23:01.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:00 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1341468800' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T18:23:01.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:00 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2140213771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:23:01.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:00 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/4029627887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:23:01.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:00 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1341468800' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T18:23:01.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.361+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2783220026 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd0d00671c0 con 0x7fd0d0100450 2023-11-16T18:23:01.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.362+0000 7fd0bf7fe700 1 -- 172.21.15.2:0/2783220026 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd0c009add0 con 0x7fd0d0100450 2023-11-16T18:23:01.364 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:23:01.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.364+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2783220026 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd0b80a1150 msgr2=0x7fd0b80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.364+0000 7fd0d4e84700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd0b80a1150 0x7fd0b80a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0067930 tx=0x7fd0c400b410 comp rx=0 tx=0).stop 2023-11-16T18:23:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.364+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2783220026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0d0100450 msgr2=0x7fd0d019ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.364+0000 7fd0d4e84700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0d0100450 0x7fd0d019ad30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00095c0 tx=0x7fd0c000f690 comp rx=0 tx=0).stop 2023-11-16T18:23:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.364+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2783220026 shutdown_connections 2023-11-16T18:23:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.364+0000 7fd0d4e84700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd0b80a1150 0x7fd0b80a3600 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.364+0000 7fd0d4e84700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0d0100e00 0x7fd0d019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.364+0000 7fd0d4e84700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0d00ff6a0 0x7fd0d00fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.364+0000 7fd0d4e84700 1 --2- 172.21.15.2:0/2783220026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0d0100450 0x7fd0d019ad30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.364+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2783220026 >> 172.21.15.2:0/2783220026 conn(0x7fd0d00f9f00 msgr2=0x7fd0d00fc200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.364+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2783220026 shutdown_connections 2023-11-16T18:23:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.364+0000 7fd0d4e84700 1 -- 172.21.15.2:0/2783220026 wait complete. 2023-11-16T18:23:01.379 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T18:23:01.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.505+0000 7ff42770b700 1 -- 172.21.15.2:0/2739074559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff420100d30 msgr2=0x7ff4201075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.505+0000 7ff42770b700 1 --2- 172.21.15.2:0/2739074559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff420100d30 0x7ff4201075f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff41800b0a0 tx=0x7ff41800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.505+0000 7ff42770b700 1 -- 172.21.15.2:0/2739074559 shutdown_connections 2023-11-16T18:23:01.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.505+0000 7ff42770b700 1 --2- 172.21.15.2:0/2739074559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4200ff1b0 0x7ff4200ff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.505+0000 7ff42770b700 1 --2- 172.21.15.2:0/2739074559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff420100d30 0x7ff4201075f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.505+0000 7ff42770b700 1 --2- 172.21.15.2:0/2739074559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4201003a0 0x7ff4201007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.505+0000 7ff42770b700 1 -- 172.21.15.2:0/2739074559 >> 172.21.15.2:0/2739074559 conn(0x7ff4200fa7b0 msgr2=0x7ff4200fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:01.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.506+0000 7ff42770b700 1 -- 172.21.15.2:0/2739074559 shutdown_connections 2023-11-16T18:23:01.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.506+0000 7ff42770b700 1 -- 172.21.15.2:0/2739074559 wait complete. 2023-11-16T18:23:01.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.506+0000 7ff42770b700 1 Processor -- start 2023-11-16T18:23:01.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.506+0000 7ff42770b700 1 -- start start 2023-11-16T18:23:01.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.506+0000 7ff42770b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4200ff1b0 0x7ff420195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:01.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.506+0000 7ff42770b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4201003a0 0x7ff420196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:01.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.506+0000 7ff42770b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff420100d30 0x7ff42019d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:01.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.506+0000 7ff42770b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42019db50 con 0x7ff4201003a0 2023-11-16T18:23:01.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.506+0000 7ff42770b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42019dc90 con 0x7ff4200ff1b0 2023-11-16T18:23:01.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.506+0000 7ff42770b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42019de20 con 0x7ff420100d30 2023-11-16T18:23:01.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.506+0000 7ff424ca6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4201003a0 0x7ff420196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:01.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff424ca6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4201003a0 0x7ff420196510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48130/0 (socket says 172.21.15.2:48130) 2023-11-16T18:23:01.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff424ca6700 1 -- 172.21.15.2:0/291588813 learned_addr learned my addr 172.21.15.2:0/291588813 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:01.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff4254a7700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4200ff1b0 0x7ff420195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:01.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff425ca8700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff420100d30 0x7ff42019d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:01.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff424ca6700 1 -- 172.21.15.2:0/291588813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff420100d30 msgr2=0x7ff42019d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff424ca6700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff420100d30 0x7ff42019d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff424ca6700 1 -- 172.21.15.2:0/291588813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4200ff1b0 msgr2=0x7ff420195fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff424ca6700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4200ff1b0 0x7ff420195fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff424ca6700 1 -- 172.21.15.2:0/291588813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff418009d00 con 0x7ff4201003a0 2023-11-16T18:23:01.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff425ca8700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff420100d30 0x7ff42019d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:01.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff424ca6700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4201003a0 0x7ff420196510 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00c920 tx=0x7ff41c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:01.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff4127fc700 1 -- 172.21.15.2:0/291588813 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41c007b00 con 0x7ff4201003a0 2023-11-16T18:23:01.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff4127fc700 1 -- 172.21.15.2:0/291588813 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff41c00ce70 con 0x7ff4201003a0 2023-11-16T18:23:01.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff42770b700 1 -- 172.21.15.2:0/291588813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42010ab30 con 0x7ff4201003a0 2023-11-16T18:23:01.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff4127fc700 1 -- 172.21.15.2:0/291588813 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41c0054e0 con 0x7ff4201003a0 2023-11-16T18:23:01.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.507+0000 7ff42770b700 1 -- 172.21.15.2:0/291588813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42010a9a0 con 0x7ff4201003a0 2023-11-16T18:23:01.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.508+0000 7ff42770b700 1 -- 172.21.15.2:0/291588813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42019e5c0 con 0x7ff4201003a0 2023-11-16T18:23:01.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.509+0000 7ff4127fc700 1 -- 172.21.15.2:0/291588813 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7ff41c004750 con 0x7ff4201003a0 2023-11-16T18:23:01.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.510+0000 7ff4127fc700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff40c0a11d0 0x7ff40c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:01.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.510+0000 7ff4254a7700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff40c0a11d0 0x7ff40c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:01.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.510+0000 7ff4127fc700 1 -- 172.21.15.2:0/291588813 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff41c0cca50 con 0x7ff4201003a0 2023-11-16T18:23:01.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.510+0000 7ff4254a7700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff40c0a11d0 0x7ff40c0a3680 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff420100200 tx=0x7ff41400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:01.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.513+0000 7ff4127fc700 1 -- 172.21.15.2:0/291588813 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff41c096030 con 0x7ff4201003a0 2023-11-16T18:23:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.714+0000 7ff42770b700 1 -- 172.21.15.2:0/291588813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff42019eb70 con 0x7ff4201003a0 2023-11-16T18:23:01.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.715+0000 7ff4127fc700 1 -- 172.21.15.2:0/291588813 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff41c01d020 con 0x7ff4201003a0 2023-11-16T18:23:01.718 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:23:01.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.717+0000 7ff42770b700 1 -- 172.21.15.2:0/291588813 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff40c0a11d0 msgr2=0x7ff40c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.717+0000 7ff42770b700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff40c0a11d0 0x7ff40c0a3680 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff420100200 tx=0x7ff41400a3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.717+0000 7ff42770b700 1 -- 172.21.15.2:0/291588813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4201003a0 msgr2=0x7ff420196510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.717+0000 7ff42770b700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4201003a0 0x7ff420196510 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00c920 tx=0x7ff41c00cc30 comp rx=0 tx=0).stop 2023-11-16T18:23:01.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.717+0000 7ff42770b700 1 -- 172.21.15.2:0/291588813 shutdown_connections 2023-11-16T18:23:01.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.717+0000 7ff42770b700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff40c0a11d0 0x7ff40c0a3680 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.717+0000 7ff42770b700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff420100d30 0x7ff42019d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.717+0000 7ff42770b700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4200ff1b0 0x7ff420195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.717+0000 7ff42770b700 1 --2- 172.21.15.2:0/291588813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4201003a0 0x7ff420196510 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.717+0000 7ff42770b700 1 -- 172.21.15.2:0/291588813 >> 172.21.15.2:0/291588813 conn(0x7ff4200fa7b0 msgr2=0x7ff420068e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:01.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.717+0000 7ff42770b700 1 -- 172.21.15.2:0/291588813 shutdown_connections 2023-11-16T18:23:01.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.717+0000 7ff42770b700 1 -- 172.21.15.2:0/291588813 wait complete. 2023-11-16T18:23:01.731 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T18:23:01.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.859+0000 7fbce38e5700 1 -- 172.21.15.2:0/4035415970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101480 msgr2=0x7fbcdc1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.859+0000 7fbce38e5700 1 --2- 172.21.15.2:0/4035415970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101480 0x7fbcdc1018d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8009990 tx=0x7fbcd8009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.859+0000 7fbce38e5700 1 -- 172.21.15.2:0/4035415970 shutdown_connections 2023-11-16T18:23:01.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.859+0000 7fbce38e5700 1 --2- 172.21.15.2:0/4035415970 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbcdc100290 0x7fbcdc1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.859+0000 7fbce38e5700 1 --2- 172.21.15.2:0/4035415970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcdc101e10 0x7fbcdc1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.859+0000 7fbce38e5700 1 --2- 172.21.15.2:0/4035415970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101480 0x7fbcdc1018d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.859+0000 7fbce38e5700 1 -- 172.21.15.2:0/4035415970 >> 172.21.15.2:0/4035415970 conn(0x7fbcdc0fb8b0 msgr2=0x7fbcdc0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:01.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.859+0000 7fbce38e5700 1 -- 172.21.15.2:0/4035415970 shutdown_connections 2023-11-16T18:23:01.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.859+0000 7fbce38e5700 1 -- 172.21.15.2:0/4035415970 wait complete. 2023-11-16T18:23:01.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.860+0000 7fbce38e5700 1 Processor -- start 2023-11-16T18:23:01.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.860+0000 7fbce38e5700 1 -- start start 2023-11-16T18:23:01.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.860+0000 7fbce38e5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcdc100290 0x7fbcdc195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:01.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.860+0000 7fbce38e5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbcdc101480 0x7fbcdc196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:01.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.860+0000 7fbce38e5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101e10 0x7fbcdc19d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:01.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.860+0000 7fbce38e5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc19dab0 con 0x7fbcdc101e10 2023-11-16T18:23:01.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.860+0000 7fbce38e5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc19dbf0 con 0x7fbcdc100290 2023-11-16T18:23:01.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.860+0000 7fbce38e5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc19dd80 con 0x7fbcdc101480 2023-11-16T18:23:01.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.860+0000 7fbce1e82700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101e10 0x7fbcdc19d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:01.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.860+0000 7fbce1e82700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101e10 0x7fbcdc19d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48146/0 (socket says 172.21.15.2:48146) 2023-11-16T18:23:01.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.860+0000 7fbce1e82700 1 -- 172.21.15.2:0/2438615159 learned_addr learned my addr 172.21.15.2:0/2438615159 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:01.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.861+0000 7fbce1681700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcdc100290 0x7fbcdc195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:01.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.861+0000 7fbce0e80700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbcdc101480 0x7fbcdc196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:01.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.861+0000 7fbce1e82700 1 -- 172.21.15.2:0/2438615159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbcdc101480 msgr2=0x7fbcdc196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.861+0000 7fbce1e82700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbcdc101480 0x7fbcdc196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.861+0000 7fbce1e82700 1 -- 172.21.15.2:0/2438615159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcdc100290 msgr2=0x7fbcdc195f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:01.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.861+0000 7fbce1e82700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcdc100290 0x7fbcdc195f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:01.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.861+0000 7fbce1e82700 1 -- 172.21.15.2:0/2438615159 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd80095f0 con 0x7fbcdc101e10 2023-11-16T18:23:01.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.861+0000 7fbce1e82700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101e10 0x7fbcdc19d4e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4007d70 tx=0x7fbcd400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:01.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.861+0000 7fbcd27fc700 1 -- 172.21.15.2:0/2438615159 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd400ad40 con 0x7fbcdc101e10 2023-11-16T18:23:01.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.861+0000 7fbcd27fc700 1 -- 172.21.15.2:0/2438615159 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcd40128e0 con 0x7fbcdc101e10 2023-11-16T18:23:01.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.861+0000 7fbce38e5700 1 -- 172.21.15.2:0/2438615159 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcdc10bc00 con 0x7fbcdc101e10 2023-11-16T18:23:01.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.861+0000 7fbcd27fc700 1 -- 172.21.15.2:0/2438615159 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4011780 con 0x7fbcdc101e10 2023-11-16T18:23:01.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.861+0000 7fbce38e5700 1 -- 172.21.15.2:0/2438615159 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcdc10ba70 con 0x7fbcdc101e10 2023-11-16T18:23:01.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.862+0000 7fbce38e5700 1 -- 172.21.15.2:0/2438615159 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcdc107820 con 0x7fbcdc101e10 2023-11-16T18:23:01.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.863+0000 7fbcd27fc700 1 -- 172.21.15.2:0/2438615159 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fbcd4012a50 con 0x7fbcdc101e10 2023-11-16T18:23:01.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.864+0000 7fbcd27fc700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbcc80a11d0 0x7fbcc80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:01.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.864+0000 7fbce1681700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbcc80a11d0 0x7fbcc80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:01.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.864+0000 7fbcd27fc700 1 -- 172.21.15.2:0/2438615159 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbcd40cd1e0 con 0x7fbcdc101e10 2023-11-16T18:23:01.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.864+0000 7fbce1681700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbcc80a11d0 0x7fbcc80a3680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbccc005950 tx=0x7fbccc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:01.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:01.868+0000 7fbcd27fc700 1 -- 172.21.15.2:0/2438615159 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbcd4096740 con 0x7fbcdc101e10 2023-11-16T18:23:02.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.066+0000 7fbce38e5700 1 -- 172.21.15.2:0/2438615159 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbcdc061700 con 0x7fbcdc101e10 2023-11-16T18:23:02.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.066+0000 7fbcd27fc700 1 -- 172.21.15.2:0/2438615159 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbcd4016020 con 0x7fbcdc101e10 2023-11-16T18:23:02.068 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T18:23:02.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.068+0000 7fbce38e5700 1 -- 172.21.15.2:0/2438615159 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbcc80a11d0 msgr2=0x7fbcc80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:02.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.068+0000 7fbce38e5700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbcc80a11d0 0x7fbcc80a3680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbccc005950 tx=0x7fbccc0058e0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.068+0000 7fbce38e5700 1 -- 172.21.15.2:0/2438615159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101e10 msgr2=0x7fbcdc19d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:02.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.068+0000 7fbce38e5700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101e10 0x7fbcdc19d4e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4007d70 tx=0x7fbcd400a340 comp rx=0 tx=0).stop 2023-11-16T18:23:02.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.068+0000 7fbce38e5700 1 -- 172.21.15.2:0/2438615159 shutdown_connections 2023-11-16T18:23:02.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.068+0000 7fbce38e5700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbcc80a11d0 0x7fbcc80a3680 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.068+0000 7fbce38e5700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101e10 0x7fbcdc19d4e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.068+0000 7fbce38e5700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcdc100290 0x7fbcdc195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.068+0000 7fbce38e5700 1 --2- 172.21.15.2:0/2438615159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbcdc101480 0x7fbcdc196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.068+0000 7fbce38e5700 1 -- 172.21.15.2:0/2438615159 >> 172.21.15.2:0/2438615159 conn(0x7fbcdc0fb8b0 msgr2=0x7fbcdc104960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:02.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.068+0000 7fbce38e5700 1 -- 172.21.15.2:0/2438615159 shutdown_connections 2023-11-16T18:23:02.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.068+0000 7fbce38e5700 1 -- 172.21.15.2:0/2438615159 wait complete. 2023-11-16T18:23:02.082 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:01 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2884103510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:02.082 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:01 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2783220026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:23:02.082 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:01 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/291588813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:23:02.082 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:01 smithi002 ceph-mon[98807]: pgmap v268: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 498 B/s wr, 0 op/s 2023-11-16T18:23:02.083 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T18:23:02.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.209+0000 7fae8801f700 1 -- 172.21.15.2:0/3396287872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae801003b0 msgr2=0x7fae80100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:02.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.209+0000 7fae8801f700 1 --2- 172.21.15.2:0/3396287872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae801003b0 0x7fae80100800 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fae7c009990 tx=0x7fae7c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.209+0000 7fae8801f700 1 -- 172.21.15.2:0/3396287872 shutdown_connections 2023-11-16T18:23:02.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.209+0000 7fae8801f700 1 --2- 172.21.15.2:0/3396287872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae80100d40 0x7fae80107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.209+0000 7fae8801f700 1 --2- 172.21.15.2:0/3396287872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae800ff1c0 0x7fae800ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.209+0000 7fae8801f700 1 --2- 172.21.15.2:0/3396287872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae801003b0 0x7fae80100800 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.209+0000 7fae8801f700 1 -- 172.21.15.2:0/3396287872 >> 172.21.15.2:0/3396287872 conn(0x7fae800fa800 msgr2=0x7fae800fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:02.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.209+0000 7fae8801f700 1 -- 172.21.15.2:0/3396287872 shutdown_connections 2023-11-16T18:23:02.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.209+0000 7fae8801f700 1 -- 172.21.15.2:0/3396287872 wait complete. 2023-11-16T18:23:02.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae8801f700 1 Processor -- start 2023-11-16T18:23:02.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae8801f700 1 -- start start 2023-11-16T18:23:02.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae8801f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae800ff1c0 0x7fae80069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:02.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae8801f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae801003b0 0x7fae8019ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:02.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae8801f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae80100d40 0x7fae8019d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:02.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae8801f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8019da30 con 0x7fae801003b0 2023-11-16T18:23:02.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae8801f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8019dba0 con 0x7fae80100d40 2023-11-16T18:23:02.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae8801f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8019dd60 con 0x7fae800ff1c0 2023-11-16T18:23:02.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae855ba700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae801003b0 0x7fae8019ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:02.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae855ba700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae801003b0 0x7fae8019ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48158/0 (socket says 172.21.15.2:48158) 2023-11-16T18:23:02.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae855ba700 1 -- 172.21.15.2:0/1026465458 learned_addr learned my addr 172.21.15.2:0/1026465458 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:02.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae85dbb700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae800ff1c0 0x7fae80069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:02.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae865bc700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae80100d40 0x7fae8019d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:02.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae855ba700 1 -- 172.21.15.2:0/1026465458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae800ff1c0 msgr2=0x7fae80069970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:02.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.210+0000 7fae855ba700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae800ff1c0 0x7fae80069970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.211+0000 7fae855ba700 1 -- 172.21.15.2:0/1026465458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae80100d40 msgr2=0x7fae8019d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:02.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.211+0000 7fae855ba700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae80100d40 0x7fae8019d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.211+0000 7fae855ba700 1 -- 172.21.15.2:0/1026465458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c0095f0 con 0x7fae801003b0 2023-11-16T18:23:02.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.211+0000 7fae855ba700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae801003b0 0x7fae8019ac10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fae7c004760 tx=0x7fae7c004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:02.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.211+0000 7fae76ffd700 1 -- 172.21.15.2:0/1026465458 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c01d070 con 0x7fae801003b0 2023-11-16T18:23:02.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.211+0000 7fae76ffd700 1 -- 172.21.15.2:0/1026465458 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae7c004cf0 con 0x7fae801003b0 2023-11-16T18:23:02.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.211+0000 7fae8801f700 1 -- 172.21.15.2:0/1026465458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8010ab30 con 0x7fae801003b0 2023-11-16T18:23:02.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.211+0000 7fae76ffd700 1 -- 172.21.15.2:0/1026465458 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c005280 con 0x7fae801003b0 2023-11-16T18:23:02.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.211+0000 7fae8801f700 1 -- 172.21.15.2:0/1026465458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8010a9a0 con 0x7fae801003b0 2023-11-16T18:23:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.212+0000 7fae8801f700 1 -- 172.21.15.2:0/1026465458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae801066b0 con 0x7fae801003b0 2023-11-16T18:23:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.213+0000 7fae76ffd700 1 -- 172.21.15.2:0/1026465458 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fae7c0053e0 con 0x7fae801003b0 2023-11-16T18:23:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.214+0000 7fae76ffd700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fae6c0a1150 0x7fae6c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.214+0000 7fae85dbb700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fae6c0a1150 0x7fae6c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.214+0000 7fae76ffd700 1 -- 172.21.15.2:0/1026465458 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fae7c0ce0e0 con 0x7fae801003b0 2023-11-16T18:23:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.214+0000 7fae85dbb700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fae6c0a1150 0x7fae6c0a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fae70005fd0 tx=0x7fae70005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.217+0000 7fae76ffd700 1 -- 172.21.15.2:0/1026465458 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fae7c097670 con 0x7fae801003b0 2023-11-16T18:23:02.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:01 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2884103510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:02.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:01 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2783220026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:23:02.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:01 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/291588813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:23:02.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:01 smithi040 ceph-mon[100251]: pgmap v268: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 498 B/s wr, 0 op/s 2023-11-16T18:23:02.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:01 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2884103510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:02.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:01 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2783220026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:23:02.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:01 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/291588813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:23:02.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:01 smithi188 ceph-mon[100639]: pgmap v268: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 498 B/s wr, 0 op/s 2023-11-16T18:23:02.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.415+0000 7fae8801f700 1 -- 172.21.15.2:0/1026465458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fae80061ad0 con 0x7fae801003b0 2023-11-16T18:23:02.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.416+0000 7fae76ffd700 1 -- 172.21.15.2:0/1026465458 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v86) v1 ==== 103+0+11 (secure 0 0 0) 0x7fae7c09ac90 con 0x7fae801003b0 2023-11-16T18:23:02.418 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:23:02.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.418+0000 7fae8801f700 1 -- 172.21.15.2:0/1026465458 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fae6c0a1150 msgr2=0x7fae6c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:02.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.418+0000 7fae8801f700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fae6c0a1150 0x7fae6c0a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fae70005fd0 tx=0x7fae70005e50 comp rx=0 tx=0).stop 2023-11-16T18:23:02.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.418+0000 7fae8801f700 1 -- 172.21.15.2:0/1026465458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae801003b0 msgr2=0x7fae8019ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:02.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.418+0000 7fae8801f700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae801003b0 0x7fae8019ac10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fae7c004760 tx=0x7fae7c004790 comp rx=0 tx=0).stop 2023-11-16T18:23:02.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.418+0000 7fae8801f700 1 -- 172.21.15.2:0/1026465458 shutdown_connections 2023-11-16T18:23:02.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.418+0000 7fae8801f700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fae6c0a1150 0x7fae6c0a3600 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.418+0000 7fae8801f700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae800ff1c0 0x7fae80069970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.418+0000 7fae8801f700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae80100d40 0x7fae8019d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.418+0000 7fae8801f700 1 --2- 172.21.15.2:0/1026465458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae801003b0 0x7fae8019ac10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.418+0000 7fae8801f700 1 -- 172.21.15.2:0/1026465458 >> 172.21.15.2:0/1026465458 conn(0x7fae800fa800 msgr2=0x7fae801037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:02.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.418+0000 7fae8801f700 1 -- 172.21.15.2:0/1026465458 shutdown_connections 2023-11-16T18:23:02.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.418+0000 7fae8801f700 1 -- 172.21.15.2:0/1026465458 wait complete. 2023-11-16T18:23:02.433 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-16T18:23:02.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.558+0000 7fa9ac498700 1 -- 172.21.15.2:0/2646572056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a4100c90 msgr2=0x7fa9a4107550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:02.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.558+0000 7fa9ac498700 1 --2- 172.21.15.2:0/2646572056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a4100c90 0x7fa9a4107550 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa99c00b0a0 tx=0x7fa99c00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.558+0000 7fa9ac498700 1 -- 172.21.15.2:0/2646572056 shutdown_connections 2023-11-16T18:23:02.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.558+0000 7fa9ac498700 1 --2- 172.21.15.2:0/2646572056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9a40ff110 0x7fa9a40ff520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.558+0000 7fa9ac498700 1 --2- 172.21.15.2:0/2646572056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a4100c90 0x7fa9a4107550 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.558+0000 7fa9ac498700 1 --2- 172.21.15.2:0/2646572056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a4100300 0x7fa9a4100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.558+0000 7fa9ac498700 1 -- 172.21.15.2:0/2646572056 >> 172.21.15.2:0/2646572056 conn(0x7fa9a40fa6f0 msgr2=0x7fa9a40fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:02.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.558+0000 7fa9ac498700 1 -- 172.21.15.2:0/2646572056 shutdown_connections 2023-11-16T18:23:02.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.558+0000 7fa9ac498700 1 -- 172.21.15.2:0/2646572056 wait complete. 2023-11-16T18:23:02.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9ac498700 1 Processor -- start 2023-11-16T18:23:02.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9ac498700 1 -- start start 2023-11-16T18:23:02.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9ac498700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a40ff110 0x7fa9a4068950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:02.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9ac498700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a4100300 0x7fa9a419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:02.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9ac498700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9a4100c90 0x7fa9a419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:02.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9ac498700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a419dc00 con 0x7fa9a4100300 2023-11-16T18:23:02.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9ac498700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a419dd70 con 0x7fa9a40ff110 2023-11-16T18:23:02.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9ac498700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a419df30 con 0x7fa9a4100c90 2023-11-16T18:23:02.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9a9a33700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a4100300 0x7fa9a419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:02.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9a9a33700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a4100300 0x7fa9a419ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48176/0 (socket says 172.21.15.2:48176) 2023-11-16T18:23:02.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9a9a33700 1 -- 172.21.15.2:0/1138592191 learned_addr learned my addr 172.21.15.2:0/1138592191 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:02.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9aa234700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a40ff110 0x7fa9a4068950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:02.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9aaa35700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9a4100c90 0x7fa9a419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:02.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9a9a33700 1 -- 172.21.15.2:0/1138592191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9a4100c90 msgr2=0x7fa9a419d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:02.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9a9a33700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9a4100c90 0x7fa9a419d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9a9a33700 1 -- 172.21.15.2:0/1138592191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a40ff110 msgr2=0x7fa9a4068950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:02.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9a9a33700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a40ff110 0x7fa9a4068950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9a9a33700 1 -- 172.21.15.2:0/1138592191 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c009d00 con 0x7fa9a4100300 2023-11-16T18:23:02.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9aaa35700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9a4100c90 0x7fa9a419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:02.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa9a9a33700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a4100300 0x7fa9a419ade0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000d950 tx=0x7fa9a000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:02.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.559+0000 7fa99b7fe700 1 -- 172.21.15.2:0/1138592191 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0009940 con 0x7fa9a4100300 2023-11-16T18:23:02.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.560+0000 7fa99b7fe700 1 -- 172.21.15.2:0/1138592191 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa9a0010460 con 0x7fa9a4100300 2023-11-16T18:23:02.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.560+0000 7fa9ac498700 1 -- 172.21.15.2:0/1138592191 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a410aad0 con 0x7fa9a4100300 2023-11-16T18:23:02.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.560+0000 7fa99b7fe700 1 -- 172.21.15.2:0/1138592191 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a000f620 con 0x7fa9a4100300 2023-11-16T18:23:02.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.560+0000 7fa9ac498700 1 -- 172.21.15.2:0/1138592191 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a410a940 con 0x7fa9a4100300 2023-11-16T18:23:02.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.560+0000 7fa9ac498700 1 -- 172.21.15.2:0/1138592191 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a4105fa0 con 0x7fa9a4100300 2023-11-16T18:23:02.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.562+0000 7fa99b7fe700 1 -- 172.21.15.2:0/1138592191 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fa9a00108d0 con 0x7fa9a4100300 2023-11-16T18:23:02.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.562+0000 7fa99b7fe700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa9900a11d0 0x7fa9900a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:02.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.562+0000 7fa9aa234700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa9900a11d0 0x7fa9900a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:02.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.562+0000 7fa99b7fe700 1 -- 172.21.15.2:0/1138592191 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa9a00cc900 con 0x7fa9a4100300 2023-11-16T18:23:02.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.563+0000 7fa9aa234700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa9900a11d0 0x7fa9900a3680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4100160 tx=0x7fa994009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:02.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.565+0000 7fa99b7fe700 1 -- 172.21.15.2:0/1138592191 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa9a0095e60 con 0x7fa9a4100300 2023-11-16T18:23:02.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.782+0000 7fa9ac498700 1 -- 172.21.15.2:0/1138592191 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa9a4061700 con 0x7fa9a4100300 2023-11-16T18:23:02.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.782+0000 7fa99b7fe700 1 -- 172.21.15.2:0/1138592191 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v86) v1 ==== 81+0+579 (secure 0 0 0) 0x7fa9a0020080 con 0x7fa9a4100300 2023-11-16T18:23:02.784 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/857143886 2023-11-17T18:16:37.723675+0000 2023-11-16T18:23:02.784 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/251514255 2023-11-17T18:16:37.723675+0000 2023-11-16T18:23:02.784 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/1605535303 2023-11-17T18:16:37.723675+0000 2023-11-16T18:23:02.785 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/1605535303 2023-11-17T18:16:37.723675+0000 2023-11-16T18:23:02.785 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/1609662822 2023-11-17T18:16:13.217812+0000 2023-11-16T18:23:02.785 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/1609662822 2023-11-17T18:16:13.217812+0000 2023-11-16T18:23:02.785 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/708195783 2023-11-17T18:16:13.217812+0000 2023-11-16T18:23:02.785 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/1158002645 2023-11-17T18:16:13.217812+0000 2023-11-16T18:23:02.785 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/1986390775 2023-11-17T18:16:13.217812+0000 2023-11-16T18:23:02.785 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/2955607810 2023-11-17T18:16:37.723675+0000 2023-11-16T18:23:02.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.784+0000 7fa9ac498700 1 -- 172.21.15.2:0/1138592191 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa9900a11d0 msgr2=0x7fa9900a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:02.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.784+0000 7fa9ac498700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa9900a11d0 0x7fa9900a3680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4100160 tx=0x7fa994009260 comp rx=0 tx=0).stop 2023-11-16T18:23:02.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.784+0000 7fa9ac498700 1 -- 172.21.15.2:0/1138592191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a4100300 msgr2=0x7fa9a419ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:02.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.784+0000 7fa9ac498700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a4100300 0x7fa9a419ade0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000d950 tx=0x7fa9a000dc60 comp rx=0 tx=0).stop 2023-11-16T18:23:02.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.784+0000 7fa9ac498700 1 -- 172.21.15.2:0/1138592191 shutdown_connections 2023-11-16T18:23:02.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.784+0000 7fa9ac498700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa9900a11d0 0x7fa9900a3680 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.784+0000 7fa9ac498700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9a4100c90 0x7fa9a419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.784+0000 7fa9ac498700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a40ff110 0x7fa9a4068950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.784+0000 7fa9ac498700 1 --2- 172.21.15.2:0/1138592191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a4100300 0x7fa9a419ade0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:02.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.784+0000 7fa9ac498700 1 -- 172.21.15.2:0/1138592191 >> 172.21.15.2:0/1138592191 conn(0x7fa9a40fa6f0 msgr2=0x7fa9a40fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:02.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.784+0000 7fa9ac498700 1 -- 172.21.15.2:0/1138592191 shutdown_connections 2023-11-16T18:23:02.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:02.784+0000 7fa9ac498700 1 -- 172.21.15.2:0/1138592191 wait complete. 2023-11-16T18:23:02.788 INFO:teuthology.orchestra.run.smithi002.stderr:listed 10 entries 2023-11-16T18:23:02.798 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi002.front.sepia.ceph.com... 2023-11-16T18:23:02.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:23:02.798 DEBUG:teuthology.orchestra.run.smithi002:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-11-16T18:23:02.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi002.stderr:ceph-fuse[135801]: fuse finished with error 0 and tester_r 0 2023-11-16T18:23:02.839 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T18:23:02.933 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T18:23:02.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:23:02.933 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-11-16T18:23:02.967 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:23:03.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:02 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2438615159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:23:03.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:02 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1026465458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:23:03.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:02 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1138592191' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T18:23:03.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.132+0000 7f1faa115700 1 -- 172.21.15.2:0/2260143913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa4100d40 msgr2=0x7f1fa4107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:03.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.132+0000 7f1faa115700 1 --2- 172.21.15.2:0/2260143913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa4100d40 0x7f1fa4107600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c00b0a0 tx=0x7f1f9c00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.132+0000 7f1faa115700 1 -- 172.21.15.2:0/2260143913 shutdown_connections 2023-11-16T18:23:03.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.132+0000 7f1faa115700 1 --2- 172.21.15.2:0/2260143913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa4100d40 0x7f1fa4107600 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.132+0000 7f1faa115700 1 --2- 172.21.15.2:0/2260143913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fa40ff1c0 0x7f1fa40ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.132+0000 7f1faa115700 1 --2- 172.21.15.2:0/2260143913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fa41003b0 0x7f1fa4100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.132+0000 7f1faa115700 1 -- 172.21.15.2:0/2260143913 >> 172.21.15.2:0/2260143913 conn(0x7f1fa40fa7c0 msgr2=0x7f1fa40fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:03.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.132+0000 7f1faa115700 1 -- 172.21.15.2:0/2260143913 shutdown_connections 2023-11-16T18:23:03.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.132+0000 7f1faa115700 1 -- 172.21.15.2:0/2260143913 wait complete. 2023-11-16T18:23:03.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.133+0000 7f1faa115700 1 Processor -- start 2023-11-16T18:23:03.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.133+0000 7f1faa115700 1 -- start start 2023-11-16T18:23:03.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.133+0000 7f1faa115700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa40ff1c0 0x7f1fa4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:03.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.133+0000 7f1faa115700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fa41003b0 0x7f1fa419aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:03.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.133+0000 7f1faa115700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fa4100d40 0x7f1fa419d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:03.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.133+0000 7f1faa115700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa419dcd0 con 0x7f1fa40ff1c0 2023-11-16T18:23:03.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.133+0000 7f1faa115700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa419de40 con 0x7f1fa41003b0 2023-11-16T18:23:03.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.133+0000 7f1faa115700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa419e000 con 0x7f1fa4100d40 2023-11-16T18:23:03.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.133+0000 7f1fa37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa40ff1c0 0x7f1fa4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:03.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.133+0000 7f1fa37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa40ff1c0 0x7f1fa4068930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48188/0 (socket says 172.21.15.2:48188) 2023-11-16T18:23:03.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.133+0000 7f1fa37fe700 1 -- 172.21.15.2:0/168142620 learned_addr learned my addr 172.21.15.2:0/168142620 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:03.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1fa3fff700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fa4100d40 0x7f1fa419d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1fa2ffd700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fa41003b0 0x7f1fa419aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1fa37fe700 1 -- 172.21.15.2:0/168142620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fa4100d40 msgr2=0x7f1fa419d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1fa37fe700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fa4100d40 0x7f1fa419d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1fa37fe700 1 -- 172.21.15.2:0/168142620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fa41003b0 msgr2=0x7f1fa419aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1fa37fe700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fa41003b0 0x7f1fa419aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1fa37fe700 1 -- 172.21.15.2:0/168142620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c009d00 con 0x7f1fa40ff1c0 2023-11-16T18:23:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1fa3fff700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fa4100d40 0x7f1fa419d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1fa37fe700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa40ff1c0 0x7f1fa4068930 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400d980 tx=0x7f1f9400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1fa0ff9700 1 -- 172.21.15.2:0/168142620 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9400de60 con 0x7f1fa40ff1c0 2023-11-16T18:23:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1fa0ff9700 1 -- 172.21.15.2:0/168142620 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f940044e0 con 0x7f1fa40ff1c0 2023-11-16T18:23:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1faa115700 1 -- 172.21.15.2:0/168142620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa410ab80 con 0x7f1fa40ff1c0 2023-11-16T18:23:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1fa0ff9700 1 -- 172.21.15.2:0/168142620 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94003be0 con 0x7f1fa40ff1c0 2023-11-16T18:23:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.134+0000 7f1faa115700 1 -- 172.21.15.2:0/168142620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa410a9f0 con 0x7f1fa40ff1c0 2023-11-16T18:23:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.135+0000 7f1faa115700 1 -- 172.21.15.2:0/168142620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa419e750 con 0x7f1fa40ff1c0 2023-11-16T18:23:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.136+0000 7f1fa0ff9700 1 -- 172.21.15.2:0/168142620 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f1f94009bb0 con 0x7f1fa40ff1c0 2023-11-16T18:23:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.137+0000 7f1fa0ff9700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1f900a1150 0x7f1f900a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.137+0000 7f1fa0ff9700 1 -- 172.21.15.2:0/168142620 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1f940ccc40 con 0x7f1fa40ff1c0 2023-11-16T18:23:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.137+0000 7f1fa2ffd700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1f900a1150 0x7f1f900a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.137+0000 7f1fa2ffd700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1f900a1150 0x7f1f900a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c005950 tx=0x7f1f8c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:03.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.140+0000 7f1fa0ff9700 1 -- 172.21.15.2:0/168142620 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1f940961a0 con 0x7f1fa40ff1c0 2023-11-16T18:23:03.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:02 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2438615159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:23:03.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:02 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1026465458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:23:03.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:02 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1138592191' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T18:23:03.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:02 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2438615159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:23:03.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:02 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1026465458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:23:03.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:02 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1138592191' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T18:23:03.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.342+0000 7f1faa115700 1 -- 172.21.15.2:0/168142620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fa41060a0 con 0x7f1fa40ff1c0 2023-11-16T18:23:03.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.344+0000 7f1fa0ff9700 1 -- 172.21.15.2:0/168142620 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23853 (secure 0 0 0) 0x7f1f94009940 con 0x7f1fa40ff1c0 2023-11-16T18:23:03.349 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:03.349 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":86,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:22:51.534683+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:23:03.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.346+0000 7f1faa115700 1 -- 172.21.15.2:0/168142620 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1f900a1150 msgr2=0x7f1f900a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:03.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.346+0000 7f1faa115700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1f900a1150 0x7f1f900a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c005950 tx=0x7f1f8c0058e0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.346+0000 7f1faa115700 1 -- 172.21.15.2:0/168142620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa40ff1c0 msgr2=0x7f1fa4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:03.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.346+0000 7f1faa115700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa40ff1c0 0x7f1fa4068930 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400d980 tx=0x7f1f9400dd40 comp rx=0 tx=0).stop 2023-11-16T18:23:03.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.346+0000 7f1faa115700 1 -- 172.21.15.2:0/168142620 shutdown_connections 2023-11-16T18:23:03.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.346+0000 7f1faa115700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1f900a1150 0x7f1f900a3600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.346+0000 7f1faa115700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fa4100d40 0x7f1fa419d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.346+0000 7f1faa115700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa40ff1c0 0x7f1fa4068930 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.346+0000 7f1faa115700 1 --2- 172.21.15.2:0/168142620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fa41003b0 0x7f1fa419aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.346+0000 7f1faa115700 1 -- 172.21.15.2:0/168142620 >> 172.21.15.2:0/168142620 conn(0x7f1fa40fa7c0 msgr2=0x7f1fa40fca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:03.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.346+0000 7f1faa115700 1 -- 172.21.15.2:0/168142620 shutdown_connections 2023-11-16T18:23:03.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.346+0000 7f1faa115700 1 -- 172.21.15.2:0/168142620 wait complete. 2023-11-16T18:23:03.361 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:23:03.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.485+0000 7f57f5259700 1 -- 172.21.15.2:0/2785390505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57f0100da0 msgr2=0x7f57f0107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:03.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.485+0000 7f57f5259700 1 --2- 172.21.15.2:0/2785390505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57f0100da0 0x7f57f0107660 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f57e800b0a0 tx=0x7f57e800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.485+0000 7f57f5259700 1 -- 172.21.15.2:0/2785390505 shutdown_connections 2023-11-16T18:23:03.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.485+0000 7f57f5259700 1 --2- 172.21.15.2:0/2785390505 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57f00ff220 0x7f57f00ff630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.485+0000 7f57f5259700 1 --2- 172.21.15.2:0/2785390505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57f0100da0 0x7f57f0107660 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.485+0000 7f57f5259700 1 --2- 172.21.15.2:0/2785390505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57f0100410 0x7f57f0100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.485+0000 7f57f5259700 1 -- 172.21.15.2:0/2785390505 >> 172.21.15.2:0/2785390505 conn(0x7f57f00fa820 msgr2=0x7f57f00fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:03.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.485+0000 7f57f5259700 1 -- 172.21.15.2:0/2785390505 shutdown_connections 2023-11-16T18:23:03.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.485+0000 7f57f5259700 1 -- 172.21.15.2:0/2785390505 wait complete. 2023-11-16T18:23:03.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.485+0000 7f57f5259700 1 Processor -- start 2023-11-16T18:23:03.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57f5259700 1 -- start start 2023-11-16T18:23:03.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57f5259700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57f00ff220 0x7f57f00698a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:03.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57f5259700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57f0100410 0x7f57f019ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:03.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57f5259700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57f0100da0 0x7f57f019d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:03.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57f5259700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f019dc10 con 0x7f57f0100da0 2023-11-16T18:23:03.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57f5259700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f019dd80 con 0x7f57f0100410 2023-11-16T18:23:03.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57f5259700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f019df40 con 0x7f57f00ff220 2023-11-16T18:23:03.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57ef59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57f0100da0 0x7f57f019d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:03.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57ef59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57f0100da0 0x7f57f019d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48204/0 (socket says 172.21.15.2:48204) 2023-11-16T18:23:03.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57ef59e700 1 -- 172.21.15.2:0/1364578706 learned_addr learned my addr 172.21.15.2:0/1364578706 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:03.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57ee59c700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57f0100410 0x7f57f019ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:03.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57eed9d700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57f00ff220 0x7f57f00698a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:03.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57ef59e700 1 -- 172.21.15.2:0/1364578706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57f00ff220 msgr2=0x7f57f00698a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:03.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57ef59e700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57f00ff220 0x7f57f00698a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57ef59e700 1 -- 172.21.15.2:0/1364578706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57f0100410 msgr2=0x7f57f019ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:03.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57ef59e700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57f0100410 0x7f57f019ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57ef59e700 1 -- 172.21.15.2:0/1364578706 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e8009d00 con 0x7f57f0100da0 2023-11-16T18:23:03.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57eed9d700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57f00ff220 0x7f57f00698a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:03.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.486+0000 7f57ef59e700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57f0100da0 0x7f57f019d470 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f57e800b780 tx=0x7f57e800bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:03.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.487+0000 7f57d7fff700 1 -- 172.21.15.2:0/1364578706 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e800e040 con 0x7f57f0100da0 2023-11-16T18:23:03.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.487+0000 7f57f5259700 1 -- 172.21.15.2:0/1364578706 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f010ab90 con 0x7f57f0100da0 2023-11-16T18:23:03.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.487+0000 7f57d7fff700 1 -- 172.21.15.2:0/1364578706 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57e8004e60 con 0x7f57f0100da0 2023-11-16T18:23:03.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.487+0000 7f57d7fff700 1 -- 172.21.15.2:0/1364578706 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e801b740 con 0x7f57f0100da0 2023-11-16T18:23:03.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.487+0000 7f57f5259700 1 -- 172.21.15.2:0/1364578706 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f010aa00 con 0x7f57f0100da0 2023-11-16T18:23:03.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.488+0000 7f57f5259700 1 -- 172.21.15.2:0/1364578706 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f019e590 con 0x7f57f0100da0 2023-11-16T18:23:03.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.489+0000 7f57d7fff700 1 -- 172.21.15.2:0/1364578706 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f57e801da40 con 0x7f57f0100da0 2023-11-16T18:23:03.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.489+0000 7f57d7fff700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f57d80a1150 0x7f57d80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:03.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.489+0000 7f57d7fff700 1 -- 172.21.15.2:0/1364578706 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f57e80ce470 con 0x7f57f0100da0 2023-11-16T18:23:03.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.489+0000 7f57eed9d700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f57d80a1150 0x7f57d80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:03.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.490+0000 7f57eed9d700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f57d80a1150 0x7f57d80a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f57e0005fd0 tx=0x7f57e0005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:03.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.494+0000 7f57d7fff700 1 -- 172.21.15.2:0/1364578706 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f57e8097a60 con 0x7f57f0100da0 2023-11-16T18:23:03.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.699+0000 7f57f5259700 1 -- 172.21.15.2:0/1364578706 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57f0105fe0 con 0x7f57f0100da0 2023-11-16T18:23:03.703 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.701+0000 7f57d7fff700 1 -- 172.21.15.2:0/1364578706 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23853 (secure 0 0 0) 0x7f57e809b080 con 0x7f57f0100da0 2023-11-16T18:23:03.703 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:03.703 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":86,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:22:51.534683+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:23:03.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.703+0000 7f57f5259700 1 -- 172.21.15.2:0/1364578706 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f57d80a1150 msgr2=0x7f57d80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:03.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.703+0000 7f57f5259700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f57d80a1150 0x7f57d80a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f57e0005fd0 tx=0x7f57e0005f00 comp rx=0 tx=0).stop 2023-11-16T18:23:03.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.703+0000 7f57f5259700 1 -- 172.21.15.2:0/1364578706 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57f0100da0 msgr2=0x7f57f019d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:03.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.703+0000 7f57f5259700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57f0100da0 0x7f57f019d470 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f57e800b780 tx=0x7f57e800bd00 comp rx=0 tx=0).stop 2023-11-16T18:23:03.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.703+0000 7f57f5259700 1 -- 172.21.15.2:0/1364578706 shutdown_connections 2023-11-16T18:23:03.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.703+0000 7f57f5259700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f57d80a1150 0x7f57d80a3600 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.703+0000 7f57f5259700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57f00ff220 0x7f57f00698a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.703+0000 7f57f5259700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57f0100da0 0x7f57f019d470 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.703+0000 7f57f5259700 1 --2- 172.21.15.2:0/1364578706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57f0100410 0x7f57f019ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.703+0000 7f57f5259700 1 -- 172.21.15.2:0/1364578706 >> 172.21.15.2:0/1364578706 conn(0x7f57f00fa820 msgr2=0x7f57f00fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:03.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.703+0000 7f57f5259700 1 -- 172.21.15.2:0/1364578706 shutdown_connections 2023-11-16T18:23:03.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.703+0000 7f57f5259700 1 -- 172.21.15.2:0/1364578706 wait complete. 2023-11-16T18:23:03.723 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-16T18:23:03.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.849+0000 7f8f05d54700 1 -- 172.21.15.2:0/1810604862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f00100d30 msgr2=0x7f8f001075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:03.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.849+0000 7f8f05d54700 1 --2- 172.21.15.2:0/1810604862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f00100d30 0x7f8f001075f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800b0a0 tx=0x7f8ef800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.849+0000 7f8f05d54700 1 -- 172.21.15.2:0/1810604862 shutdown_connections 2023-11-16T18:23:03.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.849+0000 7f8f05d54700 1 --2- 172.21.15.2:0/1810604862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f00100d30 0x7f8f001075f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.849+0000 7f8f05d54700 1 --2- 172.21.15.2:0/1810604862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f000ff1b0 0x7f8f000ff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.849+0000 7f8f05d54700 1 --2- 172.21.15.2:0/1810604862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f001003a0 0x7f8f001007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.849+0000 7f8f05d54700 1 -- 172.21.15.2:0/1810604862 >> 172.21.15.2:0/1810604862 conn(0x7f8f000fa7b0 msgr2=0x7f8f000fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:03.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.850+0000 7f8f05d54700 1 -- 172.21.15.2:0/1810604862 shutdown_connections 2023-11-16T18:23:03.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.850+0000 7f8f05d54700 1 -- 172.21.15.2:0/1810604862 wait complete. 2023-11-16T18:23:03.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.850+0000 7f8f05d54700 1 Processor -- start 2023-11-16T18:23:03.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.850+0000 7f8f05d54700 1 -- start start 2023-11-16T18:23:03.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.850+0000 7f8f05d54700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f000ff1b0 0x7f8f00069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:03.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.850+0000 7f8f05d54700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f001003a0 0x7f8f0019d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:03.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.850+0000 7f8f05d54700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f00100d30 0x7f8f0019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:03.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.850+0000 7f8f05d54700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0019fe30 con 0x7f8f000ff1b0 2023-11-16T18:23:03.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.850+0000 7f8f05d54700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0019ffa0 con 0x7f8f00100d30 2023-11-16T18:23:03.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.850+0000 7f8f05d54700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f001a0160 con 0x7f8f001003a0 2023-11-16T18:23:03.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8eff7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f000ff1b0 0x7f8f00069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:03.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8eff7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f000ff1b0 0x7f8f00069970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48214/0 (socket says 172.21.15.2:48214) 2023-11-16T18:23:03.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8eff7fe700 1 -- 172.21.15.2:0/3978401808 learned_addr learned my addr 172.21.15.2:0/3978401808 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:03.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8efffff700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f00100d30 0x7f8f0019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:03.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8eff7fe700 1 -- 172.21.15.2:0/3978401808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f001003a0 msgr2=0x7f8f0019d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:03.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8efeffd700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f001003a0 0x7f8f0019d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:03.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8eff7fe700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f001003a0 0x7f8f0019d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8eff7fe700 1 -- 172.21.15.2:0/3978401808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f00100d30 msgr2=0x7f8f0019f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:03.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8eff7fe700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f00100d30 0x7f8f0019f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:03.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8eff7fe700 1 -- 172.21.15.2:0/3978401808 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef8009d00 con 0x7f8f000ff1b0 2023-11-16T18:23:03.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8efffff700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f00100d30 0x7f8f0019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:03.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8efeffd700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f001003a0 0x7f8f0019d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:03.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8eff7fe700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f000ff1b0 0x7f8f00069970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000d980 tx=0x7f8ef000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:03.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8efcff9700 1 -- 172.21.15.2:0/3978401808 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef000de60 con 0x7f8f000ff1b0 2023-11-16T18:23:03.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8efcff9700 1 -- 172.21.15.2:0/3978401808 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ef00044e0 con 0x7f8f000ff1b0 2023-11-16T18:23:03.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8f05d54700 1 -- 172.21.15.2:0/3978401808 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0010ab70 con 0x7f8f000ff1b0 2023-11-16T18:23:03.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8efcff9700 1 -- 172.21.15.2:0/3978401808 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0003be0 con 0x7f8f000ff1b0 2023-11-16T18:23:03.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.851+0000 7f8f05d54700 1 -- 172.21.15.2:0/3978401808 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0010a9e0 con 0x7f8f000ff1b0 2023-11-16T18:23:03.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.852+0000 7f8f05d54700 1 -- 172.21.15.2:0/3978401808 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f001a08b0 con 0x7f8f000ff1b0 2023-11-16T18:23:03.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.853+0000 7f8efcff9700 1 -- 172.21.15.2:0/3978401808 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f8ef0009bb0 con 0x7f8f000ff1b0 2023-11-16T18:23:03.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.854+0000 7f8efcff9700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8ee80a1150 0x7f8ee80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:03.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.854+0000 7f8efcff9700 1 -- 172.21.15.2:0/3978401808 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8ef00ccca0 con 0x7f8f000ff1b0 2023-11-16T18:23:03.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.854+0000 7f8efeffd700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8ee80a1150 0x7f8ee80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:03.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.855+0000 7f8efeffd700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8ee80a1150 0x7f8ee80a3600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4005fd0 tx=0x7f8ef4005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:03.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:03.857+0000 7f8efcff9700 1 -- 172.21.15.2:0/3978401808 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8ef00961a0 con 0x7f8f000ff1b0 2023-11-16T18:23:04.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:03 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/168142620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:04.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:03 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1364578706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:04.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:03 smithi002 ceph-mon[98807]: pgmap v269: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-11-16T18:23:04.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.108+0000 7f8f05d54700 1 -- 172.21.15.2:0/3978401808 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8f00106050 con 0x7f8f000ff1b0 2023-11-16T18:23:04.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:03 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/168142620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:04.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:03 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1364578706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:04.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:03 smithi040 ceph-mon[100251]: pgmap v269: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-11-16T18:23:04.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:03 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/168142620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:04.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:03 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1364578706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:04.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:03 smithi188 ceph-mon[100639]: pgmap v269: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-11-16T18:23:04.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.885+0000 7f8efcff9700 1 -- 172.21.15.2:0/3978401808 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8ef0009940 con 0x7f8f000ff1b0 2023-11-16T18:23:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.888+0000 7f8f05d54700 1 -- 172.21.15.2:0/3978401808 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8ee80a1150 msgr2=0x7f8ee80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.888+0000 7f8f05d54700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8ee80a1150 0x7f8ee80a3600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4005fd0 tx=0x7f8ef4005dc0 comp rx=0 tx=0).stop 2023-11-16T18:23:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.888+0000 7f8f05d54700 1 -- 172.21.15.2:0/3978401808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f000ff1b0 msgr2=0x7f8f00069970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.888+0000 7f8f05d54700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f000ff1b0 0x7f8f00069970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000d980 tx=0x7f8ef000dd40 comp rx=0 tx=0).stop 2023-11-16T18:23:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.888+0000 7f8f05d54700 1 -- 172.21.15.2:0/3978401808 shutdown_connections 2023-11-16T18:23:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.888+0000 7f8f05d54700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8ee80a1150 0x7f8ee80a3600 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.888+0000 7f8f05d54700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f00100d30 0x7f8f0019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.888+0000 7f8f05d54700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f000ff1b0 0x7f8f00069970 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.888+0000 7f8f05d54700 1 --2- 172.21.15.2:0/3978401808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f001003a0 0x7f8f0019d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:04.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.888+0000 7f8f05d54700 1 -- 172.21.15.2:0/3978401808 >> 172.21.15.2:0/3978401808 conn(0x7f8f000fa7b0 msgr2=0x7f8f000fc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:04.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.888+0000 7f8f05d54700 1 -- 172.21.15.2:0/3978401808 shutdown_connections 2023-11-16T18:23:04.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:04.888+0000 7f8f05d54700 1 -- 172.21.15.2:0/3978401808 wait complete. 2023-11-16T18:23:04.905 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-11-16T18:23:04.905 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-11-16T18:23:05.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.039+0000 7f48e8e42700 1 -- 172.21.15.2:0/3384322526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48e4101500 msgr2=0x7f48e4101950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:05.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.039+0000 7f48e8e42700 1 --2- 172.21.15.2:0/3384322526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48e4101500 0x7f48e4101950 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f48d8009990 tx=0x7f48d8009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:05.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.039+0000 7f48e8e42700 1 -- 172.21.15.2:0/3384322526 shutdown_connections 2023-11-16T18:23:05.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.039+0000 7f48e8e42700 1 --2- 172.21.15.2:0/3384322526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f48e4100310 0x7f48e4100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:05.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.039+0000 7f48e8e42700 1 --2- 172.21.15.2:0/3384322526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e4101e90 0x7f48e4108750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:05.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.039+0000 7f48e8e42700 1 --2- 172.21.15.2:0/3384322526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48e4101500 0x7f48e4101950 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:05.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.039+0000 7f48e8e42700 1 -- 172.21.15.2:0/3384322526 >> 172.21.15.2:0/3384322526 conn(0x7f48e40fb910 msgr2=0x7f48e40fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:05.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.039+0000 7f48e8e42700 1 -- 172.21.15.2:0/3384322526 shutdown_connections 2023-11-16T18:23:05.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.039+0000 7f48e8e42700 1 -- 172.21.15.2:0/3384322526 wait complete. 2023-11-16T18:23:05.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e8e42700 1 Processor -- start 2023-11-16T18:23:05.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e8e42700 1 -- start start 2023-11-16T18:23:05.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e8e42700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f48e4100310 0x7f48e4195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:05.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e8e42700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48e4101500 0x7f48e41964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:05.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e8e42700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e4101e90 0x7f48e419d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:05.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e8e42700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e419daf0 con 0x7f48e4101500 2023-11-16T18:23:05.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e8e42700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e419dc30 con 0x7f48e4101e90 2023-11-16T18:23:05.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e8e42700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e419ddc0 con 0x7f48e4100310 2023-11-16T18:23:05.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e1d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48e4101500 0x7f48e41964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:05.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e1d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48e4101500 0x7f48e41964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48230/0 (socket says 172.21.15.2:48230) 2023-11-16T18:23:05.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e1d9b700 1 -- 172.21.15.2:0/3063721164 learned_addr learned my addr 172.21.15.2:0/3063721164 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:05.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e259c700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f48e4100310 0x7f48e4195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:05.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e2d9d700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e4101e90 0x7f48e419d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:05.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e1d9b700 1 -- 172.21.15.2:0/3063721164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f48e4100310 msgr2=0x7f48e4195f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:05.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e1d9b700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f48e4100310 0x7f48e4195f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:05.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e1d9b700 1 -- 172.21.15.2:0/3063721164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e4101e90 msgr2=0x7f48e419d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:05.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e1d9b700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e4101e90 0x7f48e419d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:05.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e1d9b700 1 -- 172.21.15.2:0/3063721164 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d80095f0 con 0x7f48e4101500 2023-11-16T18:23:05.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.040+0000 7f48e2d9d700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e4101e90 0x7f48e419d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:05.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.041+0000 7f48e1d9b700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48e4101500 0x7f48e41964b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f48d8000c00 tx=0x7f48d80047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:05.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.041+0000 7f48d37fe700 1 -- 172.21.15.2:0/3063721164 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d801d070 con 0x7f48e4101500 2023-11-16T18:23:05.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.041+0000 7f48d37fe700 1 -- 172.21.15.2:0/3063721164 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d8022470 con 0x7f48e4101500 2023-11-16T18:23:05.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.041+0000 7f48e8e42700 1 -- 172.21.15.2:0/3063721164 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e410bc10 con 0x7f48e4101500 2023-11-16T18:23:05.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.041+0000 7f48d37fe700 1 -- 172.21.15.2:0/3063721164 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d8017650 con 0x7f48e4101500 2023-11-16T18:23:05.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.041+0000 7f48e8e42700 1 -- 172.21.15.2:0/3063721164 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e410ba80 con 0x7f48e4101500 2023-11-16T18:23:05.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.042+0000 7f48e8e42700 1 -- 172.21.15.2:0/3063721164 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e419e3c0 con 0x7f48e4101500 2023-11-16T18:23:05.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.043+0000 7f48d37fe700 1 -- 172.21.15.2:0/3063721164 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f48d80225e0 con 0x7f48e4101500 2023-11-16T18:23:05.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.044+0000 7f48d37fe700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f48cc0a11d0 0x7f48cc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:05.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.044+0000 7f48d37fe700 1 -- 172.21.15.2:0/3063721164 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f48d80ce4d0 con 0x7f48e4101500 2023-11-16T18:23:05.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.044+0000 7f48e259c700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f48cc0a11d0 0x7f48cc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:05.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.044+0000 7f48e259c700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f48cc0a11d0 0x7f48cc0a3680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f48e4101360 tx=0x7f48d400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:05.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.047+0000 7f48d37fe700 1 -- 172.21.15.2:0/3063721164 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f48d8097a60 con 0x7f48e4101500 2023-11-16T18:23:05.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:04 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3978401808' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-11-16T18:23:05.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:04 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3978401808' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-11-16T18:23:05.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.295+0000 7f48e8e42700 1 -- 172.21.15.2:0/3063721164 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f48e4107260 con 0x7f48e4101500 2023-11-16T18:23:05.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:04 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3978401808' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-11-16T18:23:05.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.891+0000 7f48d37fe700 1 -- 172.21.15.2:0/3063721164 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f48d809b080 con 0x7f48e4101500 2023-11-16T18:23:05.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.894+0000 7f48e8e42700 1 -- 172.21.15.2:0/3063721164 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f48cc0a11d0 msgr2=0x7f48cc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:05.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.894+0000 7f48e8e42700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f48cc0a11d0 0x7f48cc0a3680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f48e4101360 tx=0x7f48d400b410 comp rx=0 tx=0).stop 2023-11-16T18:23:05.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.894+0000 7f48e8e42700 1 -- 172.21.15.2:0/3063721164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48e4101500 msgr2=0x7f48e41964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:05.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.894+0000 7f48e8e42700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48e4101500 0x7f48e41964b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f48d8000c00 tx=0x7f48d80047a0 comp rx=0 tx=0).stop 2023-11-16T18:23:05.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.894+0000 7f48e8e42700 1 -- 172.21.15.2:0/3063721164 shutdown_connections 2023-11-16T18:23:05.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.894+0000 7f48e8e42700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f48cc0a11d0 0x7f48cc0a3680 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:05.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.894+0000 7f48e8e42700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f48e4100310 0x7f48e4195f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:05.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.894+0000 7f48e8e42700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e4101e90 0x7f48e419d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:05.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.894+0000 7f48e8e42700 1 --2- 172.21.15.2:0/3063721164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48e4101500 0x7f48e41964b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:05.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.894+0000 7f48e8e42700 1 -- 172.21.15.2:0/3063721164 >> 172.21.15.2:0/3063721164 conn(0x7f48e40fb910 msgr2=0x7f48e40fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:05.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.894+0000 7f48e8e42700 1 -- 172.21.15.2:0/3063721164 shutdown_connections 2023-11-16T18:23:05.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:05.894+0000 7f48e8e42700 1 -- 172.21.15.2:0/3063721164 wait complete. 2023-11-16T18:23:05.915 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-11-16T18:23:06.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.049+0000 7f4d15398700 1 -- 172.21.15.2:0/1907620806 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d10100d90 msgr2=0x7f4d10103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:06.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.049+0000 7f4d15398700 1 --2- 172.21.15.2:0/1907620806 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d10100d90 0x7f4d10103170 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000b0a0 tx=0x7f4d0000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:06.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.049+0000 7f4d15398700 1 -- 172.21.15.2:0/1907620806 shutdown_connections 2023-11-16T18:23:06.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.049+0000 7f4d15398700 1 --2- 172.21.15.2:0/1907620806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d10105fd0 0x7f4d101083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:06.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.049+0000 7f4d15398700 1 --2- 172.21.15.2:0/1907620806 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d10100d90 0x7f4d10103170 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:06.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.049+0000 7f4d15398700 1 --2- 172.21.15.2:0/1907620806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d101036b0 0x7f4d10105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:06.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.049+0000 7f4d15398700 1 -- 172.21.15.2:0/1907620806 >> 172.21.15.2:0/1907620806 conn(0x7f4d100fa750 msgr2=0x7f4d100fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:06.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.049+0000 7f4d15398700 1 -- 172.21.15.2:0/1907620806 shutdown_connections 2023-11-16T18:23:06.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.049+0000 7f4d15398700 1 -- 172.21.15.2:0/1907620806 wait complete. 2023-11-16T18:23:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d15398700 1 Processor -- start 2023-11-16T18:23:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d15398700 1 -- start start 2023-11-16T18:23:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d15398700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d10100d90 0x7f4d10198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d15398700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d101036b0 0x7f4d101988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d15398700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d10105fd0 0x7f4d1019f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d15398700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1019ff10 con 0x7f4d10100d90 2023-11-16T18:23:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d15398700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d101a0050 con 0x7f4d10105fd0 2023-11-16T18:23:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d15398700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d101a01e0 con 0x7f4d101036b0 2023-11-16T18:23:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d0effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d10100d90 0x7f4d10198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d0effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d10100d90 0x7f4d10198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48242/0 (socket says 172.21.15.2:48242) 2023-11-16T18:23:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d0effd700 1 -- 172.21.15.2:0/524617146 learned_addr learned my addr 172.21.15.2:0/524617146 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d0f7fe700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d10105fd0 0x7f4d1019f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d0e7fc700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d101036b0 0x7f4d101988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:06.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d0effd700 1 -- 172.21.15.2:0/524617146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d101036b0 msgr2=0x7f4d101988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:06.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d0effd700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d101036b0 0x7f4d101988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:06.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d0effd700 1 -- 172.21.15.2:0/524617146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d10105fd0 msgr2=0x7f4d1019f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:06.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.050+0000 7f4d0effd700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d10105fd0 0x7f4d1019f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:06.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.051+0000 7f4d0effd700 1 -- 172.21.15.2:0/524617146 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d00009d00 con 0x7f4d10100d90 2023-11-16T18:23:06.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.051+0000 7f4d0e7fc700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d101036b0 0x7f4d101988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:06.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.051+0000 7f4d0f7fe700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d10105fd0 0x7f4d1019f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:06.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.051+0000 7f4d0effd700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d10100d90 0x7f4d10198390 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4d000062a0 tx=0x7f4d00004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:06.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.051+0000 7f4d07fff700 1 -- 172.21.15.2:0/524617146 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0000e040 con 0x7f4d10100d90 2023-11-16T18:23:06.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.051+0000 7f4d07fff700 1 -- 172.21.15.2:0/524617146 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d0001db50 con 0x7f4d10100d90 2023-11-16T18:23:06.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.051+0000 7f4d15398700 1 -- 172.21.15.2:0/524617146 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1010bc40 con 0x7f4d10100d90 2023-11-16T18:23:06.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.051+0000 7f4d07fff700 1 -- 172.21.15.2:0/524617146 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0001bb90 con 0x7f4d10100d90 2023-11-16T18:23:06.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.051+0000 7f4d15398700 1 -- 172.21.15.2:0/524617146 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1010bab0 con 0x7f4d10100d90 2023-11-16T18:23:06.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.052+0000 7f4d15398700 1 -- 172.21.15.2:0/524617146 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d10107b40 con 0x7f4d10100d90 2023-11-16T18:23:06.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.053+0000 7f4d07fff700 1 -- 172.21.15.2:0/524617146 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f4d0001bcf0 con 0x7f4d10100d90 2023-11-16T18:23:06.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.054+0000 7f4d07fff700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4cfc0a1150 0x7f4cfc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:06.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.054+0000 7f4d0e7fc700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4cfc0a1150 0x7f4cfc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:06.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.054+0000 7f4d07fff700 1 -- 172.21.15.2:0/524617146 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4d000ceed0 con 0x7f4d10100d90 2023-11-16T18:23:06.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.054+0000 7f4d0e7fc700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4cfc0a1150 0x7f4cfc0a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8005fd0 tx=0x7f4cf8005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:06.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.057+0000 7f4d07fff700 1 -- 172.21.15.2:0/524617146 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4d000984c0 con 0x7f4d10100d90 2023-11-16T18:23:06.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:05 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3978401808' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-11-16T18:23:06.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:05 smithi040 ceph-mon[100251]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:23:06.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:05 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3063721164' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-11-16T18:23:06.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:05 smithi040 ceph-mon[100251]: pgmap v270: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T18:23:06.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:05 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3978401808' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-11-16T18:23:06.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:05 smithi188 ceph-mon[100639]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:23:06.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:05 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3063721164' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-11-16T18:23:06.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:05 smithi188 ceph-mon[100639]: pgmap v270: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T18:23:06.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.303+0000 7f4d15398700 1 -- 172.21.15.2:0/524617146 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f4d1002cfe0 con 0x7f4d10100d90 2023-11-16T18:23:06.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:05 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3978401808' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-11-16T18:23:06.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:05 smithi002 ceph-mon[98807]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:23:06.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:05 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3063721164' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-11-16T18:23:06.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:05 smithi002 ceph-mon[98807]: pgmap v270: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T18:23:06.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.896+0000 7f4d07fff700 1 -- 172.21.15.2:0/524617146 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7f4d0009bae0 con 0x7f4d10100d90 2023-11-16T18:23:06.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.899+0000 7f4d15398700 1 -- 172.21.15.2:0/524617146 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4cfc0a1150 msgr2=0x7f4cfc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:06.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.899+0000 7f4d15398700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4cfc0a1150 0x7f4cfc0a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8005fd0 tx=0x7f4cf8005ee0 comp rx=0 tx=0).stop 2023-11-16T18:23:06.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.899+0000 7f4d15398700 1 -- 172.21.15.2:0/524617146 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d10100d90 msgr2=0x7f4d10198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:06.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.899+0000 7f4d15398700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d10100d90 0x7f4d10198390 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4d000062a0 tx=0x7f4d00004db0 comp rx=0 tx=0).stop 2023-11-16T18:23:06.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.899+0000 7f4d15398700 1 -- 172.21.15.2:0/524617146 shutdown_connections 2023-11-16T18:23:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.899+0000 7f4d15398700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4cfc0a1150 0x7f4cfc0a3600 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.899+0000 7f4d15398700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d10105fd0 0x7f4d1019f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.899+0000 7f4d15398700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d10100d90 0x7f4d10198390 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.899+0000 7f4d15398700 1 --2- 172.21.15.2:0/524617146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d101036b0 0x7f4d101988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.899+0000 7f4d15398700 1 -- 172.21.15.2:0/524617146 >> 172.21.15.2:0/524617146 conn(0x7f4d100fa750 msgr2=0x7f4d100ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.899+0000 7f4d15398700 1 -- 172.21.15.2:0/524617146 shutdown_connections 2023-11-16T18:23:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:06.899+0000 7f4d15398700 1 -- 172.21.15.2:0/524617146 wait complete. 2023-11-16T18:23:06.920 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-11-16T18:23:07.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.053+0000 7f72b44be700 1 -- 172.21.15.2:0/3579334692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72ac101e40 msgr2=0x7f72ac108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:07.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.053+0000 7f72b44be700 1 --2- 172.21.15.2:0/3579334692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72ac101e40 0x7f72ac108700 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f72a400b0a0 tx=0x7f72a400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:07.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.053+0000 7f72b44be700 1 -- 172.21.15.2:0/3579334692 shutdown_connections 2023-11-16T18:23:07.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.053+0000 7f72b44be700 1 --2- 172.21.15.2:0/3579334692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72ac101e40 0x7f72ac108700 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:07.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.053+0000 7f72b44be700 1 --2- 172.21.15.2:0/3579334692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72ac1002c0 0x7f72ac1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:07.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.053+0000 7f72b44be700 1 --2- 172.21.15.2:0/3579334692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72ac1014b0 0x7f72ac101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:07.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.054+0000 7f72b44be700 1 -- 172.21.15.2:0/3579334692 >> 172.21.15.2:0/3579334692 conn(0x7f72ac0fb8e0 msgr2=0x7f72ac0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:07.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.054+0000 7f72b44be700 1 -- 172.21.15.2:0/3579334692 shutdown_connections 2023-11-16T18:23:07.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.054+0000 7f72b44be700 1 -- 172.21.15.2:0/3579334692 wait complete. 2023-11-16T18:23:07.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.054+0000 7f72b44be700 1 Processor -- start 2023-11-16T18:23:07.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.054+0000 7f72b44be700 1 -- start start 2023-11-16T18:23:07.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b44be700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72ac1002c0 0x7f72ac1960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:07.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b44be700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72ac1014b0 0x7f72ac1965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:07.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b44be700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72ac101e40 0x7f72ac19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:07.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b44be700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac19dc20 con 0x7f72ac1014b0 2023-11-16T18:23:07.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b44be700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac19dd60 con 0x7f72ac101e40 2023-11-16T18:23:07.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b44be700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac19def0 con 0x7f72ac1002c0 2023-11-16T18:23:07.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b1a59700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72ac1014b0 0x7f72ac1965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:07.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b1a59700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72ac1014b0 0x7f72ac1965e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48256/0 (socket says 172.21.15.2:48256) 2023-11-16T18:23:07.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b1a59700 1 -- 172.21.15.2:0/3466136351 learned_addr learned my addr 172.21.15.2:0/3466136351 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:07.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b2a5b700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72ac101e40 0x7f72ac19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:07.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b225a700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72ac1002c0 0x7f72ac1960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:07.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b1a59700 1 -- 172.21.15.2:0/3466136351 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72ac1002c0 msgr2=0x7f72ac1960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:07.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b1a59700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72ac1002c0 0x7f72ac1960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:07.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b1a59700 1 -- 172.21.15.2:0/3466136351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72ac101e40 msgr2=0x7f72ac19d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:07.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b1a59700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72ac101e40 0x7f72ac19d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:07.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b1a59700 1 -- 172.21.15.2:0/3466136351 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4009d00 con 0x7f72ac1014b0 2023-11-16T18:23:07.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b1a59700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72ac1014b0 0x7f72ac1965e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f72a800d950 tx=0x7f72a800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:07.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f729f7fe700 1 -- 172.21.15.2:0/3466136351 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8009940 con 0x7f72ac1014b0 2023-11-16T18:23:07.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f729f7fe700 1 -- 172.21.15.2:0/3466136351 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f72a8010460 con 0x7f72ac1014b0 2023-11-16T18:23:07.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f72b44be700 1 -- 172.21.15.2:0/3466136351 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ac10bc50 con 0x7f72ac1014b0 2023-11-16T18:23:07.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.055+0000 7f729f7fe700 1 -- 172.21.15.2:0/3466136351 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a800f620 con 0x7f72ac1014b0 2023-11-16T18:23:07.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.056+0000 7f72b44be700 1 -- 172.21.15.2:0/3466136351 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ac10bac0 con 0x7f72ac1014b0 2023-11-16T18:23:07.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.056+0000 7f72b44be700 1 -- 172.21.15.2:0/3466136351 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ac19e5e0 con 0x7f72ac1014b0 2023-11-16T18:23:07.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.058+0000 7f729f7fe700 1 -- 172.21.15.2:0/3466136351 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f72a80108d0 con 0x7f72ac1014b0 2023-11-16T18:23:07.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.058+0000 7f729f7fe700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f72980a1150 0x7f72980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:07.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.058+0000 7f72b225a700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f72980a1150 0x7f72980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:07.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.058+0000 7f729f7fe700 1 -- 172.21.15.2:0/3466136351 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f72a80cc990 con 0x7f72ac1014b0 2023-11-16T18:23:07.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.059+0000 7f72b225a700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f72980a1150 0x7f72980a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f72a0005950 tx=0x7f72a00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:07.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.062+0000 7f729f7fe700 1 -- 172.21.15.2:0/3466136351 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f72a8095ef0 con 0x7f72ac1014b0 2023-11-16T18:23:07.172 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:06 smithi040 ceph-mon[100251]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-11-16T18:23:07.172 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:06 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3063721164' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-11-16T18:23:07.172 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:06 smithi040 ceph-mon[100251]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-11-16T18:23:07.173 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:06 smithi040 ceph-mon[100251]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:23:07.173 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:06 smithi040 ceph-mon[100251]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-11-16T18:23:07.173 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:06 smithi040 ceph-mon[100251]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-11-16T18:23:07.173 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:06 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/524617146' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T18:23:07.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:06 smithi188 ceph-mon[100639]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-11-16T18:23:07.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:06 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3063721164' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-11-16T18:23:07.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:06 smithi188 ceph-mon[100639]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-11-16T18:23:07.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:06 smithi188 ceph-mon[100639]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:23:07.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:06 smithi188 ceph-mon[100639]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-11-16T18:23:07.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:06 smithi188 ceph-mon[100639]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-11-16T18:23:07.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:06 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/524617146' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T18:23:07.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.311+0000 7f72b44be700 1 -- 172.21.15.2:0/3466136351 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f72ac107160 con 0x7f72ac1014b0 2023-11-16T18:23:07.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:06 smithi002 ceph-mon[98807]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-11-16T18:23:07.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:06 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3063721164' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-11-16T18:23:07.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:06 smithi002 ceph-mon[98807]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-11-16T18:23:07.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:06 smithi002 ceph-mon[98807]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T18:23:07.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:06 smithi002 ceph-mon[98807]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-11-16T18:23:07.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:06 smithi002 ceph-mon[98807]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-11-16T18:23:07.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:06 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/524617146' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T18:23:07.483 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:23:07 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-b[125477]: 2023-11-16T18:23:07.171+0000 7f68bf57f700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-11-16T18:23:07.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.908+0000 7f729f7fe700 1 -- 172.21.15.2:0/3466136351 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f72a80160b0 con 0x7f72ac1014b0 2023-11-16T18:23:07.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.911+0000 7f72b44be700 1 -- 172.21.15.2:0/3466136351 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f72980a1150 msgr2=0x7f72980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:07.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.911+0000 7f72b44be700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f72980a1150 0x7f72980a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f72a0005950 tx=0x7f72a00058e0 comp rx=0 tx=0).stop 2023-11-16T18:23:07.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.911+0000 7f72b44be700 1 -- 172.21.15.2:0/3466136351 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72ac1014b0 msgr2=0x7f72ac1965e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:07.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.911+0000 7f72b44be700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72ac1014b0 0x7f72ac1965e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f72a800d950 tx=0x7f72a800dc60 comp rx=0 tx=0).stop 2023-11-16T18:23:07.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.911+0000 7f72b44be700 1 -- 172.21.15.2:0/3466136351 shutdown_connections 2023-11-16T18:23:07.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.911+0000 7f72b44be700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f72980a1150 0x7f72980a3600 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:07.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.911+0000 7f72b44be700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72ac1002c0 0x7f72ac1960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:07.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.911+0000 7f72b44be700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72ac101e40 0x7f72ac19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:07.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.911+0000 7f72b44be700 1 --2- 172.21.15.2:0/3466136351 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72ac1014b0 0x7f72ac1965e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:07.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.911+0000 7f72b44be700 1 -- 172.21.15.2:0/3466136351 >> 172.21.15.2:0/3466136351 conn(0x7f72ac0fb8e0 msgr2=0x7f72ac0fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:07.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.912+0000 7f72b44be700 1 -- 172.21.15.2:0/3466136351 shutdown_connections 2023-11-16T18:23:07.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:07.912+0000 7f72b44be700 1 -- 172.21.15.2:0/3466136351 wait complete. 2023-11-16T18:23:07.930 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2023-11-16T18:23:07.930 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-11-16T18:23:08.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.066+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/272356808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6e41036b0 msgr2=0x7fa6e4105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:08.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.066+0000 7fa6e9dc6700 1 --2- 172.21.15.2:0/272356808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6e41036b0 0x7fa6e4105a90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009990 tx=0x7fa6d4009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.066+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/272356808 shutdown_connections 2023-11-16T18:23:08.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.066+0000 7fa6e9dc6700 1 --2- 172.21.15.2:0/272356808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6e4100d90 0x7fa6e4103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.066+0000 7fa6e9dc6700 1 --2- 172.21.15.2:0/272356808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6e4105fd0 0x7fa6e41083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.066+0000 7fa6e9dc6700 1 --2- 172.21.15.2:0/272356808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6e41036b0 0x7fa6e4105a90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.066+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/272356808 >> 172.21.15.2:0/272356808 conn(0x7fa6e40fa750 msgr2=0x7fa6e40fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:08.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.066+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/272356808 shutdown_connections 2023-11-16T18:23:08.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.066+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/272356808 wait complete. 2023-11-16T18:23:08.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e9dc6700 1 Processor -- start 2023-11-16T18:23:08.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e9dc6700 1 -- start start 2023-11-16T18:23:08.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e9dc6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6e4100d90 0x7fa6e4196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:08.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e9dc6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6e41036b0 0x7fa6e4196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:08.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e9dc6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6e4105fd0 0x7fa6e419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:08.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e9dc6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e419dbd0 con 0x7fa6e4100d90 2023-11-16T18:23:08.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e9dc6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e419dd10 con 0x7fa6e41036b0 2023-11-16T18:23:08.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e9dc6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e419dea0 con 0x7fa6e4105fd0 2023-11-16T18:23:08.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6e4100d90 0x7fa6e4196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:08.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6e4100d90 0x7fa6e4196050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48268/0 (socket says 172.21.15.2:48268) 2023-11-16T18:23:08.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e37fe700 1 -- 172.21.15.2:0/3996218424 learned_addr learned my addr 172.21.15.2:0/3996218424 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:08.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e3fff700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6e4105fd0 0x7fa6e419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:08.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e2ffd700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6e41036b0 0x7fa6e4196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:08.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e37fe700 1 -- 172.21.15.2:0/3996218424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6e4105fd0 msgr2=0x7fa6e419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:08.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e37fe700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6e4105fd0 0x7fa6e419d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e37fe700 1 -- 172.21.15.2:0/3996218424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6e41036b0 msgr2=0x7fa6e4196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:08.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e37fe700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6e41036b0 0x7fa6e4196590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.067+0000 7fa6e37fe700 1 -- 172.21.15.2:0/3996218424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d40095f0 con 0x7fa6e4100d90 2023-11-16T18:23:08.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.068+0000 7fa6e37fe700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6e4100d90 0x7fa6e4196050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800d980 tx=0x7fa6d800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:08.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.068+0000 7fa6e0ff9700 1 -- 172.21.15.2:0/3996218424 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8009b40 con 0x7fa6e4100d90 2023-11-16T18:23:08.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.068+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/3996218424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e410bc50 con 0x7fa6e4100d90 2023-11-16T18:23:08.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.068+0000 7fa6e0ff9700 1 -- 172.21.15.2:0/3996218424 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6d800de60 con 0x7fa6e4100d90 2023-11-16T18:23:08.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.068+0000 7fa6e0ff9700 1 -- 172.21.15.2:0/3996218424 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d800f620 con 0x7fa6e4100d90 2023-11-16T18:23:08.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.068+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/3996218424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e410bac0 con 0x7fa6e4100d90 2023-11-16T18:23:08.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.069+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/3996218424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e40fc2c0 con 0x7fa6e4100d90 2023-11-16T18:23:08.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.070+0000 7fa6e0ff9700 1 -- 172.21.15.2:0/3996218424 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fa6d8010460 con 0x7fa6e4100d90 2023-11-16T18:23:08.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.071+0000 7fa6e0ff9700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa6cc0a11d0 0x7fa6cc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:08.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.071+0000 7fa6e2ffd700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa6cc0a11d0 0x7fa6cc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:08.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.071+0000 7fa6e0ff9700 1 -- 172.21.15.2:0/3996218424 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa6d80cc670 con 0x7fa6e4100d90 2023-11-16T18:23:08.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.072+0000 7fa6e2ffd700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa6cc0a11d0 0x7fa6cc0a3680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4000c00 tx=0x7fa6d4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:08.076 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.075+0000 7fa6e0ff9700 1 -- 172.21.15.2:0/3996218424 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa6d8095c50 con 0x7fa6e4100d90 2023-11-16T18:23:08.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:07 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] up:active 2023-11-16T18:23:08.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:07 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/524617146' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-11-16T18:23:08.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:07 smithi040 ceph-mon[100251]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-11-16T18:23:08.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:07 smithi040 ceph-mon[100251]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-11-16T18:23:08.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:07 smithi040 ceph-mon[100251]: Cluster is now healthy 2023-11-16T18:23:08.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:07 smithi040 ceph-mon[100251]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-11-16T18:23:08.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:07 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-11-16T18:23:08.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:07 smithi040 ceph-mon[100251]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-11-16T18:23:08.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:07 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3466136351' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-11-16T18:23:08.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:07 smithi040 ceph-mon[100251]: pgmap v271: 129 pgs: 129 active+clean; 582 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 1 op/s 2023-11-16T18:23:08.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:07 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] up:active 2023-11-16T18:23:08.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:07 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/524617146' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-11-16T18:23:08.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:07 smithi188 ceph-mon[100639]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-11-16T18:23:08.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:07 smithi188 ceph-mon[100639]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-11-16T18:23:08.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:07 smithi188 ceph-mon[100639]: Cluster is now healthy 2023-11-16T18:23:08.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:07 smithi188 ceph-mon[100639]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-11-16T18:23:08.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:07 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-11-16T18:23:08.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:07 smithi188 ceph-mon[100639]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-11-16T18:23:08.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:07 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3466136351' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-11-16T18:23:08.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:07 smithi188 ceph-mon[100639]: pgmap v271: 129 pgs: 129 active+clean; 582 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 1 op/s 2023-11-16T18:23:08.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.306+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/3996218424 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6e4061c10 con 0x7fa6cc0a11d0 2023-11-16T18:23:08.315 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:07 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] up:active 2023-11-16T18:23:08.316 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:07 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/524617146' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-11-16T18:23:08.316 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:07 smithi002 ceph-mon[98807]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-11-16T18:23:08.316 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:07 smithi002 ceph-mon[98807]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-11-16T18:23:08.316 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:07 smithi002 ceph-mon[98807]: Cluster is now healthy 2023-11-16T18:23:08.316 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:07 smithi002 ceph-mon[98807]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-11-16T18:23:08.317 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:07 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-11-16T18:23:08.317 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:07 smithi002 ceph-mon[98807]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-11-16T18:23:08.317 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:07 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3466136351' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-11-16T18:23:08.317 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:07 smithi002 ceph-mon[98807]: pgmap v271: 129 pgs: 129 active+clean; 582 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 1 op/s 2023-11-16T18:23:08.318 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:23:08 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:23:08.312+0000 7f44d68d7700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:23:08.318 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:23:08 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:23:08.312+0000 7f44d68d7700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:23:08.318 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:23:08 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:23:08.312+0000 7f44d68d7700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:23:08.318 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:23:08 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:23:08.312+0000 7f44d68d7700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:23:08.318 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:23:08 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:23:08.312+0000 7f44d68d7700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:23:08.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.358+0000 7fa6e0ff9700 1 -- 172.21.15.2:0/3996218424 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa6e4061c10 con 0x7fa6cc0a11d0 2023-11-16T18:23:08.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.361+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/3996218424 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa6cc0a11d0 msgr2=0x7fa6cc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:08.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.361+0000 7fa6e9dc6700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa6cc0a11d0 0x7fa6cc0a3680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4000c00 tx=0x7fa6d4005fb0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.361+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/3996218424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6e4100d90 msgr2=0x7fa6e4196050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:08.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.361+0000 7fa6e9dc6700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6e4100d90 0x7fa6e4196050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800d980 tx=0x7fa6d800dd40 comp rx=0 tx=0).stop 2023-11-16T18:23:08.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.361+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/3996218424 shutdown_connections 2023-11-16T18:23:08.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.361+0000 7fa6e9dc6700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa6cc0a11d0 0x7fa6cc0a3680 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.361+0000 7fa6e9dc6700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6e4105fd0 0x7fa6e419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.361+0000 7fa6e9dc6700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6e4100d90 0x7fa6e4196050 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.361+0000 7fa6e9dc6700 1 --2- 172.21.15.2:0/3996218424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6e41036b0 0x7fa6e4196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.361+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/3996218424 >> 172.21.15.2:0/3996218424 conn(0x7fa6e40fa750 msgr2=0x7fa6e4101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:08.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.361+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/3996218424 shutdown_connections 2023-11-16T18:23:08.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.361+0000 7fa6e9dc6700 1 -- 172.21.15.2:0/3996218424 wait complete. 2023-11-16T18:23:08.376 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-11-16T18:23:08.507 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.503+0000 7f9ecc280700 1 -- 172.21.15.2:0/238713319 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ec4100d30 msgr2=0x7f9ec41075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:08.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.503+0000 7f9ecc280700 1 --2- 172.21.15.2:0/238713319 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ec4100d30 0x7f9ec41075f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc00b0a0 tx=0x7f9ebc00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.503+0000 7f9ecc280700 1 -- 172.21.15.2:0/238713319 shutdown_connections 2023-11-16T18:23:08.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.503+0000 7f9ecc280700 1 --2- 172.21.15.2:0/238713319 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ec4100d30 0x7f9ec41075f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.503+0000 7f9ecc280700 1 --2- 172.21.15.2:0/238713319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ec40ff1b0 0x7f9ec40ff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.503+0000 7f9ecc280700 1 --2- 172.21.15.2:0/238713319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec41003a0 0x7f9ec41007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.503+0000 7f9ecc280700 1 -- 172.21.15.2:0/238713319 >> 172.21.15.2:0/238713319 conn(0x7f9ec40fa7d0 msgr2=0x7f9ec40fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:08.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.503+0000 7f9ecc280700 1 -- 172.21.15.2:0/238713319 shutdown_connections 2023-11-16T18:23:08.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9ecc280700 1 -- 172.21.15.2:0/238713319 wait complete. 2023-11-16T18:23:08.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9ecc280700 1 Processor -- start 2023-11-16T18:23:08.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9ecc280700 1 -- start start 2023-11-16T18:23:08.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9ecc280700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ec40ff1b0 0x7f9ec4069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:08.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9ecc280700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec41003a0 0x7f9ec419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:08.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9ecc280700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ec4100d30 0x7f9ec419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:08.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9ecc280700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec419dc00 con 0x7f9ec40ff1b0 2023-11-16T18:23:08.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9ecc280700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec419dd70 con 0x7f9ec4100d30 2023-11-16T18:23:08.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9ecc280700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec419df30 con 0x7f9ec41003a0 2023-11-16T18:23:08.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9eca01c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ec40ff1b0 0x7f9ec4069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:08.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9eca01c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ec40ff1b0 0x7f9ec4069bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48302/0 (socket says 172.21.15.2:48302) 2023-11-16T18:23:08.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9eca01c700 1 -- 172.21.15.2:0/1153658579 learned_addr learned my addr 172.21.15.2:0/1153658579 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:08.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9eca81d700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ec4100d30 0x7f9ec419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:08.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.504+0000 7f9ec981b700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec41003a0 0x7f9ec419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:08.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.505+0000 7f9eca01c700 1 -- 172.21.15.2:0/1153658579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec41003a0 msgr2=0x7f9ec419ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:08.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.505+0000 7f9eca01c700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec41003a0 0x7f9ec419ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.505+0000 7f9eca01c700 1 -- 172.21.15.2:0/1153658579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ec4100d30 msgr2=0x7f9ec419d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:08.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.505+0000 7f9eca01c700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ec4100d30 0x7f9ec419d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.505+0000 7f9eca01c700 1 -- 172.21.15.2:0/1153658579 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc009d00 con 0x7f9ec40ff1b0 2023-11-16T18:23:08.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.505+0000 7f9eca81d700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ec4100d30 0x7f9ec419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:08.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.505+0000 7f9eca01c700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ec40ff1b0 0x7f9ec4069bc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9ec000d980 tx=0x7f9ec000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:08.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.505+0000 7f9eb77fe700 1 -- 172.21.15.2:0/1153658579 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec000de60 con 0x7f9ec40ff1b0 2023-11-16T18:23:08.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.505+0000 7f9ecc280700 1 -- 172.21.15.2:0/1153658579 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec410ab70 con 0x7f9ec40ff1b0 2023-11-16T18:23:08.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.505+0000 7f9eb77fe700 1 -- 172.21.15.2:0/1153658579 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ec00044e0 con 0x7f9ec40ff1b0 2023-11-16T18:23:08.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.505+0000 7f9eb77fe700 1 -- 172.21.15.2:0/1153658579 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec0003be0 con 0x7f9ec40ff1b0 2023-11-16T18:23:08.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.505+0000 7f9ecc280700 1 -- 172.21.15.2:0/1153658579 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec410a9e0 con 0x7f9ec40ff1b0 2023-11-16T18:23:08.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.506+0000 7f9ecc280700 1 -- 172.21.15.2:0/1153658579 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec41066a0 con 0x7f9ec40ff1b0 2023-11-16T18:23:08.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.507+0000 7f9eb77fe700 1 -- 172.21.15.2:0/1153658579 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f9ec0009bb0 con 0x7f9ec40ff1b0 2023-11-16T18:23:08.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.508+0000 7f9eb77fe700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9eb00a1150 0x7f9eb00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:08.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.508+0000 7f9eb77fe700 1 -- 172.21.15.2:0/1153658579 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9ec00ccc40 con 0x7f9ec40ff1b0 2023-11-16T18:23:08.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.508+0000 7f9ec981b700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9eb00a1150 0x7f9eb00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:08.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.508+0000 7f9ec981b700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9eb00a1150 0x7f9eb00a3600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8005950 tx=0x7f9eb80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:08.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.512+0000 7f9eb77fe700 1 -- 172.21.15.2:0/1153658579 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9ec00961a0 con 0x7f9ec40ff1b0 2023-11-16T18:23:08.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.746+0000 7f9ecc280700 1 -- 172.21.15.2:0/1153658579 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ec4061c10 con 0x7f9eb00a1150 2023-11-16T18:23:08.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.751+0000 7f9eb77fe700 1 -- 172.21.15.2:0/1153658579 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f9ec4061c10 con 0x7f9eb00a1150 2023-11-16T18:23:08.766 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv_0/500b6a74-bb83-4116-a487-1eaa50c52876 2023-11-16T18:23:08.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.753+0000 7f9ecc280700 1 -- 172.21.15.2:0/1153658579 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9eb00a1150 msgr2=0x7f9eb00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:08.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.753+0000 7f9ecc280700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9eb00a1150 0x7f9eb00a3600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8005950 tx=0x7f9eb80058e0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.753+0000 7f9ecc280700 1 -- 172.21.15.2:0/1153658579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ec40ff1b0 msgr2=0x7f9ec4069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:08.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.753+0000 7f9ecc280700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ec40ff1b0 0x7f9ec4069bc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9ec000d980 tx=0x7f9ec000dd40 comp rx=0 tx=0).stop 2023-11-16T18:23:08.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.753+0000 7f9ecc280700 1 -- 172.21.15.2:0/1153658579 shutdown_connections 2023-11-16T18:23:08.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.753+0000 7f9ecc280700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9eb00a1150 0x7f9eb00a3600 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.753+0000 7f9ecc280700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ec4100d30 0x7f9ec419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.753+0000 7f9ecc280700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ec40ff1b0 0x7f9ec4069bc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.769 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.753+0000 7f9ecc280700 1 --2- 172.21.15.2:0/1153658579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ec41003a0 0x7f9ec419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.769 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.753+0000 7f9ecc280700 1 -- 172.21.15.2:0/1153658579 >> 172.21.15.2:0/1153658579 conn(0x7f9ec40fa7d0 msgr2=0x7f9ec41037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:08.769 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.753+0000 7f9ecc280700 1 -- 172.21.15.2:0/1153658579 shutdown_connections 2023-11-16T18:23:08.769 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.753+0000 7f9ecc280700 1 -- 172.21.15.2:0/1153658579 wait complete. 2023-11-16T18:23:08.769 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-11-16T18:23:08.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.897+0000 7fe9b13a2700 1 -- 172.21.15.2:0/86478088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ac0734d0 msgr2=0x7fe9ac073940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:08.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.897+0000 7fe9b13a2700 1 --2- 172.21.15.2:0/86478088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ac0734d0 0x7fe9ac073940 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0009990 tx=0x7fe9a0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.897+0000 7fe9b13a2700 1 -- 172.21.15.2:0/86478088 shutdown_connections 2023-11-16T18:23:08.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.897+0000 7fe9b13a2700 1 --2- 172.21.15.2:0/86478088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9ac107f10 0x7fe9ac10a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.897+0000 7fe9b13a2700 1 --2- 172.21.15.2:0/86478088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9ac074ab0 0x7fe9ac072f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.897+0000 7fe9b13a2700 1 --2- 172.21.15.2:0/86478088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ac0734d0 0x7fe9ac073940 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.897+0000 7fe9b13a2700 1 -- 172.21.15.2:0/86478088 >> 172.21.15.2:0/86478088 conn(0x7fe9ac0fb6a0 msgr2=0x7fe9ac0fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:08.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.897+0000 7fe9b13a2700 1 -- 172.21.15.2:0/86478088 shutdown_connections 2023-11-16T18:23:08.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.897+0000 7fe9b13a2700 1 -- 172.21.15.2:0/86478088 wait complete. 2023-11-16T18:23:08.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.897+0000 7fe9b13a2700 1 Processor -- start 2023-11-16T18:23:08.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.897+0000 7fe9b13a2700 1 -- start start 2023-11-16T18:23:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9b13a2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9ac0734d0 0x7fe9ac198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9b13a2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ac074ab0 0x7fe9ac198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9b13a2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9ac107f10 0x7fe9ac19f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9b13a2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac19fb80 con 0x7fe9ac074ab0 2023-11-16T18:23:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9b13a2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac19fcc0 con 0x7fe9ac107f10 2023-11-16T18:23:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9b13a2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac19fe50 con 0x7fe9ac0734d0 2023-11-16T18:23:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9ab7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ac074ab0 0x7fe9ac198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9ab7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ac074ab0 0x7fe9ac198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48318/0 (socket says 172.21.15.2:48318) 2023-11-16T18:23:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9ab7fe700 1 -- 172.21.15.2:0/484685938 learned_addr learned my addr 172.21.15.2:0/484685938 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9abfff700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9ac0734d0 0x7fe9ac198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9b0ba1700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9ac107f10 0x7fe9ac19f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9ab7fe700 1 -- 172.21.15.2:0/484685938 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9ac0734d0 msgr2=0x7fe9ac198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9ab7fe700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9ac0734d0 0x7fe9ac198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9ab7fe700 1 -- 172.21.15.2:0/484685938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9ac107f10 msgr2=0x7fe9ac19f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:08.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9ab7fe700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9ac107f10 0x7fe9ac19f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:08.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9ab7fe700 1 -- 172.21.15.2:0/484685938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a00095f0 con 0x7fe9ac074ab0 2023-11-16T18:23:08.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9b0ba1700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9ac107f10 0x7fe9ac19f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:08.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9ab7fe700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ac074ab0 0x7fe9ac198540 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0005910 tx=0x7fe9a000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:08.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.898+0000 7fe9a97fa700 1 -- 172.21.15.2:0/484685938 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a001d070 con 0x7fe9ac074ab0 2023-11-16T18:23:08.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.899+0000 7fe9a97fa700 1 -- 172.21.15.2:0/484685938 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9a000fb80 con 0x7fe9ac074ab0 2023-11-16T18:23:08.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.899+0000 7fe9a97fa700 1 -- 172.21.15.2:0/484685938 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0017650 con 0x7fe9ac074ab0 2023-11-16T18:23:08.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.899+0000 7fe9b13a2700 1 -- 172.21.15.2:0/484685938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ac10dc10 con 0x7fe9ac074ab0 2023-11-16T18:23:08.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.899+0000 7fe9b13a2700 1 -- 172.21.15.2:0/484685938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ac10da80 con 0x7fe9ac074ab0 2023-11-16T18:23:08.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.900+0000 7fe9b13a2700 1 -- 172.21.15.2:0/484685938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ac106650 con 0x7fe9ac074ab0 2023-11-16T18:23:08.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.900+0000 7fe9a97fa700 1 -- 172.21.15.2:0/484685938 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fe9a0022470 con 0x7fe9ac074ab0 2023-11-16T18:23:08.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.901+0000 7fe9a97fa700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe9940a1150 0x7fe9940a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:08.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.901+0000 7fe9a97fa700 1 -- 172.21.15.2:0/484685938 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe9a00ce0a0 con 0x7fe9ac074ab0 2023-11-16T18:23:08.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.901+0000 7fe9abfff700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe9940a1150 0x7fe9940a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:08.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.902+0000 7fe9abfff700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe9940a1150 0x7fe9940a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe99c005950 tx=0x7fe99c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:08.905+0000 7fe9a97fa700 1 -- 172.21.15.2:0/484685938 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe9a0097690 con 0x7fe9ac074ab0 2023-11-16T18:23:09.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:08 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] up:active 2023-11-16T18:23:09.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:08 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3466136351' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-11-16T18:23:09.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:08 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-11-16T18:23:09.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:08 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:23:09.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:08 smithi002 ceph-mon[98807]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:23:09.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:08 smithi002 ceph-mon[98807]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:23:09.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.138+0000 7fe9b13a2700 1 -- 172.21.15.2:0/484685938 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9ac061bf0 con 0x7fe9940a1150 2023-11-16T18:23:09.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.168+0000 7fe9a97fa700 1 -- 172.21.15.2:0/484685938 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe9ac061bf0 con 0x7fe9940a1150 2023-11-16T18:23:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.171+0000 7fe9b13a2700 1 -- 172.21.15.2:0/484685938 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe9940a1150 msgr2=0x7fe9940a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.171+0000 7fe9b13a2700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe9940a1150 0x7fe9940a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe99c005950 tx=0x7fe99c00b410 comp rx=0 tx=0).stop 2023-11-16T18:23:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.171+0000 7fe9b13a2700 1 -- 172.21.15.2:0/484685938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ac074ab0 msgr2=0x7fe9ac198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.171+0000 7fe9b13a2700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ac074ab0 0x7fe9ac198540 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0005910 tx=0x7fe9a000f690 comp rx=0 tx=0).stop 2023-11-16T18:23:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.171+0000 7fe9b13a2700 1 -- 172.21.15.2:0/484685938 shutdown_connections 2023-11-16T18:23:09.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.171+0000 7fe9b13a2700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe9940a1150 0x7fe9940a3600 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.171+0000 7fe9b13a2700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9ac0734d0 0x7fe9ac198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.171+0000 7fe9b13a2700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9ac107f10 0x7fe9ac19f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.171+0000 7fe9b13a2700 1 --2- 172.21.15.2:0/484685938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ac074ab0 0x7fe9ac198540 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.171+0000 7fe9b13a2700 1 -- 172.21.15.2:0/484685938 >> 172.21.15.2:0/484685938 conn(0x7fe9ac0fb6a0 msgr2=0x7fe9ac0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:09.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.171+0000 7fe9b13a2700 1 -- 172.21.15.2:0/484685938 shutdown_connections 2023-11-16T18:23:09.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.171+0000 7fe9b13a2700 1 -- 172.21.15.2:0/484685938 wait complete. 2023-11-16T18:23:09.185 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-11-16T18:23:09.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:08 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] up:active 2023-11-16T18:23:09.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:08 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3466136351' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-11-16T18:23:09.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:08 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-11-16T18:23:09.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:08 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:23:09.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:08 smithi040 ceph-mon[100251]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:23:09.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:08 smithi040 ceph-mon[100251]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:23:09.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:08 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] up:active 2023-11-16T18:23:09.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:08 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3466136351' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-11-16T18:23:09.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:08 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-11-16T18:23:09.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:08 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:23:09.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:08 smithi188 ceph-mon[100639]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:23:09.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:08 smithi188 ceph-mon[100639]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:23:09.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.312+0000 7f6962dd5700 1 -- 172.21.15.2:0/777603588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f695c100d20 msgr2=0x7f695c1075e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:09.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.312+0000 7f6962dd5700 1 --2- 172.21.15.2:0/777603588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f695c100d20 0x7f695c1075e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f695400b0a0 tx=0x7f695400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.312+0000 7f6962dd5700 1 -- 172.21.15.2:0/777603588 shutdown_connections 2023-11-16T18:23:09.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.312+0000 7f6962dd5700 1 --2- 172.21.15.2:0/777603588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f695c100d20 0x7f695c1075e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.312+0000 7f6962dd5700 1 --2- 172.21.15.2:0/777603588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f695c0ff1a0 0x7f695c0ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.312+0000 7f6962dd5700 1 --2- 172.21.15.2:0/777603588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f695c100390 0x7f695c1007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.312+0000 7f6962dd5700 1 -- 172.21.15.2:0/777603588 >> 172.21.15.2:0/777603588 conn(0x7f695c0fa7a0 msgr2=0x7f695c0fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:09.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.312+0000 7f6962dd5700 1 -- 172.21.15.2:0/777603588 shutdown_connections 2023-11-16T18:23:09.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.312+0000 7f6962dd5700 1 -- 172.21.15.2:0/777603588 wait complete. 2023-11-16T18:23:09.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.312+0000 7f6962dd5700 1 Processor -- start 2023-11-16T18:23:09.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.312+0000 7f6962dd5700 1 -- start start 2023-11-16T18:23:09.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f6962dd5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f695c0ff1a0 0x7f695c1982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:09.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f6962dd5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f695c100390 0x7f695c1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:09.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f6962dd5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f695c100d20 0x7f695c19f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:09.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f6962dd5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c19fe20 con 0x7f695c100390 2023-11-16T18:23:09.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f6962dd5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c19ff60 con 0x7f695c0ff1a0 2023-11-16T18:23:09.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f6962dd5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c1a00f0 con 0x7f695c100d20 2023-11-16T18:23:09.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f695bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f695c100390 0x7f695c1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:09.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f695bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f695c100390 0x7f695c1987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48334/0 (socket says 172.21.15.2:48334) 2023-11-16T18:23:09.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f695bfff700 1 -- 172.21.15.2:0/1826018175 learned_addr learned my addr 172.21.15.2:0/1826018175 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:09.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f6960b71700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f695c0ff1a0 0x7f695c1982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:09.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f6961372700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f695c100d20 0x7f695c19f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:09.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f695bfff700 1 -- 172.21.15.2:0/1826018175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f695c100d20 msgr2=0x7f695c19f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:09.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f695bfff700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f695c100d20 0x7f695c19f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.319 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f695bfff700 1 -- 172.21.15.2:0/1826018175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f695c0ff1a0 msgr2=0x7f695c1982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:09.319 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f695bfff700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f695c0ff1a0 0x7f695c1982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.319 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f695bfff700 1 -- 172.21.15.2:0/1826018175 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6954009d00 con 0x7f695c100390 2023-11-16T18:23:09.319 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f695bfff700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f695c100390 0x7f695c1987e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f694800e9a0 tx=0x7f694800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:09.319 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f6959ffb700 1 -- 172.21.15.2:0/1826018175 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948009790 con 0x7f695c100390 2023-11-16T18:23:09.319 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f6959ffb700 1 -- 172.21.15.2:0/1826018175 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f694800cbd0 con 0x7f695c100390 2023-11-16T18:23:09.319 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f6962dd5700 1 -- 172.21.15.2:0/1826018175 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c10ab60 con 0x7f695c100390 2023-11-16T18:23:09.319 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.313+0000 7f6959ffb700 1 -- 172.21.15.2:0/1826018175 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948018680 con 0x7f695c100390 2023-11-16T18:23:09.319 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.314+0000 7f6962dd5700 1 -- 172.21.15.2:0/1826018175 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c10a9d0 con 0x7f695c100390 2023-11-16T18:23:09.319 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.315+0000 7f6962dd5700 1 -- 172.21.15.2:0/1826018175 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695c1a0830 con 0x7f695c100390 2023-11-16T18:23:09.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.316+0000 7f6959ffb700 1 -- 172.21.15.2:0/1826018175 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f694800cd40 con 0x7f695c100390 2023-11-16T18:23:09.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.316+0000 7f6959ffb700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f694c0a11d0 0x7f694c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:09.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.317+0000 7f6960b71700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f694c0a11d0 0x7f694c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:09.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.317+0000 7f6959ffb700 1 -- 172.21.15.2:0/1826018175 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6948014070 con 0x7f695c100390 2023-11-16T18:23:09.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.317+0000 7f6960b71700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f694c0a11d0 0x7f694c0a3680 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f695c1001f0 tx=0x7f695000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:09.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.320+0000 7f6959ffb700 1 -- 172.21.15.2:0/1826018175 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f69480964f0 con 0x7f695c100390 2023-11-16T18:23:09.555 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.553+0000 7f6962dd5700 1 -- 172.21.15.2:0/1826018175 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f695c0fb690 con 0x7f694c0a11d0 2023-11-16T18:23:09.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.557+0000 7f6959ffb700 1 -- 172.21.15.2:0/1826018175 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f695c0fb690 con 0x7f694c0a11d0 2023-11-16T18:23:09.559 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e 2023-11-16T18:23:09.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.559+0000 7f6962dd5700 1 -- 172.21.15.2:0/1826018175 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f694c0a11d0 msgr2=0x7f694c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:09.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.559+0000 7f6962dd5700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f694c0a11d0 0x7f694c0a3680 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f695c1001f0 tx=0x7f695000b410 comp rx=0 tx=0).stop 2023-11-16T18:23:09.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.559+0000 7f6962dd5700 1 -- 172.21.15.2:0/1826018175 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f695c100390 msgr2=0x7f695c1987e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.559+0000 7f6962dd5700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f695c100390 0x7f695c1987e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f694800e9a0 tx=0x7f694800ecb0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.559+0000 7f6962dd5700 1 -- 172.21.15.2:0/1826018175 shutdown_connections 2023-11-16T18:23:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.559+0000 7f6962dd5700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f694c0a11d0 0x7f694c0a3680 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.559+0000 7f6962dd5700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f695c100d20 0x7f695c19f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.559+0000 7f6962dd5700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f695c0ff1a0 0x7f695c1982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.559+0000 7f6962dd5700 1 --2- 172.21.15.2:0/1826018175 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f695c100390 0x7f695c1987e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.559+0000 7f6962dd5700 1 -- 172.21.15.2:0/1826018175 >> 172.21.15.2:0/1826018175 conn(0x7f695c0fa7a0 msgr2=0x7f695c0faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.559+0000 7f6962dd5700 1 -- 172.21.15.2:0/1826018175 shutdown_connections 2023-11-16T18:23:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.559+0000 7f6962dd5700 1 -- 172.21.15.2:0/1826018175 wait complete. 2023-11-16T18:23:09.574 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:23:09.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.697+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1411233252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd34100850 msgr2=0x7fdd34100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:09.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.697+0000 7fdd3c7f4700 1 --2- 172.21.15.2:0/1411233252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd34100850 0x7fdd34100c60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009990 tx=0x7fdd28009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.697+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1411233252 shutdown_connections 2023-11-16T18:23:09.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.697+0000 7fdd3c7f4700 1 --2- 172.21.15.2:0/1411233252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd340ffdc0 0x7fdd34107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.697+0000 7fdd3c7f4700 1 --2- 172.21.15.2:0/1411233252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd34100850 0x7fdd34100c60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.697+0000 7fdd3c7f4700 1 --2- 172.21.15.2:0/1411233252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd340ff410 0x7fdd340ff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.697+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1411233252 >> 172.21.15.2:0/1411233252 conn(0x7fdd340f9f50 msgr2=0x7fdd340fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:09.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.697+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1411233252 shutdown_connections 2023-11-16T18:23:09.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.697+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1411233252 wait complete. 2023-11-16T18:23:09.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3c7f4700 1 Processor -- start 2023-11-16T18:23:09.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3c7f4700 1 -- start start 2023-11-16T18:23:09.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3c7f4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd340ff410 0x7fdd341031c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:09.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3c7f4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd340ffdc0 0x7fdd34101800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:09.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3c7f4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd34100850 0x7fdd34101d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:09.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3c7f4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd341024e0 con 0x7fdd34100850 2023-11-16T18:23:09.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3c7f4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd34102650 con 0x7fdd340ff410 2023-11-16T18:23:09.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3c7f4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd341a0100 con 0x7fdd340ffdc0 2023-11-16T18:23:09.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3ad91700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd34100850 0x7fdd34101d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:09.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3ad91700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd34100850 0x7fdd34101d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48348/0 (socket says 172.21.15.2:48348) 2023-11-16T18:23:09.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3ad91700 1 -- 172.21.15.2:0/1188611400 learned_addr learned my addr 172.21.15.2:0/1188611400 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:09.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd39d8f700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd340ffdc0 0x7fdd34101800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:09.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3a590700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd340ff410 0x7fdd341031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:09.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3ad91700 1 -- 172.21.15.2:0/1188611400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd340ffdc0 msgr2=0x7fdd34101800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:09.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3ad91700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd340ffdc0 0x7fdd34101800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3ad91700 1 -- 172.21.15.2:0/1188611400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd340ff410 msgr2=0x7fdd341031c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:09.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3ad91700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd340ff410 0x7fdd341031c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.698+0000 7fdd3ad91700 1 -- 172.21.15.2:0/1188611400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd280095f0 con 0x7fdd34100850 2023-11-16T18:23:09.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.699+0000 7fdd3ad91700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd34100850 0x7fdd34101d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00ada0 tx=0x7fdd2c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:09.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.699+0000 7fdd277fe700 1 -- 172.21.15.2:0/1188611400 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c00f070 con 0x7fdd34100850 2023-11-16T18:23:09.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.699+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1188611400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3410abc0 con 0x7fdd34100850 2023-11-16T18:23:09.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.699+0000 7fdd277fe700 1 -- 172.21.15.2:0/1188611400 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd2c007b30 con 0x7fdd34100850 2023-11-16T18:23:09.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.699+0000 7fdd277fe700 1 -- 172.21.15.2:0/1188611400 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c005280 con 0x7fdd34100850 2023-11-16T18:23:09.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.699+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1188611400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3410aa30 con 0x7fdd34100850 2023-11-16T18:23:09.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.700+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1188611400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd34067260 con 0x7fdd34100850 2023-11-16T18:23:09.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.701+0000 7fdd277fe700 1 -- 172.21.15.2:0/1188611400 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fdd2c00c8d0 con 0x7fdd34100850 2023-11-16T18:23:09.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.702+0000 7fdd277fe700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdd200a1150 0x7fdd200a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:09.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.702+0000 7fdd3a590700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdd200a1150 0x7fdd200a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:09.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.702+0000 7fdd277fe700 1 -- 172.21.15.2:0/1188611400 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdd2c0cc040 con 0x7fdd34100850 2023-11-16T18:23:09.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.702+0000 7fdd3a590700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdd200a1150 0x7fdd200a3600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800b5c0 tx=0x7fdd28005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:09.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.706+0000 7fdd277fe700 1 -- 172.21.15.2:0/1188611400 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdd2c0990c0 con 0x7fdd34100850 2023-11-16T18:23:09.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.952+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1188611400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd341a0bc0 con 0x7fdd34100850 2023-11-16T18:23:09.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.952+0000 7fdd277fe700 1 -- 172.21.15.2:0/1188611400 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10774 (secure 0 0 0) 0x7fdd2c098c50 con 0x7fdd34100850 2023-11-16T18:23:09.954 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:09.955 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:23:07.909835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:23:09.956 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.954+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1188611400 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdd200a1150 msgr2=0x7fdd200a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:09.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.954+0000 7fdd3c7f4700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdd200a1150 0x7fdd200a3600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800b5c0 tx=0x7fdd28005fb0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.954+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1188611400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd34100850 msgr2=0x7fdd34101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:09.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.954+0000 7fdd3c7f4700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd34100850 0x7fdd34101d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00ada0 tx=0x7fdd2c00c330 comp rx=0 tx=0).stop 2023-11-16T18:23:09.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.954+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1188611400 shutdown_connections 2023-11-16T18:23:09.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.954+0000 7fdd3c7f4700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fdd200a1150 0x7fdd200a3600 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.954+0000 7fdd3c7f4700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd34100850 0x7fdd34101d40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.954+0000 7fdd3c7f4700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd340ff410 0x7fdd341031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.954+0000 7fdd3c7f4700 1 --2- 172.21.15.2:0/1188611400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd340ffdc0 0x7fdd34101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:09.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.954+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1188611400 >> 172.21.15.2:0/1188611400 conn(0x7fdd340f9f50 msgr2=0x7fdd340fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:09.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.955+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1188611400 shutdown_connections 2023-11-16T18:23:09.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:09.955+0000 7fdd3c7f4700 1 -- 172.21.15.2:0/1188611400 wait complete. 2023-11-16T18:23:09.958 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T18:23:09.969 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:23:10.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.093+0000 7f94d55df700 1 -- 172.21.15.2:0/1842501768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d0100500 msgr2=0x7f94d0106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:10.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.093+0000 7f94d55df700 1 --2- 172.21.15.2:0/1842501768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d0100500 0x7f94d0106dc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f94c800b0a0 tx=0x7f94c800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.093+0000 7f94d55df700 1 -- 172.21.15.2:0/1842501768 shutdown_connections 2023-11-16T18:23:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.093+0000 7f94d55df700 1 --2- 172.21.15.2:0/1842501768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d0100500 0x7f94d0106dc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.093+0000 7f94d55df700 1 --2- 172.21.15.2:0/1842501768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d00fe980 0x7f94d00fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.093+0000 7f94d55df700 1 --2- 172.21.15.2:0/1842501768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94d00ffb70 0x7f94d00fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.093+0000 7f94d55df700 1 -- 172.21.15.2:0/1842501768 >> 172.21.15.2:0/1842501768 conn(0x7f94d00f9f80 msgr2=0x7f94d00fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.093+0000 7f94d55df700 1 -- 172.21.15.2:0/1842501768 shutdown_connections 2023-11-16T18:23:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.093+0000 7f94d55df700 1 -- 172.21.15.2:0/1842501768 wait complete. 2023-11-16T18:23:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.093+0000 7f94d55df700 1 Processor -- start 2023-11-16T18:23:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.093+0000 7f94d55df700 1 -- start start 2023-11-16T18:23:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94d55df700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94d00fe980 0x7f94d0108260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94d55df700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d00ffb70 0x7f94d019ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94d55df700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d0100500 0x7f94d019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94d55df700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d019dbf0 con 0x7f94d00ffb70 2023-11-16T18:23:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94d55df700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d019dd60 con 0x7f94d0100500 2023-11-16T18:23:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94d55df700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d019df20 con 0x7f94d00fe980 2023-11-16T18:23:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94ce7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d00ffb70 0x7f94d019ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94ce7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d00ffb70 0x7f94d019ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48360/0 (socket says 172.21.15.2:48360) 2023-11-16T18:23:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94ce7fc700 1 -- 172.21.15.2:0/1643514571 learned_addr learned my addr 172.21.15.2:0/1643514571 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94ceffd700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94d00fe980 0x7f94d0108260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94cf7fe700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d0100500 0x7f94d019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94ce7fc700 1 -- 172.21.15.2:0/1643514571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94d00fe980 msgr2=0x7f94d0108260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94ce7fc700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94d00fe980 0x7f94d0108260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94ce7fc700 1 -- 172.21.15.2:0/1643514571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d0100500 msgr2=0x7f94d019d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94ce7fc700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d0100500 0x7f94d019d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94ce7fc700 1 -- 172.21.15.2:0/1643514571 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c8009d00 con 0x7f94d00ffb70 2023-11-16T18:23:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94cf7fe700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d0100500 0x7f94d019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94ce7fc700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d00ffb70 0x7f94d019ae10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f94c400d950 tx=0x7f94c400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:10.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94b7fff700 1 -- 172.21.15.2:0/1643514571 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4009940 con 0x7f94d00ffb70 2023-11-16T18:23:10.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94b7fff700 1 -- 172.21.15.2:0/1643514571 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94c4010460 con 0x7f94d00ffb70 2023-11-16T18:23:10.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94d55df700 1 -- 172.21.15.2:0/1643514571 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d010ab70 con 0x7f94d00ffb70 2023-11-16T18:23:10.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94b7fff700 1 -- 172.21.15.2:0/1643514571 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c400f620 con 0x7f94d00ffb70 2023-11-16T18:23:10.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.094+0000 7f94d55df700 1 -- 172.21.15.2:0/1643514571 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d010a9e0 con 0x7f94d00ffb70 2023-11-16T18:23:10.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.095+0000 7f94d55df700 1 -- 172.21.15.2:0/1643514571 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d0105dd0 con 0x7f94d00ffb70 2023-11-16T18:23:10.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.097+0000 7f94b7fff700 1 -- 172.21.15.2:0/1643514571 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f94c40108d0 con 0x7f94d00ffb70 2023-11-16T18:23:10.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.097+0000 7f94b7fff700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f94b80a11d0 0x7f94b80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:10.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.097+0000 7f94b7fff700 1 -- 172.21.15.2:0/1643514571 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f94c40cc900 con 0x7f94d00ffb70 2023-11-16T18:23:10.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.097+0000 7f94ceffd700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f94b80a11d0 0x7f94b80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:10.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.098+0000 7f94ceffd700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f94b80a11d0 0x7f94b80a3680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f94d00ff9d0 tx=0x7f94c000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:10.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.101+0000 7f94b7fff700 1 -- 172.21.15.2:0/1643514571 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f94c4095e60 con 0x7f94d00ffb70 2023-11-16T18:23:10.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:09 smithi040 ceph-mon[100251]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:23:10.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:09 smithi040 ceph-mon[100251]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:23:10.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:09 smithi040 ceph-mon[100251]: pgmap v272: 129 pgs: 129 active+clean; 582 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.6 KiB/s wr, 3 op/s 2023-11-16T18:23:10.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:09 smithi188 ceph-mon[100639]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:23:10.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:09 smithi188 ceph-mon[100639]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:23:10.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:09 smithi188 ceph-mon[100639]: pgmap v272: 129 pgs: 129 active+clean; 582 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.6 KiB/s wr, 3 op/s 2023-11-16T18:23:10.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.302+0000 7f94d55df700 1 -- 172.21.15.2:0/1643514571 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94d002cfe0 con 0x7f94d00ffb70 2023-11-16T18:23:10.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.304+0000 7f94b7fff700 1 -- 172.21.15.2:0/1643514571 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23853 (secure 0 0 0) 0x7f94c4020090 con 0x7f94d00ffb70 2023-11-16T18:23:10.306 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:10.306 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":86,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:22:51.534683+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:23:10.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.305+0000 7f94d55df700 1 -- 172.21.15.2:0/1643514571 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f94b80a11d0 msgr2=0x7f94b80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:10.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.305+0000 7f94d55df700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f94b80a11d0 0x7f94b80a3680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f94d00ff9d0 tx=0x7f94c000a380 comp rx=0 tx=0).stop 2023-11-16T18:23:10.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.305+0000 7f94d55df700 1 -- 172.21.15.2:0/1643514571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d00ffb70 msgr2=0x7f94d019ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:10.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.305+0000 7f94d55df700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d00ffb70 0x7f94d019ae10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f94c400d950 tx=0x7f94c400dc60 comp rx=0 tx=0).stop 2023-11-16T18:23:10.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.306+0000 7f94d55df700 1 -- 172.21.15.2:0/1643514571 shutdown_connections 2023-11-16T18:23:10.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.306+0000 7f94d55df700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f94b80a11d0 0x7f94b80a3680 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.306+0000 7f94d55df700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94d00fe980 0x7f94d0108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.306+0000 7f94d55df700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d0100500 0x7f94d019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.306+0000 7f94d55df700 1 --2- 172.21.15.2:0/1643514571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d00ffb70 0x7f94d019ae10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.306+0000 7f94d55df700 1 -- 172.21.15.2:0/1643514571 >> 172.21.15.2:0/1643514571 conn(0x7f94d00f9f80 msgr2=0x7f94d0102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:10.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.306+0000 7f94d55df700 1 -- 172.21.15.2:0/1643514571 shutdown_connections 2023-11-16T18:23:10.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.306+0000 7f94d55df700 1 -- 172.21.15.2:0/1643514571 wait complete. 2023-11-16T18:23:10.319 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:09 smithi002 ceph-mon[98807]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:23:10.319 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:09 smithi002 ceph-mon[98807]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:23:10.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:09 smithi002 ceph-mon[98807]: pgmap v272: 129 pgs: 129 active+clean; 582 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.6 KiB/s wr, 3 op/s 2023-11-16T18:23:10.321 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-16T18:23:10.321 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T18:23:10.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.446+0000 7f056fe42700 1 -- 172.21.15.2:0/3673770986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05681004e0 msgr2=0x7f05681008f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:10.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.446+0000 7f056fe42700 1 --2- 172.21.15.2:0/3673770986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05681004e0 0x7f05681008f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f055c009990 tx=0x7f055c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.446+0000 7f056fe42700 1 -- 172.21.15.2:0/3673770986 shutdown_connections 2023-11-16T18:23:10.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.446+0000 7f056fe42700 1 --2- 172.21.15.2:0/3673770986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0568102060 0x7f0568108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.446+0000 7f056fe42700 1 --2- 172.21.15.2:0/3673770986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05681004e0 0x7f05681008f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.446+0000 7f056fe42700 1 --2- 172.21.15.2:0/3673770986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05681016d0 0x7f0568101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.446+0000 7f056fe42700 1 -- 172.21.15.2:0/3673770986 >> 172.21.15.2:0/3673770986 conn(0x7f05680fbac0 msgr2=0x7f05680fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:10.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.446+0000 7f056fe42700 1 -- 172.21.15.2:0/3673770986 shutdown_connections 2023-11-16T18:23:10.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.447+0000 7f056fe42700 1 -- 172.21.15.2:0/3673770986 wait complete. 2023-11-16T18:23:10.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.447+0000 7f056fe42700 1 Processor -- start 2023-11-16T18:23:10.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.447+0000 7f056fe42700 1 -- start start 2023-11-16T18:23:10.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.447+0000 7f056fe42700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05681004e0 0x7f0568198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:10.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.447+0000 7f056fe42700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05681016d0 0x7f0568198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:10.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.447+0000 7f056fe42700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0568102060 0x7f056819f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:10.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.447+0000 7f056fe42700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056819fec0 con 0x7f05681004e0 2023-11-16T18:23:10.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.447+0000 7f056fe42700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05681a0000 con 0x7f05681016d0 2023-11-16T18:23:10.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.447+0000 7f056fe42700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05681a0190 con 0x7f0568102060 2023-11-16T18:23:10.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.447+0000 7f056dbde700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05681004e0 0x7f0568198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:10.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.447+0000 7f056dbde700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05681004e0 0x7f0568198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48372/0 (socket says 172.21.15.2:48372) 2023-11-16T18:23:10.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.447+0000 7f056dbde700 1 -- 172.21.15.2:0/1950142394 learned_addr learned my addr 172.21.15.2:0/1950142394 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:10.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.447+0000 7f056d3dd700 1 --2- 172.21.15.2:0/1950142394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05681016d0 0x7f0568198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:10.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.448+0000 7f056dbde700 1 -- 172.21.15.2:0/1950142394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0568102060 msgr2=0x7f056819f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:10.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.448+0000 7f056dbde700 1 --2- 172.21.15.2:0/1950142394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0568102060 0x7f056819f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.448+0000 7f056dbde700 1 -- 172.21.15.2:0/1950142394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05681016d0 msgr2=0x7f0568198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:10.455 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.448+0000 7f056dbde700 1 --2- 172.21.15.2:0/1950142394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05681016d0 0x7f0568198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.455 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.448+0000 7f056dbde700 1 -- 172.21.15.2:0/1950142394 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055c0095f0 con 0x7f05681004e0 2023-11-16T18:23:10.455 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.448+0000 7f056dbde700 1 --2- 172.21.15.2:0/1950142394 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05681004e0 0x7f0568198340 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f055c0059d0 tx=0x7f055c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:10.455 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.448+0000 7f055affd700 1 -- 172.21.15.2:0/1950142394 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c01d070 con 0x7f05681004e0 2023-11-16T18:23:10.455 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.448+0000 7f055affd700 1 -- 172.21.15.2:0/1950142394 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f055c004020 con 0x7f05681004e0 2023-11-16T18:23:10.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.448+0000 7f056fe42700 1 -- 172.21.15.2:0/1950142394 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056810be40 con 0x7f05681004e0 2023-11-16T18:23:10.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.448+0000 7f055affd700 1 -- 172.21.15.2:0/1950142394 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c022d00 con 0x7f05681004e0 2023-11-16T18:23:10.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.448+0000 7f056fe42700 1 -- 172.21.15.2:0/1950142394 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056810bcb0 con 0x7f05681004e0 2023-11-16T18:23:10.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.449+0000 7f056fe42700 1 -- 172.21.15.2:0/1950142394 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056806a2f0 con 0x7f05681004e0 2023-11-16T18:23:10.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.451+0000 7f055affd700 1 -- 172.21.15.2:0/1950142394 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f055c022470 con 0x7f05681004e0 2023-11-16T18:23:10.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.451+0000 7f055affd700 1 --2- 172.21.15.2:0/1950142394 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f05540a11d0 0x7f05540a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:10.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.451+0000 7f055affd700 1 -- 172.21.15.2:0/1950142394 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f055c0ce5c0 con 0x7f05681004e0 2023-11-16T18:23:10.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.452+0000 7f056d3dd700 1 --2- 172.21.15.2:0/1950142394 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f05540a11d0 0x7f05540a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:10.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.452+0000 7f056d3dd700 1 --2- 172.21.15.2:0/1950142394 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f05540a11d0 0x7f05540a3680 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0564005950 tx=0x7f056400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:10.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.455+0000 7f055affd700 1 -- 172.21.15.2:0/1950142394 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f055c097bb0 con 0x7f05681004e0 2023-11-16T18:23:10.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.651+0000 7f056fe42700 1 -- 172.21.15.2:0/1950142394 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0568107f70 con 0x7f05540a11d0 2023-11-16T18:23:10.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.666+0000 7f055affd700 1 -- 172.21.15.2:0/1950142394 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422937 (secure 0 0 0) 0x7f0568107f70 con 0x7f05540a11d0 2023-11-16T18:23:10.678 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.670+0000 7f056fe42700 1 -- 172.21.15.2:0/1950142394 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f05540a11d0 msgr2=0x7f05540a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.670+0000 7f056fe42700 1 --2- 172.21.15.2:0/1950142394 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f05540a11d0 0x7f05540a3680 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0564005950 tx=0x7f056400b410 comp rx=0 tx=0).stop 2023-11-16T18:23:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.670+0000 7f056fe42700 1 -- 172.21.15.2:0/1950142394 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05681004e0 msgr2=0x7f0568198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.670+0000 7f056fe42700 1 --2- 172.21.15.2:0/1950142394 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05681004e0 0x7f0568198340 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f055c0059d0 tx=0x7f055c005740 comp rx=0 tx=0).stop 2023-11-16T18:23:10.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.671+0000 7f056fe42700 1 -- 172.21.15.2:0/1950142394 shutdown_connections 2023-11-16T18:23:10.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.671+0000 7f056fe42700 1 --2- 172.21.15.2:0/1950142394 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f05540a11d0 0x7f05540a3680 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.671+0000 7f056fe42700 1 --2- 172.21.15.2:0/1950142394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0568102060 0x7f056819f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.671+0000 7f056fe42700 1 --2- 172.21.15.2:0/1950142394 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05681004e0 0x7f0568198340 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.671+0000 7f056fe42700 1 --2- 172.21.15.2:0/1950142394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05681016d0 0x7f0568198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.671+0000 7f056fe42700 1 -- 172.21.15.2:0/1950142394 >> 172.21.15.2:0/1950142394 conn(0x7f05680fbac0 msgr2=0x7f05680fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:10.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.671+0000 7f056fe42700 1 -- 172.21.15.2:0/1950142394 shutdown_connections 2023-11-16T18:23:10.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.671+0000 7f056fe42700 1 -- 172.21.15.2:0/1950142394 wait complete. 2023-11-16T18:23:10.684 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-16T18:23:10.685 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2023-11-16T18:23:09.806106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596430,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":142,"num_read_kb":125,"num_write":264,"num_write_kb":2971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79220,"kb_used_data":11156,"kb_used_omap":0,"kb_used_meta":67968,"kb_avail":1124862604,"statfs":{"total":1151940427776,"available":1151859306496,"internally_reserved":0,"allocated":11423744,"data_stored":7328562,"data_compressed":248619,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":0,"internal_metadata":69599232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3700,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":38,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003438"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635790+0000","last_change":"2023-11-16T18:22:46.829814+0000","last_active":"2023-11-16T18:22:51.635790+0000","last_peered":"2023-11-16T18:22:51.635790+0000","last_clean":"2023-11-16T18:22:51.635790+0000","last_became_active":"2023-11-16T18:22:46.829407+0000","last_became_peered":"2023-11-16T18:22:46.829407+0000","last_unstale":"2023-11-16T18:22:51.635790+0000","last_undegraded":"2023-11-16T18:22:51.635790+0000","last_fullsized":"2023-11-16T18:22:51.635790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:58:53.882493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583289+0000","last_change":"2023-11-16T18:22:46.818700+0000","last_active":"2023-11-16T18:22:51.583289+0000","last_peered":"2023-11-16T18:22:51.583289+0000","last_clean":"2023-11-16T18:22:51.583289+0000","last_became_active":"2023-11-16T18:22:46.818388+0000","last_became_peered":"2023-11-16T18:22:46.818388+0000","last_unstale":"2023-11-16T18:22:51.583289+0000","last_undegraded":"2023-11-16T18:22:51.583289+0000","last_fullsized":"2023-11-16T18:22:51.583289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:53:39.888834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582382+0000","last_change":"2023-11-16T18:22:46.827732+0000","last_active":"2023-11-16T18:22:51.582382+0000","last_peered":"2023-11-16T18:22:51.582382+0000","last_clean":"2023-11-16T18:22:51.582382+0000","last_became_active":"2023-11-16T18:22:46.827328+0000","last_became_peered":"2023-11-16T18:22:46.827328+0000","last_unstale":"2023-11-16T18:22:51.582382+0000","last_undegraded":"2023-11-16T18:22:51.582382+0000","last_fullsized":"2023-11-16T18:22:51.582382+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:36:48.387249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584389+0000","last_change":"2023-11-16T18:22:46.826837+0000","last_active":"2023-11-16T18:22:51.584389+0000","last_peered":"2023-11-16T18:22:51.584389+0000","last_clean":"2023-11-16T18:22:51.584389+0000","last_became_active":"2023-11-16T18:22:46.826532+0000","last_became_peered":"2023-11-16T18:22:46.826532+0000","last_unstale":"2023-11-16T18:22:51.584389+0000","last_undegraded":"2023-11-16T18:22:51.584389+0000","last_fullsized":"2023-11-16T18:22:51.584389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:28:11.423559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582313+0000","last_change":"2023-11-16T18:22:46.841164+0000","last_active":"2023-11-16T18:22:51.582313+0000","last_peered":"2023-11-16T18:22:51.582313+0000","last_clean":"2023-11-16T18:22:51.582313+0000","last_became_active":"2023-11-16T18:22:46.840855+0000","last_became_peered":"2023-11-16T18:22:46.840855+0000","last_unstale":"2023-11-16T18:22:51.582313+0000","last_undegraded":"2023-11-16T18:22:51.582313+0000","last_fullsized":"2023-11-16T18:22:51.582313+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:15:32.916615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583589+0000","last_change":"2023-11-16T18:22:46.822171+0000","last_active":"2023-11-16T18:22:51.583589+0000","last_peered":"2023-11-16T18:22:51.583589+0000","last_clean":"2023-11-16T18:22:51.583589+0000","last_became_active":"2023-11-16T18:22:46.821860+0000","last_became_peered":"2023-11-16T18:22:46.821860+0000","last_unstale":"2023-11-16T18:22:51.583589+0000","last_undegraded":"2023-11-16T18:22:51.583589+0000","last_fullsized":"2023-11-16T18:22:51.583589+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:16:33.605016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583209+0000","last_change":"2023-11-16T18:22:46.832463+0000","last_active":"2023-11-16T18:22:51.583209+0000","last_peered":"2023-11-16T18:22:51.583209+0000","last_clean":"2023-11-16T18:22:51.583209+0000","last_became_active":"2023-11-16T18:22:46.832141+0000","last_became_peered":"2023-11-16T18:22:46.832141+0000","last_unstale":"2023-11-16T18:22:51.583209+0000","last_undegraded":"2023-11-16T18:22:51.583209+0000","last_fullsized":"2023-11-16T18:22:51.583209+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:39:23.533316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.537754+0000","last_change":"2023-11-16T18:22:46.844239+0000","last_active":"2023-11-16T18:22:51.537754+0000","last_peered":"2023-11-16T18:22:51.537754+0000","last_clean":"2023-11-16T18:22:51.537754+0000","last_became_active":"2023-11-16T18:22:46.843906+0000","last_became_peered":"2023-11-16T18:22:46.843906+0000","last_unstale":"2023-11-16T18:22:51.537754+0000","last_undegraded":"2023-11-16T18:22:51.537754+0000","last_fullsized":"2023-11-16T18:22:51.537754+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:31:34.112452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.540316+0000","last_change":"2023-11-16T18:22:46.834625+0000","last_active":"2023-11-16T18:22:51.540316+0000","last_peered":"2023-11-16T18:22:51.540316+0000","last_clean":"2023-11-16T18:22:51.540316+0000","last_became_active":"2023-11-16T18:22:46.834145+0000","last_became_peered":"2023-11-16T18:22:46.834145+0000","last_unstale":"2023-11-16T18:22:51.540316+0000","last_undegraded":"2023-11-16T18:22:51.540316+0000","last_fullsized":"2023-11-16T18:22:51.540316+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:59:35.306111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584001+0000","last_change":"2023-11-16T18:22:46.827569+0000","last_active":"2023-11-16T18:22:51.584001+0000","last_peered":"2023-11-16T18:22:51.584001+0000","last_clean":"2023-11-16T18:22:51.584001+0000","last_became_active":"2023-11-16T18:22:46.827234+0000","last_became_peered":"2023-11-16T18:22:46.827234+0000","last_unstale":"2023-11-16T18:22:51.584001+0000","last_undegraded":"2023-11-16T18:22:51.584001+0000","last_fullsized":"2023-11-16T18:22:51.584001+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:37:08.465042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.585080+0000","last_change":"2023-11-16T18:22:46.833701+0000","last_active":"2023-11-16T18:22:51.585080+0000","last_peered":"2023-11-16T18:22:51.585080+0000","last_clean":"2023-11-16T18:22:51.585080+0000","last_became_active":"2023-11-16T18:22:46.833333+0000","last_became_peered":"2023-11-16T18:22:46.833333+0000","last_unstale":"2023-11-16T18:22:51.585080+0000","last_undegraded":"2023-11-16T18:22:51.585080+0000","last_fullsized":"2023-11-16T18:22:51.585080+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:39:03.487707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584461+0000","last_change":"2023-11-16T18:22:46.823222+0000","last_active":"2023-11-16T18:22:51.584461+0000","last_peered":"2023-11-16T18:22:51.584461+0000","last_clean":"2023-11-16T18:22:51.584461+0000","last_became_active":"2023-11-16T18:22:46.822224+0000","last_became_peered":"2023-11-16T18:22:46.822224+0000","last_unstale":"2023-11-16T18:22:51.584461+0000","last_undegraded":"2023-11-16T18:22:51.584461+0000","last_fullsized":"2023-11-16T18:22:51.584461+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:55:47.913232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583763+0000","last_change":"2023-11-16T18:22:46.818631+0000","last_active":"2023-11-16T18:22:51.583763+0000","last_peered":"2023-11-16T18:22:51.583763+0000","last_clean":"2023-11-16T18:22:51.583763+0000","last_became_active":"2023-11-16T18:22:46.818305+0000","last_became_peered":"2023-11-16T18:22:46.818305+0000","last_unstale":"2023-11-16T18:22:51.583763+0000","last_undegraded":"2023-11-16T18:22:51.583763+0000","last_fullsized":"2023-11-16T18:22:51.583763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:42:03.259127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.585367+0000","last_change":"2023-11-16T18:22:46.825439+0000","last_active":"2023-11-16T18:22:51.585367+0000","last_peered":"2023-11-16T18:22:51.585367+0000","last_clean":"2023-11-16T18:22:51.585367+0000","last_became_active":"2023-11-16T18:22:46.825062+0000","last_became_peered":"2023-11-16T18:22:46.825062+0000","last_unstale":"2023-11-16T18:22:51.585367+0000","last_undegraded":"2023-11-16T18:22:51.585367+0000","last_fullsized":"2023-11-16T18:22:51.585367+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:20:36.560885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582640+0000","last_change":"2023-11-16T18:22:46.827730+0000","last_active":"2023-11-16T18:22:51.582640+0000","last_peered":"2023-11-16T18:22:51.582640+0000","last_clean":"2023-11-16T18:22:51.582640+0000","last_became_active":"2023-11-16T18:22:46.827416+0000","last_became_peered":"2023-11-16T18:22:46.827416+0000","last_unstale":"2023-11-16T18:22:51.582640+0000","last_undegraded":"2023-11-16T18:22:51.582640+0000","last_fullsized":"2023-11-16T18:22:51.582640+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:28:41.500983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.539430+0000","last_change":"2023-11-16T18:22:46.834259+0000","last_active":"2023-11-16T18:22:51.539430+0000","last_peered":"2023-11-16T18:22:51.539430+0000","last_clean":"2023-11-16T18:22:51.539430+0000","last_became_active":"2023-11-16T18:22:46.833775+0000","last_became_peered":"2023-11-16T18:22:46.833775+0000","last_unstale":"2023-11-16T18:22:51.539430+0000","last_undegraded":"2023-11-16T18:22:51.539430+0000","last_fullsized":"2023-11-16T18:22:51.539430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:24:52.545621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635645+0000","last_change":"2023-11-16T18:22:46.829475+0000","last_active":"2023-11-16T18:22:51.635645+0000","last_peered":"2023-11-16T18:22:51.635645+0000","last_clean":"2023-11-16T18:22:51.635645+0000","last_became_active":"2023-11-16T18:22:46.829157+0000","last_became_peered":"2023-11-16T18:22:46.829157+0000","last_unstale":"2023-11-16T18:22:51.635645+0000","last_undegraded":"2023-11-16T18:22:51.635645+0000","last_fullsized":"2023-11-16T18:22:51.635645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:01:49.857829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584839+0000","last_change":"2023-11-16T18:22:46.821716+0000","last_active":"2023-11-16T18:22:51.584839+0000","last_peered":"2023-11-16T18:22:51.584839+0000","last_clean":"2023-11-16T18:22:51.584839+0000","last_became_active":"2023-11-16T18:22:46.821393+0000","last_became_peered":"2023-11-16T18:22:46.821393+0000","last_unstale":"2023-11-16T18:22:51.584839+0000","last_undegraded":"2023-11-16T18:22:51.584839+0000","last_fullsized":"2023-11-16T18:22:51.584839+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:39:50.644126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583545+0000","last_change":"2023-11-16T18:22:46.837433+0000","last_active":"2023-11-16T18:22:51.583545+0000","last_peered":"2023-11-16T18:22:51.583545+0000","last_clean":"2023-11-16T18:22:51.583545+0000","last_became_active":"2023-11-16T18:22:46.837143+0000","last_became_peered":"2023-11-16T18:22:46.837143+0000","last_unstale":"2023-11-16T18:22:51.583545+0000","last_undegraded":"2023-11-16T18:22:51.583545+0000","last_fullsized":"2023-11-16T18:22:51.583545+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:27:52.152539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582661+0000","last_change":"2023-11-16T18:22:51.123749+0000","last_active":"2023-11-16T18:22:51.582661+0000","last_peered":"2023-11-16T18:22:51.582661+0000","last_clean":"2023-11-16T18:22:51.582661+0000","last_became_active":"2023-11-16T18:22:51.123185+0000","last_became_peered":"2023-11-16T18:22:51.123185+0000","last_unstale":"2023-11-16T18:22:51.582661+0000","last_undegraded":"2023-11-16T18:22:51.582661+0000","last_fullsized":"2023-11-16T18:22:51.582661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:48:41.870442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'4","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.585154+0000","last_change":"2023-11-16T18:22:46.385440+0000","last_active":"2023-11-16T18:22:51.585154+0000","last_peered":"2023-11-16T18:22:51.585154+0000","last_clean":"2023-11-16T18:22:51.585154+0000","last_became_active":"2023-11-16T18:22:46.385098+0000","last_became_peered":"2023-11-16T18:22:46.385098+0000","last_unstale":"2023-11-16T18:22:51.585154+0000","last_undegraded":"2023-11-16T18:22:51.585154+0000","last_fullsized":"2023-11-16T18:22:51.585154+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:27:11.799990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538546+0000","last_change":"2023-11-16T18:22:46.842681+0000","last_active":"2023-11-16T18:22:51.538546+0000","last_peered":"2023-11-16T18:22:51.538546+0000","last_clean":"2023-11-16T18:22:51.538546+0000","last_became_active":"2023-11-16T18:22:46.842408+0000","last_became_peered":"2023-11-16T18:22:46.842408+0000","last_unstale":"2023-11-16T18:22:51.538546+0000","last_undegraded":"2023-11-16T18:22:51.538546+0000","last_fullsized":"2023-11-16T18:22:51.538546+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:21:02.462535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582828+0000","last_change":"2023-11-16T18:22:51.123450+0000","last_active":"2023-11-16T18:22:51.582828+0000","last_peered":"2023-11-16T18:22:51.582828+0000","last_clean":"2023-11-16T18:22:51.582828+0000","last_became_active":"2023-11-16T18:22:51.123002+0000","last_became_peered":"2023-11-16T18:22:51.123002+0000","last_unstale":"2023-11-16T18:22:51.582828+0000","last_undegraded":"2023-11-16T18:22:51.582828+0000","last_fullsized":"2023-11-16T18:22:51.582828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:04:27.030138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538660+0000","last_change":"2023-11-16T18:22:46.384768+0000","last_active":"2023-11-16T18:22:51.538660+0000","last_peered":"2023-11-16T18:22:51.538660+0000","last_clean":"2023-11-16T18:22:51.538660+0000","last_became_active":"2023-11-16T18:22:46.384246+0000","last_became_peered":"2023-11-16T18:22:46.384246+0000","last_unstale":"2023-11-16T18:22:51.538660+0000","last_undegraded":"2023-11-16T18:22:51.538660+0000","last_fullsized":"2023-11-16T18:22:51.538660+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:26:21.800521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583368+0000","last_change":"2023-11-16T18:22:46.826069+0000","last_active":"2023-11-16T18:22:51.583368+0000","last_peered":"2023-11-16T18:22:51.583368+0000","last_clean":"2023-11-16T18:22:51.583368+0000","last_became_active":"2023-11-16T18:22:46.824676+0000","last_became_peered":"2023-11-16T18:22:46.824676+0000","last_unstale":"2023-11-16T18:22:51.583368+0000","last_undegraded":"2023-11-16T18:22:51.583368+0000","last_fullsized":"2023-11-16T18:22:51.583368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:48:03.420408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584224+0000","last_change":"2023-11-16T18:22:51.127529+0000","last_active":"2023-11-16T18:22:51.584224+0000","last_peered":"2023-11-16T18:22:51.584224+0000","last_clean":"2023-11-16T18:22:51.584224+0000","last_became_active":"2023-11-16T18:22:51.126748+0000","last_became_peered":"2023-11-16T18:22:51.126748+0000","last_unstale":"2023-11-16T18:22:51.584224+0000","last_undegraded":"2023-11-16T18:22:51.584224+0000","last_fullsized":"2023-11-16T18:22:51.584224+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:31:01.008519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:06.912752+0000","last_change":"2023-11-16T18:22:46.385890+0000","last_active":"2023-11-16T18:23:06.912752+0000","last_peered":"2023-11-16T18:23:06.912752+0000","last_clean":"2023-11-16T18:23:06.912752+0000","last_became_active":"2023-11-16T18:22:46.385568+0000","last_became_peered":"2023-11-16T18:22:46.385568+0000","last_unstale":"2023-11-16T18:23:06.912752+0000","last_undegraded":"2023-11-16T18:23:06.912752+0000","last_fullsized":"2023-11-16T18:23:06.912752+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:16:48.773546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.539902+0000","last_change":"2023-11-16T18:22:46.824796+0000","last_active":"2023-11-16T18:22:51.539902+0000","last_peered":"2023-11-16T18:22:51.539902+0000","last_clean":"2023-11-16T18:22:51.539902+0000","last_became_active":"2023-11-16T18:22:46.823895+0000","last_became_peered":"2023-11-16T18:22:46.823895+0000","last_unstale":"2023-11-16T18:22:51.539902+0000","last_undegraded":"2023-11-16T18:22:51.539902+0000","last_fullsized":"2023-11-16T18:22:51.539902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:40:21.122081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582786+0000","last_change":"2023-11-16T18:22:51.118714+0000","last_active":"2023-11-16T18:22:51.582786+0000","last_peered":"2023-11-16T18:22:51.582786+0000","last_clean":"2023-11-16T18:22:51.582786+0000","last_became_active":"2023-11-16T18:22:51.118216+0000","last_became_peered":"2023-11-16T18:22:51.118216+0000","last_unstale":"2023-11-16T18:22:51.582786+0000","last_undegraded":"2023-11-16T18:22:51.582786+0000","last_fullsized":"2023-11-16T18:22:51.582786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:12:26.790005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582618+0000","last_change":"2023-11-16T18:22:46.376431+0000","last_active":"2023-11-16T18:22:51.582618+0000","last_peered":"2023-11-16T18:22:51.582618+0000","last_clean":"2023-11-16T18:22:51.582618+0000","last_became_active":"2023-11-16T18:22:46.375914+0000","last_became_peered":"2023-11-16T18:22:46.375914+0000","last_unstale":"2023-11-16T18:22:51.582618+0000","last_undegraded":"2023-11-16T18:22:51.582618+0000","last_fullsized":"2023-11-16T18:22:51.582618+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:30:55.925400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.542071+0000","last_change":"2023-11-16T18:22:46.825002+0000","last_active":"2023-11-16T18:22:51.542071+0000","last_peered":"2023-11-16T18:22:51.542071+0000","last_clean":"2023-11-16T18:22:51.542071+0000","last_became_active":"2023-11-16T18:22:46.824012+0000","last_became_peered":"2023-11-16T18:22:46.824012+0000","last_unstale":"2023-11-16T18:22:51.542071+0000","last_undegraded":"2023-11-16T18:22:51.542071+0000","last_fullsized":"2023-11-16T18:22:51.542071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:23:47.920490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583388+0000","last_change":"2023-11-16T18:22:51.117623+0000","last_active":"2023-11-16T18:22:51.583388+0000","last_peered":"2023-11-16T18:22:51.583388+0000","last_clean":"2023-11-16T18:22:51.583388+0000","last_became_active":"2023-11-16T18:22:51.117183+0000","last_became_peered":"2023-11-16T18:22:51.117183+0000","last_unstale":"2023-11-16T18:22:51.583388+0000","last_undegraded":"2023-11-16T18:22:51.583388+0000","last_fullsized":"2023-11-16T18:22:51.583388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:47:56.989813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:05.905304+0000","last_change":"2023-11-16T18:22:46.384920+0000","last_active":"2023-11-16T18:23:05.905304+0000","last_peered":"2023-11-16T18:23:05.905304+0000","last_clean":"2023-11-16T18:23:05.905304+0000","last_became_active":"2023-11-16T18:22:46.384577+0000","last_became_peered":"2023-11-16T18:22:46.384577+0000","last_unstale":"2023-11-16T18:23:05.905304+0000","last_undegraded":"2023-11-16T18:23:05.905304+0000","last_fullsized":"2023-11-16T18:23:05.905304+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:12:39.928656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582483+0000","last_change":"2023-11-16T18:22:46.820073+0000","last_active":"2023-11-16T18:22:51.582483+0000","last_peered":"2023-11-16T18:22:51.582483+0000","last_clean":"2023-11-16T18:22:51.582483+0000","last_became_active":"2023-11-16T18:22:46.819764+0000","last_became_peered":"2023-11-16T18:22:46.819764+0000","last_unstale":"2023-11-16T18:22:51.582483+0000","last_undegraded":"2023-11-16T18:22:51.582483+0000","last_fullsized":"2023-11-16T18:22:51.582483+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:08.246496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583304+0000","last_change":"2023-11-16T18:22:51.116619+0000","last_active":"2023-11-16T18:22:51.583304+0000","last_peered":"2023-11-16T18:22:51.583304+0000","last_clean":"2023-11-16T18:22:51.583304+0000","last_became_active":"2023-11-16T18:22:51.116264+0000","last_became_peered":"2023-11-16T18:22:51.116264+0000","last_unstale":"2023-11-16T18:22:51.583304+0000","last_undegraded":"2023-11-16T18:22:51.583304+0000","last_fullsized":"2023-11-16T18:22:51.583304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:33:21.580177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583423+0000","last_change":"2023-11-16T18:22:46.389590+0000","last_active":"2023-11-16T18:22:51.583423+0000","last_peered":"2023-11-16T18:22:51.583423+0000","last_clean":"2023-11-16T18:22:51.583423+0000","last_became_active":"2023-11-16T18:22:46.389354+0000","last_became_peered":"2023-11-16T18:22:46.389354+0000","last_unstale":"2023-11-16T18:22:51.583423+0000","last_undegraded":"2023-11-16T18:22:51.583423+0000","last_fullsized":"2023-11-16T18:22:51.583423+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:32:08.189457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541898+0000","last_change":"2023-11-16T18:22:46.841106+0000","last_active":"2023-11-16T18:22:51.541898+0000","last_peered":"2023-11-16T18:22:51.541898+0000","last_clean":"2023-11-16T18:22:51.541898+0000","last_became_active":"2023-11-16T18:22:46.840876+0000","last_became_peered":"2023-11-16T18:22:46.840876+0000","last_unstale":"2023-11-16T18:22:51.541898+0000","last_undegraded":"2023-11-16T18:22:51.541898+0000","last_fullsized":"2023-11-16T18:22:51.541898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:28:37.411931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635709+0000","last_change":"2023-11-16T18:22:51.125813+0000","last_active":"2023-11-16T18:22:51.635709+0000","last_peered":"2023-11-16T18:22:51.635709+0000","last_clean":"2023-11-16T18:22:51.635709+0000","last_became_active":"2023-11-16T18:22:51.125277+0000","last_became_peered":"2023-11-16T18:22:51.125277+0000","last_unstale":"2023-11-16T18:22:51.635709+0000","last_undegraded":"2023-11-16T18:22:51.635709+0000","last_fullsized":"2023-11-16T18:22:51.635709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:26:22.312004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:06.914290+0000","last_change":"2023-11-16T18:22:46.381208+0000","last_active":"2023-11-16T18:23:06.914290+0000","last_peered":"2023-11-16T18:23:06.914290+0000","last_clean":"2023-11-16T18:23:06.914290+0000","last_became_active":"2023-11-16T18:22:46.380351+0000","last_became_peered":"2023-11-16T18:22:46.380351+0000","last_unstale":"2023-11-16T18:23:06.914290+0000","last_undegraded":"2023-11-16T18:23:06.914290+0000","last_fullsized":"2023-11-16T18:23:06.914290+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:16:24.398001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.537912+0000","last_change":"2023-11-16T18:22:46.843043+0000","last_active":"2023-11-16T18:22:51.537912+0000","last_peered":"2023-11-16T18:22:51.537912+0000","last_clean":"2023-11-16T18:22:51.537912+0000","last_became_active":"2023-11-16T18:22:46.841522+0000","last_became_peered":"2023-11-16T18:22:46.841522+0000","last_unstale":"2023-11-16T18:22:51.537912+0000","last_undegraded":"2023-11-16T18:22:51.537912+0000","last_fullsized":"2023-11-16T18:22:51.537912+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:51:42.615558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583266+0000","last_change":"2023-11-16T18:22:51.127729+0000","last_active":"2023-11-16T18:22:51.583266+0000","last_peered":"2023-11-16T18:22:51.583266+0000","last_clean":"2023-11-16T18:22:51.583266+0000","last_became_active":"2023-11-16T18:22:51.127303+0000","last_became_peered":"2023-11-16T18:22:51.127303+0000","last_unstale":"2023-11-16T18:22:51.583266+0000","last_undegraded":"2023-11-16T18:22:51.583266+0000","last_fullsized":"2023-11-16T18:22:51.583266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:15:44.768952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541434+0000","last_change":"2023-11-16T18:22:46.377437+0000","last_active":"2023-11-16T18:22:51.541434+0000","last_peered":"2023-11-16T18:22:51.541434+0000","last_clean":"2023-11-16T18:22:51.541434+0000","last_became_active":"2023-11-16T18:22:46.377120+0000","last_became_peered":"2023-11-16T18:22:46.377120+0000","last_unstale":"2023-11-16T18:22:51.541434+0000","last_undegraded":"2023-11-16T18:22:51.541434+0000","last_fullsized":"2023-11-16T18:22:51.541434+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:56:43.302512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583817+0000","last_change":"2023-11-16T18:22:46.837804+0000","last_active":"2023-11-16T18:22:51.583817+0000","last_peered":"2023-11-16T18:22:51.583817+0000","last_clean":"2023-11-16T18:22:51.583817+0000","last_became_active":"2023-11-16T18:22:46.837432+0000","last_became_peered":"2023-11-16T18:22:46.837432+0000","last_unstale":"2023-11-16T18:22:51.583817+0000","last_undegraded":"2023-11-16T18:22:51.583817+0000","last_fullsized":"2023-11-16T18:22:51.583817+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:11:41.766949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538026+0000","last_change":"2023-11-16T18:22:51.127546+0000","last_active":"2023-11-16T18:22:51.538026+0000","last_peered":"2023-11-16T18:22:51.538026+0000","last_clean":"2023-11-16T18:22:51.538026+0000","last_became_active":"2023-11-16T18:22:51.127133+0000","last_became_peered":"2023-11-16T18:22:51.127133+0000","last_unstale":"2023-11-16T18:22:51.538026+0000","last_undegraded":"2023-11-16T18:22:51.538026+0000","last_fullsized":"2023-11-16T18:22:51.538026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:04:16.775399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584207+0000","last_change":"2023-11-16T18:22:46.386309+0000","last_active":"2023-11-16T18:22:51.584207+0000","last_peered":"2023-11-16T18:22:51.584207+0000","last_clean":"2023-11-16T18:22:51.584207+0000","last_became_active":"2023-11-16T18:22:46.385960+0000","last_became_peered":"2023-11-16T18:22:46.385960+0000","last_unstale":"2023-11-16T18:22:51.584207+0000","last_undegraded":"2023-11-16T18:22:51.584207+0000","last_fullsized":"2023-11-16T18:22:51.584207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:40:57.568161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.540289+0000","last_change":"2023-11-16T18:22:46.827211+0000","last_active":"2023-11-16T18:22:51.540289+0000","last_peered":"2023-11-16T18:22:51.540289+0000","last_clean":"2023-11-16T18:22:51.540289+0000","last_became_active":"2023-11-16T18:22:46.826396+0000","last_became_peered":"2023-11-16T18:22:46.826396+0000","last_unstale":"2023-11-16T18:22:51.540289+0000","last_undegraded":"2023-11-16T18:22:51.540289+0000","last_fullsized":"2023-11-16T18:22:51.540289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:27:52.671151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583449+0000","last_change":"2023-11-16T18:22:51.124816+0000","last_active":"2023-11-16T18:22:51.583449+0000","last_peered":"2023-11-16T18:22:51.583449+0000","last_clean":"2023-11-16T18:22:51.583449+0000","last_became_active":"2023-11-16T18:22:51.124296+0000","last_became_peered":"2023-11-16T18:22:51.124296+0000","last_unstale":"2023-11-16T18:22:51.583449+0000","last_undegraded":"2023-11-16T18:22:51.583449+0000","last_fullsized":"2023-11-16T18:22:51.583449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:47:31.452408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:05.904769+0000","last_change":"2023-11-16T18:22:46.384680+0000","last_active":"2023-11-16T18:23:05.904769+0000","last_peered":"2023-11-16T18:23:05.904769+0000","last_clean":"2023-11-16T18:23:05.904769+0000","last_became_active":"2023-11-16T18:22:46.384290+0000","last_became_peered":"2023-11-16T18:22:46.384290+0000","last_unstale":"2023-11-16T18:23:05.904769+0000","last_undegraded":"2023-11-16T18:23:05.904769+0000","last_fullsized":"2023-11-16T18:23:05.904769+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:28:49.556854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583498+0000","last_change":"2023-11-16T18:22:46.834726+0000","last_active":"2023-11-16T18:22:51.583498+0000","last_peered":"2023-11-16T18:22:51.583498+0000","last_clean":"2023-11-16T18:22:51.583498+0000","last_became_active":"2023-11-16T18:22:46.834424+0000","last_became_peered":"2023-11-16T18:22:46.834424+0000","last_unstale":"2023-11-16T18:22:51.583498+0000","last_undegraded":"2023-11-16T18:22:51.583498+0000","last_fullsized":"2023-11-16T18:22:51.583498+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:12:05.088718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584404+0000","last_change":"2023-11-16T18:22:51.125178+0000","last_active":"2023-11-16T18:22:51.584404+0000","last_peered":"2023-11-16T18:22:51.584404+0000","last_clean":"2023-11-16T18:22:51.584404+0000","last_became_active":"2023-11-16T18:22:51.124778+0000","last_became_peered":"2023-11-16T18:22:51.124778+0000","last_unstale":"2023-11-16T18:22:51.584404+0000","last_undegraded":"2023-11-16T18:22:51.584404+0000","last_fullsized":"2023-11-16T18:22:51.584404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:20:51.981074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541968+0000","last_change":"2023-11-16T18:22:46.380804+0000","last_active":"2023-11-16T18:22:51.541968+0000","last_peered":"2023-11-16T18:22:51.541968+0000","last_clean":"2023-11-16T18:22:51.541968+0000","last_became_active":"2023-11-16T18:22:46.380477+0000","last_became_peered":"2023-11-16T18:22:46.380477+0000","last_unstale":"2023-11-16T18:22:51.541968+0000","last_undegraded":"2023-11-16T18:22:51.541968+0000","last_fullsized":"2023-11-16T18:22:51.541968+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:19:26.230982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.540026+0000","last_change":"2023-11-16T18:22:46.825022+0000","last_active":"2023-11-16T18:22:51.540026+0000","last_peered":"2023-11-16T18:22:51.540026+0000","last_clean":"2023-11-16T18:22:51.540026+0000","last_became_active":"2023-11-16T18:22:46.824505+0000","last_became_peered":"2023-11-16T18:22:46.824505+0000","last_unstale":"2023-11-16T18:22:51.540026+0000","last_undegraded":"2023-11-16T18:22:51.540026+0000","last_fullsized":"2023-11-16T18:22:51.540026+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:01:52.602465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582852+0000","last_change":"2023-11-16T18:22:51.119711+0000","last_active":"2023-11-16T18:22:51.582852+0000","last_peered":"2023-11-16T18:22:51.582852+0000","last_clean":"2023-11-16T18:22:51.582852+0000","last_became_active":"2023-11-16T18:22:51.119304+0000","last_became_peered":"2023-11-16T18:22:51.119304+0000","last_unstale":"2023-11-16T18:22:51.582852+0000","last_undegraded":"2023-11-16T18:22:51.582852+0000","last_fullsized":"2023-11-16T18:22:51.582852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:06:35.490333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"86'10","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:02.838985+0000","last_change":"2023-11-16T18:22:46.382915+0000","last_active":"2023-11-16T18:23:02.838985+0000","last_peered":"2023-11-16T18:23:02.838985+0000","last_clean":"2023-11-16T18:23:02.838985+0000","last_became_active":"2023-11-16T18:22:46.382601+0000","last_became_peered":"2023-11-16T18:22:46.382601+0000","last_unstale":"2023-11-16T18:23:02.838985+0000","last_undegraded":"2023-11-16T18:23:02.838985+0000","last_fullsized":"2023-11-16T18:23:02.838985+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:02:18.799978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582729+0000","last_change":"2023-11-16T18:22:46.833537+0000","last_active":"2023-11-16T18:22:51.582729+0000","last_peered":"2023-11-16T18:22:51.582729+0000","last_clean":"2023-11-16T18:22:51.582729+0000","last_became_active":"2023-11-16T18:22:46.833217+0000","last_became_peered":"2023-11-16T18:22:46.833217+0000","last_unstale":"2023-11-16T18:22:51.582729+0000","last_undegraded":"2023-11-16T18:22:51.582729+0000","last_fullsized":"2023-11-16T18:22:51.582729+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:17:49.309028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.542410+0000","last_change":"2023-11-16T18:22:51.121091+0000","last_active":"2023-11-16T18:22:51.542410+0000","last_peered":"2023-11-16T18:22:51.542410+0000","last_clean":"2023-11-16T18:22:51.542410+0000","last_became_active":"2023-11-16T18:22:51.120656+0000","last_became_peered":"2023-11-16T18:22:51.120656+0000","last_unstale":"2023-11-16T18:22:51.542410+0000","last_undegraded":"2023-11-16T18:22:51.542410+0000","last_fullsized":"2023-11-16T18:22:51.542410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:53:23.944904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583180+0000","last_change":"2023-11-16T18:22:46.382812+0000","last_active":"2023-11-16T18:22:51.583180+0000","last_peered":"2023-11-16T18:22:51.583180+0000","last_clean":"2023-11-16T18:22:51.583180+0000","last_became_active":"2023-11-16T18:22:46.382499+0000","last_became_peered":"2023-11-16T18:22:46.382499+0000","last_unstale":"2023-11-16T18:22:51.583180+0000","last_undegraded":"2023-11-16T18:22:51.583180+0000","last_fullsized":"2023-11-16T18:22:51.583180+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:21:45.115121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583056+0000","last_change":"2023-11-16T18:22:46.840917+0000","last_active":"2023-11-16T18:22:51.583056+0000","last_peered":"2023-11-16T18:22:51.583056+0000","last_clean":"2023-11-16T18:22:51.583056+0000","last_became_active":"2023-11-16T18:22:46.840579+0000","last_became_peered":"2023-11-16T18:22:46.840579+0000","last_unstale":"2023-11-16T18:22:51.583056+0000","last_undegraded":"2023-11-16T18:22:51.583056+0000","last_fullsized":"2023-11-16T18:22:51.583056+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:52:31.233036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635389+0000","last_change":"2023-11-16T18:22:51.117852+0000","last_active":"2023-11-16T18:22:51.635389+0000","last_peered":"2023-11-16T18:22:51.635389+0000","last_clean":"2023-11-16T18:22:51.635389+0000","last_became_active":"2023-11-16T18:22:51.117390+0000","last_became_peered":"2023-11-16T18:22:51.117390+0000","last_unstale":"2023-11-16T18:22:51.635389+0000","last_undegraded":"2023-11-16T18:22:51.635389+0000","last_fullsized":"2023-11-16T18:22:51.635389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:34:36.312020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:05.904274+0000","last_change":"2023-11-16T18:22:46.385137+0000","last_active":"2023-11-16T18:23:05.904274+0000","last_peered":"2023-11-16T18:23:05.904274+0000","last_clean":"2023-11-16T18:23:05.904274+0000","last_became_active":"2023-11-16T18:22:46.384636+0000","last_became_peered":"2023-11-16T18:22:46.384636+0000","last_unstale":"2023-11-16T18:23:05.904274+0000","last_undegraded":"2023-11-16T18:23:05.904274+0000","last_fullsized":"2023-11-16T18:23:05.904274+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:24:08.648319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584689+0000","last_change":"2023-11-16T18:22:46.834786+0000","last_active":"2023-11-16T18:22:51.584689+0000","last_peered":"2023-11-16T18:22:51.584689+0000","last_clean":"2023-11-16T18:22:51.584689+0000","last_became_active":"2023-11-16T18:22:46.834338+0000","last_became_peered":"2023-11-16T18:22:46.834338+0000","last_unstale":"2023-11-16T18:22:51.584689+0000","last_undegraded":"2023-11-16T18:22:51.584689+0000","last_fullsized":"2023-11-16T18:22:51.584689+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:44:06.406787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584963+0000","last_change":"2023-11-16T18:22:51.124667+0000","last_active":"2023-11-16T18:22:51.584963+0000","last_peered":"2023-11-16T18:22:51.584963+0000","last_clean":"2023-11-16T18:22:51.584963+0000","last_became_active":"2023-11-16T18:22:51.124211+0000","last_became_peered":"2023-11-16T18:22:51.124211+0000","last_unstale":"2023-11-16T18:22:51.584963+0000","last_undegraded":"2023-11-16T18:22:51.584963+0000","last_fullsized":"2023-11-16T18:22:51.584963+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:03:47.792127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583038+0000","last_change":"2023-11-16T18:22:46.380829+0000","last_active":"2023-11-16T18:22:51.583038+0000","last_peered":"2023-11-16T18:22:51.583038+0000","last_clean":"2023-11-16T18:22:51.583038+0000","last_became_active":"2023-11-16T18:22:46.380466+0000","last_became_peered":"2023-11-16T18:22:46.380466+0000","last_unstale":"2023-11-16T18:22:51.583038+0000","last_undegraded":"2023-11-16T18:22:51.583038+0000","last_fullsized":"2023-11-16T18:22:51.583038+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:09:12.621584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583414+0000","last_change":"2023-11-16T18:22:46.835900+0000","last_active":"2023-11-16T18:22:51.583414+0000","last_peered":"2023-11-16T18:22:51.583414+0000","last_clean":"2023-11-16T18:22:51.583414+0000","last_became_active":"2023-11-16T18:22:46.835438+0000","last_became_peered":"2023-11-16T18:22:46.835438+0000","last_unstale":"2023-11-16T18:22:51.583414+0000","last_undegraded":"2023-11-16T18:22:51.583414+0000","last_fullsized":"2023-11-16T18:22:51.583414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:09:28.382280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582802+0000","last_change":"2023-11-16T18:22:51.120005+0000","last_active":"2023-11-16T18:22:51.582802+0000","last_peered":"2023-11-16T18:22:51.582802+0000","last_clean":"2023-11-16T18:22:51.582802+0000","last_became_active":"2023-11-16T18:22:51.119540+0000","last_became_peered":"2023-11-16T18:22:51.119540+0000","last_unstale":"2023-11-16T18:22:51.582802+0000","last_undegraded":"2023-11-16T18:22:51.582802+0000","last_fullsized":"2023-11-16T18:22:51.582802+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:49:58.798976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:06.915362+0000","last_change":"2023-11-16T18:22:46.378745+0000","last_active":"2023-11-16T18:23:06.915362+0000","last_peered":"2023-11-16T18:23:06.915362+0000","last_clean":"2023-11-16T18:23:06.915362+0000","last_became_active":"2023-11-16T18:22:46.378447+0000","last_became_peered":"2023-11-16T18:22:46.378447+0000","last_unstale":"2023-11-16T18:23:06.915362+0000","last_undegraded":"2023-11-16T18:23:06.915362+0000","last_fullsized":"2023-11-16T18:23:06.915362+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:37:35.122937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541447+0000","last_change":"2023-11-16T18:22:46.823838+0000","last_active":"2023-11-16T18:22:51.541447+0000","last_peered":"2023-11-16T18:22:51.541447+0000","last_clean":"2023-11-16T18:22:51.541447+0000","last_became_active":"2023-11-16T18:22:46.823545+0000","last_became_peered":"2023-11-16T18:22:46.823545+0000","last_unstale":"2023-11-16T18:22:51.541447+0000","last_undegraded":"2023-11-16T18:22:51.541447+0000","last_fullsized":"2023-11-16T18:22:51.541447+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:30:23.340444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541509+0000","last_change":"2023-11-16T18:22:51.127371+0000","last_active":"2023-11-16T18:22:51.541509+0000","last_peered":"2023-11-16T18:22:51.541509+0000","last_clean":"2023-11-16T18:22:51.541509+0000","last_became_active":"2023-11-16T18:22:51.126951+0000","last_became_peered":"2023-11-16T18:22:51.126951+0000","last_unstale":"2023-11-16T18:22:51.541509+0000","last_undegraded":"2023-11-16T18:22:51.541509+0000","last_fullsized":"2023-11-16T18:22:51.541509+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:19:52.597045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'1","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583488+0000","last_change":"2023-11-16T18:22:46.376561+0000","last_active":"2023-11-16T18:22:51.583488+0000","last_peered":"2023-11-16T18:22:51.583488+0000","last_clean":"2023-11-16T18:22:51.583488+0000","last_became_active":"2023-11-16T18:22:46.375426+0000","last_became_peered":"2023-11-16T18:22:46.375426+0000","last_unstale":"2023-11-16T18:22:51.583488+0000","last_undegraded":"2023-11-16T18:22:51.583488+0000","last_fullsized":"2023-11-16T18:22:51.583488+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:07:21.137762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635564+0000","last_change":"2023-11-16T18:22:46.826657+0000","last_active":"2023-11-16T18:22:51.635564+0000","last_peered":"2023-11-16T18:22:51.635564+0000","last_clean":"2023-11-16T18:22:51.635564+0000","last_became_active":"2023-11-16T18:22:46.826126+0000","last_became_peered":"2023-11-16T18:22:46.826126+0000","last_unstale":"2023-11-16T18:22:51.635564+0000","last_undegraded":"2023-11-16T18:22:51.635564+0000","last_fullsized":"2023-11-16T18:22:51.635564+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:40:32.749831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582581+0000","last_change":"2023-11-16T18:22:51.127949+0000","last_active":"2023-11-16T18:22:51.582581+0000","last_peered":"2023-11-16T18:22:51.582581+0000","last_clean":"2023-11-16T18:22:51.582581+0000","last_became_active":"2023-11-16T18:22:51.127388+0000","last_became_peered":"2023-11-16T18:22:51.127388+0000","last_unstale":"2023-11-16T18:22:51.582581+0000","last_undegraded":"2023-11-16T18:22:51.582581+0000","last_fullsized":"2023-11-16T18:22:51.582581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:53:28.858371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584118+0000","last_change":"2023-11-16T18:22:46.378568+0000","last_active":"2023-11-16T18:22:51.584118+0000","last_peered":"2023-11-16T18:22:51.584118+0000","last_clean":"2023-11-16T18:22:51.584118+0000","last_became_active":"2023-11-16T18:22:46.378018+0000","last_became_peered":"2023-11-16T18:22:46.378018+0000","last_unstale":"2023-11-16T18:22:51.584118+0000","last_undegraded":"2023-11-16T18:22:51.584118+0000","last_fullsized":"2023-11-16T18:22:51.584118+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:20:08.573909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583828+0000","last_change":"2023-11-16T18:22:46.841129+0000","last_active":"2023-11-16T18:22:51.583828+0000","last_peered":"2023-11-16T18:22:51.583828+0000","last_clean":"2023-11-16T18:22:51.583828+0000","last_became_active":"2023-11-16T18:22:46.840780+0000","last_became_peered":"2023-11-16T18:22:46.840780+0000","last_unstale":"2023-11-16T18:22:51.583828+0000","last_undegraded":"2023-11-16T18:22:51.583828+0000","last_fullsized":"2023-11-16T18:22:51.583828+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:25:22.363206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583049+0000","last_change":"2023-11-16T18:22:51.126221+0000","last_active":"2023-11-16T18:22:51.583049+0000","last_peered":"2023-11-16T18:22:51.583049+0000","last_clean":"2023-11-16T18:22:51.583049+0000","last_became_active":"2023-11-16T18:22:51.125770+0000","last_became_peered":"2023-11-16T18:22:51.125770+0000","last_unstale":"2023-11-16T18:22:51.583049+0000","last_undegraded":"2023-11-16T18:22:51.583049+0000","last_fullsized":"2023-11-16T18:22:51.583049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:15:19.731508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584284+0000","last_change":"2023-11-16T18:22:46.384872+0000","last_active":"2023-11-16T18:22:51.584284+0000","last_peered":"2023-11-16T18:22:51.584284+0000","last_clean":"2023-11-16T18:22:51.584284+0000","last_became_active":"2023-11-16T18:22:46.384499+0000","last_became_peered":"2023-11-16T18:22:46.384499+0000","last_unstale":"2023-11-16T18:22:51.584284+0000","last_undegraded":"2023-11-16T18:22:51.584284+0000","last_fullsized":"2023-11-16T18:22:51.584284+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:01:05.143223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583583+0000","last_change":"2023-11-16T18:22:46.822574+0000","last_active":"2023-11-16T18:22:51.583583+0000","last_peered":"2023-11-16T18:22:51.583583+0000","last_clean":"2023-11-16T18:22:51.583583+0000","last_became_active":"2023-11-16T18:22:46.822230+0000","last_became_peered":"2023-11-16T18:22:46.822230+0000","last_unstale":"2023-11-16T18:22:51.583583+0000","last_undegraded":"2023-11-16T18:22:51.583583+0000","last_fullsized":"2023-11-16T18:22:51.583583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:05:08.158509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582682+0000","last_change":"2023-11-16T18:22:51.117838+0000","last_active":"2023-11-16T18:22:51.582682+0000","last_peered":"2023-11-16T18:22:51.582682+0000","last_clean":"2023-11-16T18:22:51.582682+0000","last_became_active":"2023-11-16T18:22:51.117275+0000","last_became_peered":"2023-11-16T18:22:51.117275+0000","last_unstale":"2023-11-16T18:22:51.582682+0000","last_undegraded":"2023-11-16T18:22:51.582682+0000","last_fullsized":"2023-11-16T18:22:51.582682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:32:52.147084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541546+0000","last_change":"2023-11-16T18:22:46.827499+0000","last_active":"2023-11-16T18:22:51.541546+0000","last_peered":"2023-11-16T18:22:51.541546+0000","last_clean":"2023-11-16T18:22:51.541546+0000","last_became_active":"2023-11-16T18:22:46.827166+0000","last_became_peered":"2023-11-16T18:22:46.827166+0000","last_unstale":"2023-11-16T18:22:51.541546+0000","last_undegraded":"2023-11-16T18:22:51.541546+0000","last_fullsized":"2023-11-16T18:22:51.541546+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:04:21.034867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:06.914914+0000","last_change":"2023-11-16T18:22:46.381397+0000","last_active":"2023-11-16T18:23:06.914914+0000","last_peered":"2023-11-16T18:23:06.914914+0000","last_clean":"2023-11-16T18:23:06.914914+0000","last_became_active":"2023-11-16T18:22:46.380939+0000","last_became_peered":"2023-11-16T18:22:46.380939+0000","last_unstale":"2023-11-16T18:23:06.914914+0000","last_undegraded":"2023-11-16T18:23:06.914914+0000","last_fullsized":"2023-11-16T18:23:06.914914+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:05:23.498482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584902+0000","last_change":"2023-11-16T18:22:51.125044+0000","last_active":"2023-11-16T18:22:51.584902+0000","last_peered":"2023-11-16T18:22:51.584902+0000","last_clean":"2023-11-16T18:22:51.584902+0000","last_became_active":"2023-11-16T18:22:51.124422+0000","last_became_peered":"2023-11-16T18:22:51.124422+0000","last_unstale":"2023-11-16T18:22:51.584902+0000","last_undegraded":"2023-11-16T18:22:51.584902+0000","last_fullsized":"2023-11-16T18:22:51.584902+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:42:15.443481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.537935+0000","last_change":"2023-11-16T18:22:46.841500+0000","last_active":"2023-11-16T18:22:51.537935+0000","last_peered":"2023-11-16T18:22:51.537935+0000","last_clean":"2023-11-16T18:22:51.537935+0000","last_became_active":"2023-11-16T18:22:46.841186+0000","last_became_peered":"2023-11-16T18:22:46.841186+0000","last_unstale":"2023-11-16T18:22:51.537935+0000","last_undegraded":"2023-11-16T18:22:51.537935+0000","last_fullsized":"2023-11-16T18:22:51.537935+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:42:31.304146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584247+0000","last_change":"2023-11-16T18:22:46.380988+0000","last_active":"2023-11-16T18:22:51.584247+0000","last_peered":"2023-11-16T18:22:51.584247+0000","last_clean":"2023-11-16T18:22:51.584247+0000","last_became_active":"2023-11-16T18:22:46.380616+0000","last_became_peered":"2023-11-16T18:22:46.380616+0000","last_unstale":"2023-11-16T18:22:51.584247+0000","last_undegraded":"2023-11-16T18:22:51.584247+0000","last_fullsized":"2023-11-16T18:22:51.584247+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:17:42.747589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583007+0000","last_change":"2023-11-16T18:21:44.282706+0000","last_active":"2023-11-16T18:22:51.583007+0000","last_peered":"2023-11-16T18:22:51.583007+0000","last_clean":"2023-11-16T18:22:51.583007+0000","last_became_active":"2023-11-16T18:21:44.282201+0000","last_became_peered":"2023-11-16T18:21:44.282201+0000","last_unstale":"2023-11-16T18:22:51.583007+0000","last_undegraded":"2023-11-16T18:22:51.583007+0000","last_fullsized":"2023-11-16T18:22:51.583007+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_clean_scrub_stamp":"2023-11-16T18:18:46.711191+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:57:10.073628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.542133+0000","last_change":"2023-11-16T18:22:51.123051+0000","last_active":"2023-11-16T18:22:51.542133+0000","last_peered":"2023-11-16T18:22:51.542133+0000","last_clean":"2023-11-16T18:22:51.542133+0000","last_became_active":"2023-11-16T18:22:51.122635+0000","last_became_peered":"2023-11-16T18:22:51.122635+0000","last_unstale":"2023-11-16T18:22:51.542133+0000","last_undegraded":"2023-11-16T18:22:51.542133+0000","last_fullsized":"2023-11-16T18:22:51.542133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:21:44.306952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582426+0000","last_change":"2023-11-16T18:22:46.822625+0000","last_active":"2023-11-16T18:22:51.582426+0000","last_peered":"2023-11-16T18:22:51.582426+0000","last_clean":"2023-11-16T18:22:51.582426+0000","last_became_active":"2023-11-16T18:22:46.822296+0000","last_became_peered":"2023-11-16T18:22:46.822296+0000","last_unstale":"2023-11-16T18:22:51.582426+0000","last_undegraded":"2023-11-16T18:22:51.582426+0000","last_fullsized":"2023-11-16T18:22:51.582426+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:07:27.988390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583315+0000","last_change":"2023-11-16T18:22:46.380445+0000","last_active":"2023-11-16T18:22:51.583315+0000","last_peered":"2023-11-16T18:22:51.583315+0000","last_clean":"2023-11-16T18:22:51.583315+0000","last_became_active":"2023-11-16T18:22:46.380113+0000","last_became_peered":"2023-11-16T18:22:46.380113+0000","last_unstale":"2023-11-16T18:22:51.583315+0000","last_undegraded":"2023-11-16T18:22:51.583315+0000","last_fullsized":"2023-11-16T18:22:51.583315+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:18:58.685352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635377+0000","last_change":"2023-11-16T18:22:51.118457+0000","last_active":"2023-11-16T18:22:51.635377+0000","last_peered":"2023-11-16T18:22:51.635377+0000","last_clean":"2023-11-16T18:22:51.635377+0000","last_became_active":"2023-11-16T18:22:51.118013+0000","last_became_peered":"2023-11-16T18:22:51.118013+0000","last_unstale":"2023-11-16T18:22:51.635377+0000","last_undegraded":"2023-11-16T18:22:51.635377+0000","last_fullsized":"2023-11-16T18:22:51.635377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:39:13.358035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:05.904986+0000","last_change":"2023-11-16T18:22:46.384799+0000","last_active":"2023-11-16T18:23:05.904986+0000","last_peered":"2023-11-16T18:23:05.904986+0000","last_clean":"2023-11-16T18:23:05.904986+0000","last_became_active":"2023-11-16T18:22:46.384482+0000","last_became_peered":"2023-11-16T18:22:46.384482+0000","last_unstale":"2023-11-16T18:23:05.904986+0000","last_undegraded":"2023-11-16T18:23:05.904986+0000","last_fullsized":"2023-11-16T18:23:05.904986+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:43:28.699940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583195+0000","last_change":"2023-11-16T18:22:46.829541+0000","last_active":"2023-11-16T18:22:51.583195+0000","last_peered":"2023-11-16T18:22:51.583195+0000","last_clean":"2023-11-16T18:22:51.583195+0000","last_became_active":"2023-11-16T18:22:46.829006+0000","last_became_peered":"2023-11-16T18:22:46.829006+0000","last_unstale":"2023-11-16T18:22:51.583195+0000","last_undegraded":"2023-11-16T18:22:51.583195+0000","last_fullsized":"2023-11-16T18:22:51.583195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:43:56.291096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538010+0000","last_change":"2023-11-16T18:22:51.122751+0000","last_active":"2023-11-16T18:22:51.538010+0000","last_peered":"2023-11-16T18:22:51.538010+0000","last_clean":"2023-11-16T18:22:51.538010+0000","last_became_active":"2023-11-16T18:22:51.122423+0000","last_became_peered":"2023-11-16T18:22:51.122423+0000","last_unstale":"2023-11-16T18:22:51.538010+0000","last_undegraded":"2023-11-16T18:22:51.538010+0000","last_fullsized":"2023-11-16T18:22:51.538010+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:24:08.649040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584727+0000","last_change":"2023-11-16T18:22:46.381856+0000","last_active":"2023-11-16T18:22:51.584727+0000","last_peered":"2023-11-16T18:22:51.584727+0000","last_clean":"2023-11-16T18:22:51.584727+0000","last_became_active":"2023-11-16T18:22:46.381518+0000","last_became_peered":"2023-11-16T18:22:46.381518+0000","last_unstale":"2023-11-16T18:22:51.584727+0000","last_undegraded":"2023-11-16T18:22:51.584727+0000","last_fullsized":"2023-11-16T18:22:51.584727+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:09:47.615061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538288+0000","last_change":"2023-11-16T18:22:46.843802+0000","last_active":"2023-11-16T18:22:51.538288+0000","last_peered":"2023-11-16T18:22:51.538288+0000","last_clean":"2023-11-16T18:22:51.538288+0000","last_became_active":"2023-11-16T18:22:46.842210+0000","last_became_peered":"2023-11-16T18:22:46.842210+0000","last_unstale":"2023-11-16T18:22:51.538288+0000","last_undegraded":"2023-11-16T18:22:51.538288+0000","last_fullsized":"2023-11-16T18:22:51.538288+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:33:56.708900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582275+0000","last_change":"2023-11-16T18:22:51.124632+0000","last_active":"2023-11-16T18:22:51.582275+0000","last_peered":"2023-11-16T18:22:51.582275+0000","last_clean":"2023-11-16T18:22:51.582275+0000","last_became_active":"2023-11-16T18:22:51.124220+0000","last_became_peered":"2023-11-16T18:22:51.124220+0000","last_unstale":"2023-11-16T18:22:51.582275+0000","last_undegraded":"2023-11-16T18:22:51.582275+0000","last_fullsized":"2023-11-16T18:22:51.582275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:33:27.384018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584105+0000","last_change":"2023-11-16T18:22:46.381456+0000","last_active":"2023-11-16T18:22:51.584105+0000","last_peered":"2023-11-16T18:22:51.584105+0000","last_clean":"2023-11-16T18:22:51.584105+0000","last_became_active":"2023-11-16T18:22:46.381137+0000","last_became_peered":"2023-11-16T18:22:46.381137+0000","last_unstale":"2023-11-16T18:22:51.584105+0000","last_undegraded":"2023-11-16T18:22:51.584105+0000","last_fullsized":"2023-11-16T18:22:51.584105+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:04:47.876051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583353+0000","last_change":"2023-11-16T18:22:46.838249+0000","last_active":"2023-11-16T18:22:51.583353+0000","last_peered":"2023-11-16T18:22:51.583353+0000","last_clean":"2023-11-16T18:22:51.583353+0000","last_became_active":"2023-11-16T18:22:46.837925+0000","last_became_peered":"2023-11-16T18:22:46.837925+0000","last_unstale":"2023-11-16T18:22:51.583353+0000","last_undegraded":"2023-11-16T18:22:51.583353+0000","last_fullsized":"2023-11-16T18:22:51.583353+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:05:01.131026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583670+0000","last_change":"2023-11-16T18:22:51.123687+0000","last_active":"2023-11-16T18:22:51.583670+0000","last_peered":"2023-11-16T18:22:51.583670+0000","last_clean":"2023-11-16T18:22:51.583670+0000","last_became_active":"2023-11-16T18:22:51.123209+0000","last_became_peered":"2023-11-16T18:22:51.123209+0000","last_unstale":"2023-11-16T18:22:51.583670+0000","last_undegraded":"2023-11-16T18:22:51.583670+0000","last_fullsized":"2023-11-16T18:22:51.583670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:43:25.846371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.539539+0000","last_change":"2023-11-16T18:22:46.378038+0000","last_active":"2023-11-16T18:22:51.539539+0000","last_peered":"2023-11-16T18:22:51.539539+0000","last_clean":"2023-11-16T18:22:51.539539+0000","last_became_active":"2023-11-16T18:22:46.377680+0000","last_became_peered":"2023-11-16T18:22:46.377680+0000","last_unstale":"2023-11-16T18:22:51.539539+0000","last_undegraded":"2023-11-16T18:22:51.539539+0000","last_fullsized":"2023-11-16T18:22:51.539539+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:19:09.447833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582149+0000","last_change":"2023-11-16T18:22:46.821888+0000","last_active":"2023-11-16T18:22:51.582149+0000","last_peered":"2023-11-16T18:22:51.582149+0000","last_clean":"2023-11-16T18:22:51.582149+0000","last_became_active":"2023-11-16T18:22:46.821446+0000","last_became_peered":"2023-11-16T18:22:46.821446+0000","last_unstale":"2023-11-16T18:22:51.582149+0000","last_undegraded":"2023-11-16T18:22:51.582149+0000","last_fullsized":"2023-11-16T18:22:51.582149+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:36:05.450444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.585099+0000","last_change":"2023-11-16T18:22:51.119717+0000","last_active":"2023-11-16T18:22:51.585099+0000","last_peered":"2023-11-16T18:22:51.585099+0000","last_clean":"2023-11-16T18:22:51.585099+0000","last_became_active":"2023-11-16T18:22:51.119256+0000","last_became_peered":"2023-11-16T18:22:51.119256+0000","last_unstale":"2023-11-16T18:22:51.585099+0000","last_undegraded":"2023-11-16T18:22:51.585099+0000","last_fullsized":"2023-11-16T18:22:51.585099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:26:59.741857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:05.906947+0000","last_change":"2023-11-16T18:22:46.375387+0000","last_active":"2023-11-16T18:23:05.906947+0000","last_peered":"2023-11-16T18:23:05.906947+0000","last_clean":"2023-11-16T18:23:05.906947+0000","last_became_active":"2023-11-16T18:22:46.374886+0000","last_became_peered":"2023-11-16T18:22:46.374886+0000","last_unstale":"2023-11-16T18:23:05.906947+0000","last_undegraded":"2023-11-16T18:23:05.906947+0000","last_fullsized":"2023-11-16T18:23:05.906947+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:51:03.887363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635094+0000","last_change":"2023-11-16T18:22:46.820315+0000","last_active":"2023-11-16T18:22:51.635094+0000","last_peered":"2023-11-16T18:22:51.635094+0000","last_clean":"2023-11-16T18:22:51.635094+0000","last_became_active":"2023-11-16T18:22:46.819592+0000","last_became_peered":"2023-11-16T18:22:46.819592+0000","last_unstale":"2023-11-16T18:22:51.635094+0000","last_undegraded":"2023-11-16T18:22:51.635094+0000","last_fullsized":"2023-11-16T18:22:51.635094+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:39:18.272375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584121+0000","last_change":"2023-11-16T18:22:51.123645+0000","last_active":"2023-11-16T18:22:51.584121+0000","last_peered":"2023-11-16T18:22:51.584121+0000","last_clean":"2023-11-16T18:22:51.584121+0000","last_became_active":"2023-11-16T18:22:51.123099+0000","last_became_peered":"2023-11-16T18:22:51.123099+0000","last_unstale":"2023-11-16T18:22:51.584121+0000","last_undegraded":"2023-11-16T18:22:51.584121+0000","last_fullsized":"2023-11-16T18:22:51.584121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:44:09.664575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582962+0000","last_change":"2023-11-16T18:22:46.378638+0000","last_active":"2023-11-16T18:22:51.582962+0000","last_peered":"2023-11-16T18:22:51.582962+0000","last_clean":"2023-11-16T18:22:51.582962+0000","last_became_active":"2023-11-16T18:22:46.378341+0000","last_became_peered":"2023-11-16T18:22:46.378341+0000","last_unstale":"2023-11-16T18:22:51.582962+0000","last_undegraded":"2023-11-16T18:22:51.582962+0000","last_fullsized":"2023-11-16T18:22:51.582962+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:40:53.983774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538491+0000","last_change":"2023-11-16T18:22:46.843431+0000","last_active":"2023-11-16T18:22:51.538491+0000","last_peered":"2023-11-16T18:22:51.538491+0000","last_clean":"2023-11-16T18:22:51.538491+0000","last_became_active":"2023-11-16T18:22:46.842992+0000","last_became_peered":"2023-11-16T18:22:46.842992+0000","last_unstale":"2023-11-16T18:22:51.538491+0000","last_undegraded":"2023-11-16T18:22:51.538491+0000","last_fullsized":"2023-11-16T18:22:51.538491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:18:53.495119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583896+0000","last_change":"2023-11-16T18:22:51.122640+0000","last_active":"2023-11-16T18:22:51.583896+0000","last_peered":"2023-11-16T18:22:51.583896+0000","last_clean":"2023-11-16T18:22:51.583896+0000","last_became_active":"2023-11-16T18:22:51.122006+0000","last_became_peered":"2023-11-16T18:22:51.122006+0000","last_unstale":"2023-11-16T18:22:51.583896+0000","last_undegraded":"2023-11-16T18:22:51.583896+0000","last_fullsized":"2023-11-16T18:22:51.583896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:10:12.659920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:05.905977+0000","last_change":"2023-11-16T18:22:46.384837+0000","last_active":"2023-11-16T18:23:05.905977+0000","last_peered":"2023-11-16T18:23:05.905977+0000","last_clean":"2023-11-16T18:23:05.905977+0000","last_became_active":"2023-11-16T18:22:46.384541+0000","last_became_peered":"2023-11-16T18:22:46.384541+0000","last_unstale":"2023-11-16T18:23:05.905977+0000","last_undegraded":"2023-11-16T18:23:05.905977+0000","last_fullsized":"2023-11-16T18:23:05.905977+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:10:01.746531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584631+0000","last_change":"2023-11-16T18:22:46.839967+0000","last_active":"2023-11-16T18:22:51.584631+0000","last_peered":"2023-11-16T18:22:51.584631+0000","last_clean":"2023-11-16T18:22:51.584631+0000","last_became_active":"2023-11-16T18:22:46.839645+0000","last_became_peered":"2023-11-16T18:22:46.839645+0000","last_unstale":"2023-11-16T18:22:51.584631+0000","last_undegraded":"2023-11-16T18:22:51.584631+0000","last_fullsized":"2023-11-16T18:22:51.584631+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:51:40.823551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.539711+0000","last_change":"2023-11-16T18:22:51.126548+0000","last_active":"2023-11-16T18:22:51.539711+0000","last_peered":"2023-11-16T18:22:51.539711+0000","last_clean":"2023-11-16T18:22:51.539711+0000","last_became_active":"2023-11-16T18:22:51.126039+0000","last_became_peered":"2023-11-16T18:22:51.126039+0000","last_unstale":"2023-11-16T18:22:51.539711+0000","last_undegraded":"2023-11-16T18:22:51.539711+0000","last_fullsized":"2023-11-16T18:22:51.539711+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:26:40.925200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583458+0000","last_change":"2023-11-16T18:22:46.383947+0000","last_active":"2023-11-16T18:22:51.583458+0000","last_peered":"2023-11-16T18:22:51.583458+0000","last_clean":"2023-11-16T18:22:51.583458+0000","last_became_active":"2023-11-16T18:22:46.383661+0000","last_became_peered":"2023-11-16T18:22:46.383661+0000","last_unstale":"2023-11-16T18:22:51.583458+0000","last_undegraded":"2023-11-16T18:22:51.583458+0000","last_fullsized":"2023-11-16T18:22:51.583458+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:26:19.213002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.585153+0000","last_change":"2023-11-16T18:22:46.836877+0000","last_active":"2023-11-16T18:22:51.585153+0000","last_peered":"2023-11-16T18:22:51.585153+0000","last_clean":"2023-11-16T18:22:51.585153+0000","last_became_active":"2023-11-16T18:22:46.836477+0000","last_became_peered":"2023-11-16T18:22:46.836477+0000","last_unstale":"2023-11-16T18:22:51.585153+0000","last_undegraded":"2023-11-16T18:22:51.585153+0000","last_fullsized":"2023-11-16T18:22:51.585153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:00:14.928162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582862+0000","last_change":"2023-11-16T18:22:51.126930+0000","last_active":"2023-11-16T18:22:51.582862+0000","last_peered":"2023-11-16T18:22:51.582862+0000","last_clean":"2023-11-16T18:22:51.582862+0000","last_became_active":"2023-11-16T18:22:51.126409+0000","last_became_peered":"2023-11-16T18:22:51.126409+0000","last_unstale":"2023-11-16T18:22:51.582862+0000","last_undegraded":"2023-11-16T18:22:51.582862+0000","last_fullsized":"2023-11-16T18:22:51.582862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:39:20.438296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.542073+0000","last_change":"2023-11-16T18:22:46.380820+0000","last_active":"2023-11-16T18:22:51.542073+0000","last_peered":"2023-11-16T18:22:51.542073+0000","last_clean":"2023-11-16T18:22:51.542073+0000","last_became_active":"2023-11-16T18:22:46.380446+0000","last_became_peered":"2023-11-16T18:22:46.380446+0000","last_unstale":"2023-11-16T18:22:51.542073+0000","last_undegraded":"2023-11-16T18:22:51.542073+0000","last_fullsized":"2023-11-16T18:22:51.542073+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:25:56.255338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584787+0000","last_change":"2023-11-16T18:22:46.838153+0000","last_active":"2023-11-16T18:22:51.584787+0000","last_peered":"2023-11-16T18:22:51.584787+0000","last_clean":"2023-11-16T18:22:51.584787+0000","last_became_active":"2023-11-16T18:22:46.837836+0000","last_became_peered":"2023-11-16T18:22:46.837836+0000","last_unstale":"2023-11-16T18:22:51.584787+0000","last_undegraded":"2023-11-16T18:22:51.584787+0000","last_fullsized":"2023-11-16T18:22:51.584787+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:33:16.340646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541226+0000","last_change":"2023-11-16T18:22:51.119993+0000","last_active":"2023-11-16T18:22:51.541226+0000","last_peered":"2023-11-16T18:22:51.541226+0000","last_clean":"2023-11-16T18:22:51.541226+0000","last_became_active":"2023-11-16T18:22:51.119401+0000","last_became_peered":"2023-11-16T18:22:51.119401+0000","last_unstale":"2023-11-16T18:22:51.541226+0000","last_undegraded":"2023-11-16T18:22:51.541226+0000","last_fullsized":"2023-11-16T18:22:51.541226+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:16:01.027816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582901+0000","last_change":"2023-11-16T18:22:46.383399+0000","last_active":"2023-11-16T18:22:51.582901+0000","last_peered":"2023-11-16T18:22:51.582901+0000","last_clean":"2023-11-16T18:22:51.582901+0000","last_became_active":"2023-11-16T18:22:46.382938+0000","last_became_peered":"2023-11-16T18:22:46.382938+0000","last_unstale":"2023-11-16T18:22:51.582901+0000","last_undegraded":"2023-11-16T18:22:51.582901+0000","last_fullsized":"2023-11-16T18:22:51.582901+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:18:33.772254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.585416+0000","last_change":"2023-11-16T18:22:46.836035+0000","last_active":"2023-11-16T18:22:51.585416+0000","last_peered":"2023-11-16T18:22:51.585416+0000","last_clean":"2023-11-16T18:22:51.585416+0000","last_became_active":"2023-11-16T18:22:46.835709+0000","last_became_peered":"2023-11-16T18:22:46.835709+0000","last_unstale":"2023-11-16T18:22:51.585416+0000","last_undegraded":"2023-11-16T18:22:51.585416+0000","last_fullsized":"2023-11-16T18:22:51.585416+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:04:15.508385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635093+0000","last_change":"2023-11-16T18:22:46.820535+0000","last_active":"2023-11-16T18:22:51.635093+0000","last_peered":"2023-11-16T18:22:51.635093+0000","last_clean":"2023-11-16T18:22:51.635093+0000","last_became_active":"2023-11-16T18:22:46.820009+0000","last_became_peered":"2023-11-16T18:22:46.820009+0000","last_unstale":"2023-11-16T18:22:51.635093+0000","last_undegraded":"2023-11-16T18:22:51.635093+0000","last_fullsized":"2023-11-16T18:22:51.635093+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:03:50.304489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582866+0000","last_change":"2023-11-16T18:22:46.831762+0000","last_active":"2023-11-16T18:22:51.582866+0000","last_peered":"2023-11-16T18:22:51.582866+0000","last_clean":"2023-11-16T18:22:51.582866+0000","last_became_active":"2023-11-16T18:22:46.831429+0000","last_became_peered":"2023-11-16T18:22:46.831429+0000","last_unstale":"2023-11-16T18:22:51.582866+0000","last_undegraded":"2023-11-16T18:22:51.582866+0000","last_fullsized":"2023-11-16T18:22:51.582866+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:15:09.682384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582832+0000","last_change":"2023-11-16T18:22:46.838133+0000","last_active":"2023-11-16T18:22:51.582832+0000","last_peered":"2023-11-16T18:22:51.582832+0000","last_clean":"2023-11-16T18:22:51.582832+0000","last_became_active":"2023-11-16T18:22:46.837845+0000","last_became_peered":"2023-11-16T18:22:46.837845+0000","last_unstale":"2023-11-16T18:22:51.582832+0000","last_undegraded":"2023-11-16T18:22:51.582832+0000","last_fullsized":"2023-11-16T18:22:51.582832+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:05:42.510958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583143+0000","last_change":"2023-11-16T18:22:46.833813+0000","last_active":"2023-11-16T18:22:51.583143+0000","last_peered":"2023-11-16T18:22:51.583143+0000","last_clean":"2023-11-16T18:22:51.583143+0000","last_became_active":"2023-11-16T18:22:46.833379+0000","last_became_peered":"2023-11-16T18:22:46.833379+0000","last_unstale":"2023-11-16T18:22:51.583143+0000","last_undegraded":"2023-11-16T18:22:51.583143+0000","last_fullsized":"2023-11-16T18:22:51.583143+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:47:34.426651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583312+0000","last_change":"2023-11-16T18:22:46.823192+0000","last_active":"2023-11-16T18:22:51.583312+0000","last_peered":"2023-11-16T18:22:51.583312+0000","last_clean":"2023-11-16T18:22:51.583312+0000","last_became_active":"2023-11-16T18:22:46.822930+0000","last_became_peered":"2023-11-16T18:22:46.822930+0000","last_unstale":"2023-11-16T18:22:51.583312+0000","last_undegraded":"2023-11-16T18:22:51.583312+0000","last_fullsized":"2023-11-16T18:22:51.583312+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:02:02.465742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635585+0000","last_change":"2023-11-16T18:22:46.825815+0000","last_active":"2023-11-16T18:22:51.635585+0000","last_peered":"2023-11-16T18:22:51.635585+0000","last_clean":"2023-11-16T18:22:51.635585+0000","last_became_active":"2023-11-16T18:22:46.825517+0000","last_became_peered":"2023-11-16T18:22:46.825517+0000","last_unstale":"2023-11-16T18:22:51.635585+0000","last_undegraded":"2023-11-16T18:22:51.635585+0000","last_fullsized":"2023-11-16T18:22:51.635585+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:00:54.827931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583678+0000","last_change":"2023-11-16T18:22:46.826825+0000","last_active":"2023-11-16T18:22:51.583678+0000","last_peered":"2023-11-16T18:22:51.583678+0000","last_clean":"2023-11-16T18:22:51.583678+0000","last_became_active":"2023-11-16T18:22:46.826495+0000","last_became_peered":"2023-11-16T18:22:46.826495+0000","last_unstale":"2023-11-16T18:22:51.583678+0000","last_undegraded":"2023-11-16T18:22:51.583678+0000","last_fullsized":"2023-11-16T18:22:51.583678+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:34:04.792163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538413+0000","last_change":"2023-11-16T18:22:46.844090+0000","last_active":"2023-11-16T18:22:51.538413+0000","last_peered":"2023-11-16T18:22:51.538413+0000","last_clean":"2023-11-16T18:22:51.538413+0000","last_became_active":"2023-11-16T18:22:46.843641+0000","last_became_peered":"2023-11-16T18:22:46.843641+0000","last_unstale":"2023-11-16T18:22:51.538413+0000","last_undegraded":"2023-11-16T18:22:51.538413+0000","last_fullsized":"2023-11-16T18:22:51.538413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:57:37.160782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541384+0000","last_change":"2023-11-16T18:22:46.838340+0000","last_active":"2023-11-16T18:22:51.541384+0000","last_peered":"2023-11-16T18:22:51.541384+0000","last_clean":"2023-11-16T18:22:51.541384+0000","last_became_active":"2023-11-16T18:22:46.838039+0000","last_became_peered":"2023-11-16T18:22:46.838039+0000","last_unstale":"2023-11-16T18:22:51.541384+0000","last_undegraded":"2023-11-16T18:22:51.541384+0000","last_fullsized":"2023-11-16T18:22:51.541384+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:24:21.228784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538151+0000","last_change":"2023-11-16T18:22:46.841739+0000","last_active":"2023-11-16T18:22:51.538151+0000","last_peered":"2023-11-16T18:22:51.538151+0000","last_clean":"2023-11-16T18:22:51.538151+0000","last_became_active":"2023-11-16T18:22:46.841450+0000","last_became_peered":"2023-11-16T18:22:46.841450+0000","last_unstale":"2023-11-16T18:22:51.538151+0000","last_undegraded":"2023-11-16T18:22:51.538151+0000","last_fullsized":"2023-11-16T18:22:51.538151+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:17:38.521808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.540157+0000","last_change":"2023-11-16T18:22:46.839060+0000","last_active":"2023-11-16T18:22:51.540157+0000","last_peered":"2023-11-16T18:22:51.540157+0000","last_clean":"2023-11-16T18:22:51.540157+0000","last_became_active":"2023-11-16T18:22:46.838725+0000","last_became_peered":"2023-11-16T18:22:46.838725+0000","last_unstale":"2023-11-16T18:22:51.540157+0000","last_undegraded":"2023-11-16T18:22:51.540157+0000","last_fullsized":"2023-11-16T18:22:51.540157+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:36:04.190543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583894+0000","last_change":"2023-11-16T18:22:46.833439+0000","last_active":"2023-11-16T18:22:51.583894+0000","last_peered":"2023-11-16T18:22:51.583894+0000","last_clean":"2023-11-16T18:22:51.583894+0000","last_became_active":"2023-11-16T18:22:46.833111+0000","last_became_peered":"2023-11-16T18:22:46.833111+0000","last_unstale":"2023-11-16T18:22:51.583894+0000","last_undegraded":"2023-11-16T18:22:51.583894+0000","last_fullsized":"2023-11-16T18:22:51.583894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:15:52.910452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635322+0000","last_change":"2023-11-16T18:22:46.825185+0000","last_active":"2023-11-16T18:22:51.635322+0000","last_peered":"2023-11-16T18:22:51.635322+0000","last_clean":"2023-11-16T18:22:51.635322+0000","last_became_active":"2023-11-16T18:22:46.824866+0000","last_became_peered":"2023-11-16T18:22:46.824866+0000","last_unstale":"2023-11-16T18:22:51.635322+0000","last_undegraded":"2023-11-16T18:22:51.635322+0000","last_fullsized":"2023-11-16T18:22:51.635322+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:14:35.877637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":215374,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6062,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":83,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":17970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481808,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6040,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739112,"statfs":{"total":95995035648,"available":95988850688,"internally_reserved":0,"allocated":933888,"data_stored":462514,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67100000000000004}]},{"osd":2,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.6539999999999999,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.7130000000000001,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.70899999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.68400000000000005}]},{"osd":10,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.75},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.69699999999999995}]}]},{"osd":10,"up_from":69,"seq":296352743443,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6300,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":1003520,"data_stored":1056288,"data_compressed":47845,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":1,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.68999999999999995}]},{"osd":4,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.67200000000000004}]},{"osd":5,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.81499999999999995}]},{"osd":7,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55000000000000004}]},{"osd":8,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.80200000000000005}]},{"osd":9,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.84699999999999998}]},{"osd":11,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.73799999999999999}]}]},{"osd":9,"up_from":64,"seq":274877906968,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6232,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738920,"statfs":{"total":95995035648,"available":95988654080,"internally_reserved":0,"allocated":933888,"data_stored":462514,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.51600000000000001}]},{"osd":1,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.621},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.79400000000000004}]},{"osd":2,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.63900000000000001}]},{"osd":3,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.55200000000000005}]},{"osd":4,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.74099999999999999}]},{"osd":5,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.78100000000000003}]},{"osd":6,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.379},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.42699999999999999}]},{"osd":7,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.753}]},{"osd":8,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.874},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.81000000000000005}]},{"osd":10,"last update":"Thu Nov 16 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.82099999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]}]},{"osd":8,"up_from":57,"seq":244813135901,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6376,"kb_used_data":928,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738776,"statfs":{"total":95995035648,"available":95988506624,"internally_reserved":0,"allocated":950272,"data_stored":462670,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.58299999999999996,"15min":0.58799999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.81699999999999995,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.42499999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.876},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.443,"5min":0.316,"15min":0.316},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.83799999999999997}]},{"osd":2,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.36599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.626},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.48599999999999999,"15min":0.47599999999999998},"min":{"1min":0.39000000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.72799999999999998}]},{"osd":3,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.41799999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.628,"15min":0.628},"min":{"1min":0.41299999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.77300000000000002}]},{"osd":5,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52500000000000002,"15min":0.51300000000000001},"min":{"1min":0.49399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73899999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.37,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.77400000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.84499999999999997}]},{"osd":6,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65900000000000003,"15min":0.65700000000000003},"min":{"1min":0.51600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.52700000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.84799999999999998,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.86399999999999999}]},{"osd":7,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.68000000000000005,"15min":0.66700000000000004},"min":{"1min":0.52000000000000002,"5min":0.44,"15min":0.44},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.54300000000000004,"5min":0.434,"15min":0.434},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.91600000000000004}]},{"osd":9,"last update":"Thu Nov 16 18:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.90300000000000002}]},{"osd":10,"last update":"Thu Nov 16 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":1.016}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.999}]}]},{"osd":7,"up_from":49,"seq":210453397537,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6472,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738680,"statfs":{"total":95995035648,"available":95988408320,"internally_reserved":0,"allocated":983040,"data_stored":1052882,"data_compressed":47845,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.60699999999999998,"15min":0.59499999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81999999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81699999999999995,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.436,"15min":0.42499999999999999},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.47199999999999998,"15min":0.46300000000000002},"min":{"1min":0.28299999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.80500000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.71899999999999997}]},{"osd":2,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.626,"15min":0.61399999999999999},"min":{"1min":0.47999999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.63400000000000001,"15min":0.61699999999999999},"min":{"1min":0.46200000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.745}]},{"osd":3,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.83599999999999997}]},{"osd":4,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.55900000000000005,"15min":0.54600000000000004},"min":{"1min":0.36199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56299999999999994,"15min":0.55800000000000005},"min":{"1min":0.33400000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.68799999999999994,"15min":0.67500000000000004},"min":{"1min":0.52400000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.872},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.75900000000000001,"15min":0.749},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.79900000000000004}]},{"osd":6,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.70699999999999996,"15min":0.69699999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.73999999999999999,"15min":0.72999999999999998},"min":{"1min":0.44700000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0820000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.91400000000000003}]},{"osd":8,"last update":"Thu Nov 16 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.747,"15min":0.74099999999999999},"min":{"1min":0.501,"5min":0.433,"15min":0.433},"max":{"1min":1.0529999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.77900000000000003,"15min":0.77700000000000002},"min":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0409999999999999,"5min":1.107,"15min":1.107},"last":1.0049999999999999}]},{"osd":9,"last update":"Thu Nov 16 18:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.94599999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.98599999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]}]},{"osd":6,"up_from":44,"seq":188978561062,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6360,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":933888,"data_stored":462022,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.379,"15min":0.36299999999999999},"min":{"1min":0.34399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.39400000000000002,"15min":0.378},"min":{"1min":0.30199999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Nov 16 18:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.61199999999999999,"15min":0.59199999999999997},"min":{"1min":0.44900000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.54100000000000004,"15min":0.52900000000000003},"min":{"1min":0.38900000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.59599999999999997}]},{"osd":2,"last update":"Thu Nov 16 18:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53200000000000003,"15min":0.51900000000000002},"min":{"1min":0.442,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.76500000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.59099999999999997,"15min":0.56599999999999995},"min":{"1min":0.51100000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.503}]},{"osd":3,"last update":"Thu Nov 16 18:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.40300000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.443},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.45200000000000001,"15min":0.436},"min":{"1min":0.33100000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.45900000000000002}]},{"osd":4,"last update":"Thu Nov 16 18:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.50900000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94599999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.68700000000000006,"15min":0.66300000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.92000000000000004}]},{"osd":5,"last update":"Thu Nov 16 18:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.58799999999999997,"15min":0.57599999999999996},"min":{"1min":0.42999999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":1.004},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.63700000000000001,"15min":0.623},"min":{"1min":0.52400000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.83999999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.76200000000000001,"15min":0.752},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.79000000000000004,"15min":0.77600000000000002},"min":{"1min":0.57999999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.80800000000000005}]},{"osd":8,"last update":"Thu Nov 16 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.82399999999999995,"15min":0.82599999999999996},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.70199999999999996,"15min":0.70699999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85799999999999998,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.89500000000000002}]},{"osd":9,"last update":"Thu Nov 16 18:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.629},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.70299999999999996}]},{"osd":10,"last update":"Thu Nov 16 18:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.85699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7260,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737892,"statfs":{"total":95995035648,"available":95987601408,"internally_reserved":0,"allocated":937984,"data_stored":465453,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:31 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51800000000000002,"15min":0.53100000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53100000000000003,"15min":0.53700000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55800000000000005,"15min":0.53300000000000003},"min":{"1min":0.41099999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78100000000000003,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56999999999999995,"15min":0.54600000000000004},"min":{"1min":0.36199999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.74399999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Nov 16 18:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.57499999999999996,"15min":0.56299999999999994},"min":{"1min":0.435,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.57899999999999996,"15min":0.56799999999999995},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.80500000000000005,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Nov 16 18:22:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.42099999999999999,"15min":0.39800000000000002},"min":{"1min":0.32200000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.438,"15min":0.40799999999999997},"min":{"1min":0.33300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.438}]},{"osd":5,"last update":"Thu Nov 16 18:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.63600000000000001,"15min":0.624},"min":{"1min":0.49099999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85299999999999998,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.81999999999999995}]},{"osd":6,"last update":"Thu Nov 16 18:22:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.60799999999999998,"15min":0.59699999999999998},"min":{"1min":0.42099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.47999999999999998}]},{"osd":7,"last update":"Thu Nov 16 18:22:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.59399999999999997,"15min":0.57299999999999995},"min":{"1min":0.38700000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.63100000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.66800000000000004,"15min":0.65800000000000003},"min":{"1min":0.51900000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.751},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67000000000000004,"15min":0.66000000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.61799999999999999}]},{"osd":9,"last update":"Thu Nov 16 18:22:20 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.496},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.69399999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.66800000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7196,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":6272,"kb_avail":93737956,"statfs":{"total":95995035648,"available":95987666944,"internally_reserved":0,"allocated":937984,"data_stored":462353,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6422528},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.52500000000000002,"15min":0.49099999999999999},"min":{"1min":0.376,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.52300000000000002,"15min":0.48899999999999999},"min":{"1min":0.42399999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.69099999999999995}]},{"osd":2,"last update":"Thu Nov 16 18:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.64900000000000002,"15min":0.72899999999999998},"min":{"1min":0.379,"5min":0.371,"15min":0.371},"max":{"1min":0.81399999999999995,"5min":5.3650000000000002,"15min":5.3650000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.624,"15min":0.67000000000000004},"min":{"1min":0.374,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.872,"5min":3.4009999999999998,"15min":3.4009999999999998},"last":0.60699999999999998}]},{"osd":3,"last update":"Thu Nov 16 18:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55800000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.45100000000000001},"min":{"1min":0.32700000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.57299999999999995}]},{"osd":4,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.57999999999999996,"15min":0.55400000000000005},"min":{"1min":0.39800000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.57699999999999996,"15min":0.56699999999999995},"min":{"1min":0.41599999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89200000000000002,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.72699999999999998}]},{"osd":5,"last update":"Thu Nov 16 18:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58099999999999996,"15min":0.57499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82099999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.59999999999999998,"15min":0.58599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84099999999999997,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.55400000000000005,"15min":0.53300000000000003},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.44900000000000001}]},{"osd":7,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.95099999999999996,"5min":1.004,"15min":1.004},"last":0.82499999999999996}]},{"osd":8,"last update":"Thu Nov 16 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89600000000000002,"5min":1.052,"15min":1.052},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.96699999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.64900000000000002}]},{"osd":9,"last update":"Thu Nov 16 18:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.622},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.63}]},{"osd":10,"last update":"Thu Nov 16 18:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.81299999999999994}]},{"osd":11,"last update":"Thu Nov 16 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.73699999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7268,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737884,"statfs":{"total":95995035648,"available":95987593216,"internally_reserved":0,"allocated":946176,"data_stored":465760,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.50700000000000001,"15min":0.46999999999999997},"min":{"1min":0.30399999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.49399999999999999,"15min":0.46100000000000002},"min":{"1min":0.35599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56499999999999995}]},{"osd":1,"last update":"Thu Nov 16 18:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.53900000000000003,"15min":0.51300000000000001},"min":{"1min":0.33000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.56399999999999995,"15min":0.53900000000000003},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Nov 16 18:22:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.56000000000000005,"15min":0.53600000000000003},"min":{"1min":0.378,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.56399999999999995,"15min":0.53600000000000003},"min":{"1min":0.41499999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.72999999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.54500000000000004,"15min":0.52700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.75},{"interface":"front","average":{"1min":0.63,"5min":0.56299999999999994,"15min":0.54400000000000004},"min":{"1min":0.36799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Nov 16 18:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.44,"15min":0.41299999999999998},"min":{"1min":0.34699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.497,"15min":0.47099999999999997},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.61799999999999999}]},{"osd":6,"last update":"Thu Nov 16 18:22:15 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.627,"15min":0.61199999999999999},"min":{"1min":0.44700000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.63600000000000001,"15min":0.62},"min":{"1min":0.50600000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.95299999999999996}]},{"osd":7,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.61899999999999999,"15min":0.61399999999999999},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.81899999999999995}]},{"osd":8,"last update":"Thu Nov 16 18:22:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.71299999999999997}]},{"osd":9,"last update":"Thu Nov 16 18:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.84199999999999997}]},{"osd":10,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.93999999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182451,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6624,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738528,"statfs":{"total":95995035648,"available":95988252672,"internally_reserved":0,"allocated":942080,"data_stored":461943,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.379,"15min":0.36499999999999999},"min":{"1min":0.33700000000000002,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.39500000000000002,"15min":0.38600000000000001},"min":{"1min":0.30299999999999999,"5min":0.185,"15min":0.185},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.38300000000000001}]},{"osd":1,"last update":"Thu Nov 16 18:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.53800000000000003,"15min":0.52000000000000002},"min":{"1min":0.375,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.92400000000000004,"5min":1.129,"15min":1.129},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.55100000000000005,"15min":0.53300000000000003},"min":{"1min":0.437,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Nov 16 18:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.58399999999999996,"15min":0.56399999999999995},"min":{"1min":0.38100000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.41499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.873,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.61899999999999999,"15min":0.59699999999999998},"min":{"1min":0.501,"5min":0.378,"15min":0.378},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.627},"min":{"1min":0.47699999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.94799999999999995,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.68700000000000006}]},{"osd":5,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77400000000000002,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.35999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98799999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.505}]},{"osd":6,"last update":"Thu Nov 16 18:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.52100000000000002,"15min":0.48699999999999999},"min":{"1min":0.41899999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.33800000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.41199999999999998}]},{"osd":7,"last update":"Thu Nov 16 18:22:37 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.54100000000000004,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.621},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.70299999999999996,"15min":0.68899999999999995},"min":{"1min":0.45700000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.56599999999999995}]},{"osd":8,"last update":"Thu Nov 16 18:22:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.60799999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.48899999999999999}]},{"osd":10,"last update":"Thu Nov 16 18:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.64000000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953518,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6484,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738668,"statfs":{"total":95995035648,"available":95988396032,"internally_reserved":0,"allocated":929792,"data_stored":461681,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.504,"15min":0.48699999999999999},"min":{"1min":0.39900000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.752,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.48899999999999999,"15min":0.46400000000000002},"min":{"1min":0.41799999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.502}]},{"osd":1,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.39200000000000002,"15min":0.372},"min":{"1min":0.28799999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.42899999999999999,"15min":0.40999999999999998},"min":{"1min":0.312,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.434}]},{"osd":2,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.372,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.59599999999999997,"15min":0.56799999999999995},"min":{"1min":0.48599999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.55800000000000005,"15min":0.53500000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.54500000000000004,"15min":0.52700000000000002},"min":{"1min":0.40000000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80400000000000005,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.51700000000000002}]},{"osd":5,"last update":"Thu Nov 16 18:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.63900000000000001,"15min":0.60999999999999999},"min":{"1min":0.46500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.98599999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.65000000000000002,"15min":0.63400000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.72799999999999998}]},{"osd":6,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.46200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.749,"5min":0.65100000000000002,"15min":0.63500000000000001},"min":{"1min":0.499,"5min":0.432,"15min":0.432},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.53500000000000003}]},{"osd":7,"last update":"Thu Nov 16 18:22:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.58799999999999997,"15min":0.56299999999999994},"min":{"1min":0.37,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.55800000000000005,"15min":0.54700000000000004},"min":{"1min":0.443,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.56000000000000005}]},{"osd":8,"last update":"Thu Nov 16 18:23:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.84799999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.54500000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.91900000000000004,"5min":1.109,"15min":1.109},"last":0.624}]},{"osd":9,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.69299999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.67400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]},{"osd":5,"up_from":37,"seq":158913789993,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6608,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738544,"statfs":{"total":95995035648,"available":95988269056,"internally_reserved":0,"allocated":991232,"data_stored":1052482,"data_compressed":47845,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52400000000000002,"15min":0.51200000000000001},"min":{"1min":0.38500000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.751,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.501,"5min":0.59899999999999998,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64800000000000002,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.69199999999999995}]},{"osd":1,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53300000000000003,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.37,"15min":0.37},"max":{"1min":0.78600000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.52700000000000002,"15min":0.51400000000000001},"min":{"1min":0.36799999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.746,"5min":0.873,"15min":0.873},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39500000000000002,"15min":0.39200000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39600000000000002,"15min":0.38},"min":{"1min":0.28100000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84699999999999998,"5min":0.88,"15min":0.88},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.375,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56799999999999995,"15min":0.56299999999999994},"min":{"1min":0.41399999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82199999999999995,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.54900000000000004}]},{"osd":6,"last update":"Thu Nov 16 18:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.875,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59599999999999997,"15min":0.59499999999999997},"min":{"1min":0.47599999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.71099999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:22:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67100000000000004}]},{"osd":8,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.83599999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.753,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.505}]},{"osd":9,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.79600000000000004}]},{"osd":10,"last update":"Thu Nov 16 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.747}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34529,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T18:23:10.695 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T18:23:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.854+0000 7ff399d02700 1 -- 172.21.15.2:0/1603045433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3941004e0 msgr2=0x7ff3941008f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.854+0000 7ff399d02700 1 --2- 172.21.15.2:0/1603045433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3941004e0 0x7ff3941008f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff388009990 tx=0x7ff388009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.854+0000 7ff399d02700 1 -- 172.21.15.2:0/1603045433 shutdown_connections 2023-11-16T18:23:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.854+0000 7ff399d02700 1 --2- 172.21.15.2:0/1603045433 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff394102060 0x7ff394108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.854+0000 7ff399d02700 1 --2- 172.21.15.2:0/1603045433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3941004e0 0x7ff3941008f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.854+0000 7ff399d02700 1 --2- 172.21.15.2:0/1603045433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3941016d0 0x7ff394101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.854+0000 7ff399d02700 1 -- 172.21.15.2:0/1603045433 >> 172.21.15.2:0/1603045433 conn(0x7ff3940fbac0 msgr2=0x7ff3940fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.854+0000 7ff399d02700 1 -- 172.21.15.2:0/1603045433 shutdown_connections 2023-11-16T18:23:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.854+0000 7ff399d02700 1 -- 172.21.15.2:0/1603045433 wait complete. 2023-11-16T18:23:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.854+0000 7ff399d02700 1 Processor -- start 2023-11-16T18:23:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.854+0000 7ff399d02700 1 -- start start 2023-11-16T18:23:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff399d02700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3941004e0 0x7ff394196210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff399d02700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3941016d0 0x7ff394196750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff399d02700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff394102060 0x7ff39419d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff399d02700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39419dd90 con 0x7ff3941004e0 2023-11-16T18:23:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff399d02700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39419ded0 con 0x7ff394102060 2023-11-16T18:23:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff399d02700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39419e060 con 0x7ff3941016d0 2023-11-16T18:23:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff3937fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3941004e0 0x7ff394196210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff3937fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3941004e0 0x7ff394196210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48386/0 (socket says 172.21.15.2:48386) 2023-11-16T18:23:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff3937fe700 1 -- 172.21.15.2:0/2827735696 learned_addr learned my addr 172.21.15.2:0/2827735696 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:10.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff3937fe700 1 -- 172.21.15.2:0/2827735696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3941016d0 msgr2=0x7ff394196750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:10.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff3937fe700 1 --2- 172.21.15.2:0/2827735696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3941016d0 0x7ff394196750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff3937fe700 1 -- 172.21.15.2:0/2827735696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff394102060 msgr2=0x7ff39419d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:10.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff3937fe700 1 --2- 172.21.15.2:0/2827735696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff394102060 0x7ff39419d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:10.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff3937fe700 1 -- 172.21.15.2:0/2827735696 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3880095f0 con 0x7ff3941004e0 2023-11-16T18:23:10.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff3937fe700 1 --2- 172.21.15.2:0/2827735696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3941004e0 0x7ff394196210 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff388000c00 tx=0x7ff38800f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:10.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff390ff9700 1 -- 172.21.15.2:0/2827735696 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38801d070 con 0x7ff3941004e0 2023-11-16T18:23:10.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff390ff9700 1 -- 172.21.15.2:0/2827735696 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff38800fbe0 con 0x7ff3941004e0 2023-11-16T18:23:10.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff399d02700 1 -- 172.21.15.2:0/2827735696 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff39410be40 con 0x7ff3941004e0 2023-11-16T18:23:10.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff390ff9700 1 -- 172.21.15.2:0/2827735696 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff388017650 con 0x7ff3941004e0 2023-11-16T18:23:10.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.855+0000 7ff399d02700 1 -- 172.21.15.2:0/2827735696 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff39410bcb0 con 0x7ff3941004e0 2023-11-16T18:23:10.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.856+0000 7ff399d02700 1 -- 172.21.15.2:0/2827735696 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3941079d0 con 0x7ff3941004e0 2023-11-16T18:23:10.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.857+0000 7ff390ff9700 1 -- 172.21.15.2:0/2827735696 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7ff388022470 con 0x7ff3941004e0 2023-11-16T18:23:10.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.858+0000 7ff390ff9700 1 --2- 172.21.15.2:0/2827735696 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff37c0a1150 0x7ff37c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:10.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.858+0000 7ff392ffd700 1 --2- 172.21.15.2:0/2827735696 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff37c0a1150 0x7ff37c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:10.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.858+0000 7ff390ff9700 1 -- 172.21.15.2:0/2827735696 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff3880cdf40 con 0x7ff3941004e0 2023-11-16T18:23:10.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.858+0000 7ff392ffd700 1 --2- 172.21.15.2:0/2827735696 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff37c0a1150 0x7ff37c0a3600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff384005fd0 tx=0x7ff384005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:10.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.862+0000 7ff390ff9700 1 -- 172.21.15.2:0/2827735696 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff3880975c0 con 0x7ff3941004e0 2023-11-16T18:23:10.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:10.929+0000 7ff390ff9700 1 -- 172.21.15.2:0/2827735696 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ff388097990 con 0x7ff3941004e0 2023-11-16T18:23:11.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.062+0000 7ff399d02700 1 -- 172.21.15.2:0/2827735696 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff394060f30 con 0x7ff37c0a1150 2023-11-16T18:23:11.079 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.077+0000 7ff390ff9700 1 -- 172.21.15.2:0/2827735696 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422937 (secure 0 0 0) 0x7ff394060f30 con 0x7ff37c0a1150 2023-11-16T18:23:11.084 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:11.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.080+0000 7ff399d02700 1 -- 172.21.15.2:0/2827735696 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff37c0a1150 msgr2=0x7ff37c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:11.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.080+0000 7ff399d02700 1 --2- 172.21.15.2:0/2827735696 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff37c0a1150 0x7ff37c0a3600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff384005fd0 tx=0x7ff384005d00 comp rx=0 tx=0).stop 2023-11-16T18:23:11.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.080+0000 7ff399d02700 1 -- 172.21.15.2:0/2827735696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3941004e0 msgr2=0x7ff394196210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:11.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.080+0000 7ff399d02700 1 --2- 172.21.15.2:0/2827735696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3941004e0 0x7ff394196210 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff388000c00 tx=0x7ff38800f510 comp rx=0 tx=0).stop 2023-11-16T18:23:11.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.080+0000 7ff399d02700 1 -- 172.21.15.2:0/2827735696 shutdown_connections 2023-11-16T18:23:11.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.080+0000 7ff399d02700 1 --2- 172.21.15.2:0/2827735696 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff37c0a1150 0x7ff37c0a3600 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:11.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.080+0000 7ff399d02700 1 --2- 172.21.15.2:0/2827735696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff394102060 0x7ff39419d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:11.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.080+0000 7ff399d02700 1 --2- 172.21.15.2:0/2827735696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3941004e0 0x7ff394196210 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:11.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.080+0000 7ff399d02700 1 --2- 172.21.15.2:0/2827735696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3941016d0 0x7ff394196750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:11.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.080+0000 7ff399d02700 1 -- 172.21.15.2:0/2827735696 >> 172.21.15.2:0/2827735696 conn(0x7ff3940fbac0 msgr2=0x7ff394104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:11.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.080+0000 7ff399d02700 1 -- 172.21.15.2:0/2827735696 shutdown_connections 2023-11-16T18:23:11.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:11.081+0000 7ff399d02700 1 -- 172.21.15.2:0/2827735696 wait complete. 2023-11-16T18:23:11.090 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-16T18:23:11.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:10 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1188611400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:23:11.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:10 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1643514571' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:11.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:10 smithi002 ceph-mon[98807]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:23:11.095 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2023-11-16T18:23:09.806106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596430,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":142,"num_read_kb":125,"num_write":264,"num_write_kb":2971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79220,"kb_used_data":11156,"kb_used_omap":0,"kb_used_meta":67968,"kb_avail":1124862604,"statfs":{"total":1151940427776,"available":1151859306496,"internally_reserved":0,"allocated":11423744,"data_stored":7328562,"data_compressed":248619,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":0,"internal_metadata":69599232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3700,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":38,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003438"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635790+0000","last_change":"2023-11-16T18:22:46.829814+0000","last_active":"2023-11-16T18:22:51.635790+0000","last_peered":"2023-11-16T18:22:51.635790+0000","last_clean":"2023-11-16T18:22:51.635790+0000","last_became_active":"2023-11-16T18:22:46.829407+0000","last_became_peered":"2023-11-16T18:22:46.829407+0000","last_unstale":"2023-11-16T18:22:51.635790+0000","last_undegraded":"2023-11-16T18:22:51.635790+0000","last_fullsized":"2023-11-16T18:22:51.635790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:58:53.882493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583289+0000","last_change":"2023-11-16T18:22:46.818700+0000","last_active":"2023-11-16T18:22:51.583289+0000","last_peered":"2023-11-16T18:22:51.583289+0000","last_clean":"2023-11-16T18:22:51.583289+0000","last_became_active":"2023-11-16T18:22:46.818388+0000","last_became_peered":"2023-11-16T18:22:46.818388+0000","last_unstale":"2023-11-16T18:22:51.583289+0000","last_undegraded":"2023-11-16T18:22:51.583289+0000","last_fullsized":"2023-11-16T18:22:51.583289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:53:39.888834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582382+0000","last_change":"2023-11-16T18:22:46.827732+0000","last_active":"2023-11-16T18:22:51.582382+0000","last_peered":"2023-11-16T18:22:51.582382+0000","last_clean":"2023-11-16T18:22:51.582382+0000","last_became_active":"2023-11-16T18:22:46.827328+0000","last_became_peered":"2023-11-16T18:22:46.827328+0000","last_unstale":"2023-11-16T18:22:51.582382+0000","last_undegraded":"2023-11-16T18:22:51.582382+0000","last_fullsized":"2023-11-16T18:22:51.582382+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:36:48.387249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584389+0000","last_change":"2023-11-16T18:22:46.826837+0000","last_active":"2023-11-16T18:22:51.584389+0000","last_peered":"2023-11-16T18:22:51.584389+0000","last_clean":"2023-11-16T18:22:51.584389+0000","last_became_active":"2023-11-16T18:22:46.826532+0000","last_became_peered":"2023-11-16T18:22:46.826532+0000","last_unstale":"2023-11-16T18:22:51.584389+0000","last_undegraded":"2023-11-16T18:22:51.584389+0000","last_fullsized":"2023-11-16T18:22:51.584389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:28:11.423559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582313+0000","last_change":"2023-11-16T18:22:46.841164+0000","last_active":"2023-11-16T18:22:51.582313+0000","last_peered":"2023-11-16T18:22:51.582313+0000","last_clean":"2023-11-16T18:22:51.582313+0000","last_became_active":"2023-11-16T18:22:46.840855+0000","last_became_peered":"2023-11-16T18:22:46.840855+0000","last_unstale":"2023-11-16T18:22:51.582313+0000","last_undegraded":"2023-11-16T18:22:51.582313+0000","last_fullsized":"2023-11-16T18:22:51.582313+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:15:32.916615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583589+0000","last_change":"2023-11-16T18:22:46.822171+0000","last_active":"2023-11-16T18:22:51.583589+0000","last_peered":"2023-11-16T18:22:51.583589+0000","last_clean":"2023-11-16T18:22:51.583589+0000","last_became_active":"2023-11-16T18:22:46.821860+0000","last_became_peered":"2023-11-16T18:22:46.821860+0000","last_unstale":"2023-11-16T18:22:51.583589+0000","last_undegraded":"2023-11-16T18:22:51.583589+0000","last_fullsized":"2023-11-16T18:22:51.583589+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:16:33.605016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583209+0000","last_change":"2023-11-16T18:22:46.832463+0000","last_active":"2023-11-16T18:22:51.583209+0000","last_peered":"2023-11-16T18:22:51.583209+0000","last_clean":"2023-11-16T18:22:51.583209+0000","last_became_active":"2023-11-16T18:22:46.832141+0000","last_became_peered":"2023-11-16T18:22:46.832141+0000","last_unstale":"2023-11-16T18:22:51.583209+0000","last_undegraded":"2023-11-16T18:22:51.583209+0000","last_fullsized":"2023-11-16T18:22:51.583209+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:39:23.533316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.537754+0000","last_change":"2023-11-16T18:22:46.844239+0000","last_active":"2023-11-16T18:22:51.537754+0000","last_peered":"2023-11-16T18:22:51.537754+0000","last_clean":"2023-11-16T18:22:51.537754+0000","last_became_active":"2023-11-16T18:22:46.843906+0000","last_became_peered":"2023-11-16T18:22:46.843906+0000","last_unstale":"2023-11-16T18:22:51.537754+0000","last_undegraded":"2023-11-16T18:22:51.537754+0000","last_fullsized":"2023-11-16T18:22:51.537754+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:31:34.112452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.540316+0000","last_change":"2023-11-16T18:22:46.834625+0000","last_active":"2023-11-16T18:22:51.540316+0000","last_peered":"2023-11-16T18:22:51.540316+0000","last_clean":"2023-11-16T18:22:51.540316+0000","last_became_active":"2023-11-16T18:22:46.834145+0000","last_became_peered":"2023-11-16T18:22:46.834145+0000","last_unstale":"2023-11-16T18:22:51.540316+0000","last_undegraded":"2023-11-16T18:22:51.540316+0000","last_fullsized":"2023-11-16T18:22:51.540316+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:59:35.306111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584001+0000","last_change":"2023-11-16T18:22:46.827569+0000","last_active":"2023-11-16T18:22:51.584001+0000","last_peered":"2023-11-16T18:22:51.584001+0000","last_clean":"2023-11-16T18:22:51.584001+0000","last_became_active":"2023-11-16T18:22:46.827234+0000","last_became_peered":"2023-11-16T18:22:46.827234+0000","last_unstale":"2023-11-16T18:22:51.584001+0000","last_undegraded":"2023-11-16T18:22:51.584001+0000","last_fullsized":"2023-11-16T18:22:51.584001+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:37:08.465042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.585080+0000","last_change":"2023-11-16T18:22:46.833701+0000","last_active":"2023-11-16T18:22:51.585080+0000","last_peered":"2023-11-16T18:22:51.585080+0000","last_clean":"2023-11-16T18:22:51.585080+0000","last_became_active":"2023-11-16T18:22:46.833333+0000","last_became_peered":"2023-11-16T18:22:46.833333+0000","last_unstale":"2023-11-16T18:22:51.585080+0000","last_undegraded":"2023-11-16T18:22:51.585080+0000","last_fullsized":"2023-11-16T18:22:51.585080+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:39:03.487707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584461+0000","last_change":"2023-11-16T18:22:46.823222+0000","last_active":"2023-11-16T18:22:51.584461+0000","last_peered":"2023-11-16T18:22:51.584461+0000","last_clean":"2023-11-16T18:22:51.584461+0000","last_became_active":"2023-11-16T18:22:46.822224+0000","last_became_peered":"2023-11-16T18:22:46.822224+0000","last_unstale":"2023-11-16T18:22:51.584461+0000","last_undegraded":"2023-11-16T18:22:51.584461+0000","last_fullsized":"2023-11-16T18:22:51.584461+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:55:47.913232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583763+0000","last_change":"2023-11-16T18:22:46.818631+0000","last_active":"2023-11-16T18:22:51.583763+0000","last_peered":"2023-11-16T18:22:51.583763+0000","last_clean":"2023-11-16T18:22:51.583763+0000","last_became_active":"2023-11-16T18:22:46.818305+0000","last_became_peered":"2023-11-16T18:22:46.818305+0000","last_unstale":"2023-11-16T18:22:51.583763+0000","last_undegraded":"2023-11-16T18:22:51.583763+0000","last_fullsized":"2023-11-16T18:22:51.583763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:42:03.259127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.585367+0000","last_change":"2023-11-16T18:22:46.825439+0000","last_active":"2023-11-16T18:22:51.585367+0000","last_peered":"2023-11-16T18:22:51.585367+0000","last_clean":"2023-11-16T18:22:51.585367+0000","last_became_active":"2023-11-16T18:22:46.825062+0000","last_became_peered":"2023-11-16T18:22:46.825062+0000","last_unstale":"2023-11-16T18:22:51.585367+0000","last_undegraded":"2023-11-16T18:22:51.585367+0000","last_fullsized":"2023-11-16T18:22:51.585367+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:20:36.560885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582640+0000","last_change":"2023-11-16T18:22:46.827730+0000","last_active":"2023-11-16T18:22:51.582640+0000","last_peered":"2023-11-16T18:22:51.582640+0000","last_clean":"2023-11-16T18:22:51.582640+0000","last_became_active":"2023-11-16T18:22:46.827416+0000","last_became_peered":"2023-11-16T18:22:46.827416+0000","last_unstale":"2023-11-16T18:22:51.582640+0000","last_undegraded":"2023-11-16T18:22:51.582640+0000","last_fullsized":"2023-11-16T18:22:51.582640+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:28:41.500983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.539430+0000","last_change":"2023-11-16T18:22:46.834259+0000","last_active":"2023-11-16T18:22:51.539430+0000","last_peered":"2023-11-16T18:22:51.539430+0000","last_clean":"2023-11-16T18:22:51.539430+0000","last_became_active":"2023-11-16T18:22:46.833775+0000","last_became_peered":"2023-11-16T18:22:46.833775+0000","last_unstale":"2023-11-16T18:22:51.539430+0000","last_undegraded":"2023-11-16T18:22:51.539430+0000","last_fullsized":"2023-11-16T18:22:51.539430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:24:52.545621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635645+0000","last_change":"2023-11-16T18:22:46.829475+0000","last_active":"2023-11-16T18:22:51.635645+0000","last_peered":"2023-11-16T18:22:51.635645+0000","last_clean":"2023-11-16T18:22:51.635645+0000","last_became_active":"2023-11-16T18:22:46.829157+0000","last_became_peered":"2023-11-16T18:22:46.829157+0000","last_unstale":"2023-11-16T18:22:51.635645+0000","last_undegraded":"2023-11-16T18:22:51.635645+0000","last_fullsized":"2023-11-16T18:22:51.635645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:01:49.857829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584839+0000","last_change":"2023-11-16T18:22:46.821716+0000","last_active":"2023-11-16T18:22:51.584839+0000","last_peered":"2023-11-16T18:22:51.584839+0000","last_clean":"2023-11-16T18:22:51.584839+0000","last_became_active":"2023-11-16T18:22:46.821393+0000","last_became_peered":"2023-11-16T18:22:46.821393+0000","last_unstale":"2023-11-16T18:22:51.584839+0000","last_undegraded":"2023-11-16T18:22:51.584839+0000","last_fullsized":"2023-11-16T18:22:51.584839+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:39:50.644126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583545+0000","last_change":"2023-11-16T18:22:46.837433+0000","last_active":"2023-11-16T18:22:51.583545+0000","last_peered":"2023-11-16T18:22:51.583545+0000","last_clean":"2023-11-16T18:22:51.583545+0000","last_became_active":"2023-11-16T18:22:46.837143+0000","last_became_peered":"2023-11-16T18:22:46.837143+0000","last_unstale":"2023-11-16T18:22:51.583545+0000","last_undegraded":"2023-11-16T18:22:51.583545+0000","last_fullsized":"2023-11-16T18:22:51.583545+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:27:52.152539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582661+0000","last_change":"2023-11-16T18:22:51.123749+0000","last_active":"2023-11-16T18:22:51.582661+0000","last_peered":"2023-11-16T18:22:51.582661+0000","last_clean":"2023-11-16T18:22:51.582661+0000","last_became_active":"2023-11-16T18:22:51.123185+0000","last_became_peered":"2023-11-16T18:22:51.123185+0000","last_unstale":"2023-11-16T18:22:51.582661+0000","last_undegraded":"2023-11-16T18:22:51.582661+0000","last_fullsized":"2023-11-16T18:22:51.582661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:48:41.870442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'4","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.585154+0000","last_change":"2023-11-16T18:22:46.385440+0000","last_active":"2023-11-16T18:22:51.585154+0000","last_peered":"2023-11-16T18:22:51.585154+0000","last_clean":"2023-11-16T18:22:51.585154+0000","last_became_active":"2023-11-16T18:22:46.385098+0000","last_became_peered":"2023-11-16T18:22:46.385098+0000","last_unstale":"2023-11-16T18:22:51.585154+0000","last_undegraded":"2023-11-16T18:22:51.585154+0000","last_fullsized":"2023-11-16T18:22:51.585154+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:27:11.799990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538546+0000","last_change":"2023-11-16T18:22:46.842681+0000","last_active":"2023-11-16T18:22:51.538546+0000","last_peered":"2023-11-16T18:22:51.538546+0000","last_clean":"2023-11-16T18:22:51.538546+0000","last_became_active":"2023-11-16T18:22:46.842408+0000","last_became_peered":"2023-11-16T18:22:46.842408+0000","last_unstale":"2023-11-16T18:22:51.538546+0000","last_undegraded":"2023-11-16T18:22:51.538546+0000","last_fullsized":"2023-11-16T18:22:51.538546+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:21:02.462535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582828+0000","last_change":"2023-11-16T18:22:51.123450+0000","last_active":"2023-11-16T18:22:51.582828+0000","last_peered":"2023-11-16T18:22:51.582828+0000","last_clean":"2023-11-16T18:22:51.582828+0000","last_became_active":"2023-11-16T18:22:51.123002+0000","last_became_peered":"2023-11-16T18:22:51.123002+0000","last_unstale":"2023-11-16T18:22:51.582828+0000","last_undegraded":"2023-11-16T18:22:51.582828+0000","last_fullsized":"2023-11-16T18:22:51.582828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:04:27.030138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538660+0000","last_change":"2023-11-16T18:22:46.384768+0000","last_active":"2023-11-16T18:22:51.538660+0000","last_peered":"2023-11-16T18:22:51.538660+0000","last_clean":"2023-11-16T18:22:51.538660+0000","last_became_active":"2023-11-16T18:22:46.384246+0000","last_became_peered":"2023-11-16T18:22:46.384246+0000","last_unstale":"2023-11-16T18:22:51.538660+0000","last_undegraded":"2023-11-16T18:22:51.538660+0000","last_fullsized":"2023-11-16T18:22:51.538660+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:26:21.800521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583368+0000","last_change":"2023-11-16T18:22:46.826069+0000","last_active":"2023-11-16T18:22:51.583368+0000","last_peered":"2023-11-16T18:22:51.583368+0000","last_clean":"2023-11-16T18:22:51.583368+0000","last_became_active":"2023-11-16T18:22:46.824676+0000","last_became_peered":"2023-11-16T18:22:46.824676+0000","last_unstale":"2023-11-16T18:22:51.583368+0000","last_undegraded":"2023-11-16T18:22:51.583368+0000","last_fullsized":"2023-11-16T18:22:51.583368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:48:03.420408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584224+0000","last_change":"2023-11-16T18:22:51.127529+0000","last_active":"2023-11-16T18:22:51.584224+0000","last_peered":"2023-11-16T18:22:51.584224+0000","last_clean":"2023-11-16T18:22:51.584224+0000","last_became_active":"2023-11-16T18:22:51.126748+0000","last_became_peered":"2023-11-16T18:22:51.126748+0000","last_unstale":"2023-11-16T18:22:51.584224+0000","last_undegraded":"2023-11-16T18:22:51.584224+0000","last_fullsized":"2023-11-16T18:22:51.584224+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:31:01.008519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:06.912752+0000","last_change":"2023-11-16T18:22:46.385890+0000","last_active":"2023-11-16T18:23:06.912752+0000","last_peered":"2023-11-16T18:23:06.912752+0000","last_clean":"2023-11-16T18:23:06.912752+0000","last_became_active":"2023-11-16T18:22:46.385568+0000","last_became_peered":"2023-11-16T18:22:46.385568+0000","last_unstale":"2023-11-16T18:23:06.912752+0000","last_undegraded":"2023-11-16T18:23:06.912752+0000","last_fullsized":"2023-11-16T18:23:06.912752+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:16:48.773546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.539902+0000","last_change":"2023-11-16T18:22:46.824796+0000","last_active":"2023-11-16T18:22:51.539902+0000","last_peered":"2023-11-16T18:22:51.539902+0000","last_clean":"2023-11-16T18:22:51.539902+0000","last_became_active":"2023-11-16T18:22:46.823895+0000","last_became_peered":"2023-11-16T18:22:46.823895+0000","last_unstale":"2023-11-16T18:22:51.539902+0000","last_undegraded":"2023-11-16T18:22:51.539902+0000","last_fullsized":"2023-11-16T18:22:51.539902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:40:21.122081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582786+0000","last_change":"2023-11-16T18:22:51.118714+0000","last_active":"2023-11-16T18:22:51.582786+0000","last_peered":"2023-11-16T18:22:51.582786+0000","last_clean":"2023-11-16T18:22:51.582786+0000","last_became_active":"2023-11-16T18:22:51.118216+0000","last_became_peered":"2023-11-16T18:22:51.118216+0000","last_unstale":"2023-11-16T18:22:51.582786+0000","last_undegraded":"2023-11-16T18:22:51.582786+0000","last_fullsized":"2023-11-16T18:22:51.582786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:12:26.790005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582618+0000","last_change":"2023-11-16T18:22:46.376431+0000","last_active":"2023-11-16T18:22:51.582618+0000","last_peered":"2023-11-16T18:22:51.582618+0000","last_clean":"2023-11-16T18:22:51.582618+0000","last_became_active":"2023-11-16T18:22:46.375914+0000","last_became_peered":"2023-11-16T18:22:46.375914+0000","last_unstale":"2023-11-16T18:22:51.582618+0000","last_undegraded":"2023-11-16T18:22:51.582618+0000","last_fullsized":"2023-11-16T18:22:51.582618+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:30:55.925400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.542071+0000","last_change":"2023-11-16T18:22:46.825002+0000","last_active":"2023-11-16T18:22:51.542071+0000","last_peered":"2023-11-16T18:22:51.542071+0000","last_clean":"2023-11-16T18:22:51.542071+0000","last_became_active":"2023-11-16T18:22:46.824012+0000","last_became_peered":"2023-11-16T18:22:46.824012+0000","last_unstale":"2023-11-16T18:22:51.542071+0000","last_undegraded":"2023-11-16T18:22:51.542071+0000","last_fullsized":"2023-11-16T18:22:51.542071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:23:47.920490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583388+0000","last_change":"2023-11-16T18:22:51.117623+0000","last_active":"2023-11-16T18:22:51.583388+0000","last_peered":"2023-11-16T18:22:51.583388+0000","last_clean":"2023-11-16T18:22:51.583388+0000","last_became_active":"2023-11-16T18:22:51.117183+0000","last_became_peered":"2023-11-16T18:22:51.117183+0000","last_unstale":"2023-11-16T18:22:51.583388+0000","last_undegraded":"2023-11-16T18:22:51.583388+0000","last_fullsized":"2023-11-16T18:22:51.583388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:47:56.989813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:05.905304+0000","last_change":"2023-11-16T18:22:46.384920+0000","last_active":"2023-11-16T18:23:05.905304+0000","last_peered":"2023-11-16T18:23:05.905304+0000","last_clean":"2023-11-16T18:23:05.905304+0000","last_became_active":"2023-11-16T18:22:46.384577+0000","last_became_peered":"2023-11-16T18:22:46.384577+0000","last_unstale":"2023-11-16T18:23:05.905304+0000","last_undegraded":"2023-11-16T18:23:05.905304+0000","last_fullsized":"2023-11-16T18:23:05.905304+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:12:39.928656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582483+0000","last_change":"2023-11-16T18:22:46.820073+0000","last_active":"2023-11-16T18:22:51.582483+0000","last_peered":"2023-11-16T18:22:51.582483+0000","last_clean":"2023-11-16T18:22:51.582483+0000","last_became_active":"2023-11-16T18:22:46.819764+0000","last_became_peered":"2023-11-16T18:22:46.819764+0000","last_unstale":"2023-11-16T18:22:51.582483+0000","last_undegraded":"2023-11-16T18:22:51.582483+0000","last_fullsized":"2023-11-16T18:22:51.582483+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:08.246496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583304+0000","last_change":"2023-11-16T18:22:51.116619+0000","last_active":"2023-11-16T18:22:51.583304+0000","last_peered":"2023-11-16T18:22:51.583304+0000","last_clean":"2023-11-16T18:22:51.583304+0000","last_became_active":"2023-11-16T18:22:51.116264+0000","last_became_peered":"2023-11-16T18:22:51.116264+0000","last_unstale":"2023-11-16T18:22:51.583304+0000","last_undegraded":"2023-11-16T18:22:51.583304+0000","last_fullsized":"2023-11-16T18:22:51.583304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:33:21.580177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583423+0000","last_change":"2023-11-16T18:22:46.389590+0000","last_active":"2023-11-16T18:22:51.583423+0000","last_peered":"2023-11-16T18:22:51.583423+0000","last_clean":"2023-11-16T18:22:51.583423+0000","last_became_active":"2023-11-16T18:22:46.389354+0000","last_became_peered":"2023-11-16T18:22:46.389354+0000","last_unstale":"2023-11-16T18:22:51.583423+0000","last_undegraded":"2023-11-16T18:22:51.583423+0000","last_fullsized":"2023-11-16T18:22:51.583423+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:32:08.189457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541898+0000","last_change":"2023-11-16T18:22:46.841106+0000","last_active":"2023-11-16T18:22:51.541898+0000","last_peered":"2023-11-16T18:22:51.541898+0000","last_clean":"2023-11-16T18:22:51.541898+0000","last_became_active":"2023-11-16T18:22:46.840876+0000","last_became_peered":"2023-11-16T18:22:46.840876+0000","last_unstale":"2023-11-16T18:22:51.541898+0000","last_undegraded":"2023-11-16T18:22:51.541898+0000","last_fullsized":"2023-11-16T18:22:51.541898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:28:37.411931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635709+0000","last_change":"2023-11-16T18:22:51.125813+0000","last_active":"2023-11-16T18:22:51.635709+0000","last_peered":"2023-11-16T18:22:51.635709+0000","last_clean":"2023-11-16T18:22:51.635709+0000","last_became_active":"2023-11-16T18:22:51.125277+0000","last_became_peered":"2023-11-16T18:22:51.125277+0000","last_unstale":"2023-11-16T18:22:51.635709+0000","last_undegraded":"2023-11-16T18:22:51.635709+0000","last_fullsized":"2023-11-16T18:22:51.635709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:26:22.312004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:06.914290+0000","last_change":"2023-11-16T18:22:46.381208+0000","last_active":"2023-11-16T18:23:06.914290+0000","last_peered":"2023-11-16T18:23:06.914290+0000","last_clean":"2023-11-16T18:23:06.914290+0000","last_became_active":"2023-11-16T18:22:46.380351+0000","last_became_peered":"2023-11-16T18:22:46.380351+0000","last_unstale":"2023-11-16T18:23:06.914290+0000","last_undegraded":"2023-11-16T18:23:06.914290+0000","last_fullsized":"2023-11-16T18:23:06.914290+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:16:24.398001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.537912+0000","last_change":"2023-11-16T18:22:46.843043+0000","last_active":"2023-11-16T18:22:51.537912+0000","last_peered":"2023-11-16T18:22:51.537912+0000","last_clean":"2023-11-16T18:22:51.537912+0000","last_became_active":"2023-11-16T18:22:46.841522+0000","last_became_peered":"2023-11-16T18:22:46.841522+0000","last_unstale":"2023-11-16T18:22:51.537912+0000","last_undegraded":"2023-11-16T18:22:51.537912+0000","last_fullsized":"2023-11-16T18:22:51.537912+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:51:42.615558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583266+0000","last_change":"2023-11-16T18:22:51.127729+0000","last_active":"2023-11-16T18:22:51.583266+0000","last_peered":"2023-11-16T18:22:51.583266+0000","last_clean":"2023-11-16T18:22:51.583266+0000","last_became_active":"2023-11-16T18:22:51.127303+0000","last_became_peered":"2023-11-16T18:22:51.127303+0000","last_unstale":"2023-11-16T18:22:51.583266+0000","last_undegraded":"2023-11-16T18:22:51.583266+0000","last_fullsized":"2023-11-16T18:22:51.583266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:15:44.768952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541434+0000","last_change":"2023-11-16T18:22:46.377437+0000","last_active":"2023-11-16T18:22:51.541434+0000","last_peered":"2023-11-16T18:22:51.541434+0000","last_clean":"2023-11-16T18:22:51.541434+0000","last_became_active":"2023-11-16T18:22:46.377120+0000","last_became_peered":"2023-11-16T18:22:46.377120+0000","last_unstale":"2023-11-16T18:22:51.541434+0000","last_undegraded":"2023-11-16T18:22:51.541434+0000","last_fullsized":"2023-11-16T18:22:51.541434+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:56:43.302512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583817+0000","last_change":"2023-11-16T18:22:46.837804+0000","last_active":"2023-11-16T18:22:51.583817+0000","last_peered":"2023-11-16T18:22:51.583817+0000","last_clean":"2023-11-16T18:22:51.583817+0000","last_became_active":"2023-11-16T18:22:46.837432+0000","last_became_peered":"2023-11-16T18:22:46.837432+0000","last_unstale":"2023-11-16T18:22:51.583817+0000","last_undegraded":"2023-11-16T18:22:51.583817+0000","last_fullsized":"2023-11-16T18:22:51.583817+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:11:41.766949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538026+0000","last_change":"2023-11-16T18:22:51.127546+0000","last_active":"2023-11-16T18:22:51.538026+0000","last_peered":"2023-11-16T18:22:51.538026+0000","last_clean":"2023-11-16T18:22:51.538026+0000","last_became_active":"2023-11-16T18:22:51.127133+0000","last_became_peered":"2023-11-16T18:22:51.127133+0000","last_unstale":"2023-11-16T18:22:51.538026+0000","last_undegraded":"2023-11-16T18:22:51.538026+0000","last_fullsized":"2023-11-16T18:22:51.538026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:04:16.775399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584207+0000","last_change":"2023-11-16T18:22:46.386309+0000","last_active":"2023-11-16T18:22:51.584207+0000","last_peered":"2023-11-16T18:22:51.584207+0000","last_clean":"2023-11-16T18:22:51.584207+0000","last_became_active":"2023-11-16T18:22:46.385960+0000","last_became_peered":"2023-11-16T18:22:46.385960+0000","last_unstale":"2023-11-16T18:22:51.584207+0000","last_undegraded":"2023-11-16T18:22:51.584207+0000","last_fullsized":"2023-11-16T18:22:51.584207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:40:57.568161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.540289+0000","last_change":"2023-11-16T18:22:46.827211+0000","last_active":"2023-11-16T18:22:51.540289+0000","last_peered":"2023-11-16T18:22:51.540289+0000","last_clean":"2023-11-16T18:22:51.540289+0000","last_became_active":"2023-11-16T18:22:46.826396+0000","last_became_peered":"2023-11-16T18:22:46.826396+0000","last_unstale":"2023-11-16T18:22:51.540289+0000","last_undegraded":"2023-11-16T18:22:51.540289+0000","last_fullsized":"2023-11-16T18:22:51.540289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:27:52.671151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583449+0000","last_change":"2023-11-16T18:22:51.124816+0000","last_active":"2023-11-16T18:22:51.583449+0000","last_peered":"2023-11-16T18:22:51.583449+0000","last_clean":"2023-11-16T18:22:51.583449+0000","last_became_active":"2023-11-16T18:22:51.124296+0000","last_became_peered":"2023-11-16T18:22:51.124296+0000","last_unstale":"2023-11-16T18:22:51.583449+0000","last_undegraded":"2023-11-16T18:22:51.583449+0000","last_fullsized":"2023-11-16T18:22:51.583449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:47:31.452408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:05.904769+0000","last_change":"2023-11-16T18:22:46.384680+0000","last_active":"2023-11-16T18:23:05.904769+0000","last_peered":"2023-11-16T18:23:05.904769+0000","last_clean":"2023-11-16T18:23:05.904769+0000","last_became_active":"2023-11-16T18:22:46.384290+0000","last_became_peered":"2023-11-16T18:22:46.384290+0000","last_unstale":"2023-11-16T18:23:05.904769+0000","last_undegraded":"2023-11-16T18:23:05.904769+0000","last_fullsized":"2023-11-16T18:23:05.904769+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:28:49.556854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583498+0000","last_change":"2023-11-16T18:22:46.834726+0000","last_active":"2023-11-16T18:22:51.583498+0000","last_peered":"2023-11-16T18:22:51.583498+0000","last_clean":"2023-11-16T18:22:51.583498+0000","last_became_active":"2023-11-16T18:22:46.834424+0000","last_became_peered":"2023-11-16T18:22:46.834424+0000","last_unstale":"2023-11-16T18:22:51.583498+0000","last_undegraded":"2023-11-16T18:22:51.583498+0000","last_fullsized":"2023-11-16T18:22:51.583498+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:12:05.088718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584404+0000","last_change":"2023-11-16T18:22:51.125178+0000","last_active":"2023-11-16T18:22:51.584404+0000","last_peered":"2023-11-16T18:22:51.584404+0000","last_clean":"2023-11-16T18:22:51.584404+0000","last_became_active":"2023-11-16T18:22:51.124778+0000","last_became_peered":"2023-11-16T18:22:51.124778+0000","last_unstale":"2023-11-16T18:22:51.584404+0000","last_undegraded":"2023-11-16T18:22:51.584404+0000","last_fullsized":"2023-11-16T18:22:51.584404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:20:51.981074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541968+0000","last_change":"2023-11-16T18:22:46.380804+0000","last_active":"2023-11-16T18:22:51.541968+0000","last_peered":"2023-11-16T18:22:51.541968+0000","last_clean":"2023-11-16T18:22:51.541968+0000","last_became_active":"2023-11-16T18:22:46.380477+0000","last_became_peered":"2023-11-16T18:22:46.380477+0000","last_unstale":"2023-11-16T18:22:51.541968+0000","last_undegraded":"2023-11-16T18:22:51.541968+0000","last_fullsized":"2023-11-16T18:22:51.541968+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:19:26.230982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.540026+0000","last_change":"2023-11-16T18:22:46.825022+0000","last_active":"2023-11-16T18:22:51.540026+0000","last_peered":"2023-11-16T18:22:51.540026+0000","last_clean":"2023-11-16T18:22:51.540026+0000","last_became_active":"2023-11-16T18:22:46.824505+0000","last_became_peered":"2023-11-16T18:22:46.824505+0000","last_unstale":"2023-11-16T18:22:51.540026+0000","last_undegraded":"2023-11-16T18:22:51.540026+0000","last_fullsized":"2023-11-16T18:22:51.540026+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:01:52.602465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582852+0000","last_change":"2023-11-16T18:22:51.119711+0000","last_active":"2023-11-16T18:22:51.582852+0000","last_peered":"2023-11-16T18:22:51.582852+0000","last_clean":"2023-11-16T18:22:51.582852+0000","last_became_active":"2023-11-16T18:22:51.119304+0000","last_became_peered":"2023-11-16T18:22:51.119304+0000","last_unstale":"2023-11-16T18:22:51.582852+0000","last_undegraded":"2023-11-16T18:22:51.582852+0000","last_fullsized":"2023-11-16T18:22:51.582852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:06:35.490333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"86'10","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:02.838985+0000","last_change":"2023-11-16T18:22:46.382915+0000","last_active":"2023-11-16T18:23:02.838985+0000","last_peered":"2023-11-16T18:23:02.838985+0000","last_clean":"2023-11-16T18:23:02.838985+0000","last_became_active":"2023-11-16T18:22:46.382601+0000","last_became_peered":"2023-11-16T18:22:46.382601+0000","last_unstale":"2023-11-16T18:23:02.838985+0000","last_undegraded":"2023-11-16T18:23:02.838985+0000","last_fullsized":"2023-11-16T18:23:02.838985+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:02:18.799978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582729+0000","last_change":"2023-11-16T18:22:46.833537+0000","last_active":"2023-11-16T18:22:51.582729+0000","last_peered":"2023-11-16T18:22:51.582729+0000","last_clean":"2023-11-16T18:22:51.582729+0000","last_became_active":"2023-11-16T18:22:46.833217+0000","last_became_peered":"2023-11-16T18:22:46.833217+0000","last_unstale":"2023-11-16T18:22:51.582729+0000","last_undegraded":"2023-11-16T18:22:51.582729+0000","last_fullsized":"2023-11-16T18:22:51.582729+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:17:49.309028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.542410+0000","last_change":"2023-11-16T18:22:51.121091+0000","last_active":"2023-11-16T18:22:51.542410+0000","last_peered":"2023-11-16T18:22:51.542410+0000","last_clean":"2023-11-16T18:22:51.542410+0000","last_became_active":"2023-11-16T18:22:51.120656+0000","last_became_peered":"2023-11-16T18:22:51.120656+0000","last_unstale":"2023-11-16T18:22:51.542410+0000","last_undegraded":"2023-11-16T18:22:51.542410+0000","last_fullsized":"2023-11-16T18:22:51.542410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:53:23.944904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583180+0000","last_change":"2023-11-16T18:22:46.382812+0000","last_active":"2023-11-16T18:22:51.583180+0000","last_peered":"2023-11-16T18:22:51.583180+0000","last_clean":"2023-11-16T18:22:51.583180+0000","last_became_active":"2023-11-16T18:22:46.382499+0000","last_became_peered":"2023-11-16T18:22:46.382499+0000","last_unstale":"2023-11-16T18:22:51.583180+0000","last_undegraded":"2023-11-16T18:22:51.583180+0000","last_fullsized":"2023-11-16T18:22:51.583180+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:21:45.115121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583056+0000","last_change":"2023-11-16T18:22:46.840917+0000","last_active":"2023-11-16T18:22:51.583056+0000","last_peered":"2023-11-16T18:22:51.583056+0000","last_clean":"2023-11-16T18:22:51.583056+0000","last_became_active":"2023-11-16T18:22:46.840579+0000","last_became_peered":"2023-11-16T18:22:46.840579+0000","last_unstale":"2023-11-16T18:22:51.583056+0000","last_undegraded":"2023-11-16T18:22:51.583056+0000","last_fullsized":"2023-11-16T18:22:51.583056+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:52:31.233036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635389+0000","last_change":"2023-11-16T18:22:51.117852+0000","last_active":"2023-11-16T18:22:51.635389+0000","last_peered":"2023-11-16T18:22:51.635389+0000","last_clean":"2023-11-16T18:22:51.635389+0000","last_became_active":"2023-11-16T18:22:51.117390+0000","last_became_peered":"2023-11-16T18:22:51.117390+0000","last_unstale":"2023-11-16T18:22:51.635389+0000","last_undegraded":"2023-11-16T18:22:51.635389+0000","last_fullsized":"2023-11-16T18:22:51.635389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:34:36.312020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:05.904274+0000","last_change":"2023-11-16T18:22:46.385137+0000","last_active":"2023-11-16T18:23:05.904274+0000","last_peered":"2023-11-16T18:23:05.904274+0000","last_clean":"2023-11-16T18:23:05.904274+0000","last_became_active":"2023-11-16T18:22:46.384636+0000","last_became_peered":"2023-11-16T18:22:46.384636+0000","last_unstale":"2023-11-16T18:23:05.904274+0000","last_undegraded":"2023-11-16T18:23:05.904274+0000","last_fullsized":"2023-11-16T18:23:05.904274+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:24:08.648319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584689+0000","last_change":"2023-11-16T18:22:46.834786+0000","last_active":"2023-11-16T18:22:51.584689+0000","last_peered":"2023-11-16T18:22:51.584689+0000","last_clean":"2023-11-16T18:22:51.584689+0000","last_became_active":"2023-11-16T18:22:46.834338+0000","last_became_peered":"2023-11-16T18:22:46.834338+0000","last_unstale":"2023-11-16T18:22:51.584689+0000","last_undegraded":"2023-11-16T18:22:51.584689+0000","last_fullsized":"2023-11-16T18:22:51.584689+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:44:06.406787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584963+0000","last_change":"2023-11-16T18:22:51.124667+0000","last_active":"2023-11-16T18:22:51.584963+0000","last_peered":"2023-11-16T18:22:51.584963+0000","last_clean":"2023-11-16T18:22:51.584963+0000","last_became_active":"2023-11-16T18:22:51.124211+0000","last_became_peered":"2023-11-16T18:22:51.124211+0000","last_unstale":"2023-11-16T18:22:51.584963+0000","last_undegraded":"2023-11-16T18:22:51.584963+0000","last_fullsized":"2023-11-16T18:22:51.584963+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:03:47.792127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583038+0000","last_change":"2023-11-16T18:22:46.380829+0000","last_active":"2023-11-16T18:22:51.583038+0000","last_peered":"2023-11-16T18:22:51.583038+0000","last_clean":"2023-11-16T18:22:51.583038+0000","last_became_active":"2023-11-16T18:22:46.380466+0000","last_became_peered":"2023-11-16T18:22:46.380466+0000","last_unstale":"2023-11-16T18:22:51.583038+0000","last_undegraded":"2023-11-16T18:22:51.583038+0000","last_fullsized":"2023-11-16T18:22:51.583038+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:09:12.621584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583414+0000","last_change":"2023-11-16T18:22:46.835900+0000","last_active":"2023-11-16T18:22:51.583414+0000","last_peered":"2023-11-16T18:22:51.583414+0000","last_clean":"2023-11-16T18:22:51.583414+0000","last_became_active":"2023-11-16T18:22:46.835438+0000","last_became_peered":"2023-11-16T18:22:46.835438+0000","last_unstale":"2023-11-16T18:22:51.583414+0000","last_undegraded":"2023-11-16T18:22:51.583414+0000","last_fullsized":"2023-11-16T18:22:51.583414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:09:28.382280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582802+0000","last_change":"2023-11-16T18:22:51.120005+0000","last_active":"2023-11-16T18:22:51.582802+0000","last_peered":"2023-11-16T18:22:51.582802+0000","last_clean":"2023-11-16T18:22:51.582802+0000","last_became_active":"2023-11-16T18:22:51.119540+0000","last_became_peered":"2023-11-16T18:22:51.119540+0000","last_unstale":"2023-11-16T18:22:51.582802+0000","last_undegraded":"2023-11-16T18:22:51.582802+0000","last_fullsized":"2023-11-16T18:22:51.582802+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:49:58.798976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:06.915362+0000","last_change":"2023-11-16T18:22:46.378745+0000","last_active":"2023-11-16T18:23:06.915362+0000","last_peered":"2023-11-16T18:23:06.915362+0000","last_clean":"2023-11-16T18:23:06.915362+0000","last_became_active":"2023-11-16T18:22:46.378447+0000","last_became_peered":"2023-11-16T18:22:46.378447+0000","last_unstale":"2023-11-16T18:23:06.915362+0000","last_undegraded":"2023-11-16T18:23:06.915362+0000","last_fullsized":"2023-11-16T18:23:06.915362+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:37:35.122937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541447+0000","last_change":"2023-11-16T18:22:46.823838+0000","last_active":"2023-11-16T18:22:51.541447+0000","last_peered":"2023-11-16T18:22:51.541447+0000","last_clean":"2023-11-16T18:22:51.541447+0000","last_became_active":"2023-11-16T18:22:46.823545+0000","last_became_peered":"2023-11-16T18:22:46.823545+0000","last_unstale":"2023-11-16T18:22:51.541447+0000","last_undegraded":"2023-11-16T18:22:51.541447+0000","last_fullsized":"2023-11-16T18:22:51.541447+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:30:23.340444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541509+0000","last_change":"2023-11-16T18:22:51.127371+0000","last_active":"2023-11-16T18:22:51.541509+0000","last_peered":"2023-11-16T18:22:51.541509+0000","last_clean":"2023-11-16T18:22:51.541509+0000","last_became_active":"2023-11-16T18:22:51.126951+0000","last_became_peered":"2023-11-16T18:22:51.126951+0000","last_unstale":"2023-11-16T18:22:51.541509+0000","last_undegraded":"2023-11-16T18:22:51.541509+0000","last_fullsized":"2023-11-16T18:22:51.541509+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:19:52.597045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'1","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583488+0000","last_change":"2023-11-16T18:22:46.376561+0000","last_active":"2023-11-16T18:22:51.583488+0000","last_peered":"2023-11-16T18:22:51.583488+0000","last_clean":"2023-11-16T18:22:51.583488+0000","last_became_active":"2023-11-16T18:22:46.375426+0000","last_became_peered":"2023-11-16T18:22:46.375426+0000","last_unstale":"2023-11-16T18:22:51.583488+0000","last_undegraded":"2023-11-16T18:22:51.583488+0000","last_fullsized":"2023-11-16T18:22:51.583488+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:07:21.137762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635564+0000","last_change":"2023-11-16T18:22:46.826657+0000","last_active":"2023-11-16T18:22:51.635564+0000","last_peered":"2023-11-16T18:22:51.635564+0000","last_clean":"2023-11-16T18:22:51.635564+0000","last_became_active":"2023-11-16T18:22:46.826126+0000","last_became_peered":"2023-11-16T18:22:46.826126+0000","last_unstale":"2023-11-16T18:22:51.635564+0000","last_undegraded":"2023-11-16T18:22:51.635564+0000","last_fullsized":"2023-11-16T18:22:51.635564+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:40:32.749831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582581+0000","last_change":"2023-11-16T18:22:51.127949+0000","last_active":"2023-11-16T18:22:51.582581+0000","last_peered":"2023-11-16T18:22:51.582581+0000","last_clean":"2023-11-16T18:22:51.582581+0000","last_became_active":"2023-11-16T18:22:51.127388+0000","last_became_peered":"2023-11-16T18:22:51.127388+0000","last_unstale":"2023-11-16T18:22:51.582581+0000","last_undegraded":"2023-11-16T18:22:51.582581+0000","last_fullsized":"2023-11-16T18:22:51.582581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:53:28.858371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584118+0000","last_change":"2023-11-16T18:22:46.378568+0000","last_active":"2023-11-16T18:22:51.584118+0000","last_peered":"2023-11-16T18:22:51.584118+0000","last_clean":"2023-11-16T18:22:51.584118+0000","last_became_active":"2023-11-16T18:22:46.378018+0000","last_became_peered":"2023-11-16T18:22:46.378018+0000","last_unstale":"2023-11-16T18:22:51.584118+0000","last_undegraded":"2023-11-16T18:22:51.584118+0000","last_fullsized":"2023-11-16T18:22:51.584118+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:20:08.573909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583828+0000","last_change":"2023-11-16T18:22:46.841129+0000","last_active":"2023-11-16T18:22:51.583828+0000","last_peered":"2023-11-16T18:22:51.583828+0000","last_clean":"2023-11-16T18:22:51.583828+0000","last_became_active":"2023-11-16T18:22:46.840780+0000","last_became_peered":"2023-11-16T18:22:46.840780+0000","last_unstale":"2023-11-16T18:22:51.583828+0000","last_undegraded":"2023-11-16T18:22:51.583828+0000","last_fullsized":"2023-11-16T18:22:51.583828+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:25:22.363206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583049+0000","last_change":"2023-11-16T18:22:51.126221+0000","last_active":"2023-11-16T18:22:51.583049+0000","last_peered":"2023-11-16T18:22:51.583049+0000","last_clean":"2023-11-16T18:22:51.583049+0000","last_became_active":"2023-11-16T18:22:51.125770+0000","last_became_peered":"2023-11-16T18:22:51.125770+0000","last_unstale":"2023-11-16T18:22:51.583049+0000","last_undegraded":"2023-11-16T18:22:51.583049+0000","last_fullsized":"2023-11-16T18:22:51.583049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:15:19.731508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584284+0000","last_change":"2023-11-16T18:22:46.384872+0000","last_active":"2023-11-16T18:22:51.584284+0000","last_peered":"2023-11-16T18:22:51.584284+0000","last_clean":"2023-11-16T18:22:51.584284+0000","last_became_active":"2023-11-16T18:22:46.384499+0000","last_became_peered":"2023-11-16T18:22:46.384499+0000","last_unstale":"2023-11-16T18:22:51.584284+0000","last_undegraded":"2023-11-16T18:22:51.584284+0000","last_fullsized":"2023-11-16T18:22:51.584284+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:01:05.143223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583583+0000","last_change":"2023-11-16T18:22:46.822574+0000","last_active":"2023-11-16T18:22:51.583583+0000","last_peered":"2023-11-16T18:22:51.583583+0000","last_clean":"2023-11-16T18:22:51.583583+0000","last_became_active":"2023-11-16T18:22:46.822230+0000","last_became_peered":"2023-11-16T18:22:46.822230+0000","last_unstale":"2023-11-16T18:22:51.583583+0000","last_undegraded":"2023-11-16T18:22:51.583583+0000","last_fullsized":"2023-11-16T18:22:51.583583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:05:08.158509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582682+0000","last_change":"2023-11-16T18:22:51.117838+0000","last_active":"2023-11-16T18:22:51.582682+0000","last_peered":"2023-11-16T18:22:51.582682+0000","last_clean":"2023-11-16T18:22:51.582682+0000","last_became_active":"2023-11-16T18:22:51.117275+0000","last_became_peered":"2023-11-16T18:22:51.117275+0000","last_unstale":"2023-11-16T18:22:51.582682+0000","last_undegraded":"2023-11-16T18:22:51.582682+0000","last_fullsized":"2023-11-16T18:22:51.582682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:32:52.147084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541546+0000","last_change":"2023-11-16T18:22:46.827499+0000","last_active":"2023-11-16T18:22:51.541546+0000","last_peered":"2023-11-16T18:22:51.541546+0000","last_clean":"2023-11-16T18:22:51.541546+0000","last_became_active":"2023-11-16T18:22:46.827166+0000","last_became_peered":"2023-11-16T18:22:46.827166+0000","last_unstale":"2023-11-16T18:22:51.541546+0000","last_undegraded":"2023-11-16T18:22:51.541546+0000","last_fullsized":"2023-11-16T18:22:51.541546+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:04:21.034867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:06.914914+0000","last_change":"2023-11-16T18:22:46.381397+0000","last_active":"2023-11-16T18:23:06.914914+0000","last_peered":"2023-11-16T18:23:06.914914+0000","last_clean":"2023-11-16T18:23:06.914914+0000","last_became_active":"2023-11-16T18:22:46.380939+0000","last_became_peered":"2023-11-16T18:22:46.380939+0000","last_unstale":"2023-11-16T18:23:06.914914+0000","last_undegraded":"2023-11-16T18:23:06.914914+0000","last_fullsized":"2023-11-16T18:23:06.914914+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:05:23.498482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584902+0000","last_change":"2023-11-16T18:22:51.125044+0000","last_active":"2023-11-16T18:22:51.584902+0000","last_peered":"2023-11-16T18:22:51.584902+0000","last_clean":"2023-11-16T18:22:51.584902+0000","last_became_active":"2023-11-16T18:22:51.124422+0000","last_became_peered":"2023-11-16T18:22:51.124422+0000","last_unstale":"2023-11-16T18:22:51.584902+0000","last_undegraded":"2023-11-16T18:22:51.584902+0000","last_fullsized":"2023-11-16T18:22:51.584902+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:42:15.443481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.537935+0000","last_change":"2023-11-16T18:22:46.841500+0000","last_active":"2023-11-16T18:22:51.537935+0000","last_peered":"2023-11-16T18:22:51.537935+0000","last_clean":"2023-11-16T18:22:51.537935+0000","last_became_active":"2023-11-16T18:22:46.841186+0000","last_became_peered":"2023-11-16T18:22:46.841186+0000","last_unstale":"2023-11-16T18:22:51.537935+0000","last_undegraded":"2023-11-16T18:22:51.537935+0000","last_fullsized":"2023-11-16T18:22:51.537935+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:42:31.304146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584247+0000","last_change":"2023-11-16T18:22:46.380988+0000","last_active":"2023-11-16T18:22:51.584247+0000","last_peered":"2023-11-16T18:22:51.584247+0000","last_clean":"2023-11-16T18:22:51.584247+0000","last_became_active":"2023-11-16T18:22:46.380616+0000","last_became_peered":"2023-11-16T18:22:46.380616+0000","last_unstale":"2023-11-16T18:22:51.584247+0000","last_undegraded":"2023-11-16T18:22:51.584247+0000","last_fullsized":"2023-11-16T18:22:51.584247+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:17:42.747589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583007+0000","last_change":"2023-11-16T18:21:44.282706+0000","last_active":"2023-11-16T18:22:51.583007+0000","last_peered":"2023-11-16T18:22:51.583007+0000","last_clean":"2023-11-16T18:22:51.583007+0000","last_became_active":"2023-11-16T18:21:44.282201+0000","last_became_peered":"2023-11-16T18:21:44.282201+0000","last_unstale":"2023-11-16T18:22:51.583007+0000","last_undegraded":"2023-11-16T18:22:51.583007+0000","last_fullsized":"2023-11-16T18:22:51.583007+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_clean_scrub_stamp":"2023-11-16T18:18:46.711191+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:57:10.073628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.542133+0000","last_change":"2023-11-16T18:22:51.123051+0000","last_active":"2023-11-16T18:22:51.542133+0000","last_peered":"2023-11-16T18:22:51.542133+0000","last_clean":"2023-11-16T18:22:51.542133+0000","last_became_active":"2023-11-16T18:22:51.122635+0000","last_became_peered":"2023-11-16T18:22:51.122635+0000","last_unstale":"2023-11-16T18:22:51.542133+0000","last_undegraded":"2023-11-16T18:22:51.542133+0000","last_fullsized":"2023-11-16T18:22:51.542133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:21:44.306952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582426+0000","last_change":"2023-11-16T18:22:46.822625+0000","last_active":"2023-11-16T18:22:51.582426+0000","last_peered":"2023-11-16T18:22:51.582426+0000","last_clean":"2023-11-16T18:22:51.582426+0000","last_became_active":"2023-11-16T18:22:46.822296+0000","last_became_peered":"2023-11-16T18:22:46.822296+0000","last_unstale":"2023-11-16T18:22:51.582426+0000","last_undegraded":"2023-11-16T18:22:51.582426+0000","last_fullsized":"2023-11-16T18:22:51.582426+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:07:27.988390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583315+0000","last_change":"2023-11-16T18:22:46.380445+0000","last_active":"2023-11-16T18:22:51.583315+0000","last_peered":"2023-11-16T18:22:51.583315+0000","last_clean":"2023-11-16T18:22:51.583315+0000","last_became_active":"2023-11-16T18:22:46.380113+0000","last_became_peered":"2023-11-16T18:22:46.380113+0000","last_unstale":"2023-11-16T18:22:51.583315+0000","last_undegraded":"2023-11-16T18:22:51.583315+0000","last_fullsized":"2023-11-16T18:22:51.583315+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:18:58.685352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635377+0000","last_change":"2023-11-16T18:22:51.118457+0000","last_active":"2023-11-16T18:22:51.635377+0000","last_peered":"2023-11-16T18:22:51.635377+0000","last_clean":"2023-11-16T18:22:51.635377+0000","last_became_active":"2023-11-16T18:22:51.118013+0000","last_became_peered":"2023-11-16T18:22:51.118013+0000","last_unstale":"2023-11-16T18:22:51.635377+0000","last_undegraded":"2023-11-16T18:22:51.635377+0000","last_fullsized":"2023-11-16T18:22:51.635377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:39:13.358035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:05.904986+0000","last_change":"2023-11-16T18:22:46.384799+0000","last_active":"2023-11-16T18:23:05.904986+0000","last_peered":"2023-11-16T18:23:05.904986+0000","last_clean":"2023-11-16T18:23:05.904986+0000","last_became_active":"2023-11-16T18:22:46.384482+0000","last_became_peered":"2023-11-16T18:22:46.384482+0000","last_unstale":"2023-11-16T18:23:05.904986+0000","last_undegraded":"2023-11-16T18:23:05.904986+0000","last_fullsized":"2023-11-16T18:23:05.904986+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:43:28.699940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583195+0000","last_change":"2023-11-16T18:22:46.829541+0000","last_active":"2023-11-16T18:22:51.583195+0000","last_peered":"2023-11-16T18:22:51.583195+0000","last_clean":"2023-11-16T18:22:51.583195+0000","last_became_active":"2023-11-16T18:22:46.829006+0000","last_became_peered":"2023-11-16T18:22:46.829006+0000","last_unstale":"2023-11-16T18:22:51.583195+0000","last_undegraded":"2023-11-16T18:22:51.583195+0000","last_fullsized":"2023-11-16T18:22:51.583195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:43:56.291096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538010+0000","last_change":"2023-11-16T18:22:51.122751+0000","last_active":"2023-11-16T18:22:51.538010+0000","last_peered":"2023-11-16T18:22:51.538010+0000","last_clean":"2023-11-16T18:22:51.538010+0000","last_became_active":"2023-11-16T18:22:51.122423+0000","last_became_peered":"2023-11-16T18:22:51.122423+0000","last_unstale":"2023-11-16T18:22:51.538010+0000","last_undegraded":"2023-11-16T18:22:51.538010+0000","last_fullsized":"2023-11-16T18:22:51.538010+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:24:08.649040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584727+0000","last_change":"2023-11-16T18:22:46.381856+0000","last_active":"2023-11-16T18:22:51.584727+0000","last_peered":"2023-11-16T18:22:51.584727+0000","last_clean":"2023-11-16T18:22:51.584727+0000","last_became_active":"2023-11-16T18:22:46.381518+0000","last_became_peered":"2023-11-16T18:22:46.381518+0000","last_unstale":"2023-11-16T18:22:51.584727+0000","last_undegraded":"2023-11-16T18:22:51.584727+0000","last_fullsized":"2023-11-16T18:22:51.584727+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:09:47.615061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538288+0000","last_change":"2023-11-16T18:22:46.843802+0000","last_active":"2023-11-16T18:22:51.538288+0000","last_peered":"2023-11-16T18:22:51.538288+0000","last_clean":"2023-11-16T18:22:51.538288+0000","last_became_active":"2023-11-16T18:22:46.842210+0000","last_became_peered":"2023-11-16T18:22:46.842210+0000","last_unstale":"2023-11-16T18:22:51.538288+0000","last_undegraded":"2023-11-16T18:22:51.538288+0000","last_fullsized":"2023-11-16T18:22:51.538288+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:33:56.708900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582275+0000","last_change":"2023-11-16T18:22:51.124632+0000","last_active":"2023-11-16T18:22:51.582275+0000","last_peered":"2023-11-16T18:22:51.582275+0000","last_clean":"2023-11-16T18:22:51.582275+0000","last_became_active":"2023-11-16T18:22:51.124220+0000","last_became_peered":"2023-11-16T18:22:51.124220+0000","last_unstale":"2023-11-16T18:22:51.582275+0000","last_undegraded":"2023-11-16T18:22:51.582275+0000","last_fullsized":"2023-11-16T18:22:51.582275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:33:27.384018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584105+0000","last_change":"2023-11-16T18:22:46.381456+0000","last_active":"2023-11-16T18:22:51.584105+0000","last_peered":"2023-11-16T18:22:51.584105+0000","last_clean":"2023-11-16T18:22:51.584105+0000","last_became_active":"2023-11-16T18:22:46.381137+0000","last_became_peered":"2023-11-16T18:22:46.381137+0000","last_unstale":"2023-11-16T18:22:51.584105+0000","last_undegraded":"2023-11-16T18:22:51.584105+0000","last_fullsized":"2023-11-16T18:22:51.584105+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:04:47.876051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583353+0000","last_change":"2023-11-16T18:22:46.838249+0000","last_active":"2023-11-16T18:22:51.583353+0000","last_peered":"2023-11-16T18:22:51.583353+0000","last_clean":"2023-11-16T18:22:51.583353+0000","last_became_active":"2023-11-16T18:22:46.837925+0000","last_became_peered":"2023-11-16T18:22:46.837925+0000","last_unstale":"2023-11-16T18:22:51.583353+0000","last_undegraded":"2023-11-16T18:22:51.583353+0000","last_fullsized":"2023-11-16T18:22:51.583353+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:05:01.131026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583670+0000","last_change":"2023-11-16T18:22:51.123687+0000","last_active":"2023-11-16T18:22:51.583670+0000","last_peered":"2023-11-16T18:22:51.583670+0000","last_clean":"2023-11-16T18:22:51.583670+0000","last_became_active":"2023-11-16T18:22:51.123209+0000","last_became_peered":"2023-11-16T18:22:51.123209+0000","last_unstale":"2023-11-16T18:22:51.583670+0000","last_undegraded":"2023-11-16T18:22:51.583670+0000","last_fullsized":"2023-11-16T18:22:51.583670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:43:25.846371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.539539+0000","last_change":"2023-11-16T18:22:46.378038+0000","last_active":"2023-11-16T18:22:51.539539+0000","last_peered":"2023-11-16T18:22:51.539539+0000","last_clean":"2023-11-16T18:22:51.539539+0000","last_became_active":"2023-11-16T18:22:46.377680+0000","last_became_peered":"2023-11-16T18:22:46.377680+0000","last_unstale":"2023-11-16T18:22:51.539539+0000","last_undegraded":"2023-11-16T18:22:51.539539+0000","last_fullsized":"2023-11-16T18:22:51.539539+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:19:09.447833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582149+0000","last_change":"2023-11-16T18:22:46.821888+0000","last_active":"2023-11-16T18:22:51.582149+0000","last_peered":"2023-11-16T18:22:51.582149+0000","last_clean":"2023-11-16T18:22:51.582149+0000","last_became_active":"2023-11-16T18:22:46.821446+0000","last_became_peered":"2023-11-16T18:22:46.821446+0000","last_unstale":"2023-11-16T18:22:51.582149+0000","last_undegraded":"2023-11-16T18:22:51.582149+0000","last_fullsized":"2023-11-16T18:22:51.582149+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:36:05.450444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.585099+0000","last_change":"2023-11-16T18:22:51.119717+0000","last_active":"2023-11-16T18:22:51.585099+0000","last_peered":"2023-11-16T18:22:51.585099+0000","last_clean":"2023-11-16T18:22:51.585099+0000","last_became_active":"2023-11-16T18:22:51.119256+0000","last_became_peered":"2023-11-16T18:22:51.119256+0000","last_unstale":"2023-11-16T18:22:51.585099+0000","last_undegraded":"2023-11-16T18:22:51.585099+0000","last_fullsized":"2023-11-16T18:22:51.585099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:26:59.741857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:05.906947+0000","last_change":"2023-11-16T18:22:46.375387+0000","last_active":"2023-11-16T18:23:05.906947+0000","last_peered":"2023-11-16T18:23:05.906947+0000","last_clean":"2023-11-16T18:23:05.906947+0000","last_became_active":"2023-11-16T18:22:46.374886+0000","last_became_peered":"2023-11-16T18:22:46.374886+0000","last_unstale":"2023-11-16T18:23:05.906947+0000","last_undegraded":"2023-11-16T18:23:05.906947+0000","last_fullsized":"2023-11-16T18:23:05.906947+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:51:03.887363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635094+0000","last_change":"2023-11-16T18:22:46.820315+0000","last_active":"2023-11-16T18:22:51.635094+0000","last_peered":"2023-11-16T18:22:51.635094+0000","last_clean":"2023-11-16T18:22:51.635094+0000","last_became_active":"2023-11-16T18:22:46.819592+0000","last_became_peered":"2023-11-16T18:22:46.819592+0000","last_unstale":"2023-11-16T18:22:51.635094+0000","last_undegraded":"2023-11-16T18:22:51.635094+0000","last_fullsized":"2023-11-16T18:22:51.635094+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:39:18.272375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584121+0000","last_change":"2023-11-16T18:22:51.123645+0000","last_active":"2023-11-16T18:22:51.584121+0000","last_peered":"2023-11-16T18:22:51.584121+0000","last_clean":"2023-11-16T18:22:51.584121+0000","last_became_active":"2023-11-16T18:22:51.123099+0000","last_became_peered":"2023-11-16T18:22:51.123099+0000","last_unstale":"2023-11-16T18:22:51.584121+0000","last_undegraded":"2023-11-16T18:22:51.584121+0000","last_fullsized":"2023-11-16T18:22:51.584121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:44:09.664575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582962+0000","last_change":"2023-11-16T18:22:46.378638+0000","last_active":"2023-11-16T18:22:51.582962+0000","last_peered":"2023-11-16T18:22:51.582962+0000","last_clean":"2023-11-16T18:22:51.582962+0000","last_became_active":"2023-11-16T18:22:46.378341+0000","last_became_peered":"2023-11-16T18:22:46.378341+0000","last_unstale":"2023-11-16T18:22:51.582962+0000","last_undegraded":"2023-11-16T18:22:51.582962+0000","last_fullsized":"2023-11-16T18:22:51.582962+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:40:53.983774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538491+0000","last_change":"2023-11-16T18:22:46.843431+0000","last_active":"2023-11-16T18:22:51.538491+0000","last_peered":"2023-11-16T18:22:51.538491+0000","last_clean":"2023-11-16T18:22:51.538491+0000","last_became_active":"2023-11-16T18:22:46.842992+0000","last_became_peered":"2023-11-16T18:22:46.842992+0000","last_unstale":"2023-11-16T18:22:51.538491+0000","last_undegraded":"2023-11-16T18:22:51.538491+0000","last_fullsized":"2023-11-16T18:22:51.538491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:18:53.495119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583896+0000","last_change":"2023-11-16T18:22:51.122640+0000","last_active":"2023-11-16T18:22:51.583896+0000","last_peered":"2023-11-16T18:22:51.583896+0000","last_clean":"2023-11-16T18:22:51.583896+0000","last_became_active":"2023-11-16T18:22:51.122006+0000","last_became_peered":"2023-11-16T18:22:51.122006+0000","last_unstale":"2023-11-16T18:22:51.583896+0000","last_undegraded":"2023-11-16T18:22:51.583896+0000","last_fullsized":"2023-11-16T18:22:51.583896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:10:12.659920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:23:05.905977+0000","last_change":"2023-11-16T18:22:46.384837+0000","last_active":"2023-11-16T18:23:05.905977+0000","last_peered":"2023-11-16T18:23:05.905977+0000","last_clean":"2023-11-16T18:23:05.905977+0000","last_became_active":"2023-11-16T18:22:46.384541+0000","last_became_peered":"2023-11-16T18:22:46.384541+0000","last_unstale":"2023-11-16T18:23:05.905977+0000","last_undegraded":"2023-11-16T18:23:05.905977+0000","last_fullsized":"2023-11-16T18:23:05.905977+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:10:01.746531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584631+0000","last_change":"2023-11-16T18:22:46.839967+0000","last_active":"2023-11-16T18:22:51.584631+0000","last_peered":"2023-11-16T18:22:51.584631+0000","last_clean":"2023-11-16T18:22:51.584631+0000","last_became_active":"2023-11-16T18:22:46.839645+0000","last_became_peered":"2023-11-16T18:22:46.839645+0000","last_unstale":"2023-11-16T18:22:51.584631+0000","last_undegraded":"2023-11-16T18:22:51.584631+0000","last_fullsized":"2023-11-16T18:22:51.584631+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:51:40.823551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.539711+0000","last_change":"2023-11-16T18:22:51.126548+0000","last_active":"2023-11-16T18:22:51.539711+0000","last_peered":"2023-11-16T18:22:51.539711+0000","last_clean":"2023-11-16T18:22:51.539711+0000","last_became_active":"2023-11-16T18:22:51.126039+0000","last_became_peered":"2023-11-16T18:22:51.126039+0000","last_unstale":"2023-11-16T18:22:51.539711+0000","last_undegraded":"2023-11-16T18:22:51.539711+0000","last_fullsized":"2023-11-16T18:22:51.539711+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:26:40.925200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583458+0000","last_change":"2023-11-16T18:22:46.383947+0000","last_active":"2023-11-16T18:22:51.583458+0000","last_peered":"2023-11-16T18:22:51.583458+0000","last_clean":"2023-11-16T18:22:51.583458+0000","last_became_active":"2023-11-16T18:22:46.383661+0000","last_became_peered":"2023-11-16T18:22:46.383661+0000","last_unstale":"2023-11-16T18:22:51.583458+0000","last_undegraded":"2023-11-16T18:22:51.583458+0000","last_fullsized":"2023-11-16T18:22:51.583458+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:26:19.213002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.585153+0000","last_change":"2023-11-16T18:22:46.836877+0000","last_active":"2023-11-16T18:22:51.585153+0000","last_peered":"2023-11-16T18:22:51.585153+0000","last_clean":"2023-11-16T18:22:51.585153+0000","last_became_active":"2023-11-16T18:22:46.836477+0000","last_became_peered":"2023-11-16T18:22:46.836477+0000","last_unstale":"2023-11-16T18:22:51.585153+0000","last_undegraded":"2023-11-16T18:22:51.585153+0000","last_fullsized":"2023-11-16T18:22:51.585153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:00:14.928162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582862+0000","last_change":"2023-11-16T18:22:51.126930+0000","last_active":"2023-11-16T18:22:51.582862+0000","last_peered":"2023-11-16T18:22:51.582862+0000","last_clean":"2023-11-16T18:22:51.582862+0000","last_became_active":"2023-11-16T18:22:51.126409+0000","last_became_peered":"2023-11-16T18:22:51.126409+0000","last_unstale":"2023-11-16T18:22:51.582862+0000","last_undegraded":"2023-11-16T18:22:51.582862+0000","last_fullsized":"2023-11-16T18:22:51.582862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:39:20.438296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.542073+0000","last_change":"2023-11-16T18:22:46.380820+0000","last_active":"2023-11-16T18:22:51.542073+0000","last_peered":"2023-11-16T18:22:51.542073+0000","last_clean":"2023-11-16T18:22:51.542073+0000","last_became_active":"2023-11-16T18:22:46.380446+0000","last_became_peered":"2023-11-16T18:22:46.380446+0000","last_unstale":"2023-11-16T18:22:51.542073+0000","last_undegraded":"2023-11-16T18:22:51.542073+0000","last_fullsized":"2023-11-16T18:22:51.542073+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:25:56.255338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.584787+0000","last_change":"2023-11-16T18:22:46.838153+0000","last_active":"2023-11-16T18:22:51.584787+0000","last_peered":"2023-11-16T18:22:51.584787+0000","last_clean":"2023-11-16T18:22:51.584787+0000","last_became_active":"2023-11-16T18:22:46.837836+0000","last_became_peered":"2023-11-16T18:22:46.837836+0000","last_unstale":"2023-11-16T18:22:51.584787+0000","last_undegraded":"2023-11-16T18:22:51.584787+0000","last_fullsized":"2023-11-16T18:22:51.584787+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:33:16.340646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541226+0000","last_change":"2023-11-16T18:22:51.119993+0000","last_active":"2023-11-16T18:22:51.541226+0000","last_peered":"2023-11-16T18:22:51.541226+0000","last_clean":"2023-11-16T18:22:51.541226+0000","last_became_active":"2023-11-16T18:22:51.119401+0000","last_became_peered":"2023-11-16T18:22:51.119401+0000","last_unstale":"2023-11-16T18:22:51.541226+0000","last_undegraded":"2023-11-16T18:22:51.541226+0000","last_fullsized":"2023-11-16T18:22:51.541226+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:16:01.027816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582901+0000","last_change":"2023-11-16T18:22:46.383399+0000","last_active":"2023-11-16T18:22:51.582901+0000","last_peered":"2023-11-16T18:22:51.582901+0000","last_clean":"2023-11-16T18:22:51.582901+0000","last_became_active":"2023-11-16T18:22:46.382938+0000","last_became_peered":"2023-11-16T18:22:46.382938+0000","last_unstale":"2023-11-16T18:22:51.582901+0000","last_undegraded":"2023-11-16T18:22:51.582901+0000","last_fullsized":"2023-11-16T18:22:51.582901+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:18:33.772254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.585416+0000","last_change":"2023-11-16T18:22:46.836035+0000","last_active":"2023-11-16T18:22:51.585416+0000","last_peered":"2023-11-16T18:22:51.585416+0000","last_clean":"2023-11-16T18:22:51.585416+0000","last_became_active":"2023-11-16T18:22:46.835709+0000","last_became_peered":"2023-11-16T18:22:46.835709+0000","last_unstale":"2023-11-16T18:22:51.585416+0000","last_undegraded":"2023-11-16T18:22:51.585416+0000","last_fullsized":"2023-11-16T18:22:51.585416+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:04:15.508385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635093+0000","last_change":"2023-11-16T18:22:46.820535+0000","last_active":"2023-11-16T18:22:51.635093+0000","last_peered":"2023-11-16T18:22:51.635093+0000","last_clean":"2023-11-16T18:22:51.635093+0000","last_became_active":"2023-11-16T18:22:46.820009+0000","last_became_peered":"2023-11-16T18:22:46.820009+0000","last_unstale":"2023-11-16T18:22:51.635093+0000","last_undegraded":"2023-11-16T18:22:51.635093+0000","last_fullsized":"2023-11-16T18:22:51.635093+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:03:50.304489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582866+0000","last_change":"2023-11-16T18:22:46.831762+0000","last_active":"2023-11-16T18:22:51.582866+0000","last_peered":"2023-11-16T18:22:51.582866+0000","last_clean":"2023-11-16T18:22:51.582866+0000","last_became_active":"2023-11-16T18:22:46.831429+0000","last_became_peered":"2023-11-16T18:22:46.831429+0000","last_unstale":"2023-11-16T18:22:51.582866+0000","last_undegraded":"2023-11-16T18:22:51.582866+0000","last_fullsized":"2023-11-16T18:22:51.582866+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:15:09.682384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.582832+0000","last_change":"2023-11-16T18:22:46.838133+0000","last_active":"2023-11-16T18:22:51.582832+0000","last_peered":"2023-11-16T18:22:51.582832+0000","last_clean":"2023-11-16T18:22:51.582832+0000","last_became_active":"2023-11-16T18:22:46.837845+0000","last_became_peered":"2023-11-16T18:22:46.837845+0000","last_unstale":"2023-11-16T18:22:51.582832+0000","last_undegraded":"2023-11-16T18:22:51.582832+0000","last_fullsized":"2023-11-16T18:22:51.582832+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:05:42.510958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583143+0000","last_change":"2023-11-16T18:22:46.833813+0000","last_active":"2023-11-16T18:22:51.583143+0000","last_peered":"2023-11-16T18:22:51.583143+0000","last_clean":"2023-11-16T18:22:51.583143+0000","last_became_active":"2023-11-16T18:22:46.833379+0000","last_became_peered":"2023-11-16T18:22:46.833379+0000","last_unstale":"2023-11-16T18:22:51.583143+0000","last_undegraded":"2023-11-16T18:22:51.583143+0000","last_fullsized":"2023-11-16T18:22:51.583143+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:47:34.426651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583312+0000","last_change":"2023-11-16T18:22:46.823192+0000","last_active":"2023-11-16T18:22:51.583312+0000","last_peered":"2023-11-16T18:22:51.583312+0000","last_clean":"2023-11-16T18:22:51.583312+0000","last_became_active":"2023-11-16T18:22:46.822930+0000","last_became_peered":"2023-11-16T18:22:46.822930+0000","last_unstale":"2023-11-16T18:22:51.583312+0000","last_undegraded":"2023-11-16T18:22:51.583312+0000","last_fullsized":"2023-11-16T18:22:51.583312+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:02:02.465742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635585+0000","last_change":"2023-11-16T18:22:46.825815+0000","last_active":"2023-11-16T18:22:51.635585+0000","last_peered":"2023-11-16T18:22:51.635585+0000","last_clean":"2023-11-16T18:22:51.635585+0000","last_became_active":"2023-11-16T18:22:46.825517+0000","last_became_peered":"2023-11-16T18:22:46.825517+0000","last_unstale":"2023-11-16T18:22:51.635585+0000","last_undegraded":"2023-11-16T18:22:51.635585+0000","last_fullsized":"2023-11-16T18:22:51.635585+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:00:54.827931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583678+0000","last_change":"2023-11-16T18:22:46.826825+0000","last_active":"2023-11-16T18:22:51.583678+0000","last_peered":"2023-11-16T18:22:51.583678+0000","last_clean":"2023-11-16T18:22:51.583678+0000","last_became_active":"2023-11-16T18:22:46.826495+0000","last_became_peered":"2023-11-16T18:22:46.826495+0000","last_unstale":"2023-11-16T18:22:51.583678+0000","last_undegraded":"2023-11-16T18:22:51.583678+0000","last_fullsized":"2023-11-16T18:22:51.583678+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:34:04.792163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538413+0000","last_change":"2023-11-16T18:22:46.844090+0000","last_active":"2023-11-16T18:22:51.538413+0000","last_peered":"2023-11-16T18:22:51.538413+0000","last_clean":"2023-11-16T18:22:51.538413+0000","last_became_active":"2023-11-16T18:22:46.843641+0000","last_became_peered":"2023-11-16T18:22:46.843641+0000","last_unstale":"2023-11-16T18:22:51.538413+0000","last_undegraded":"2023-11-16T18:22:51.538413+0000","last_fullsized":"2023-11-16T18:22:51.538413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:57:37.160782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.541384+0000","last_change":"2023-11-16T18:22:46.838340+0000","last_active":"2023-11-16T18:22:51.541384+0000","last_peered":"2023-11-16T18:22:51.541384+0000","last_clean":"2023-11-16T18:22:51.541384+0000","last_became_active":"2023-11-16T18:22:46.838039+0000","last_became_peered":"2023-11-16T18:22:46.838039+0000","last_unstale":"2023-11-16T18:22:51.541384+0000","last_undegraded":"2023-11-16T18:22:51.541384+0000","last_fullsized":"2023-11-16T18:22:51.541384+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:24:21.228784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.538151+0000","last_change":"2023-11-16T18:22:46.841739+0000","last_active":"2023-11-16T18:22:51.538151+0000","last_peered":"2023-11-16T18:22:51.538151+0000","last_clean":"2023-11-16T18:22:51.538151+0000","last_became_active":"2023-11-16T18:22:46.841450+0000","last_became_peered":"2023-11-16T18:22:46.841450+0000","last_unstale":"2023-11-16T18:22:51.538151+0000","last_undegraded":"2023-11-16T18:22:51.538151+0000","last_fullsized":"2023-11-16T18:22:51.538151+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:17:38.521808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.540157+0000","last_change":"2023-11-16T18:22:46.839060+0000","last_active":"2023-11-16T18:22:51.540157+0000","last_peered":"2023-11-16T18:22:51.540157+0000","last_clean":"2023-11-16T18:22:51.540157+0000","last_became_active":"2023-11-16T18:22:46.838725+0000","last_became_peered":"2023-11-16T18:22:46.838725+0000","last_unstale":"2023-11-16T18:22:51.540157+0000","last_undegraded":"2023-11-16T18:22:51.540157+0000","last_fullsized":"2023-11-16T18:22:51.540157+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:36:04.190543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.583894+0000","last_change":"2023-11-16T18:22:46.833439+0000","last_active":"2023-11-16T18:22:51.583894+0000","last_peered":"2023-11-16T18:22:51.583894+0000","last_clean":"2023-11-16T18:22:51.583894+0000","last_became_active":"2023-11-16T18:22:46.833111+0000","last_became_peered":"2023-11-16T18:22:46.833111+0000","last_unstale":"2023-11-16T18:22:51.583894+0000","last_undegraded":"2023-11-16T18:22:51.583894+0000","last_fullsized":"2023-11-16T18:22:51.583894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:15:52.910452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-11-16T18:22:51.635322+0000","last_change":"2023-11-16T18:22:46.825185+0000","last_active":"2023-11-16T18:22:51.635322+0000","last_peered":"2023-11-16T18:22:51.635322+0000","last_clean":"2023-11-16T18:22:51.635322+0000","last_became_active":"2023-11-16T18:22:46.824866+0000","last_became_peered":"2023-11-16T18:22:46.824866+0000","last_unstale":"2023-11-16T18:22:51.635322+0000","last_undegraded":"2023-11-16T18:22:51.635322+0000","last_fullsized":"2023-11-16T18:22:51.635322+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:14:35.877637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":215374,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6062,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":83,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":17970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481808,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6040,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739112,"statfs":{"total":95995035648,"available":95988850688,"internally_reserved":0,"allocated":933888,"data_stored":462514,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67100000000000004}]},{"osd":2,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.6539999999999999,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.7130000000000001,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.70899999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.68400000000000005}]},{"osd":10,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.75},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.69699999999999995}]}]},{"osd":10,"up_from":69,"seq":296352743443,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6300,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":1003520,"data_stored":1056288,"data_compressed":47845,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":1,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.68999999999999995}]},{"osd":4,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.67200000000000004}]},{"osd":5,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.81499999999999995}]},{"osd":7,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55000000000000004}]},{"osd":8,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.80200000000000005}]},{"osd":9,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.84699999999999998}]},{"osd":11,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.73799999999999999}]}]},{"osd":9,"up_from":64,"seq":274877906968,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6232,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738920,"statfs":{"total":95995035648,"available":95988654080,"internally_reserved":0,"allocated":933888,"data_stored":462514,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.51600000000000001}]},{"osd":1,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.621},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.79400000000000004}]},{"osd":2,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.63900000000000001}]},{"osd":3,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.55200000000000005}]},{"osd":4,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.74099999999999999}]},{"osd":5,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.78100000000000003}]},{"osd":6,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.379},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.42699999999999999}]},{"osd":7,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.753}]},{"osd":8,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.874},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.81000000000000005}]},{"osd":10,"last update":"Thu Nov 16 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.82099999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]}]},{"osd":8,"up_from":57,"seq":244813135901,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6376,"kb_used_data":928,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738776,"statfs":{"total":95995035648,"available":95988506624,"internally_reserved":0,"allocated":950272,"data_stored":462670,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.58299999999999996,"15min":0.58799999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.81699999999999995,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.42499999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.876},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.443,"5min":0.316,"15min":0.316},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.83799999999999997}]},{"osd":2,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.36599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.626},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.48599999999999999,"15min":0.47599999999999998},"min":{"1min":0.39000000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.72799999999999998}]},{"osd":3,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.41799999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.628,"15min":0.628},"min":{"1min":0.41299999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.77300000000000002}]},{"osd":5,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52500000000000002,"15min":0.51300000000000001},"min":{"1min":0.49399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73899999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.37,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.77400000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.84499999999999997}]},{"osd":6,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65900000000000003,"15min":0.65700000000000003},"min":{"1min":0.51600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.52700000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.84799999999999998,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.86399999999999999}]},{"osd":7,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.68000000000000005,"15min":0.66700000000000004},"min":{"1min":0.52000000000000002,"5min":0.44,"15min":0.44},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.54300000000000004,"5min":0.434,"15min":0.434},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.91600000000000004}]},{"osd":9,"last update":"Thu Nov 16 18:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.90300000000000002}]},{"osd":10,"last update":"Thu Nov 16 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":1.016}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.999}]}]},{"osd":7,"up_from":49,"seq":210453397537,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6472,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738680,"statfs":{"total":95995035648,"available":95988408320,"internally_reserved":0,"allocated":983040,"data_stored":1052882,"data_compressed":47845,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.60699999999999998,"15min":0.59499999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81999999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81699999999999995,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.436,"15min":0.42499999999999999},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.47199999999999998,"15min":0.46300000000000002},"min":{"1min":0.28299999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.80500000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.71899999999999997}]},{"osd":2,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.626,"15min":0.61399999999999999},"min":{"1min":0.47999999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.63400000000000001,"15min":0.61699999999999999},"min":{"1min":0.46200000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.745}]},{"osd":3,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.83599999999999997}]},{"osd":4,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.55900000000000005,"15min":0.54600000000000004},"min":{"1min":0.36199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56299999999999994,"15min":0.55800000000000005},"min":{"1min":0.33400000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.68799999999999994,"15min":0.67500000000000004},"min":{"1min":0.52400000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.872},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.75900000000000001,"15min":0.749},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.79900000000000004}]},{"osd":6,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.70699999999999996,"15min":0.69699999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.73999999999999999,"15min":0.72999999999999998},"min":{"1min":0.44700000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0820000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.91400000000000003}]},{"osd":8,"last update":"Thu Nov 16 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.747,"15min":0.74099999999999999},"min":{"1min":0.501,"5min":0.433,"15min":0.433},"max":{"1min":1.0529999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.77900000000000003,"15min":0.77700000000000002},"min":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0409999999999999,"5min":1.107,"15min":1.107},"last":1.0049999999999999}]},{"osd":9,"last update":"Thu Nov 16 18:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.94599999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.98599999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]}]},{"osd":6,"up_from":44,"seq":188978561062,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6360,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":933888,"data_stored":462022,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.379,"15min":0.36299999999999999},"min":{"1min":0.34399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.39400000000000002,"15min":0.378},"min":{"1min":0.30199999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Nov 16 18:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.61199999999999999,"15min":0.59199999999999997},"min":{"1min":0.44900000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.54100000000000004,"15min":0.52900000000000003},"min":{"1min":0.38900000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.59599999999999997}]},{"osd":2,"last update":"Thu Nov 16 18:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53200000000000003,"15min":0.51900000000000002},"min":{"1min":0.442,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.76500000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.59099999999999997,"15min":0.56599999999999995},"min":{"1min":0.51100000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.503}]},{"osd":3,"last update":"Thu Nov 16 18:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.40300000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.443},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.45200000000000001,"15min":0.436},"min":{"1min":0.33100000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.45900000000000002}]},{"osd":4,"last update":"Thu Nov 16 18:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.50900000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94599999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.68700000000000006,"15min":0.66300000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.92000000000000004}]},{"osd":5,"last update":"Thu Nov 16 18:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.58799999999999997,"15min":0.57599999999999996},"min":{"1min":0.42999999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":1.004},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.63700000000000001,"15min":0.623},"min":{"1min":0.52400000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.83999999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.76200000000000001,"15min":0.752},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.79000000000000004,"15min":0.77600000000000002},"min":{"1min":0.57999999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.80800000000000005}]},{"osd":8,"last update":"Thu Nov 16 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.82399999999999995,"15min":0.82599999999999996},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.70199999999999996,"15min":0.70699999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85799999999999998,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.89500000000000002}]},{"osd":9,"last update":"Thu Nov 16 18:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.629},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.70299999999999996}]},{"osd":10,"last update":"Thu Nov 16 18:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.85699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7260,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737892,"statfs":{"total":95995035648,"available":95987601408,"internally_reserved":0,"allocated":937984,"data_stored":465453,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:31 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51800000000000002,"15min":0.53100000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53100000000000003,"15min":0.53700000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55800000000000005,"15min":0.53300000000000003},"min":{"1min":0.41099999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78100000000000003,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56999999999999995,"15min":0.54600000000000004},"min":{"1min":0.36199999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.74399999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Nov 16 18:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.57499999999999996,"15min":0.56299999999999994},"min":{"1min":0.435,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.57899999999999996,"15min":0.56799999999999995},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.80500000000000005,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Nov 16 18:22:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.42099999999999999,"15min":0.39800000000000002},"min":{"1min":0.32200000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.438,"15min":0.40799999999999997},"min":{"1min":0.33300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.438}]},{"osd":5,"last update":"Thu Nov 16 18:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.63600000000000001,"15min":0.624},"min":{"1min":0.49099999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85299999999999998,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.81999999999999995}]},{"osd":6,"last update":"Thu Nov 16 18:22:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.60799999999999998,"15min":0.59699999999999998},"min":{"1min":0.42099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.47999999999999998}]},{"osd":7,"last update":"Thu Nov 16 18:22:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.59399999999999997,"15min":0.57299999999999995},"min":{"1min":0.38700000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.63100000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.66800000000000004,"15min":0.65800000000000003},"min":{"1min":0.51900000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.751},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67000000000000004,"15min":0.66000000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.61799999999999999}]},{"osd":9,"last update":"Thu Nov 16 18:22:20 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.496},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.69399999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.66800000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7196,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":6272,"kb_avail":93737956,"statfs":{"total":95995035648,"available":95987666944,"internally_reserved":0,"allocated":937984,"data_stored":462353,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6422528},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.52500000000000002,"15min":0.49099999999999999},"min":{"1min":0.376,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.52300000000000002,"15min":0.48899999999999999},"min":{"1min":0.42399999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.69099999999999995}]},{"osd":2,"last update":"Thu Nov 16 18:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.64900000000000002,"15min":0.72899999999999998},"min":{"1min":0.379,"5min":0.371,"15min":0.371},"max":{"1min":0.81399999999999995,"5min":5.3650000000000002,"15min":5.3650000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.624,"15min":0.67000000000000004},"min":{"1min":0.374,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.872,"5min":3.4009999999999998,"15min":3.4009999999999998},"last":0.60699999999999998}]},{"osd":3,"last update":"Thu Nov 16 18:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55800000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.45100000000000001},"min":{"1min":0.32700000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.57299999999999995}]},{"osd":4,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.57999999999999996,"15min":0.55400000000000005},"min":{"1min":0.39800000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.57699999999999996,"15min":0.56699999999999995},"min":{"1min":0.41599999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89200000000000002,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.72699999999999998}]},{"osd":5,"last update":"Thu Nov 16 18:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58099999999999996,"15min":0.57499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82099999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.59999999999999998,"15min":0.58599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84099999999999997,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.55400000000000005,"15min":0.53300000000000003},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.44900000000000001}]},{"osd":7,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.95099999999999996,"5min":1.004,"15min":1.004},"last":0.82499999999999996}]},{"osd":8,"last update":"Thu Nov 16 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89600000000000002,"5min":1.052,"15min":1.052},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.96699999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.64900000000000002}]},{"osd":9,"last update":"Thu Nov 16 18:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.622},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.63}]},{"osd":10,"last update":"Thu Nov 16 18:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.81299999999999994}]},{"osd":11,"last update":"Thu Nov 16 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.73699999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7268,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737884,"statfs":{"total":95995035648,"available":95987593216,"internally_reserved":0,"allocated":946176,"data_stored":465760,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.50700000000000001,"15min":0.46999999999999997},"min":{"1min":0.30399999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.49399999999999999,"15min":0.46100000000000002},"min":{"1min":0.35599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56499999999999995}]},{"osd":1,"last update":"Thu Nov 16 18:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.53900000000000003,"15min":0.51300000000000001},"min":{"1min":0.33000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.56399999999999995,"15min":0.53900000000000003},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Nov 16 18:22:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.56000000000000005,"15min":0.53600000000000003},"min":{"1min":0.378,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.56399999999999995,"15min":0.53600000000000003},"min":{"1min":0.41499999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.72999999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.54500000000000004,"15min":0.52700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.75},{"interface":"front","average":{"1min":0.63,"5min":0.56299999999999994,"15min":0.54400000000000004},"min":{"1min":0.36799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Nov 16 18:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.44,"15min":0.41299999999999998},"min":{"1min":0.34699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.497,"15min":0.47099999999999997},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.61799999999999999}]},{"osd":6,"last update":"Thu Nov 16 18:22:15 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.627,"15min":0.61199999999999999},"min":{"1min":0.44700000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.63600000000000001,"15min":0.62},"min":{"1min":0.50600000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.95299999999999996}]},{"osd":7,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.61899999999999999,"15min":0.61399999999999999},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.81899999999999995}]},{"osd":8,"last update":"Thu Nov 16 18:22:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.71299999999999997}]},{"osd":9,"last update":"Thu Nov 16 18:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.84199999999999997}]},{"osd":10,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.93999999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182451,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6624,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738528,"statfs":{"total":95995035648,"available":95988252672,"internally_reserved":0,"allocated":942080,"data_stored":461943,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.379,"15min":0.36499999999999999},"min":{"1min":0.33700000000000002,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.39500000000000002,"15min":0.38600000000000001},"min":{"1min":0.30299999999999999,"5min":0.185,"15min":0.185},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.38300000000000001}]},{"osd":1,"last update":"Thu Nov 16 18:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.53800000000000003,"15min":0.52000000000000002},"min":{"1min":0.375,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.92400000000000004,"5min":1.129,"15min":1.129},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.55100000000000005,"15min":0.53300000000000003},"min":{"1min":0.437,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Nov 16 18:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.58399999999999996,"15min":0.56399999999999995},"min":{"1min":0.38100000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.41499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.873,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.61899999999999999,"15min":0.59699999999999998},"min":{"1min":0.501,"5min":0.378,"15min":0.378},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.627},"min":{"1min":0.47699999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.94799999999999995,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.68700000000000006}]},{"osd":5,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77400000000000002,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.35999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98799999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.505}]},{"osd":6,"last update":"Thu Nov 16 18:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.52100000000000002,"15min":0.48699999999999999},"min":{"1min":0.41899999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.33800000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.41199999999999998}]},{"osd":7,"last update":"Thu Nov 16 18:22:37 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.54100000000000004,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.621},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.70299999999999996,"15min":0.68899999999999995},"min":{"1min":0.45700000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.56599999999999995}]},{"osd":8,"last update":"Thu Nov 16 18:22:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.60799999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.48899999999999999}]},{"osd":10,"last update":"Thu Nov 16 18:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.64000000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953518,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6484,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738668,"statfs":{"total":95995035648,"available":95988396032,"internally_reserved":0,"allocated":929792,"data_stored":461681,"data_compressed":11676,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.504,"15min":0.48699999999999999},"min":{"1min":0.39900000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.752,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.48899999999999999,"15min":0.46400000000000002},"min":{"1min":0.41799999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.502}]},{"osd":1,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.39200000000000002,"15min":0.372},"min":{"1min":0.28799999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.42899999999999999,"15min":0.40999999999999998},"min":{"1min":0.312,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.434}]},{"osd":2,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.372,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.59599999999999997,"15min":0.56799999999999995},"min":{"1min":0.48599999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.55800000000000005,"15min":0.53500000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.54500000000000004,"15min":0.52700000000000002},"min":{"1min":0.40000000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80400000000000005,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.51700000000000002}]},{"osd":5,"last update":"Thu Nov 16 18:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.63900000000000001,"15min":0.60999999999999999},"min":{"1min":0.46500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.98599999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.65000000000000002,"15min":0.63400000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.72799999999999998}]},{"osd":6,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.46200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.749,"5min":0.65100000000000002,"15min":0.63500000000000001},"min":{"1min":0.499,"5min":0.432,"15min":0.432},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.53500000000000003}]},{"osd":7,"last update":"Thu Nov 16 18:22:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.58799999999999997,"15min":0.56299999999999994},"min":{"1min":0.37,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.55800000000000005,"15min":0.54700000000000004},"min":{"1min":0.443,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.56000000000000005}]},{"osd":8,"last update":"Thu Nov 16 18:23:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.84799999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.54500000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.91900000000000004,"5min":1.109,"15min":1.109},"last":0.624}]},{"osd":9,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.69299999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.67400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]},{"osd":5,"up_from":37,"seq":158913789993,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6608,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738544,"statfs":{"total":95995035648,"available":95988269056,"internally_reserved":0,"allocated":991232,"data_stored":1052482,"data_compressed":47845,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52400000000000002,"15min":0.51200000000000001},"min":{"1min":0.38500000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.751,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.501,"5min":0.59899999999999998,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64800000000000002,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.69199999999999995}]},{"osd":1,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53300000000000003,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.37,"15min":0.37},"max":{"1min":0.78600000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.52700000000000002,"15min":0.51400000000000001},"min":{"1min":0.36799999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.746,"5min":0.873,"15min":0.873},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39500000000000002,"15min":0.39200000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39600000000000002,"15min":0.38},"min":{"1min":0.28100000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84699999999999998,"5min":0.88,"15min":0.88},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.375,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56799999999999995,"15min":0.56299999999999994},"min":{"1min":0.41399999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82199999999999995,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.54900000000000004}]},{"osd":6,"last update":"Thu Nov 16 18:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.875,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59599999999999997,"15min":0.59499999999999997},"min":{"1min":0.47599999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.71099999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:22:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67100000000000004}]},{"osd":8,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.83599999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.753,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.505}]},{"osd":9,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.79600000000000004}]},{"osd":10,"last update":"Thu Nov 16 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.747}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34529,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T18:23:11.104 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-16T18:23:11.104 INFO:tasks.cephadm:Setting up client nodes... 2023-11-16T18:23:11.105 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-11-16T18:23:11.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:10 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1188611400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:23:11.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:10 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1643514571' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:11.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:10 smithi040 ceph-mon[100251]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:23:11.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:10 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1188611400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:23:11.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:10 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1643514571' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:11.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:10 smithi188 ceph-mon[100639]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:23:11.790 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:12.233 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:23:12 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-b[125477]: 2023-11-16T18:23:12.171+0000 7f68bf57f700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-11-16T18:23:12.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:11 smithi040 ceph-mon[100251]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-11-16T18:23:12.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:11 smithi040 ceph-mon[100251]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:23:12.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:11 smithi040 ceph-mon[100251]: pgmap v273: 129 pgs: 129 active+clean; 619 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2023-11-16T18:23:12.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:11 smithi188 ceph-mon[100639]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-11-16T18:23:12.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:11 smithi188 ceph-mon[100639]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:23:12.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:11 smithi188 ceph-mon[100639]: pgmap v273: 129 pgs: 129 active+clean; 619 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2023-11-16T18:23:12.302 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:11 smithi002 ceph-mon[98807]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-11-16T18:23:12.302 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:11 smithi002 ceph-mon[98807]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:23:12.302 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:11 smithi002 ceph-mon[98807]: pgmap v273: 129 pgs: 129 active+clean; 619 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2023-11-16T18:23:13.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.006+0000 7f3a5dd62700 1 -- 172.21.15.2:0/2717745254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a58100310 msgr2=0x7f3a58100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:13.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.006+0000 7f3a5dd62700 1 --2- 172.21.15.2:0/2717745254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a58100310 0x7f3a58100720 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009990 tx=0x7f3a40009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:13.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.006+0000 7f3a5dd62700 1 -- 172.21.15.2:0/2717745254 shutdown_connections 2023-11-16T18:23:13.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.006+0000 7f3a5dd62700 1 --2- 172.21.15.2:0/2717745254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a58101ea0 0x7f3a58108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:13.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.006+0000 7f3a5dd62700 1 --2- 172.21.15.2:0/2717745254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a58100310 0x7f3a58100720 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:13.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.006+0000 7f3a5dd62700 1 --2- 172.21.15.2:0/2717745254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a58101510 0x7f3a58101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:13.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.006+0000 7f3a5dd62700 1 -- 172.21.15.2:0/2717745254 >> 172.21.15.2:0/2717745254 conn(0x7f3a580fb960 msgr2=0x7f3a580fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:13.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.006+0000 7f3a5dd62700 1 -- 172.21.15.2:0/2717745254 shutdown_connections 2023-11-16T18:23:13.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.006+0000 7f3a5dd62700 1 -- 172.21.15.2:0/2717745254 wait complete. 2023-11-16T18:23:13.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.007+0000 7f3a5dd62700 1 Processor -- start 2023-11-16T18:23:13.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a5dd62700 1 -- start start 2023-11-16T18:23:13.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a5dd62700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a58100310 0x7f3a58196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:13.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a5dd62700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a58101510 0x7f3a581965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:13.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a5dd62700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a58101ea0 0x7f3a5819d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:13.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a5dd62700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5819dc20 con 0x7f3a58100310 2023-11-16T18:23:13.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a5dd62700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5819dd60 con 0x7f3a58101ea0 2023-11-16T18:23:13.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a5dd62700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5819def0 con 0x7f3a58101510 2023-11-16T18:23:13.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a58100310 0x7f3a58196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:13.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a58100310 0x7f3a58196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48400/0 (socket says 172.21.15.2:48400) 2023-11-16T18:23:13.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a577fe700 1 -- 172.21.15.2:0/532253479 learned_addr learned my addr 172.21.15.2:0/532253479 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:13.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a57fff700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a58101ea0 0x7f3a5819d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:13.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a577fe700 1 -- 172.21.15.2:0/532253479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a58101510 msgr2=0x7f3a581965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:13.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a56ffd700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a58101510 0x7f3a581965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:13.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a577fe700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a58101510 0x7f3a581965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:13.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a577fe700 1 -- 172.21.15.2:0/532253479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a58101ea0 msgr2=0x7f3a5819d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:13.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a577fe700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a58101ea0 0x7f3a5819d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:13.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a577fe700 1 -- 172.21.15.2:0/532253479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a400095f0 con 0x7f3a58100310 2023-11-16T18:23:13.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a56ffd700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a58101510 0x7f3a581965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:13.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a57fff700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a58101ea0 0x7f3a5819d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:13.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.008+0000 7f3a577fe700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a58100310 0x7f3a58196090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3a40000c00 tx=0x7f3a4000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:13.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.009+0000 7f3a54ff9700 1 -- 172.21.15.2:0/532253479 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4001d070 con 0x7f3a58100310 2023-11-16T18:23:13.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.009+0000 7f3a5dd62700 1 -- 172.21.15.2:0/532253479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5810bcf0 con 0x7f3a58100310 2023-11-16T18:23:13.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.009+0000 7f3a54ff9700 1 -- 172.21.15.2:0/532253479 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a4000fb70 con 0x7f3a58100310 2023-11-16T18:23:13.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.009+0000 7f3a54ff9700 1 -- 172.21.15.2:0/532253479 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40017650 con 0x7f3a58100310 2023-11-16T18:23:13.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.009+0000 7f3a5dd62700 1 -- 172.21.15.2:0/532253479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5810bb60 con 0x7f3a58100310 2023-11-16T18:23:13.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.010+0000 7f3a5dd62700 1 -- 172.21.15.2:0/532253479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a5819e480 con 0x7f3a58100310 2023-11-16T18:23:13.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.011+0000 7f3a54ff9700 1 -- 172.21.15.2:0/532253479 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f3a4000fce0 con 0x7f3a58100310 2023-11-16T18:23:13.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.011+0000 7f3a54ff9700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3a440a11e0 0x7f3a440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:13.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.012+0000 7f3a54ff9700 1 -- 172.21.15.2:0/532253479 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3a400ce150 con 0x7f3a58100310 2023-11-16T18:23:13.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.012+0000 7f3a56ffd700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3a440a11e0 0x7f3a440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:13.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.012+0000 7f3a56ffd700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3a440a11e0 0x7f3a440a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3a48007900 tx=0x7f3a48008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:13.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.016+0000 7f3a54ff9700 1 -- 172.21.15.2:0/532253479 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3a40097690 con 0x7f3a58100310 2023-11-16T18:23:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.293+0000 7f3a5dd62700 1 -- 172.21.15.2:0/532253479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f3a58107260 con 0x7f3a58100310 2023-11-16T18:23:13.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.298+0000 7f3a54ff9700 1 -- 172.21.15.2:0/532253479 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7f3a4009acb0 con 0x7f3a58100310 2023-11-16T18:23:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2023-11-16T18:23:13.304 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQARXlZlxHOjERAAC3bsbajbAKZZu4luXH/4VQ== 2023-11-16T18:23:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.300+0000 7f3a5dd62700 1 -- 172.21.15.2:0/532253479 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3a440a11e0 msgr2=0x7f3a440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.300+0000 7f3a5dd62700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3a440a11e0 0x7f3a440a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3a48007900 tx=0x7f3a48008040 comp rx=0 tx=0).stop 2023-11-16T18:23:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.300+0000 7f3a5dd62700 1 -- 172.21.15.2:0/532253479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a58100310 msgr2=0x7f3a58196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.300+0000 7f3a5dd62700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a58100310 0x7f3a58196090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3a40000c00 tx=0x7f3a4000f740 comp rx=0 tx=0).stop 2023-11-16T18:23:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.300+0000 7f3a5dd62700 1 -- 172.21.15.2:0/532253479 shutdown_connections 2023-11-16T18:23:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.300+0000 7f3a5dd62700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3a440a11e0 0x7f3a440a3690 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.300+0000 7f3a5dd62700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a58101ea0 0x7f3a5819d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.300+0000 7f3a5dd62700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a58100310 0x7f3a58196090 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.300+0000 7f3a5dd62700 1 --2- 172.21.15.2:0/532253479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a58101510 0x7f3a581965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:13.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.300+0000 7f3a5dd62700 1 -- 172.21.15.2:0/532253479 >> 172.21.15.2:0/532253479 conn(0x7f3a580fb960 msgr2=0x7f3a580fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:13.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.300+0000 7f3a5dd62700 1 -- 172.21.15.2:0/532253479 shutdown_connections 2023-11-16T18:23:13.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:13.300+0000 7f3a5dd62700 1 -- 172.21.15.2:0/532253479 wait complete. 2023-11-16T18:23:13.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:13 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/532253479' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-16T18:23:13.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:13 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/532253479' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-16T18:23:13.658 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:23:13.658 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-11-16T18:23:13.658 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-16T18:23:13.701 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-16T18:23:13.701 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-11-16T18:23:13.701 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph mgr dump --format=json 2023-11-16T18:23:13.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:13 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/532253479' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-16T18:23:13.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:13 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/532253479' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-16T18:23:13.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:13 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/532253479' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-16T18:23:13.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:13 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/532253479' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-16T18:23:14.390 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:14.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:14 smithi002 ceph-mon[98807]: pgmap v274: 129 pgs: 129 active+clean; 619 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2023-11-16T18:23:14.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:14 smithi040 ceph-mon[100251]: pgmap v274: 129 pgs: 129 active+clean; 619 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2023-11-16T18:23:14.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:14 smithi188 ceph-mon[100639]: pgmap v274: 129 pgs: 129 active+clean; 619 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2023-11-16T18:23:15.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.618+0000 7f47db7a5700 1 -- 172.21.15.2:0/236623210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47d40ff460 msgr2=0x7f47d40ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:15.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.618+0000 7f47db7a5700 1 --2- 172.21.15.2:0/236623210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47d40ff460 0x7f47d40ff830 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f47c80099c0 tx=0x7f47c8009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:15.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.618+0000 7f47db7a5700 1 -- 172.21.15.2:0/236623210 shutdown_connections 2023-11-16T18:23:15.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.618+0000 7f47db7a5700 1 --2- 172.21.15.2:0/236623210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47d4104500 0x7f47d410c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:15.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.618+0000 7f47db7a5700 1 --2- 172.21.15.2:0/236623210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47d40ff460 0x7f47d40ff830 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:15.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.618+0000 7f47db7a5700 1 --2- 172.21.15.2:0/236623210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d40ffe00 0x7f47d4103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:15.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.618+0000 7f47db7a5700 1 -- 172.21.15.2:0/236623210 >> 172.21.15.2:0/236623210 conn(0x7f47d40770a0 msgr2=0x7f47d40774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:15.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.618+0000 7f47db7a5700 1 -- 172.21.15.2:0/236623210 shutdown_connections 2023-11-16T18:23:15.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.618+0000 7f47db7a5700 1 -- 172.21.15.2:0/236623210 wait complete. 2023-11-16T18:23:15.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47db7a5700 1 Processor -- start 2023-11-16T18:23:15.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47db7a5700 1 -- start start 2023-11-16T18:23:15.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47db7a5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47d40ff460 0x7f47d4100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:15.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47db7a5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47d40ffe00 0x7f47d41013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:15.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47db7a5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d4104500 0x7f47d4101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:15.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47db7a5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d41a4890 con 0x7f47d40ff460 2023-11-16T18:23:15.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47db7a5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d41a4a00 con 0x7f47d4104500 2023-11-16T18:23:15.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47db7a5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d41a4bc0 con 0x7f47d40ffe00 2023-11-16T18:23:15.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47d9541700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47d40ff460 0x7f47d4100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:15.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47d9541700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47d40ff460 0x7f47d4100e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48414/0 (socket says 172.21.15.2:48414) 2023-11-16T18:23:15.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47d9541700 1 -- 172.21.15.2:0/2212821076 learned_addr learned my addr 172.21.15.2:0/2212821076 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:15.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47d9d42700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d4104500 0x7f47d4101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:15.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47d8d40700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47d40ffe00 0x7f47d41013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:15.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47d9541700 1 -- 172.21.15.2:0/2212821076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47d40ffe00 msgr2=0x7f47d41013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:15.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47d9541700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47d40ffe00 0x7f47d41013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:15.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47d9541700 1 -- 172.21.15.2:0/2212821076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d4104500 msgr2=0x7f47d4101930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:15.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47d9541700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d4104500 0x7f47d4101930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:15.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.619+0000 7f47d9541700 1 -- 172.21.15.2:0/2212821076 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c80095f0 con 0x7f47d40ff460 2023-11-16T18:23:15.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.620+0000 7f47d8d40700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47d40ffe00 0x7f47d41013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:15.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.620+0000 7f47d9d42700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d4104500 0x7f47d4101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:15.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.620+0000 7f47d9541700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47d40ff460 0x7f47d4100e70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f47c8005910 tx=0x7f47c80036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:15.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.620+0000 7f47c67fc700 1 -- 172.21.15.2:0/2212821076 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c801d070 con 0x7f47d40ff460 2023-11-16T18:23:15.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.620+0000 7f47c67fc700 1 -- 172.21.15.2:0/2212821076 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47c8004200 con 0x7f47d40ff460 2023-11-16T18:23:15.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.620+0000 7f47db7a5700 1 -- 172.21.15.2:0/2212821076 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d410f2c0 con 0x7f47d40ff460 2023-11-16T18:23:15.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.620+0000 7f47c67fc700 1 -- 172.21.15.2:0/2212821076 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c800f460 con 0x7f47d40ff460 2023-11-16T18:23:15.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.620+0000 7f47db7a5700 1 -- 172.21.15.2:0/2212821076 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d410f130 con 0x7f47d40ff460 2023-11-16T18:23:15.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.621+0000 7f47db7a5700 1 -- 172.21.15.2:0/2212821076 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d410f450 con 0x7f47d40ff460 2023-11-16T18:23:15.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.622+0000 7f47c67fc700 1 -- 172.21.15.2:0/2212821076 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f47c80226c0 con 0x7f47d40ff460 2023-11-16T18:23:15.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.622+0000 7f47c67fc700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f47c00a11e0 0x7f47c00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:15.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.623+0000 7f47d8d40700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f47c00a11e0 0x7f47c00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:15.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.623+0000 7f47c67fc700 1 -- 172.21.15.2:0/2212821076 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f47c80ce1a0 con 0x7f47d40ff460 2023-11-16T18:23:15.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.623+0000 7f47d8d40700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f47c00a11e0 0x7f47c00a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f47d0005fd0 tx=0x7f47d0005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:15.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.627+0000 7f47c67fc700 1 -- 172.21.15.2:0/2212821076 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f47c80976d0 con 0x7f47d40ff460 2023-11-16T18:23:15.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.879+0000 7f47db7a5700 1 -- 172.21.15.2:0/2212821076 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f47d4066ea0 con 0x7f47d40ff460 2023-11-16T18:23:15.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.884+0000 7f47c67fc700 1 -- 172.21.15.2:0/2212821076 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260117 (secure 0 0 0) 0x7f47c809acf0 con 0x7f47d40ff460 2023-11-16T18:23:15.888 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.887+0000 7f47db7a5700 1 -- 172.21.15.2:0/2212821076 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f47c00a11e0 msgr2=0x7f47c00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.887+0000 7f47db7a5700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f47c00a11e0 0x7f47c00a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f47d0005fd0 tx=0x7f47d0005dc0 comp rx=0 tx=0).stop 2023-11-16T18:23:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.887+0000 7f47db7a5700 1 -- 172.21.15.2:0/2212821076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47d40ff460 msgr2=0x7f47d4100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.887+0000 7f47db7a5700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47d40ff460 0x7f47d4100e70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f47c8005910 tx=0x7f47c80036f0 comp rx=0 tx=0).stop 2023-11-16T18:23:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.887+0000 7f47db7a5700 1 -- 172.21.15.2:0/2212821076 shutdown_connections 2023-11-16T18:23:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.887+0000 7f47db7a5700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f47c00a11e0 0x7f47c00a3690 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.887+0000 7f47db7a5700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d4104500 0x7f47d4101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.887+0000 7f47db7a5700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47d40ff460 0x7f47d4100e70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.887+0000 7f47db7a5700 1 --2- 172.21.15.2:0/2212821076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47d40ffe00 0x7f47d41013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.887+0000 7f47db7a5700 1 -- 172.21.15.2:0/2212821076 >> 172.21.15.2:0/2212821076 conn(0x7f47d40770a0 msgr2=0x7f47d4068730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.887+0000 7f47db7a5700 1 -- 172.21.15.2:0/2212821076 shutdown_connections 2023-11-16T18:23:15.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:15.887+0000 7f47db7a5700 1 -- 172.21.15.2:0/2212821076 wait complete. 2023-11-16T18:23:16.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:15 smithi002 ceph-mon[98807]: pgmap v275: 129 pgs: 129 active+clean; 623 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.1 KiB/s wr, 13 op/s 2023-11-16T18:23:16.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:15 smithi040 ceph-mon[100251]: pgmap v275: 129 pgs: 129 active+clean; 623 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.1 KiB/s wr, 13 op/s 2023-11-16T18:23:16.256 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":17,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3369431713},{"type":"v1","addr":"172.21.15.2:6801","nonce":3369431713}]},"active_addr":"172.21.15.2:6801/3369431713","active_change":"2023-11-16T18:16:37.723793+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24107,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.2:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3335793429}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3821858662}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1347603606}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1472878779}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2020256697}]}]} 2023-11-16T18:23:16.261 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-11-16T18:23:16.262 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-11-16T18:23:16.262 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-16T18:23:16.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:15 smithi188 ceph-mon[100639]: pgmap v275: 129 pgs: 129 active+clean; 623 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.1 KiB/s wr, 13 op/s 2023-11-16T18:23:16.917 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:17.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:16 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2212821076' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-16T18:23:17.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:16 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2212821076' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-16T18:23:17.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:16 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2212821076' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-16T18:23:18.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.126+0000 7ffa5471e700 1 -- 172.21.15.2:0/1368885522 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa4c0ffe00 msgr2=0x7ffa4c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:18.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.126+0000 7ffa5471e700 1 --2- 172.21.15.2:0/1368885522 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa4c0ffe00 0x7ffa4c103dd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0099c0 tx=0x7ffa3c009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:18.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.126+0000 7ffa5471e700 1 -- 172.21.15.2:0/1368885522 shutdown_connections 2023-11-16T18:23:18.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.126+0000 7ffa5471e700 1 --2- 172.21.15.2:0/1368885522 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa4c104500 0x7ffa4c10c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:18.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.126+0000 7ffa5471e700 1 --2- 172.21.15.2:0/1368885522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0ff460 0x7ffa4c0ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:18.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.126+0000 7ffa5471e700 1 --2- 172.21.15.2:0/1368885522 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa4c0ffe00 0x7ffa4c103dd0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:18.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.126+0000 7ffa5471e700 1 -- 172.21.15.2:0/1368885522 >> 172.21.15.2:0/1368885522 conn(0x7ffa4c0770a0 msgr2=0x7ffa4c0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:18.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.126+0000 7ffa5471e700 1 -- 172.21.15.2:0/1368885522 shutdown_connections 2023-11-16T18:23:18.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.126+0000 7ffa5471e700 1 -- 172.21.15.2:0/1368885522 wait complete. 2023-11-16T18:23:18.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa5471e700 1 Processor -- start 2023-11-16T18:23:18.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa5471e700 1 -- start start 2023-11-16T18:23:18.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa5471e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa4c0ff460 0x7ffa4c100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:18.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa5471e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0ffe00 0x7ffa4c1013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:18.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa5471e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa4c104500 0x7ffa4c101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:18.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa5471e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c1a4940 con 0x7ffa4c0ff460 2023-11-16T18:23:18.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa5471e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c1a4ab0 con 0x7ffa4c0ffe00 2023-11-16T18:23:18.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa5471e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c1a4c70 con 0x7ffa4c104500 2023-11-16T18:23:18.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa524ba700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa4c0ff460 0x7ffa4c100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:18.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa524ba700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa4c0ff460 0x7ffa4c100e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48426/0 (socket says 172.21.15.2:48426) 2023-11-16T18:23:18.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa524ba700 1 -- 172.21.15.2:0/688164872 learned_addr learned my addr 172.21.15.2:0/688164872 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:18.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa51cb9700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0ffe00 0x7ffa4c1013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:18.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa52cbb700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa4c104500 0x7ffa4c101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:18.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa524ba700 1 -- 172.21.15.2:0/688164872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa4c104500 msgr2=0x7ffa4c101930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:18.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa524ba700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa4c104500 0x7ffa4c101930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:18.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa524ba700 1 -- 172.21.15.2:0/688164872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0ffe00 msgr2=0x7ffa4c1013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:18.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa524ba700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0ffe00 0x7ffa4c1013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:18.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa524ba700 1 -- 172.21.15.2:0/688164872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa3c0095f0 con 0x7ffa4c0ff460 2023-11-16T18:23:18.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.127+0000 7ffa524ba700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa4c0ff460 0x7ffa4c100e70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800d950 tx=0x7ffa4800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:18.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.128+0000 7ffa437fe700 1 -- 172.21.15.2:0/688164872 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa48009940 con 0x7ffa4c0ff460 2023-11-16T18:23:18.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.128+0000 7ffa437fe700 1 -- 172.21.15.2:0/688164872 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa48010460 con 0x7ffa4c0ff460 2023-11-16T18:23:18.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.128+0000 7ffa437fe700 1 -- 172.21.15.2:0/688164872 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4800f620 con 0x7ffa4c0ff460 2023-11-16T18:23:18.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.128+0000 7ffa5471e700 1 -- 172.21.15.2:0/688164872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c10f230 con 0x7ffa4c0ff460 2023-11-16T18:23:18.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.128+0000 7ffa5471e700 1 -- 172.21.15.2:0/688164872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c10f0a0 con 0x7ffa4c0ff460 2023-11-16T18:23:18.134 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.129+0000 7ffa5471e700 1 -- 172.21.15.2:0/688164872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c10f3c0 con 0x7ffa4c0ff460 2023-11-16T18:23:18.134 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.130+0000 7ffa437fe700 1 -- 172.21.15.2:0/688164872 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ffa480108d0 con 0x7ffa4c0ff460 2023-11-16T18:23:18.134 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.130+0000 7ffa437fe700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ffa380a1260 0x7ffa380a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:18.134 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.131+0000 7ffa51cb9700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ffa380a1260 0x7ffa380a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:18.134 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.131+0000 7ffa437fe700 1 -- 172.21.15.2:0/688164872 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ffa480cca40 con 0x7ffa4c0ff460 2023-11-16T18:23:18.134 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.131+0000 7ffa51cb9700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ffa380a1260 0x7ffa380a3710 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00b580 tx=0x7ffa3c003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:18.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.134+0000 7ffa437fe700 1 -- 172.21.15.2:0/688164872 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ffa48095e40 con 0x7ffa4c0ff460 2023-11-16T18:23:18.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:17 smithi040 ceph-mon[100251]: pgmap v276: 129 pgs: 129 active+clean; 623 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2023-11-16T18:23:18.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:17 smithi188 ceph-mon[100639]: pgmap v276: 129 pgs: 129 active+clean; 623 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2023-11-16T18:23:18.332 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.331+0000 7ffa5471e700 1 -- 172.21.15.2:0/688164872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa4c066ea0 con 0x7ffa4c0ff460 2023-11-16T18:23:18.333 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.332+0000 7ffa437fe700 1 -- 172.21.15.2:0/688164872 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23853 (secure 0 0 0) 0x7ffa48020090 con 0x7ffa4c0ff460 2023-11-16T18:23:18.334 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":86,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:22:51.534683+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:23:18.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.334+0000 7ffa5471e700 1 -- 172.21.15.2:0/688164872 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ffa380a1260 msgr2=0x7ffa380a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:18.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.334+0000 7ffa5471e700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ffa380a1260 0x7ffa380a3710 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00b580 tx=0x7ffa3c003680 comp rx=0 tx=0).stop 2023-11-16T18:23:18.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.334+0000 7ffa5471e700 1 -- 172.21.15.2:0/688164872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa4c0ff460 msgr2=0x7ffa4c100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:18.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.334+0000 7ffa5471e700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa4c0ff460 0x7ffa4c100e70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800d950 tx=0x7ffa4800dc60 comp rx=0 tx=0).stop 2023-11-16T18:23:18.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.334+0000 7ffa5471e700 1 -- 172.21.15.2:0/688164872 shutdown_connections 2023-11-16T18:23:18.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.334+0000 7ffa5471e700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ffa380a1260 0x7ffa380a3710 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:18.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.334+0000 7ffa5471e700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa4c104500 0x7ffa4c101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:18.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.334+0000 7ffa5471e700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa4c0ff460 0x7ffa4c100e70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:18.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.334+0000 7ffa5471e700 1 --2- 172.21.15.2:0/688164872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0ffe00 0x7ffa4c1013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:18.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.334+0000 7ffa5471e700 1 -- 172.21.15.2:0/688164872 >> 172.21.15.2:0/688164872 conn(0x7ffa4c0770a0 msgr2=0x7ffa4c068790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:18.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.335+0000 7ffa5471e700 1 -- 172.21.15.2:0/688164872 shutdown_connections 2023-11-16T18:23:18.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:18.335+0000 7ffa5471e700 1 -- 172.21.15.2:0/688164872 wait complete. 2023-11-16T18:23:18.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:17 smithi002 ceph-mon[98807]: pgmap v276: 129 pgs: 129 active+clean; 623 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2023-11-16T18:23:18.784 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-11-16T18:23:18.785 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-16T18:23:19.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:18 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/688164872' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:19.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:18 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/688164872' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:19.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:18 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/688164872' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:19.443 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:19.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:19 smithi002 ceph-mon[98807]: pgmap v277: 129 pgs: 129 active+clean; 623 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2023-11-16T18:23:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:19 smithi040 ceph-mon[100251]: pgmap v277: 129 pgs: 129 active+clean; 623 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2023-11-16T18:23:20.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:19 smithi188 ceph-mon[100639]: pgmap v277: 129 pgs: 129 active+clean; 623 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2023-11-16T18:23:20.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.651+0000 7f7f13e74700 1 -- 172.21.15.2:0/429700427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f0c1049b0 msgr2=0x7f7f0c104d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:20.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.651+0000 7f7f13e74700 1 --2- 172.21.15.2:0/429700427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f0c1049b0 0x7f7f0c104d80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7f08009990 tx=0x7f7f08009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:20.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.651+0000 7f7f13e74700 1 -- 172.21.15.2:0/429700427 shutdown_connections 2023-11-16T18:23:20.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.651+0000 7f7f13e74700 1 --2- 172.21.15.2:0/429700427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f0c0ff240 0x7f7f0c10b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:20.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.651+0000 7f7f13e74700 1 --2- 172.21.15.2:0/429700427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f0c1049b0 0x7f7f0c104d80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:20.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.651+0000 7f7f13e74700 1 --2- 172.21.15.2:0/429700427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f0c105350 0x7f7f0c0feb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:20.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.651+0000 7f7f13e74700 1 -- 172.21.15.2:0/429700427 >> 172.21.15.2:0/429700427 conn(0x7f7f0c074930 msgr2=0x7f7f0c074d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:20.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.651+0000 7f7f13e74700 1 -- 172.21.15.2:0/429700427 shutdown_connections 2023-11-16T18:23:20.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.651+0000 7f7f13e74700 1 -- 172.21.15.2:0/429700427 wait complete. 2023-11-16T18:23:20.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f13e74700 1 Processor -- start 2023-11-16T18:23:20.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f13e74700 1 -- start start 2023-11-16T18:23:20.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f13e74700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f0c0ff240 0x7f7f0c1963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:20.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f13e74700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f0c1049b0 0x7f7f0c1968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:20.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f13e74700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f0c19ad50 0x7f7f0c19b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:20.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f13e74700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c19b730 con 0x7f7f0c1049b0 2023-11-16T18:23:20.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f13e74700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c19b8a0 con 0x7f7f0c19ad50 2023-11-16T18:23:20.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f13e74700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c19ba60 con 0x7f7f0c0ff240 2023-11-16T18:23:20.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f1140f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f0c1049b0 0x7f7f0c1968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:20.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f1140f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f0c1049b0 0x7f7f0c1968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48440/0 (socket says 172.21.15.2:48440) 2023-11-16T18:23:20.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f1140f700 1 -- 172.21.15.2:0/2269989911 learned_addr learned my addr 172.21.15.2:0/2269989911 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:20.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f11c10700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f0c0ff240 0x7f7f0c1963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:20.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f12411700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f0c19ad50 0x7f7f0c19b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:20.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f1140f700 1 -- 172.21.15.2:0/2269989911 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f0c0ff240 msgr2=0x7f7f0c1963a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:20.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f1140f700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f0c0ff240 0x7f7f0c1963a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:20.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f1140f700 1 -- 172.21.15.2:0/2269989911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f0c19ad50 msgr2=0x7f7f0c19b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:20.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f1140f700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f0c19ad50 0x7f7f0c19b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:20.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.652+0000 7f7f1140f700 1 -- 172.21.15.2:0/2269989911 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f080095f0 con 0x7f7f0c1049b0 2023-11-16T18:23:20.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.653+0000 7f7f1140f700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f0c1049b0 0x7f7f0c1968e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00d950 tx=0x7f7efc00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:20.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.653+0000 7f7f02ffd700 1 -- 172.21.15.2:0/2269989911 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc00bb60 con 0x7f7f0c1049b0 2023-11-16T18:23:20.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.653+0000 7f7f02ffd700 1 -- 172.21.15.2:0/2269989911 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7efc00de30 con 0x7f7f0c1049b0 2023-11-16T18:23:20.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.653+0000 7f7f13e74700 1 -- 172.21.15.2:0/2269989911 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0c10ea30 con 0x7f7f0c1049b0 2023-11-16T18:23:20.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.653+0000 7f7f02ffd700 1 -- 172.21.15.2:0/2269989911 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc0055a0 con 0x7f7f0c1049b0 2023-11-16T18:23:20.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.653+0000 7f7f13e74700 1 -- 172.21.15.2:0/2269989911 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f0c10e8a0 con 0x7f7f0c1049b0 2023-11-16T18:23:20.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.654+0000 7f7f13e74700 1 -- 172.21.15.2:0/2269989911 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f0c10ebc0 con 0x7f7f0c1049b0 2023-11-16T18:23:20.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.655+0000 7f7f02ffd700 1 -- 172.21.15.2:0/2269989911 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f7efc004750 con 0x7f7f0c1049b0 2023-11-16T18:23:20.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.656+0000 7f7f02ffd700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7ef80a1260 0x7f7ef80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:20.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.656+0000 7f7f11c10700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7ef80a1260 0x7f7ef80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:20.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.656+0000 7f7f02ffd700 1 -- 172.21.15.2:0/2269989911 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7efc0ccd00 con 0x7f7f0c1049b0 2023-11-16T18:23:20.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.656+0000 7f7f11c10700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7ef80a1260 0x7f7ef80a3710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7f08006010 tx=0x7f7f0800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:20.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.660+0000 7f7f02ffd700 1 -- 172.21.15.2:0/2269989911 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7efc096100 con 0x7f7f0c1049b0 2023-11-16T18:23:20.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.858+0000 7f7f13e74700 1 -- 172.21.15.2:0/2269989911 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f0c066ea0 con 0x7f7f0c1049b0 2023-11-16T18:23:20.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.860+0000 7f7f02ffd700 1 -- 172.21.15.2:0/2269989911 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23853 (secure 0 0 0) 0x7f7efc016090 con 0x7f7f0c1049b0 2023-11-16T18:23:20.861 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:20.861 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":86,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:22:51.534683+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:23:20.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.862+0000 7f7f13e74700 1 -- 172.21.15.2:0/2269989911 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7ef80a1260 msgr2=0x7f7ef80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:20.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.862+0000 7f7f13e74700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7ef80a1260 0x7f7ef80a3710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7f08006010 tx=0x7f7f0800b540 comp rx=0 tx=0).stop 2023-11-16T18:23:20.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.862+0000 7f7f13e74700 1 -- 172.21.15.2:0/2269989911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f0c1049b0 msgr2=0x7f7f0c1968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:20.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.862+0000 7f7f13e74700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f0c1049b0 0x7f7f0c1968e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00d950 tx=0x7f7efc00dd10 comp rx=0 tx=0).stop 2023-11-16T18:23:20.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.862+0000 7f7f13e74700 1 -- 172.21.15.2:0/2269989911 shutdown_connections 2023-11-16T18:23:20.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.862+0000 7f7f13e74700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7ef80a1260 0x7f7ef80a3710 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:20.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.862+0000 7f7f13e74700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f0c0ff240 0x7f7f0c1963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:20.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.862+0000 7f7f13e74700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f0c19ad50 0x7f7f0c19b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:20.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.862+0000 7f7f13e74700 1 --2- 172.21.15.2:0/2269989911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f0c1049b0 0x7f7f0c1968e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:20.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.862+0000 7f7f13e74700 1 -- 172.21.15.2:0/2269989911 >> 172.21.15.2:0/2269989911 conn(0x7f7f0c074930 msgr2=0x7f7f0c0fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:20.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.862+0000 7f7f13e74700 1 -- 172.21.15.2:0/2269989911 shutdown_connections 2023-11-16T18:23:20.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:20.862+0000 7f7f13e74700 1 -- 172.21.15.2:0/2269989911 wait complete. 2023-11-16T18:23:21.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:20 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2269989911' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:21.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:20 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2269989911' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:21.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:20 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2269989911' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:21.312 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph tell osd.0 flush_pg_stats 2023-11-16T18:23:21.312 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph tell osd.1 flush_pg_stats 2023-11-16T18:23:21.312 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph tell osd.2 flush_pg_stats 2023-11-16T18:23:21.313 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph tell osd.3 flush_pg_stats 2023-11-16T18:23:21.313 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph tell osd.4 flush_pg_stats 2023-11-16T18:23:21.314 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph tell osd.5 flush_pg_stats 2023-11-16T18:23:21.314 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph tell osd.6 flush_pg_stats 2023-11-16T18:23:21.315 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph tell osd.7 flush_pg_stats 2023-11-16T18:23:21.315 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph tell osd.8 flush_pg_stats 2023-11-16T18:23:21.315 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph tell osd.9 flush_pg_stats 2023-11-16T18:23:21.316 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph tell osd.10 flush_pg_stats 2023-11-16T18:23:21.316 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph tell osd.11 flush_pg_stats 2023-11-16T18:23:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:22 smithi040 ceph-mon[100251]: pgmap v278: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2023-11-16T18:23:22.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:22 smithi188 ceph-mon[100639]: pgmap v278: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2023-11-16T18:23:22.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:22 smithi002 ceph-mon[98807]: pgmap v278: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2023-11-16T18:23:22.854 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:22.975 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:23.090 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:23.109 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:23.228 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:23.270 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:23.272 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:23.291 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:23.292 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:23.424 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:23.425 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:23.476 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:24.086 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:23 smithi002 ceph-mon[98807]: pgmap v279: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2023-11-16T18:23:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:23 smithi040 ceph-mon[100251]: pgmap v279: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2023-11-16T18:23:24.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:23 smithi188 ceph-mon[100639]: pgmap v279: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2023-11-16T18:23:26.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:25 smithi002 ceph-mon[98807]: pgmap v280: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-11-16T18:23:26.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:25 smithi040 ceph-mon[100251]: pgmap v280: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-11-16T18:23:26.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:25 smithi188 ceph-mon[100639]: pgmap v280: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-11-16T18:23:26.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.591+0000 7fe5033c4700 1 -- 172.21.15.2:0/2467301524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4fc10cc80 msgr2=0x7fe4fc10f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:26.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.591+0000 7fe5033c4700 1 --2- 172.21.15.2:0/2467301524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4fc10cc80 0x7fe4fc10f060 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe4f400b0a0 tx=0x7fe4f400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:26.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.591+0000 7fe5033c4700 1 -- 172.21.15.2:0/2467301524 shutdown_connections 2023-11-16T18:23:26.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.591+0000 7fe5033c4700 1 --2- 172.21.15.2:0/2467301524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4fc0ff0d0 0x7fe4fc0ff4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:26.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.591+0000 7fe5033c4700 1 --2- 172.21.15.2:0/2467301524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4fc10cc80 0x7fe4fc10f060 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:26.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.591+0000 7fe5033c4700 1 --2- 172.21.15.2:0/2467301524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4fc0ff9e0 0x7fe4fc10c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:26.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.591+0000 7fe5033c4700 1 -- 172.21.15.2:0/2467301524 >> 172.21.15.2:0/2467301524 conn(0x7fe4fc0770a0 msgr2=0x7fe4fc0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:26.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.591+0000 7fe5033c4700 1 -- 172.21.15.2:0/2467301524 shutdown_connections 2023-11-16T18:23:26.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.591+0000 7fe5033c4700 1 -- 172.21.15.2:0/2467301524 wait complete. 2023-11-16T18:23:26.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.592+0000 7fe5033c4700 1 Processor -- start 2023-11-16T18:23:26.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.592+0000 7fe5033c4700 1 -- start start 2023-11-16T18:23:26.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.592+0000 7fe5033c4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4fc0ff0d0 0x7fe4fc19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:26.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.592+0000 7fe5033c4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4fc0ff9e0 0x7fe4fc19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:26.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.592+0000 7fe5033c4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4fc10cc80 0x7fe4fc198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:26.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.592+0000 7fe5033c4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc198c70 con 0x7fe4fc0ff0d0 2023-11-16T18:23:26.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.592+0000 7fe5033c4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc198de0 con 0x7fe4fc0ff9e0 2023-11-16T18:23:26.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.592+0000 7fe5033c4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc198fa0 con 0x7fe4fc10cc80 2023-11-16T18:23:26.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.592+0000 7fe501160700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4fc0ff0d0 0x7fe4fc19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:26.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.592+0000 7fe501160700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4fc0ff0d0 0x7fe4fc19e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48450/0 (socket says 172.21.15.2:48450) 2023-11-16T18:23:26.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.592+0000 7fe501160700 1 -- 172.21.15.2:0/1562295519 learned_addr learned my addr 172.21.15.2:0/1562295519 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:26.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.593+0000 7fe501961700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4fc10cc80 0x7fe4fc198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:26.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.593+0000 7fe50095f700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4fc0ff9e0 0x7fe4fc19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:26.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.593+0000 7fe501160700 1 -- 172.21.15.2:0/1562295519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4fc10cc80 msgr2=0x7fe4fc198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:26.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.593+0000 7fe501160700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4fc10cc80 0x7fe4fc198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:26.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.593+0000 7fe501160700 1 -- 172.21.15.2:0/1562295519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4fc0ff9e0 msgr2=0x7fe4fc19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:26.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.593+0000 7fe501160700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4fc0ff9e0 0x7fe4fc19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:26.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.593+0000 7fe501160700 1 -- 172.21.15.2:0/1562295519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f4009d00 con 0x7fe4fc0ff0d0 2023-11-16T18:23:26.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.593+0000 7fe501160700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4fc0ff0d0 0x7fe4fc19e580 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe4f800b530 tx=0x7fe4f800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:26.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.593+0000 7fe4f27fc700 1 -- 172.21.15.2:0/1562295519 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f8011830 con 0x7fe4fc0ff0d0 2023-11-16T18:23:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.593+0000 7fe5033c4700 1 -- 172.21.15.2:0/1562295519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4fc1125a0 con 0x7fe4fc0ff0d0 2023-11-16T18:23:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.593+0000 7fe4f27fc700 1 -- 172.21.15.2:0/1562295519 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe4f8011e50 con 0x7fe4fc0ff0d0 2023-11-16T18:23:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.593+0000 7fe4f27fc700 1 -- 172.21.15.2:0/1562295519 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f800f2e0 con 0x7fe4fc0ff0d0 2023-11-16T18:23:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.593+0000 7fe5033c4700 1 -- 172.21.15.2:0/1562295519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4fc112410 con 0x7fe4fc0ff0d0 2023-11-16T18:23:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.594+0000 7fe5033c4700 1 -- 172.21.15.2:0/1562295519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe4fc10eb00 con 0x7fe4fc0ff0d0 2023-11-16T18:23:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.595+0000 7fe4f27fc700 1 -- 172.21.15.2:0/1562295519 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fe4f8010580 con 0x7fe4fc0ff0d0 2023-11-16T18:23:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.596+0000 7fe4f27fc700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe4e80a11e0 0x7fe4e80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.596+0000 7fe50095f700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe4e80a11e0 0x7fe4e80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.597+0000 7fe4f27fc700 1 -- 172.21.15.2:0/1562295519 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe4f80cccb0 con 0x7fe4fc0ff0d0 2023-11-16T18:23:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.597+0000 7fe4f27fc700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264] conn(0x7fe4e80abd50 0x7fe4e80ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.597+0000 7fe4f27fc700 1 -- 172.21.15.2:0/1562295519 --> [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe4e80ae850 con 0x7fe4e80abd50 2023-11-16T18:23:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.597+0000 7fe4f27fc700 1 -- 172.21.15.2:0/1562295519 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4f80cd0c0 con 0x7fe4fc0ff0d0 2023-11-16T18:23:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.597+0000 7fe50095f700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe4e80a11e0 0x7fe4e80a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec005950 tx=0x7fe4ec0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.597+0000 7fe501961700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264] conn(0x7fe4e80abd50 0x7fe4e80ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.598+0000 7fe501961700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264] conn(0x7fe4e80abd50 0x7fe4e80ae180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.598+0000 7fe4f27fc700 1 -- 172.21.15.2:0/1562295519 <== osd.8 v2:172.21.15.188:6816/3337313264 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe4e80ae850 con 0x7fe4e80abd50 2023-11-16T18:23:26.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.615+0000 7fe5033c4700 1 -- 172.21.15.2:0/1562295519 --> [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe4fc112730 con 0x7fe4e80abd50 2023-11-16T18:23:26.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.616+0000 7fe4f27fc700 1 -- 172.21.15.2:0/1562295519 <== osd.8 v2:172.21.15.188:6816/3337313264 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe4fc112730 con 0x7fe4e80abd50 2023-11-16T18:23:26.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.616+0000 7fe5033c4700 1 -- 172.21.15.2:0/1562295519 >> [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264] conn(0x7fe4e80abd50 msgr2=0x7fe4e80ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:26.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.616+0000 7fe5033c4700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264] conn(0x7fe4e80abd50 0x7fe4e80ae180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:26.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.617+0000 7fe5033c4700 1 -- 172.21.15.2:0/1562295519 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe4e80a11e0 msgr2=0x7fe4e80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:26.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.617+0000 7fe5033c4700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fe4e80a11e0 0x7fe4e80a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec005950 tx=0x7fe4ec0058e0 comp rx=0 tx=0).stop 2023-11-16T18:23:26.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.617+0000 7fe5033c4700 1 -- 172.21.15.2:0/1562295519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4fc0ff0d0 msgr2=0x7fe4fc19e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:26.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.617+0000 7fe5033c4700 1 --2- 172.21.15.2:0/1562295519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4fc0ff0d0 0x7fe4fc19e580 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe4f800b530 tx=0x7fe4f800b8f0 comp rx=0 tx=0).stop 2023-11-16T18:23:26.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.617+0000 7fe501961700 1 -- 172.21.15.2:0/1562295519 reap_dead start 2023-11-16T18:23:26.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.617+0000 7fe5033c4700 1 -- 172.21.15.2:0/1562295519 shutdown_connections 2023-11-16T18:23:26.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.617+0000 7fe5033c4700 1 -- 172.21.15.2:0/1562295519 >> 172.21.15.2:0/1562295519 conn(0x7fe4fc0770a0 msgr2=0x7fe4fc10e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:26.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.617+0000 7fe5033c4700 1 -- 172.21.15.2:0/1562295519 shutdown_connections 2023-11-16T18:23:26.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:26.617+0000 7fe5033c4700 1 -- 172.21.15.2:0/1562295519 wait complete. 2023-11-16T18:23:27.792 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.790+0000 7fae0db12700 1 -- 172.21.15.2:0/1758543229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae080ffe00 msgr2=0x7fae08103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:27.792 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.791+0000 7fae0db12700 1 --2- 172.21.15.2:0/1758543229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae080ffe00 0x7fae08103dd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fadfc0099c0 tx=0x7fadfc009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:27.792 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.791+0000 7fae0db12700 1 -- 172.21.15.2:0/1758543229 shutdown_connections 2023-11-16T18:23:27.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.791+0000 7fae0db12700 1 --2- 172.21.15.2:0/1758543229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae08104500 0x7fae0810c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:27.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.791+0000 7fae0db12700 1 --2- 172.21.15.2:0/1758543229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae080ff460 0x7fae080ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:27.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.791+0000 7fae0db12700 1 --2- 172.21.15.2:0/1758543229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae080ffe00 0x7fae08103dd0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:27.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.791+0000 7fae0db12700 1 -- 172.21.15.2:0/1758543229 >> 172.21.15.2:0/1758543229 conn(0x7fae080770a0 msgr2=0x7fae080774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:27.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.791+0000 7fae0db12700 1 -- 172.21.15.2:0/1758543229 shutdown_connections 2023-11-16T18:23:27.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.791+0000 7fae0db12700 1 -- 172.21.15.2:0/1758543229 wait complete. 2023-11-16T18:23:27.795 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.791+0000 7fae0db12700 1 Processor -- start 2023-11-16T18:23:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae0db12700 1 -- start start 2023-11-16T18:23:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae0db12700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae080ff460 0x7fae08100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae0db12700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae080ffe00 0x7fae081013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae0db12700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae08104500 0x7fae08101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae0db12700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae081a4940 con 0x7fae080ff460 2023-11-16T18:23:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae0db12700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae081a4ab0 con 0x7fae080ffe00 2023-11-16T18:23:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae0db12700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae081a4c70 con 0x7fae08104500 2023-11-16T18:23:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae077fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae080ff460 0x7fae08100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae077fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae080ff460 0x7fae08100e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48466/0 (socket says 172.21.15.2:48466) 2023-11-16T18:23:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae077fe700 1 -- 172.21.15.2:0/1832237013 learned_addr learned my addr 172.21.15.2:0/1832237013 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae06ffd700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae080ffe00 0x7fae081013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae07fff700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae08104500 0x7fae08101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae077fe700 1 -- 172.21.15.2:0/1832237013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae08104500 msgr2=0x7fae08101930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae077fe700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae08104500 0x7fae08101930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae077fe700 1 -- 172.21.15.2:0/1832237013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae080ffe00 msgr2=0x7fae081013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae077fe700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae080ffe00 0x7fae081013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae077fe700 1 -- 172.21.15.2:0/1832237013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc0095f0 con 0x7fae080ff460 2023-11-16T18:23:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae06ffd700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae080ffe00 0x7fae081013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.792+0000 7fae077fe700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae080ff460 0x7fae08100e70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fadf800d980 tx=0x7fadf800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.793+0000 7fae04ff9700 1 -- 172.21.15.2:0/1832237013 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf800bb60 con 0x7fae080ff460 2023-11-16T18:23:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.793+0000 7fae04ff9700 1 -- 172.21.15.2:0/1832237013 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fadf8007640 con 0x7fae080ff460 2023-11-16T18:23:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.793+0000 7fae04ff9700 1 -- 172.21.15.2:0/1832237013 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8004e90 con 0x7fae080ff460 2023-11-16T18:23:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.793+0000 7fae0db12700 1 -- 172.21.15.2:0/1832237013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0810f230 con 0x7fae080ff460 2023-11-16T18:23:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.793+0000 7fae0db12700 1 -- 172.21.15.2:0/1832237013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0810f0a0 con 0x7fae080ff460 2023-11-16T18:23:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.794+0000 7fae0db12700 1 -- 172.21.15.2:0/1832237013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fae080690e0 con 0x7fae080ff460 2023-11-16T18:23:27.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.795+0000 7fae04ff9700 1 -- 172.21.15.2:0/1832237013 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fadf800bcc0 con 0x7fae080ff460 2023-11-16T18:23:27.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.795+0000 7fae04ff9700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fadf00a1260 0x7fadf00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:27.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.796+0000 7fae06ffd700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fadf00a1260 0x7fadf00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:27.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.796+0000 7fae04ff9700 1 -- 172.21.15.2:0/1832237013 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fadf80cd110 con 0x7fae080ff460 2023-11-16T18:23:27.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.796+0000 7fae04ff9700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247] conn(0x7fadf00abc60 0x7fadf00ae090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:27.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.796+0000 7fae04ff9700 1 -- 172.21.15.2:0/1832237013 --> [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadf00ae760 con 0x7fadf00abc60 2023-11-16T18:23:27.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.796+0000 7fae04ff9700 1 -- 172.21.15.2:0/1832237013 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fadf80cd520 con 0x7fae080ff460 2023-11-16T18:23:27.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.796+0000 7fae06ffd700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fadf00a1260 0x7fadf00a3710 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fadfc000c00 tx=0x7fadfc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:27.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.796+0000 7fae07fff700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247] conn(0x7fadf00abc60 0x7fadf00ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:27.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.797+0000 7fae07fff700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247] conn(0x7fadf00abc60 0x7fadf00ae090 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:27.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.797+0000 7fae04ff9700 1 -- 172.21.15.2:0/1832237013 <== osd.1 v2:172.21.15.40:6800/3016934247 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fadf00ae760 con 0x7fadf00abc60 2023-11-16T18:23:27.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.813+0000 7fae0db12700 1 -- 172.21.15.2:0/1832237013 --> [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fae0810f3c0 con 0x7fadf00abc60 2023-11-16T18:23:27.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.814+0000 7fae04ff9700 1 -- 172.21.15.2:0/1832237013 <== osd.1 v2:172.21.15.40:6800/3016934247 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fae0810f3c0 con 0x7fadf00abc60 2023-11-16T18:23:27.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.814+0000 7fae0db12700 1 -- 172.21.15.2:0/1832237013 >> [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247] conn(0x7fadf00abc60 msgr2=0x7fadf00ae090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:27.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.814+0000 7fae0db12700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247] conn(0x7fadf00abc60 0x7fadf00ae090 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:27.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.815+0000 7fae0db12700 1 -- 172.21.15.2:0/1832237013 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fadf00a1260 msgr2=0x7fadf00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:27.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.815+0000 7fae0db12700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fadf00a1260 0x7fadf00a3710 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fadfc000c00 tx=0x7fadfc005fb0 comp rx=0 tx=0).stop 2023-11-16T18:23:27.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.815+0000 7fae0db12700 1 -- 172.21.15.2:0/1832237013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae080ff460 msgr2=0x7fae08100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:27.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.815+0000 7fae0db12700 1 --2- 172.21.15.2:0/1832237013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae080ff460 0x7fae08100e70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fadf800d980 tx=0x7fadf800dd40 comp rx=0 tx=0).stop 2023-11-16T18:23:27.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.815+0000 7fae07fff700 1 -- 172.21.15.2:0/1832237013 reap_dead start 2023-11-16T18:23:27.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.815+0000 7fae0db12700 1 -- 172.21.15.2:0/1832237013 shutdown_connections 2023-11-16T18:23:27.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.815+0000 7fae0db12700 1 -- 172.21.15.2:0/1832237013 >> 172.21.15.2:0/1832237013 conn(0x7fae080770a0 msgr2=0x7fae08068790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:27.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.815+0000 7fae0db12700 1 -- 172.21.15.2:0/1832237013 shutdown_connections 2023-11-16T18:23:27.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:27.815+0000 7fae0db12700 1 -- 172.21.15.2:0/1832237013 wait complete. 2023-11-16T18:23:28.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:27 smithi002 ceph-mon[98807]: pgmap v281: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-11-16T18:23:28.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:27 smithi040 ceph-mon[100251]: pgmap v281: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-11-16T18:23:28.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:27 smithi188 ceph-mon[100639]: pgmap v281: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-11-16T18:23:28.372 INFO:teuthology.orchestra.run.smithi002.stdout:244813135905 2023-11-16T18:23:28.373 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.8 2023-11-16T18:23:30.048 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:29 smithi002 ceph-mon[98807]: pgmap v282: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-11-16T18:23:30.152 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.827+0000 7f5ac6b40700 1 -- 172.21.15.2:0/26862715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ac00ffe00 msgr2=0x7f5ac0103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.152 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.827+0000 7f5ac6b40700 1 --2- 172.21.15.2:0/26862715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ac00ffe00 0x7f5ac0103dd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5aac0099c0 tx=0x7f5aac009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.152 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.827+0000 7f5ac6b40700 1 -- 172.21.15.2:0/26862715 shutdown_connections 2023-11-16T18:23:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.827+0000 7f5ac6b40700 1 --2- 172.21.15.2:0/26862715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ac00ff460 0x7f5ac00ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.827+0000 7f5ac6b40700 1 --2- 172.21.15.2:0/26862715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ac0104500 0x7f5ac010c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.827+0000 7f5ac6b40700 1 --2- 172.21.15.2:0/26862715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ac00ffe00 0x7f5ac0103dd0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.827+0000 7f5ac6b40700 1 -- 172.21.15.2:0/26862715 >> 172.21.15.2:0/26862715 conn(0x7f5ac00770a0 msgr2=0x7f5ac00774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.827+0000 7f5ac6b40700 1 -- 172.21.15.2:0/26862715 shutdown_connections 2023-11-16T18:23:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.827+0000 7f5ac6b40700 1 -- 172.21.15.2:0/26862715 wait complete. 2023-11-16T18:23:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.828+0000 7f5ac6b40700 1 Processor -- start 2023-11-16T18:23:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.828+0000 7f5ac6b40700 1 -- start start 2023-11-16T18:23:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.828+0000 7f5ac6b40700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ac00ff460 0x7f5ac0198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.828+0000 7f5ac6b40700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ac00ffe00 0x7f5ac0198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.828+0000 7f5ac6b40700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ac0104500 0x7f5ac019cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.828+0000 7f5ac6b40700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac019d5d0 con 0x7f5ac00ffe00 2023-11-16T18:23:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.828+0000 7f5ac6b40700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac019d740 con 0x7f5ac0104500 2023-11-16T18:23:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.828+0000 7f5ac6b40700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac019d900 con 0x7f5ac00ff460 2023-11-16T18:23:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.828+0000 7f5abffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ac00ffe00 0x7f5ac0198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.828+0000 7f5abffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ac00ffe00 0x7f5ac0198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48482/0 (socket says 172.21.15.2:48482) 2023-11-16T18:23:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.828+0000 7f5abffff700 1 -- 172.21.15.2:0/1432378101 learned_addr learned my addr 172.21.15.2:0/1432378101 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.829+0000 7f5ac48dc700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ac00ff460 0x7f5ac0198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.829+0000 7f5ac50dd700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ac0104500 0x7f5ac019cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.829+0000 7f5abffff700 1 -- 172.21.15.2:0/1432378101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ac00ff460 msgr2=0x7f5ac0198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.829+0000 7f5abffff700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ac00ff460 0x7f5ac0198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.829+0000 7f5abffff700 1 -- 172.21.15.2:0/1432378101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ac0104500 msgr2=0x7f5ac019cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.829+0000 7f5abffff700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ac0104500 0x7f5ac019cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.829+0000 7f5abffff700 1 -- 172.21.15.2:0/1432378101 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aac0095f0 con 0x7f5ac00ffe00 2023-11-16T18:23:30.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.829+0000 7f5abffff700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ac00ffe00 0x7f5ac0198b50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5aac009f50 tx=0x7f5aac005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:30.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.829+0000 7f5abdffb700 1 -- 172.21.15.2:0/1432378101 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac01d070 con 0x7f5ac00ffe00 2023-11-16T18:23:30.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.829+0000 7f5abdffb700 1 -- 172.21.15.2:0/1432378101 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5aac004010 con 0x7f5ac00ffe00 2023-11-16T18:23:30.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.829+0000 7f5abdffb700 1 -- 172.21.15.2:0/1432378101 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac00fdc0 con 0x7f5ac00ffe00 2023-11-16T18:23:30.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.829+0000 7f5ac6b40700 1 -- 172.21.15.2:0/1432378101 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac010f280 con 0x7f5ac00ffe00 2023-11-16T18:23:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.829+0000 7f5ac6b40700 1 -- 172.21.15.2:0/1432378101 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac010f0f0 con 0x7f5ac00ffe00 2023-11-16T18:23:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.830+0000 7f5ac6b40700 1 -- 172.21.15.2:0/1432378101 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5ac010bce0 con 0x7f5ac00ffe00 2023-11-16T18:23:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.831+0000 7f5abdffb700 1 -- 172.21.15.2:0/1432378101 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f5aac00f4f0 con 0x7f5ac00ffe00 2023-11-16T18:23:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.832+0000 7f5abdffb700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5ab00a1260 0x7f5ab00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.832+0000 7f5ac48dc700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5ab00a1260 0x7f5ab00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.832+0000 7f5abdffb700 1 -- 172.21.15.2:0/1432378101 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5aac0ce2a0 con 0x7f5ac00ffe00 2023-11-16T18:23:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.832+0000 7f5abdffb700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884] conn(0x7f5ab00abdd0 0x7f5ab00ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.832+0000 7f5abdffb700 1 -- 172.21.15.2:0/1432378101 --> [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ab00ae8d0 con 0x7f5ab00abdd0 2023-11-16T18:23:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.832+0000 7f5abdffb700 1 -- 172.21.15.2:0/1432378101 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5aac0ce6b0 con 0x7f5ac00ffe00 2023-11-16T18:23:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.832+0000 7f5ac50dd700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884] conn(0x7f5ab00abdd0 0x7f5ab00ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.832+0000 7f5ac48dc700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5ab00a1260 0x7f5ab00a3710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4005ea0 tx=0x7f5ab4005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.833+0000 7f5ac50dd700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884] conn(0x7f5ab00abdd0 0x7f5ab00ae200 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.833+0000 7f5abdffb700 1 -- 172.21.15.2:0/1432378101 <== osd.0 v2:172.21.15.2:6802/4123843884 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f5ab00ae8d0 con 0x7f5ab00abdd0 2023-11-16T18:23:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.849+0000 7f5ac6b40700 1 -- 172.21.15.2:0/1432378101 --> [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5ac010f410 con 0x7f5ab00abdd0 2023-11-16T18:23:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.850+0000 7f5abdffb700 1 -- 172.21.15.2:0/1432378101 <== osd.0 v2:172.21.15.2:6802/4123843884 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5ac010f410 con 0x7f5ab00abdd0 2023-11-16T18:23:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.850+0000 7f5ac6b40700 1 -- 172.21.15.2:0/1432378101 >> [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884] conn(0x7f5ab00abdd0 msgr2=0x7f5ab00ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.850+0000 7f5ac6b40700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884] conn(0x7f5ab00abdd0 0x7f5ab00ae200 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.850+0000 7f5ac6b40700 1 -- 172.21.15.2:0/1432378101 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5ab00a1260 msgr2=0x7f5ab00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.850+0000 7f5ac6b40700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5ab00a1260 0x7f5ab00a3710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4005ea0 tx=0x7f5ab4005e30 comp rx=0 tx=0).stop 2023-11-16T18:23:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.850+0000 7f5ac6b40700 1 -- 172.21.15.2:0/1432378101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ac00ffe00 msgr2=0x7f5ac0198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.850+0000 7f5ac6b40700 1 --2- 172.21.15.2:0/1432378101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ac00ffe00 0x7f5ac0198b50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5aac009f50 tx=0x7f5aac005ef0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.850+0000 7f5ac50dd700 1 -- 172.21.15.2:0/1432378101 reap_dead start 2023-11-16T18:23:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.850+0000 7f5ac6b40700 1 -- 172.21.15.2:0/1432378101 shutdown_connections 2023-11-16T18:23:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.850+0000 7f5ac6b40700 1 -- 172.21.15.2:0/1432378101 >> 172.21.15.2:0/1432378101 conn(0x7f5ac00770a0 msgr2=0x7f5ac0068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:30.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.850+0000 7f5ac6b40700 1 -- 172.21.15.2:0/1432378101 shutdown_connections 2023-11-16T18:23:30.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:29.850+0000 7f5ac6b40700 1 -- 172.21.15.2:0/1432378101 wait complete. 2023-11-16T18:23:30.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:29 smithi040 ceph-mon[100251]: pgmap v282: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-11-16T18:23:30.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:29 smithi188 ceph-mon[100639]: pgmap v282: 129 pgs: 129 active+clean; 635 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-11-16T18:23:30.333 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.332+0000 7f7a46040700 1 -- 172.21.15.2:0/363918848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a401055e0 msgr2=0x7f7a4010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.333 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.332+0000 7f7a46040700 1 --2- 172.21.15.2:0/363918848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a401055e0 0x7f7a4010d160 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800b0d0 tx=0x7f7a3800b3e0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.332+0000 7f7a46040700 1 -- 172.21.15.2:0/363918848 shutdown_connections 2023-11-16T18:23:30.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.332+0000 7f7a46040700 1 --2- 172.21.15.2:0/363918848 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a40100540 0x7f7a40100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.332+0000 7f7a46040700 1 --2- 172.21.15.2:0/363918848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a401055e0 0x7f7a4010d160 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.332+0000 7f7a46040700 1 --2- 172.21.15.2:0/363918848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a40100ee0 0x7f7a40104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.332+0000 7f7a46040700 1 -- 172.21.15.2:0/363918848 >> 172.21.15.2:0/363918848 conn(0x7f7a400fbdc0 msgr2=0x7f7a400fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:30.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.332+0000 7f7a46040700 1 -- 172.21.15.2:0/363918848 shutdown_connections 2023-11-16T18:23:30.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.332+0000 7f7a46040700 1 -- 172.21.15.2:0/363918848 wait complete. 2023-11-16T18:23:30.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.333+0000 7f7a46040700 1 Processor -- start 2023-11-16T18:23:30.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.333+0000 7f7a46040700 1 -- start start 2023-11-16T18:23:30.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.333+0000 7f7a46040700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a40100540 0x7f7a40072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.333+0000 7f7a46040700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a40100ee0 0x7f7a40073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.333+0000 7f7a46040700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a401055e0 0x7f7a401a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.333+0000 7f7a46040700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a40073c20 con 0x7f7a40100540 2023-11-16T18:23:30.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.333+0000 7f7a46040700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a401a4c10 con 0x7f7a401055e0 2023-11-16T18:23:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.333+0000 7f7a46040700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a401a4d80 con 0x7f7a40100ee0 2023-11-16T18:23:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.333+0000 7f7a3f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a40100540 0x7f7a40072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.333+0000 7f7a3f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a40100540 0x7f7a40072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48512/0 (socket says 172.21.15.2:48512) 2023-11-16T18:23:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.333+0000 7f7a3f7fe700 1 -- 172.21.15.2:0/1278692132 learned_addr learned my addr 172.21.15.2:0/1278692132 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.333+0000 7f7a3effd700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a40100ee0 0x7f7a40073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.334+0000 7f7a3ffff700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a401055e0 0x7f7a401a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.334+0000 7f7a3f7fe700 1 -- 172.21.15.2:0/1278692132 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a40100ee0 msgr2=0x7f7a40073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.334+0000 7f7a3f7fe700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a40100ee0 0x7f7a40073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.334+0000 7f7a3f7fe700 1 -- 172.21.15.2:0/1278692132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a401055e0 msgr2=0x7f7a401a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.334+0000 7f7a3f7fe700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a401055e0 0x7f7a401a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.334+0000 7f7a3f7fe700 1 -- 172.21.15.2:0/1278692132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a38009d00 con 0x7f7a40100540 2023-11-16T18:23:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.334+0000 7f7a3f7fe700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a40100540 0x7f7a40072df0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000e9a0 tx=0x7f7a3000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.334+0000 7f7a3cff9700 1 -- 172.21.15.2:0/1278692132 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30009790 con 0x7f7a40100540 2023-11-16T18:23:30.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.334+0000 7f7a3cff9700 1 -- 172.21.15.2:0/1278692132 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a3000cbd0 con 0x7f7a40100540 2023-11-16T18:23:30.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.334+0000 7f7a46040700 1 -- 172.21.15.2:0/1278692132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a40110320 con 0x7f7a40100540 2023-11-16T18:23:30.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.334+0000 7f7a3cff9700 1 -- 172.21.15.2:0/1278692132 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30018680 con 0x7f7a40100540 2023-11-16T18:23:30.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.334+0000 7f7a46040700 1 -- 172.21.15.2:0/1278692132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a40110190 con 0x7f7a40100540 2023-11-16T18:23:30.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.335+0000 7f7a46040700 1 -- 172.21.15.2:0/1278692132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7a400fd0f0 con 0x7f7a40100540 2023-11-16T18:23:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.336+0000 7f7a3cff9700 1 -- 172.21.15.2:0/1278692132 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f7a3000cd40 con 0x7f7a40100540 2023-11-16T18:23:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.337+0000 7f7a3cff9700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7a2c0a11e0 0x7f7a2c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.337+0000 7f7a3cff9700 1 -- 172.21.15.2:0/1278692132 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7a30014070 con 0x7f7a40100540 2023-11-16T18:23:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.337+0000 7f7a3effd700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7a2c0a11e0 0x7f7a2c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.337+0000 7f7a3cff9700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675] conn(0x7f7a2c0abd50 0x7f7a2c0ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.337+0000 7f7a3cff9700 1 -- 172.21.15.2:0/1278692132 --> [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a2c0ae850 con 0x7f7a2c0abd50 2023-11-16T18:23:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.337+0000 7f7a3cff9700 1 -- 172.21.15.2:0/1278692132 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a300cd6e0 con 0x7f7a40100540 2023-11-16T18:23:30.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.338+0000 7f7a3effd700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7a2c0a11e0 0x7f7a2c0a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7a400745e0 tx=0x7f7a28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:30.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.338+0000 7f7a3ffff700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675] conn(0x7f7a2c0abd50 0x7f7a2c0ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.338+0000 7f7a3ffff700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675] conn(0x7f7a2c0abd50 0x7f7a2c0ae180 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:30.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.339+0000 7f7a3cff9700 1 -- 172.21.15.2:0/1278692132 <== osd.11 v2:172.21.15.188:6824/2926471675 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f7a2c0ae850 con 0x7f7a2c0abd50 2023-11-16T18:23:30.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.355+0000 7f7a46040700 1 -- 172.21.15.2:0/1278692132 --> [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7a401104b0 con 0x7f7a2c0abd50 2023-11-16T18:23:30.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.356+0000 7f7a3cff9700 1 -- 172.21.15.2:0/1278692132 <== osd.11 v2:172.21.15.188:6824/2926471675 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7a401104b0 con 0x7f7a2c0abd50 2023-11-16T18:23:30.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.356+0000 7f7a46040700 1 -- 172.21.15.2:0/1278692132 >> [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675] conn(0x7f7a2c0abd50 msgr2=0x7f7a2c0ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.356+0000 7f7a46040700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675] conn(0x7f7a2c0abd50 0x7f7a2c0ae180 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.356+0000 7f7a46040700 1 -- 172.21.15.2:0/1278692132 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7a2c0a11e0 msgr2=0x7f7a2c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.356+0000 7f7a46040700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7a2c0a11e0 0x7f7a2c0a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7a400745e0 tx=0x7f7a28009260 comp rx=0 tx=0).stop 2023-11-16T18:23:30.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.356+0000 7f7a46040700 1 -- 172.21.15.2:0/1278692132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a40100540 msgr2=0x7f7a40072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.356+0000 7f7a46040700 1 --2- 172.21.15.2:0/1278692132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a40100540 0x7f7a40072df0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000e9a0 tx=0x7f7a3000ecb0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.356+0000 7f7a3ffff700 1 -- 172.21.15.2:0/1278692132 reap_dead start 2023-11-16T18:23:30.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.357+0000 7f7a46040700 1 -- 172.21.15.2:0/1278692132 shutdown_connections 2023-11-16T18:23:30.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.357+0000 7f7a46040700 1 -- 172.21.15.2:0/1278692132 >> 172.21.15.2:0/1278692132 conn(0x7f7a400fbdc0 msgr2=0x7f7a400fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:30.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.357+0000 7f7a46040700 1 -- 172.21.15.2:0/1278692132 shutdown_connections 2023-11-16T18:23:30.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.357+0000 7f7a46040700 1 -- 172.21.15.2:0/1278692132 wait complete. 2023-11-16T18:23:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.482+0000 7f2c450d9700 1 -- 172.21.15.2:0/2728270689 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c401013e0 msgr2=0x7f2c401017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.482+0000 7f2c450d9700 1 --2- 172.21.15.2:0/2728270689 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c401013e0 0x7f2c401017b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009990 tx=0x7f2c30009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.482+0000 7f2c450d9700 1 -- 172.21.15.2:0/2728270689 shutdown_connections 2023-11-16T18:23:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.482+0000 7f2c450d9700 1 --2- 172.21.15.2:0/2728270689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c4010a900 0x7f2c4010cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.482+0000 7f2c450d9700 1 --2- 172.21.15.2:0/2728270689 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c401013e0 0x7f2c401017b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.482+0000 7f2c450d9700 1 --2- 172.21.15.2:0/2728270689 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c40101cf0 0x7f2c4010a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.482+0000 7f2c450d9700 1 -- 172.21.15.2:0/2728270689 >> 172.21.15.2:0/2728270689 conn(0x7f2c400770d0 msgr2=0x7f2c400774d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:30.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.482+0000 7f2c450d9700 1 -- 172.21.15.2:0/2728270689 shutdown_connections 2023-11-16T18:23:30.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.482+0000 7f2c450d9700 1 -- 172.21.15.2:0/2728270689 wait complete. 2023-11-16T18:23:30.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.483+0000 7f2c450d9700 1 Processor -- start 2023-11-16T18:23:30.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.483+0000 7f2c450d9700 1 -- start start 2023-11-16T18:23:30.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.483+0000 7f2c450d9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c401013e0 0x7f2c40196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.483+0000 7f2c450d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40101cf0 0x7f2c40196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.483+0000 7f2c450d9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c4010a900 0x7f2c4019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.483+0000 7f2c450d9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4019b3f0 con 0x7f2c40101cf0 2023-11-16T18:23:30.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.483+0000 7f2c450d9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4019b560 con 0x7f2c401013e0 2023-11-16T18:23:30.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.483+0000 7f2c450d9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4019b720 con 0x7f2c4010a900 2023-11-16T18:23:30.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.484+0000 7f2c3e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40101cf0 0x7f2c40196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.484+0000 7f2c3e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40101cf0 0x7f2c40196970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48528/0 (socket says 172.21.15.2:48528) 2023-11-16T18:23:30.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.484+0000 7f2c3e59c700 1 -- 172.21.15.2:0/1520145788 learned_addr learned my addr 172.21.15.2:0/1520145788 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:30.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.484+0000 7f2c3ed9d700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c401013e0 0x7f2c40196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.484+0000 7f2c3f59e700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c4010a900 0x7f2c4019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.484+0000 7f2c3ed9d700 1 -- 172.21.15.2:0/1520145788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c4010a900 msgr2=0x7f2c4019ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.484+0000 7f2c3ed9d700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c4010a900 0x7f2c4019ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.484+0000 7f2c3ed9d700 1 -- 172.21.15.2:0/1520145788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40101cf0 msgr2=0x7f2c40196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.484+0000 7f2c3ed9d700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40101cf0 0x7f2c40196970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.484+0000 7f2c3ed9d700 1 -- 172.21.15.2:0/1520145788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c300095f0 con 0x7f2c401013e0 2023-11-16T18:23:30.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.484+0000 7f2c3e59c700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40101cf0 0x7f2c40196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:23:30.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.484+0000 7f2c3ed9d700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c401013e0 0x7f2c40196430 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c300059d0 tx=0x7f2c3000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:30.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.485+0000 7f2c27fff700 1 -- 172.21.15.2:0/1520145788 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3001d070 con 0x7f2c401013e0 2023-11-16T18:23:30.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.485+0000 7f2c27fff700 1 -- 172.21.15.2:0/1520145788 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c30005510 con 0x7f2c401013e0 2023-11-16T18:23:30.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.485+0000 7f2c450d9700 1 -- 172.21.15.2:0/1520145788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c40110370 con 0x7f2c401013e0 2023-11-16T18:23:30.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.485+0000 7f2c27fff700 1 -- 172.21.15.2:0/1520145788 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30004a00 con 0x7f2c401013e0 2023-11-16T18:23:30.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.485+0000 7f2c450d9700 1 -- 172.21.15.2:0/1520145788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c401101e0 con 0x7f2c401013e0 2023-11-16T18:23:30.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.486+0000 7f2c450d9700 1 -- 172.21.15.2:0/1520145788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2c400fc7d0 con 0x7f2c401013e0 2023-11-16T18:23:30.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.488+0000 7f2c27fff700 1 -- 172.21.15.2:0/1520145788 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f2c3000fb50 con 0x7f2c401013e0 2023-11-16T18:23:30.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.488+0000 7f2c27fff700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2c280a1260 0x7f2c280a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.489+0000 7f2c27fff700 1 -- 172.21.15.2:0/1520145788 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2c300ce470 con 0x7f2c401013e0 2023-11-16T18:23:30.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.489+0000 7f2c3e59c700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2c280a1260 0x7f2c280a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.489+0000 7f2c27fff700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130] conn(0x7f2c280abdd0 0x7f2c280ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:30.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.489+0000 7f2c27fff700 1 -- 172.21.15.2:0/1520145788 --> [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c280ae8d0 con 0x7f2c280abdd0 2023-11-16T18:23:30.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.489+0000 7f2c27fff700 1 -- 172.21.15.2:0/1520145788 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c300ce880 con 0x7f2c401013e0 2023-11-16T18:23:30.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.489+0000 7f2c3e59c700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2c280a1260 0x7f2c280a3710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2c40197a50 tx=0x7f2c3400b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:30.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.489+0000 7f2c3f59e700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130] conn(0x7f2c280abdd0 0x7f2c280ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:30.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.490+0000 7f2c3f59e700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130] conn(0x7f2c280abdd0 0x7f2c280ae200 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:30.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.490+0000 7f2c27fff700 1 -- 172.21.15.2:0/1520145788 <== osd.4 v2:172.21.15.40:6808/865046130 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f2c280ae8d0 con 0x7f2c280abdd0 2023-11-16T18:23:30.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.508+0000 7f2c450d9700 1 -- 172.21.15.2:0/1520145788 --> [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2c40110500 con 0x7f2c280abdd0 2023-11-16T18:23:30.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.509+0000 7f2c27fff700 1 -- 172.21.15.2:0/1520145788 <== osd.4 v2:172.21.15.40:6808/865046130 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2c40110500 con 0x7f2c280abdd0 2023-11-16T18:23:30.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.509+0000 7f2c450d9700 1 -- 172.21.15.2:0/1520145788 >> [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130] conn(0x7f2c280abdd0 msgr2=0x7f2c280ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.509+0000 7f2c450d9700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130] conn(0x7f2c280abdd0 0x7f2c280ae200 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:30.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.509+0000 7f2c450d9700 1 -- 172.21.15.2:0/1520145788 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2c280a1260 msgr2=0x7f2c280a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.509+0000 7f2c450d9700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2c280a1260 0x7f2c280a3710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2c40197a50 tx=0x7f2c3400b500 comp rx=0 tx=0).stop 2023-11-16T18:23:30.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.509+0000 7f2c450d9700 1 -- 172.21.15.2:0/1520145788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c401013e0 msgr2=0x7f2c40196430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:30.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.509+0000 7f2c450d9700 1 --2- 172.21.15.2:0/1520145788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c401013e0 0x7f2c40196430 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c300059d0 tx=0x7f2c3000f690 comp rx=0 tx=0).stop 2023-11-16T18:23:30.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.509+0000 7f2c3f59e700 1 -- 172.21.15.2:0/1520145788 reap_dead start 2023-11-16T18:23:30.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.509+0000 7f2c450d9700 1 -- 172.21.15.2:0/1520145788 shutdown_connections 2023-11-16T18:23:30.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.509+0000 7f2c450d9700 1 -- 172.21.15.2:0/1520145788 >> 172.21.15.2:0/1520145788 conn(0x7f2c400770d0 msgr2=0x7f2c400fbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:30.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.509+0000 7f2c450d9700 1 -- 172.21.15.2:0/1520145788 shutdown_connections 2023-11-16T18:23:30.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.509+0000 7f2c450d9700 1 -- 172.21.15.2:0/1520145788 wait complete. 2023-11-16T18:23:31.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.254+0000 7f17810f7700 1 -- 172.21.15.2:0/2602450775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f177c1084a0 msgr2=0x7f177c1109a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.254+0000 7f17810f7700 1 --2- 172.21.15.2:0/2602450775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f177c1084a0 0x7f177c1109a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1770009990 tx=0x7f1770009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.254+0000 7f17810f7700 1 -- 172.21.15.2:0/2602450775 shutdown_connections 2023-11-16T18:23:31.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.254+0000 7f17810f7700 1 --2- 172.21.15.2:0/2602450775 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f177c107b90 0x7f177c107f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.254+0000 7f17810f7700 1 --2- 172.21.15.2:0/2602450775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f177c110f70 0x7f177c113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.254+0000 7f17810f7700 1 --2- 172.21.15.2:0/2602450775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f177c1084a0 0x7f177c1109a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.254+0000 7f17810f7700 1 -- 172.21.15.2:0/2602450775 >> 172.21.15.2:0/2602450775 conn(0x7f177c076020 msgr2=0x7f177c078430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:31.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.255+0000 7f17810f7700 1 -- 172.21.15.2:0/2602450775 shutdown_connections 2023-11-16T18:23:31.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.255+0000 7f17810f7700 1 -- 172.21.15.2:0/2602450775 wait complete. 2023-11-16T18:23:31.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.255+0000 7f17810f7700 1 Processor -- start 2023-11-16T18:23:31.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f17810f7700 1 -- start start 2023-11-16T18:23:31.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f17810f7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f177c107b90 0x7f177c1a27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f17810f7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f177c1084a0 0x7f177c1a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f17810f7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f177c110f70 0x7f177c19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f17810f7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c19ced0 con 0x7f177c107b90 2023-11-16T18:23:31.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f17810f7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c19d040 con 0x7f177c110f70 2023-11-16T18:23:31.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f17810f7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c19d200 con 0x7f177c1084a0 2023-11-16T18:23:31.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f177ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f177c107b90 0x7f177c1a27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f177ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f177c107b90 0x7f177c1a27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48496/0 (socket says 172.21.15.2:48496) 2023-11-16T18:23:31.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f177ad9d700 1 -- 172.21.15.2:0/635110443 learned_addr learned my addr 172.21.15.2:0/635110443 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:31.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f177b59e700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f177c110f70 0x7f177c19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f177a59c700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f177c1084a0 0x7f177c1a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f177ad9d700 1 -- 172.21.15.2:0/635110443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f177c1084a0 msgr2=0x7f177c1a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f177ad9d700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f177c1084a0 0x7f177c1a2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f177ad9d700 1 -- 172.21.15.2:0/635110443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f177c110f70 msgr2=0x7f177c19c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f177ad9d700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f177c110f70 0x7f177c19c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.256+0000 7f177ad9d700 1 -- 172.21.15.2:0/635110443 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17700095f0 con 0x7f177c107b90 2023-11-16T18:23:31.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.257+0000 7f177ad9d700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f177c107b90 0x7f177c1a27e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f176c00e9a0 tx=0x7f176c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:31.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.257+0000 7f1763fff700 1 -- 172.21.15.2:0/635110443 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c009790 con 0x7f177c107b90 2023-11-16T18:23:31.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.257+0000 7f1763fff700 1 -- 172.21.15.2:0/635110443 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f176c00cbd0 con 0x7f177c107b90 2023-11-16T18:23:31.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.257+0000 7f1763fff700 1 -- 172.21.15.2:0/635110443 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c018680 con 0x7f177c107b90 2023-11-16T18:23:31.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.257+0000 7f17810f7700 1 -- 172.21.15.2:0/635110443 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f177c116a20 con 0x7f177c107b90 2023-11-16T18:23:31.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.257+0000 7f17810f7700 1 -- 172.21.15.2:0/635110443 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f177c116890 con 0x7f177c107b90 2023-11-16T18:23:31.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.258+0000 7f17810f7700 1 -- 172.21.15.2:0/635110443 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f177c0786d0 con 0x7f177c107b90 2023-11-16T18:23:31.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.260+0000 7f1763fff700 1 -- 172.21.15.2:0/635110443 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f176c00cd40 con 0x7f177c107b90 2023-11-16T18:23:31.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.261+0000 7f1763fff700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f17640a11e0 0x7f17640a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.261+0000 7f177a59c700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f17640a11e0 0x7f17640a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.261+0000 7f1763fff700 1 -- 172.21.15.2:0/635110443 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f176c014070 con 0x7f177c107b90 2023-11-16T18:23:31.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.262+0000 7f1763fff700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812] conn(0x7f17640abd50 0x7f17640ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.262+0000 7f1763fff700 1 -- 172.21.15.2:0/635110443 --> [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f17640ae850 con 0x7f17640abd50 2023-11-16T18:23:31.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.262+0000 7f1763fff700 1 -- 172.21.15.2:0/635110443 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f176c0cd690 con 0x7f177c107b90 2023-11-16T18:23:31.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.262+0000 7f177b59e700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812] conn(0x7f17640abd50 0x7f17640ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.262+0000 7f177a59c700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f17640a11e0 0x7f17640a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f177000b2e0 tx=0x7f1770005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:31.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.262+0000 7f177b59e700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812] conn(0x7f17640abd50 0x7f17640ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:31.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.263+0000 7f1763fff700 1 -- 172.21.15.2:0/635110443 <== osd.6 v2:172.21.15.2:6818/1942175812 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f17640ae850 con 0x7f17640abd50 2023-11-16T18:23:31.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.283+0000 7f17810f7700 1 -- 172.21.15.2:0/635110443 --> [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f177c116bb0 con 0x7f17640abd50 2023-11-16T18:23:31.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.284+0000 7f1763fff700 1 -- 172.21.15.2:0/635110443 <== osd.6 v2:172.21.15.2:6818/1942175812 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f177c116bb0 con 0x7f17640abd50 2023-11-16T18:23:31.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.284+0000 7f17810f7700 1 -- 172.21.15.2:0/635110443 >> [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812] conn(0x7f17640abd50 msgr2=0x7f17640ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.284+0000 7f17810f7700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812] conn(0x7f17640abd50 0x7f17640ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.284+0000 7f17810f7700 1 -- 172.21.15.2:0/635110443 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f17640a11e0 msgr2=0x7f17640a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.284+0000 7f17810f7700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f17640a11e0 0x7f17640a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f177000b2e0 tx=0x7f1770005fb0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.284+0000 7f17810f7700 1 -- 172.21.15.2:0/635110443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f177c107b90 msgr2=0x7f177c1a27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.284+0000 7f17810f7700 1 --2- 172.21.15.2:0/635110443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f177c107b90 0x7f177c1a27e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f176c00e9a0 tx=0x7f176c00ecb0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.284+0000 7f177b59e700 1 -- 172.21.15.2:0/635110443 reap_dead start 2023-11-16T18:23:31.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.284+0000 7f17810f7700 1 -- 172.21.15.2:0/635110443 shutdown_connections 2023-11-16T18:23:31.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.284+0000 7f17810f7700 1 -- 172.21.15.2:0/635110443 >> 172.21.15.2:0/635110443 conn(0x7f177c076020 msgr2=0x7f177c077c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:31.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.285+0000 7f17810f7700 1 -- 172.21.15.2:0/635110443 shutdown_connections 2023-11-16T18:23:31.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:30.285+0000 7f17810f7700 1 -- 172.21.15.2:0/635110443 wait complete. 2023-11-16T18:23:31.218 INFO:teuthology.orchestra.run.smithi002.stdout:330712481813 2023-11-16T18:23:31.218 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.11 2023-11-16T18:23:31.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.636+0000 7efd86f3e700 1 -- 172.21.15.2:0/4183122143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd8010a890 msgr2=0x7efd8010ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.636+0000 7efd86f3e700 1 --2- 172.21.15.2:0/4183122143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd8010a890 0x7efd8010ccd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7efd7800b0a0 tx=0x7efd7800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.636+0000 7efd86f3e700 1 -- 172.21.15.2:0/4183122143 shutdown_connections 2023-11-16T18:23:31.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.636+0000 7efd86f3e700 1 --2- 172.21.15.2:0/4183122143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd8010a890 0x7efd8010ccd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.636+0000 7efd86f3e700 1 --2- 172.21.15.2:0/4183122143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd80101540 0x7efd80101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.636+0000 7efd86f3e700 1 --2- 172.21.15.2:0/4183122143 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd80101e50 0x7efd8010a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.636+0000 7efd86f3e700 1 -- 172.21.15.2:0/4183122143 >> 172.21.15.2:0/4183122143 conn(0x7efd800770a0 msgr2=0x7efd800774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:31.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.636+0000 7efd86f3e700 1 -- 172.21.15.2:0/4183122143 shutdown_connections 2023-11-16T18:23:31.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.636+0000 7efd86f3e700 1 -- 172.21.15.2:0/4183122143 wait complete. 2023-11-16T18:23:31.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.636+0000 7efd86f3e700 1 Processor -- start 2023-11-16T18:23:31.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.636+0000 7efd86f3e700 1 -- start start 2023-11-16T18:23:31.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd86f3e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd80101540 0x7efd800ff1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd86f3e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd80101e50 0x7efd800ff6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd86f3e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd8010a890 0x7efd800ffc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd86f3e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd801a8ec0 con 0x7efd80101540 2023-11-16T18:23:31.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd86f3e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd801a9000 con 0x7efd80101e50 2023-11-16T18:23:31.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd86f3e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd801a9140 con 0x7efd8010a890 2023-11-16T18:23:31.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd84cda700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd80101540 0x7efd800ff1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd84cda700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd80101540 0x7efd800ff1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48558/0 (socket says 172.21.15.2:48558) 2023-11-16T18:23:31.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd84cda700 1 -- 172.21.15.2:0/3452241640 learned_addr learned my addr 172.21.15.2:0/3452241640 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:31.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd854db700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd8010a890 0x7efd800ffc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd7ffff700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd80101e50 0x7efd800ff6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd84cda700 1 -- 172.21.15.2:0/3452241640 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd8010a890 msgr2=0x7efd800ffc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd84cda700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd8010a890 0x7efd800ffc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd84cda700 1 -- 172.21.15.2:0/3452241640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd80101e50 msgr2=0x7efd800ff6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd84cda700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd80101e50 0x7efd800ff6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd84cda700 1 -- 172.21.15.2:0/3452241640 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd78009d00 con 0x7efd80101540 2023-11-16T18:23:31.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd7ffff700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd80101e50 0x7efd800ff6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:31.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd84cda700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd80101540 0x7efd800ff1a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7efd7400d980 tx=0x7efd7400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:31.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd7dffb700 1 -- 172.21.15.2:0/3452241640 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74009b40 con 0x7efd80101540 2023-11-16T18:23:31.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd86f3e700 1 -- 172.21.15.2:0/3452241640 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd80110300 con 0x7efd80101540 2023-11-16T18:23:31.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.637+0000 7efd7dffb700 1 -- 172.21.15.2:0/3452241640 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd7400b6e0 con 0x7efd80101540 2023-11-16T18:23:31.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.638+0000 7efd7dffb700 1 -- 172.21.15.2:0/3452241640 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7400f620 con 0x7efd80101540 2023-11-16T18:23:31.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.638+0000 7efd86f3e700 1 -- 172.21.15.2:0/3452241640 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd80110170 con 0x7efd80101540 2023-11-16T18:23:31.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.638+0000 7efd86f3e700 1 -- 172.21.15.2:0/3452241640 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efd800fd760 con 0x7efd80101540 2023-11-16T18:23:31.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.640+0000 7efd7dffb700 1 -- 172.21.15.2:0/3452241640 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7efd74010460 con 0x7efd80101540 2023-11-16T18:23:31.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.640+0000 7efd7dffb700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efd700a11e0 0x7efd700a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.641+0000 7efd7ffff700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efd700a11e0 0x7efd700a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.641+0000 7efd7dffb700 1 -- 172.21.15.2:0/3452241640 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7efd740cd080 con 0x7efd80101540 2023-11-16T18:23:31.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.641+0000 7efd7dffb700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732] conn(0x7efd700abd50 0x7efd700ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.641+0000 7efd7dffb700 1 -- 172.21.15.2:0/3452241640 --> [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd700ae850 con 0x7efd700abd50 2023-11-16T18:23:31.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.641+0000 7efd7dffb700 1 -- 172.21.15.2:0/3452241640 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd740cd3f0 con 0x7efd80101540 2023-11-16T18:23:31.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.641+0000 7efd854db700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732] conn(0x7efd700abd50 0x7efd700ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.641+0000 7efd7ffff700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efd700a11e0 0x7efd700a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efd80100990 tx=0x7efd6c009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:31.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.641+0000 7efd854db700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732] conn(0x7efd700abd50 0x7efd700ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:31.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.642+0000 7efd7dffb700 1 -- 172.21.15.2:0/3452241640 <== osd.3 v2:172.21.15.2:6810/1081663732 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7efd700ae850 con 0x7efd700abd50 2023-11-16T18:23:31.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.658+0000 7efd86f3e700 1 -- 172.21.15.2:0/3452241640 --> [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efd80110490 con 0x7efd700abd50 2023-11-16T18:23:31.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.659+0000 7efd7dffb700 1 -- 172.21.15.2:0/3452241640 <== osd.3 v2:172.21.15.2:6810/1081663732 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7efd80110490 con 0x7efd700abd50 2023-11-16T18:23:31.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.659+0000 7efd86f3e700 1 -- 172.21.15.2:0/3452241640 >> [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732] conn(0x7efd700abd50 msgr2=0x7efd700ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.659+0000 7efd86f3e700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732] conn(0x7efd700abd50 0x7efd700ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.659+0000 7efd86f3e700 1 -- 172.21.15.2:0/3452241640 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efd700a11e0 msgr2=0x7efd700a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.659+0000 7efd86f3e700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efd700a11e0 0x7efd700a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efd80100990 tx=0x7efd6c009550 comp rx=0 tx=0).stop 2023-11-16T18:23:31.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.659+0000 7efd86f3e700 1 -- 172.21.15.2:0/3452241640 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd80101540 msgr2=0x7efd800ff1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.659+0000 7efd86f3e700 1 --2- 172.21.15.2:0/3452241640 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd80101540 0x7efd800ff1a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7efd7400d980 tx=0x7efd7400dd40 comp rx=0 tx=0).stop 2023-11-16T18:23:31.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.659+0000 7efd854db700 1 -- 172.21.15.2:0/3452241640 reap_dead start 2023-11-16T18:23:31.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.659+0000 7efd86f3e700 1 -- 172.21.15.2:0/3452241640 shutdown_connections 2023-11-16T18:23:31.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.659+0000 7efd86f3e700 1 -- 172.21.15.2:0/3452241640 >> 172.21.15.2:0/3452241640 conn(0x7efd800770a0 msgr2=0x7efd800fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:31.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.659+0000 7efd86f3e700 1 -- 172.21.15.2:0/3452241640 shutdown_connections 2023-11-16T18:23:31.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.659+0000 7efd86f3e700 1 -- 172.21.15.2:0/3452241640 wait complete. 2023-11-16T18:23:31.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.178+0000 7f68135f5700 1 -- 172.21.15.2:0/2935986011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c0ffe00 msgr2=0x7f680c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.178+0000 7f68135f5700 1 --2- 172.21.15.2:0/2935986011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c0ffe00 0x7f680c103dd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f67fc0099c0 tx=0x7f67fc009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.178+0000 7f68135f5700 1 -- 172.21.15.2:0/2935986011 shutdown_connections 2023-11-16T18:23:31.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.178+0000 7f68135f5700 1 --2- 172.21.15.2:0/2935986011 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f680c104500 0x7f680c10c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.178+0000 7f68135f5700 1 --2- 172.21.15.2:0/2935986011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680c0ff460 0x7f680c0ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.178+0000 7f68135f5700 1 --2- 172.21.15.2:0/2935986011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c0ffe00 0x7f680c103dd0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.178+0000 7f68135f5700 1 -- 172.21.15.2:0/2935986011 >> 172.21.15.2:0/2935986011 conn(0x7f680c0770a0 msgr2=0x7f680c0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:31.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.178+0000 7f68135f5700 1 -- 172.21.15.2:0/2935986011 shutdown_connections 2023-11-16T18:23:31.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.179+0000 7f68135f5700 1 -- 172.21.15.2:0/2935986011 wait complete. 2023-11-16T18:23:31.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.179+0000 7f68135f5700 1 Processor -- start 2023-11-16T18:23:31.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.179+0000 7f68135f5700 1 -- start start 2023-11-16T18:23:31.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.179+0000 7f68135f5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680c0ff460 0x7f680c195330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.179+0000 7f68135f5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f680c0ffe00 0x7f680c195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.179+0000 7f68135f5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c104500 0x7f680c199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.179+0000 7f68135f5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c19a2f0 con 0x7f680c104500 2023-11-16T18:23:31.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.179+0000 7f68135f5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c19a460 con 0x7f680c0ff460 2023-11-16T18:23:31.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.179+0000 7f68135f5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c19a620 con 0x7f680c0ffe00 2023-11-16T18:23:31.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.179+0000 7f6811b92700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c104500 0x7f680c199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f6811b92700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c104500 0x7f680c199ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48546/0 (socket says 172.21.15.2:48546) 2023-11-16T18:23:31.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f6811b92700 1 -- 172.21.15.2:0/623961521 learned_addr learned my addr 172.21.15.2:0/623961521 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:31.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f6811391700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680c0ff460 0x7f680c195330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f6810b90700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f680c0ffe00 0x7f680c195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f6811b92700 1 -- 172.21.15.2:0/623961521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f680c0ffe00 msgr2=0x7f680c195870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f6811b92700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f680c0ffe00 0x7f680c195870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f6811b92700 1 -- 172.21.15.2:0/623961521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680c0ff460 msgr2=0x7f680c195330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f6811b92700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680c0ff460 0x7f680c195330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f6811b92700 1 -- 172.21.15.2:0/623961521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc0095f0 con 0x7f680c104500 2023-11-16T18:23:31.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f6811b92700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c104500 0x7f680c199ce0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f680400b970 tx=0x7f680400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:31.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f68027fc700 1 -- 172.21.15.2:0/623961521 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680400d860 con 0x7f680c104500 2023-11-16T18:23:31.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f68027fc700 1 -- 172.21.15.2:0/623961521 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f680400de80 con 0x7f680c104500 2023-11-16T18:23:31.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f68027fc700 1 -- 172.21.15.2:0/623961521 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6804017400 con 0x7f680c104500 2023-11-16T18:23:31.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f68135f5700 1 -- 172.21.15.2:0/623961521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680c10f2d0 con 0x7f680c104500 2023-11-16T18:23:31.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.180+0000 7f68135f5700 1 -- 172.21.15.2:0/623961521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680c10f140 con 0x7f680c104500 2023-11-16T18:23:31.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.181+0000 7f68135f5700 1 -- 172.21.15.2:0/623961521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f680c0fd6b0 con 0x7f680c104500 2023-11-16T18:23:31.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.182+0000 7f68027fc700 1 -- 172.21.15.2:0/623961521 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f680400e630 con 0x7f680c104500 2023-11-16T18:23:31.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.183+0000 7f68027fc700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f67f80a11e0 0x7f67f80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.183+0000 7f6811391700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f67f80a11e0 0x7f67f80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.183+0000 7f68027fc700 1 -- 172.21.15.2:0/623961521 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6804006e40 con 0x7f680c104500 2023-11-16T18:23:31.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.183+0000 7f68027fc700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472] conn(0x7f67f80abd50 0x7f67f80ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.183+0000 7f68027fc700 1 -- 172.21.15.2:0/623961521 --> [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f67f80ae850 con 0x7f67f80abd50 2023-11-16T18:23:31.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.183+0000 7f68027fc700 1 -- 172.21.15.2:0/623961521 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68040cce80 con 0x7f680c104500 2023-11-16T18:23:31.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.183+0000 7f6811391700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f67f80a11e0 0x7f67f80a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6808005950 tx=0x7f680800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:31.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.183+0000 7f6810b90700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472] conn(0x7f67f80abd50 0x7f67f80ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.184+0000 7f6810b90700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472] conn(0x7f67f80abd50 0x7f67f80ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:31.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.185+0000 7f68027fc700 1 -- 172.21.15.2:0/623961521 <== osd.7 v2:172.21.15.40:6816/139194472 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f67f80ae850 con 0x7f67f80abd50 2023-11-16T18:23:31.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.202+0000 7f68135f5700 1 -- 172.21.15.2:0/623961521 --> [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f680c10f460 con 0x7f67f80abd50 2023-11-16T18:23:31.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.202+0000 7f68027fc700 1 -- 172.21.15.2:0/623961521 <== osd.7 v2:172.21.15.40:6816/139194472 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f680c10f460 con 0x7f67f80abd50 2023-11-16T18:23:31.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.203+0000 7f68135f5700 1 -- 172.21.15.2:0/623961521 >> [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472] conn(0x7f67f80abd50 msgr2=0x7f67f80ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.203+0000 7f68135f5700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472] conn(0x7f67f80abd50 0x7f67f80ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.203+0000 7f68135f5700 1 -- 172.21.15.2:0/623961521 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f67f80a11e0 msgr2=0x7f67f80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.203+0000 7f68135f5700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f67f80a11e0 0x7f67f80a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6808005950 tx=0x7f680800b410 comp rx=0 tx=0).stop 2023-11-16T18:23:31.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.203+0000 7f68135f5700 1 -- 172.21.15.2:0/623961521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c104500 msgr2=0x7f680c199ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.203+0000 7f68135f5700 1 --2- 172.21.15.2:0/623961521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c104500 0x7f680c199ce0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f680400b970 tx=0x7f680400bc80 comp rx=0 tx=0).stop 2023-11-16T18:23:31.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.203+0000 7f6811b92700 1 -- 172.21.15.2:0/623961521 reap_dead start 2023-11-16T18:23:31.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.203+0000 7f68135f5700 1 -- 172.21.15.2:0/623961521 shutdown_connections 2023-11-16T18:23:31.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.203+0000 7f68135f5700 1 -- 172.21.15.2:0/623961521 >> 172.21.15.2:0/623961521 conn(0x7f680c0770a0 msgr2=0x7f680c0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:31.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.203+0000 7f68135f5700 1 -- 172.21.15.2:0/623961521 shutdown_connections 2023-11-16T18:23:31.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.203+0000 7f68135f5700 1 -- 172.21.15.2:0/623961521 wait complete. 2023-11-16T18:23:31.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.812+0000 7f54e37ef700 1 -- 172.21.15.2:0/1120653402 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54dc104500 msgr2=0x7f54dc10c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.812+0000 7f54e37ef700 1 --2- 172.21.15.2:0/1120653402 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54dc104500 0x7f54dc10c080 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f54d400b0d0 tx=0x7f54d400b3e0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.812+0000 7f54e37ef700 1 -- 172.21.15.2:0/1120653402 shutdown_connections 2023-11-16T18:23:31.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.812+0000 7f54e37ef700 1 --2- 172.21.15.2:0/1120653402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54dc0ff460 0x7f54dc0ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.812+0000 7f54e37ef700 1 --2- 172.21.15.2:0/1120653402 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54dc104500 0x7f54dc10c080 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.812+0000 7f54e37ef700 1 --2- 172.21.15.2:0/1120653402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54dc0ffe00 0x7f54dc103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.812+0000 7f54e37ef700 1 -- 172.21.15.2:0/1120653402 >> 172.21.15.2:0/1120653402 conn(0x7f54dc0770a0 msgr2=0x7f54dc0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:31.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.812+0000 7f54e37ef700 1 -- 172.21.15.2:0/1120653402 shutdown_connections 2023-11-16T18:23:31.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.812+0000 7f54e37ef700 1 -- 172.21.15.2:0/1120653402 wait complete. 2023-11-16T18:23:31.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e37ef700 1 Processor -- start 2023-11-16T18:23:31.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e37ef700 1 -- start start 2023-11-16T18:23:31.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e37ef700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54dc0ff460 0x7f54dc10c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e37ef700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54dc0ffe00 0x7f54dc10ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e37ef700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54dc104500 0x7f54dc10d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e37ef700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc1a4980 con 0x7f54dc0ffe00 2023-11-16T18:23:31.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e37ef700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc1a4af0 con 0x7f54dc0ff460 2023-11-16T18:23:31.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e37ef700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc1a4cb0 con 0x7f54dc104500 2023-11-16T18:23:31.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e0d8a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54dc0ffe00 0x7f54dc10ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e0d8a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54dc0ffe00 0x7f54dc10ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48592/0 (socket says 172.21.15.2:48592) 2023-11-16T18:23:31.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e0d8a700 1 -- 172.21.15.2:0/1736410785 learned_addr learned my addr 172.21.15.2:0/1736410785 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:31.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e158b700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54dc0ff460 0x7f54dc10c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e1d8c700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54dc104500 0x7f54dc10d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e0d8a700 1 -- 172.21.15.2:0/1736410785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54dc104500 msgr2=0x7f54dc10d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e0d8a700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54dc104500 0x7f54dc10d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e0d8a700 1 -- 172.21.15.2:0/1736410785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54dc0ff460 msgr2=0x7f54dc10c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e0d8a700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54dc0ff460 0x7f54dc10c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e0d8a700 1 -- 172.21.15.2:0/1736410785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d4009d00 con 0x7f54dc0ffe00 2023-11-16T18:23:31.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.813+0000 7f54e158b700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54dc0ff460 0x7f54dc10c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:31.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.814+0000 7f54e0d8a700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54dc0ffe00 0x7f54dc10ced0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00e970 tx=0x7f54cc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:31.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.814+0000 7f54d27fc700 1 -- 172.21.15.2:0/1736410785 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc009790 con 0x7f54dc0ffe00 2023-11-16T18:23:31.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.814+0000 7f54e37ef700 1 -- 172.21.15.2:0/1736410785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54dc1103a0 con 0x7f54dc0ffe00 2023-11-16T18:23:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.814+0000 7f54d27fc700 1 -- 172.21.15.2:0/1736410785 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54cc004cf0 con 0x7f54dc0ffe00 2023-11-16T18:23:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.814+0000 7f54d27fc700 1 -- 172.21.15.2:0/1736410785 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc005680 con 0x7f54dc0ffe00 2023-11-16T18:23:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.815+0000 7f54e37ef700 1 -- 172.21.15.2:0/1736410785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54dc110210 con 0x7f54dc0ffe00 2023-11-16T18:23:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.815+0000 7f54e37ef700 1 -- 172.21.15.2:0/1736410785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f54dc0fd670 con 0x7f54dc0ffe00 2023-11-16T18:23:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.817+0000 7f54d27fc700 1 -- 172.21.15.2:0/1736410785 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f54cc004750 con 0x7f54dc0ffe00 2023-11-16T18:23:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.817+0000 7f54d27fc700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f54c80a11e0 0x7f54c80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.817+0000 7f54d27fc700 1 -- 172.21.15.2:0/1736410785 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f54cc014070 con 0x7f54dc0ffe00 2023-11-16T18:23:31.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.817+0000 7f54e158b700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f54c80a11e0 0x7f54c80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.818+0000 7f54d27fc700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944] conn(0x7f54c80abd50 0x7f54c80ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:31.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.818+0000 7f54d27fc700 1 -- 172.21.15.2:0/1736410785 --> [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54c80ae850 con 0x7f54c80abd50 2023-11-16T18:23:31.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.818+0000 7f54d27fc700 1 -- 172.21.15.2:0/1736410785 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54cc0cd5a0 con 0x7f54dc0ffe00 2023-11-16T18:23:31.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.818+0000 7f54e1d8c700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944] conn(0x7f54c80abd50 0x7f54c80ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:31.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.818+0000 7f54e158b700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f54c80a11e0 0x7f54c80a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009be0 tx=0x7f54d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:31.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.818+0000 7f54e1d8c700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944] conn(0x7f54c80abd50 0x7f54c80ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:31.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.819+0000 7f54d27fc700 1 -- 172.21.15.2:0/1736410785 <== osd.10 v2:172.21.15.40:6824/4286013944 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f54c80ae850 con 0x7f54c80abd50 2023-11-16T18:23:31.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.836+0000 7f54e37ef700 1 -- 172.21.15.2:0/1736410785 --> [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f54dc110530 con 0x7f54c80abd50 2023-11-16T18:23:31.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.837+0000 7f54d27fc700 1 -- 172.21.15.2:0/1736410785 <== osd.10 v2:172.21.15.40:6824/4286013944 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f54dc110530 con 0x7f54c80abd50 2023-11-16T18:23:31.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.837+0000 7f54e37ef700 1 -- 172.21.15.2:0/1736410785 >> [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944] conn(0x7f54c80abd50 msgr2=0x7f54c80ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.837+0000 7f54e37ef700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944] conn(0x7f54c80abd50 0x7f54c80ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:31.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.837+0000 7f54e37ef700 1 -- 172.21.15.2:0/1736410785 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f54c80a11e0 msgr2=0x7f54c80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.837+0000 7f54e37ef700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f54c80a11e0 0x7f54c80a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009be0 tx=0x7f54d8009260 comp rx=0 tx=0).stop 2023-11-16T18:23:31.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.837+0000 7f54e37ef700 1 -- 172.21.15.2:0/1736410785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54dc0ffe00 msgr2=0x7f54dc10ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:31.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.837+0000 7f54e37ef700 1 --2- 172.21.15.2:0/1736410785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54dc0ffe00 0x7f54dc10ced0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00e970 tx=0x7f54cc00ed30 comp rx=0 tx=0).stop 2023-11-16T18:23:31.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.837+0000 7f54e1d8c700 1 -- 172.21.15.2:0/1736410785 reap_dead start 2023-11-16T18:23:31.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.837+0000 7f54e37ef700 1 -- 172.21.15.2:0/1736410785 shutdown_connections 2023-11-16T18:23:31.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.837+0000 7f54e37ef700 1 -- 172.21.15.2:0/1736410785 >> 172.21.15.2:0/1736410785 conn(0x7f54dc0770a0 msgr2=0x7f54dc0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:31.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.838+0000 7f54e37ef700 1 -- 172.21.15.2:0/1736410785 shutdown_connections 2023-11-16T18:23:31.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.838+0000 7f54e37ef700 1 -- 172.21.15.2:0/1736410785 wait complete. 2023-11-16T18:23:32.213 INFO:teuthology.orchestra.run.smithi002.stdout:137438953523 2023-11-16T18:23:32.213 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.4 2023-11-16T18:23:32.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:32 smithi040 ceph-mon[100251]: pgmap v283: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-11-16T18:23:32.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:32 smithi188 ceph-mon[100639]: pgmap v283: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-11-16T18:23:32.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:32 smithi002 ceph-mon[98807]: pgmap v283: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-11-16T18:23:32.691 INFO:teuthology.orchestra.run.smithi002.stdout:34359738437 2023-11-16T18:23:32.692 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.0 2023-11-16T18:23:32.721 INFO:teuthology.orchestra.run.smithi002.stdout:55834574911 2023-11-16T18:23:32.721 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.1 2023-11-16T18:23:32.730 INFO:teuthology.orchestra.run.smithi002.stdout:188978561067 2023-11-16T18:23:32.730 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.6 2023-11-16T18:23:32.737 INFO:teuthology.orchestra.run.smithi002.stdout:296352743449 2023-11-16T18:23:32.737 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.10 2023-11-16T18:23:33.022 INFO:teuthology.orchestra.run.smithi002.stdout:107374182456 2023-11-16T18:23:33.023 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.3 2023-11-16T18:23:33.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.871+0000 7f0be5503700 1 -- 172.21.15.2:0/839040332 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0be00ffe10 msgr2=0x7f0be0103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.871+0000 7f0be5503700 1 --2- 172.21.15.2:0/839040332 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0be00ffe10 0x7f0be0103de0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0bc80099c0 tx=0x7f0bc8009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.871+0000 7f0be5503700 1 -- 172.21.15.2:0/839040332 shutdown_connections 2023-11-16T18:23:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.871+0000 7f0be5503700 1 --2- 172.21.15.2:0/839040332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0be0104510 0x7f0be010c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.871+0000 7f0be5503700 1 --2- 172.21.15.2:0/839040332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0be00ff470 0x7f0be00ff840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.871+0000 7f0be5503700 1 --2- 172.21.15.2:0/839040332 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0be00ffe10 0x7f0be0103de0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.871+0000 7f0be5503700 1 -- 172.21.15.2:0/839040332 >> 172.21.15.2:0/839040332 conn(0x7f0be00770d0 msgr2=0x7f0be00774d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.871+0000 7f0be5503700 1 -- 172.21.15.2:0/839040332 shutdown_connections 2023-11-16T18:23:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.871+0000 7f0be5503700 1 -- 172.21.15.2:0/839040332 wait complete. 2023-11-16T18:23:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.872+0000 7f0be5503700 1 Processor -- start 2023-11-16T18:23:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.872+0000 7f0be5503700 1 -- start start 2023-11-16T18:23:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.872+0000 7f0be5503700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0be00ff470 0x7f0be0195370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.872+0000 7f0be5503700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0be00ffe10 0x7f0be01958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.872+0000 7f0be5503700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0be0104510 0x7f0be0199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.872+0000 7f0be5503700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be019a330 con 0x7f0be00ffe10 2023-11-16T18:23:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.872+0000 7f0be5503700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be019a4a0 con 0x7f0be0104510 2023-11-16T18:23:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.872+0000 7f0be5503700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be019a660 con 0x7f0be00ff470 2023-11-16T18:23:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bd67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0be00ffe10 0x7f0be01958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bd67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0be00ffe10 0x7f0be01958b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48608/0 (socket says 172.21.15.2:48608) 2023-11-16T18:23:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bd67fc700 1 -- 172.21.15.2:0/2315207951 learned_addr learned my addr 172.21.15.2:0/2315207951 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bdeffd700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0be00ff470 0x7f0be0195370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bdf7fe700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0be0104510 0x7f0be0199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bd67fc700 1 -- 172.21.15.2:0/2315207951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0be00ff470 msgr2=0x7f0be0195370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bd67fc700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0be00ff470 0x7f0be0195370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bd67fc700 1 -- 172.21.15.2:0/2315207951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0be0104510 msgr2=0x7f0be0199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bd67fc700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0be0104510 0x7f0be0199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bd67fc700 1 -- 172.21.15.2:0/2315207951 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc80095f0 con 0x7f0be00ffe10 2023-11-16T18:23:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bd67fc700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0be00ffe10 0x7f0be01958b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009990 tx=0x7f0bc80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bdcff9700 1 -- 172.21.15.2:0/2315207951 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc801f070 con 0x7f0be00ffe10 2023-11-16T18:23:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bdcff9700 1 -- 172.21.15.2:0/2315207951 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bc801ad30 con 0x7f0be00ffe10 2023-11-16T18:23:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0be5503700 1 -- 172.21.15.2:0/2315207951 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be010f320 con 0x7f0be00ffe10 2023-11-16T18:23:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0bdcff9700 1 -- 172.21.15.2:0/2315207951 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8004200 con 0x7f0be00ffe10 2023-11-16T18:23:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.873+0000 7f0be5503700 1 -- 172.21.15.2:0/2315207951 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be010f190 con 0x7f0be00ffe10 2023-11-16T18:23:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.874+0000 7f0be5503700 1 -- 172.21.15.2:0/2315207951 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0be00fd6e0 con 0x7f0be00ffe10 2023-11-16T18:23:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.876+0000 7f0bdcff9700 1 -- 172.21.15.2:0/2315207951 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f0bc8023070 con 0x7f0be00ffe10 2023-11-16T18:23:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.876+0000 7f0bdcff9700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0bcc0a1260 0x7f0bcc0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.876+0000 7f0bdeffd700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0bcc0a1260 0x7f0bcc0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.876+0000 7f0bdcff9700 1 -- 172.21.15.2:0/2315207951 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0bc80d0a60 con 0x7f0be00ffe10 2023-11-16T18:23:33.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.876+0000 7f0bdcff9700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401] conn(0x7f0bcc0abdd0 0x7f0bcc0ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.876+0000 7f0bdcff9700 1 -- 172.21.15.2:0/2315207951 --> [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0bcc0ae8d0 con 0x7f0bcc0abdd0 2023-11-16T18:23:33.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.876+0000 7f0bdcff9700 1 -- 172.21.15.2:0/2315207951 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bc80d0e70 con 0x7f0be00ffe10 2023-11-16T18:23:33.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.877+0000 7f0bdeffd700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0bcc0a1260 0x7f0bcc0a3710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0006fd0 tx=0x7f0bd0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.877+0000 7f0bdf7fe700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401] conn(0x7f0bcc0abdd0 0x7f0bcc0ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.877+0000 7f0bdf7fe700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401] conn(0x7f0bcc0abdd0 0x7f0bcc0ae200 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.878+0000 7f0bdcff9700 1 -- 172.21.15.2:0/2315207951 <== osd.2 v2:172.21.15.188:6800/3338843401 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f0bcc0ae8d0 con 0x7f0bcc0abdd0 2023-11-16T18:23:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.894+0000 7f0be5503700 1 -- 172.21.15.2:0/2315207951 --> [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0be010f4b0 con 0x7f0bcc0abdd0 2023-11-16T18:23:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.895+0000 7f0bdcff9700 1 -- 172.21.15.2:0/2315207951 <== osd.2 v2:172.21.15.188:6800/3338843401 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0be010f4b0 con 0x7f0bcc0abdd0 2023-11-16T18:23:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.895+0000 7f0be5503700 1 -- 172.21.15.2:0/2315207951 >> [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401] conn(0x7f0bcc0abdd0 msgr2=0x7f0bcc0ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.895+0000 7f0be5503700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401] conn(0x7f0bcc0abdd0 0x7f0bcc0ae200 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.895+0000 7f0be5503700 1 -- 172.21.15.2:0/2315207951 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0bcc0a1260 msgr2=0x7f0bcc0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.895+0000 7f0be5503700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0bcc0a1260 0x7f0bcc0a3710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0006fd0 tx=0x7f0bd0008040 comp rx=0 tx=0).stop 2023-11-16T18:23:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.895+0000 7f0be5503700 1 -- 172.21.15.2:0/2315207951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0be00ffe10 msgr2=0x7f0be01958b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.895+0000 7f0be5503700 1 --2- 172.21.15.2:0/2315207951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0be00ffe10 0x7f0be01958b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009990 tx=0x7f0bc80047c0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.895+0000 7f0bdf7fe700 1 -- 172.21.15.2:0/2315207951 reap_dead start 2023-11-16T18:23:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.895+0000 7f0be5503700 1 -- 172.21.15.2:0/2315207951 shutdown_connections 2023-11-16T18:23:33.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.895+0000 7f0be5503700 1 -- 172.21.15.2:0/2315207951 >> 172.21.15.2:0/2315207951 conn(0x7f0be00770d0 msgr2=0x7f0be00fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:33.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.895+0000 7f0be5503700 1 -- 172.21.15.2:0/2315207951 shutdown_connections 2023-11-16T18:23:33.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.895+0000 7f0be5503700 1 -- 172.21.15.2:0/2315207951 wait complete. 2023-11-16T18:23:33.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.723+0000 7f096a460700 1 -- 172.21.15.2:0/1858314295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0964100540 msgr2=0x7f0964100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.723+0000 7f096a460700 1 --2- 172.21.15.2:0/1858314295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0964100540 0x7f0964100910 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f09540099c0 tx=0x7f0954009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.723+0000 7f096a460700 1 -- 172.21.15.2:0/1858314295 shutdown_connections 2023-11-16T18:23:33.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.723+0000 7f096a460700 1 --2- 172.21.15.2:0/1858314295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09641055e0 0x7f096410d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.723+0000 7f096a460700 1 --2- 172.21.15.2:0/1858314295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0964100540 0x7f0964100910 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.723+0000 7f096a460700 1 --2- 172.21.15.2:0/1858314295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0964100ee0 0x7f0964104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.723+0000 7f096a460700 1 -- 172.21.15.2:0/1858314295 >> 172.21.15.2:0/1858314295 conn(0x7f09640fbdc0 msgr2=0x7f09640fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:33.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.723+0000 7f096a460700 1 -- 172.21.15.2:0/1858314295 shutdown_connections 2023-11-16T18:23:33.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.723+0000 7f096a460700 1 -- 172.21.15.2:0/1858314295 wait complete. 2023-11-16T18:23:33.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.724+0000 7f096a460700 1 Processor -- start 2023-11-16T18:23:33.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.724+0000 7f096a460700 1 -- start start 2023-11-16T18:23:33.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.724+0000 7f096a460700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0964100540 0x7f0964072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f096a460700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0964100ee0 0x7f0964073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f096a460700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09641055e0 0x7f09641a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f096a460700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0964073b60 con 0x7f09641055e0 2023-11-16T18:23:33.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f096a460700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09641a4b40 con 0x7f0964100ee0 2023-11-16T18:23:33.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f096a460700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09641a4c80 con 0x7f0964100540 2023-11-16T18:23:33.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f09689fd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09641055e0 0x7f09641a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f09689fd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09641055e0 0x7f09641a4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48578/0 (socket says 172.21.15.2:48578) 2023-11-16T18:23:33.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f09689fd700 1 -- 172.21.15.2:0/2794847932 learned_addr learned my addr 172.21.15.2:0/2794847932 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:33.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f09637fe700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0964100ee0 0x7f0964073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f0963fff700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0964100540 0x7f0964072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f09689fd700 1 -- 172.21.15.2:0/2794847932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0964100540 msgr2=0x7f0964072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f09689fd700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0964100540 0x7f0964072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f09689fd700 1 -- 172.21.15.2:0/2794847932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0964100ee0 msgr2=0x7f0964073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f09689fd700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0964100ee0 0x7f0964073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f09689fd700 1 -- 172.21.15.2:0/2794847932 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09540095f0 con 0x7f09641055e0 2023-11-16T18:23:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f09637fe700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0964100ee0 0x7f0964073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.725+0000 7f09689fd700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09641055e0 0x7f09641a4600 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f095c00ef30 tx=0x7f095c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.726+0000 7f09617fa700 1 -- 172.21.15.2:0/2794847932 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c018070 con 0x7f09641055e0 2023-11-16T18:23:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.726+0000 7f09617fa700 1 -- 172.21.15.2:0/2794847932 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f095c00f040 con 0x7f09641055e0 2023-11-16T18:23:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.726+0000 7f096a460700 1 -- 172.21.15.2:0/2794847932 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0964110320 con 0x7f09641055e0 2023-11-16T18:23:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.726+0000 7f09617fa700 1 -- 172.21.15.2:0/2794847932 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c00cdb0 con 0x7f09641055e0 2023-11-16T18:23:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.726+0000 7f096a460700 1 -- 172.21.15.2:0/2794847932 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0964110190 con 0x7f09641055e0 2023-11-16T18:23:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.727+0000 7f096a460700 1 -- 172.21.15.2:0/2794847932 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0964104820 con 0x7f09641055e0 2023-11-16T18:23:33.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.729+0000 7f09617fa700 1 -- 172.21.15.2:0/2794847932 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f095c003a00 con 0x7f09641055e0 2023-11-16T18:23:33.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.730+0000 7f09617fa700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f094c0a11e0 0x7f094c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.730+0000 7f09617fa700 1 -- 172.21.15.2:0/2794847932 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f095c0cd180 con 0x7f09641055e0 2023-11-16T18:23:33.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.730+0000 7f0963fff700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f094c0a11e0 0x7f094c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.730+0000 7f09617fa700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840] conn(0x7f094c0abd50 0x7f094c0ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.730+0000 7f09617fa700 1 -- 172.21.15.2:0/2794847932 --> [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f094c0ae850 con 0x7f094c0abd50 2023-11-16T18:23:33.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.730+0000 7f09617fa700 1 -- 172.21.15.2:0/2794847932 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f095c0cd590 con 0x7f09641055e0 2023-11-16T18:23:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.730+0000 7f09637fe700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840] conn(0x7f094c0abd50 0x7f094c0ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.730+0000 7f0963fff700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f094c0a11e0 0x7f094c0a3690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0954009990 tx=0x7f0954005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.731+0000 7f09637fe700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840] conn(0x7f094c0abd50 0x7f094c0ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.732+0000 7f09617fa700 1 -- 172.21.15.2:0/2794847932 <== osd.5 v2:172.21.15.188:6808/325018840 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f094c0ae850 con 0x7f094c0abd50 2023-11-16T18:23:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.750+0000 7f096a460700 1 -- 172.21.15.2:0/2794847932 --> [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f09641104b0 con 0x7f094c0abd50 2023-11-16T18:23:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.751+0000 7f09617fa700 1 -- 172.21.15.2:0/2794847932 <== osd.5 v2:172.21.15.188:6808/325018840 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f09641104b0 con 0x7f094c0abd50 2023-11-16T18:23:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.751+0000 7f096a460700 1 -- 172.21.15.2:0/2794847932 >> [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840] conn(0x7f094c0abd50 msgr2=0x7f094c0ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.751+0000 7f096a460700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840] conn(0x7f094c0abd50 0x7f094c0ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.751+0000 7f096a460700 1 -- 172.21.15.2:0/2794847932 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f094c0a11e0 msgr2=0x7f094c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.751+0000 7f096a460700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f094c0a11e0 0x7f094c0a3690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0954009990 tx=0x7f0954005fb0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.751+0000 7f096a460700 1 -- 172.21.15.2:0/2794847932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09641055e0 msgr2=0x7f09641a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.751+0000 7f096a460700 1 --2- 172.21.15.2:0/2794847932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09641055e0 0x7f09641a4600 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f095c00ef30 tx=0x7f095c00c3a0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.751+0000 7f09689fd700 1 -- 172.21.15.2:0/2794847932 reap_dead start 2023-11-16T18:23:33.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.751+0000 7f096a460700 1 -- 172.21.15.2:0/2794847932 shutdown_connections 2023-11-16T18:23:33.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.751+0000 7f096a460700 1 -- 172.21.15.2:0/2794847932 >> 172.21.15.2:0/2794847932 conn(0x7f09640fbdc0 msgr2=0x7f0964103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:33.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.751+0000 7f096a460700 1 -- 172.21.15.2:0/2794847932 shutdown_connections 2023-11-16T18:23:33.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:31.751+0000 7f096a460700 1 -- 172.21.15.2:0/2794847932 wait complete. 2023-11-16T18:23:33.194 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.176+0000 7f7e7619d700 1 -- 172.21.15.2:0/302900633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e70104310 msgr2=0x7f7e7010be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.194 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.176+0000 7f7e7619d700 1 --2- 172.21.15.2:0/302900633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e70104310 0x7f7e7010be90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800b0d0 tx=0x7f7e6800b3e0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.176+0000 7f7e7619d700 1 -- 172.21.15.2:0/302900633 shutdown_connections 2023-11-16T18:23:33.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.176+0000 7f7e7619d700 1 --2- 172.21.15.2:0/302900633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e700ff270 0x7f7e700ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.176+0000 7f7e7619d700 1 --2- 172.21.15.2:0/302900633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e70104310 0x7f7e7010be90 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.176+0000 7f7e7619d700 1 --2- 172.21.15.2:0/302900633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e700ffc10 0x7f7e70103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.176+0000 7f7e7619d700 1 -- 172.21.15.2:0/302900633 >> 172.21.15.2:0/302900633 conn(0x7f7e700fab10 msgr2=0x7f7e700fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:33.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.176+0000 7f7e7619d700 1 -- 172.21.15.2:0/302900633 shutdown_connections 2023-11-16T18:23:33.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.176+0000 7f7e7619d700 1 -- 172.21.15.2:0/302900633 wait complete. 2023-11-16T18:23:33.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.177+0000 7f7e7619d700 1 Processor -- start 2023-11-16T18:23:33.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.177+0000 7f7e7619d700 1 -- start start 2023-11-16T18:23:33.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.177+0000 7f7e7619d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e700ff270 0x7f7e701962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.177+0000 7f7e7619d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e700ffc10 0x7f7e701967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.177+0000 7f7e7619d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e70104310 0x7f7e7019ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.177+0000 7f7e7619d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7019b270 con 0x7f7e700ff270 2023-11-16T18:23:33.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.177+0000 7f7e7619d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7019b3e0 con 0x7f7e700ffc10 2023-11-16T18:23:33.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.177+0000 7f7e7619d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7019b5a0 con 0x7f7e70104310 2023-11-16T18:23:33.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.177+0000 7f7e7519b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e700ff270 0x7f7e701962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.177+0000 7f7e7519b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e700ff270 0x7f7e701962b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48620/0 (socket says 172.21.15.2:48620) 2023-11-16T18:23:33.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.177+0000 7f7e7519b700 1 -- 172.21.15.2:0/3415780220 learned_addr learned my addr 172.21.15.2:0/3415780220 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:33.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.177+0000 7f7e7499a700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e700ffc10 0x7f7e701967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.178+0000 7f7e7599c700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e70104310 0x7f7e7019ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.178+0000 7f7e7519b700 1 -- 172.21.15.2:0/3415780220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e70104310 msgr2=0x7f7e7019ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.178+0000 7f7e7519b700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e70104310 0x7f7e7019ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.178+0000 7f7e7519b700 1 -- 172.21.15.2:0/3415780220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e700ffc10 msgr2=0x7f7e701967f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.178+0000 7f7e7519b700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e700ffc10 0x7f7e701967f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.178+0000 7f7e7519b700 1 -- 172.21.15.2:0/3415780220 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e68009d00 con 0x7f7e700ff270 2023-11-16T18:23:33.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.178+0000 7f7e7519b700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e700ff270 0x7f7e701962b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00d950 tx=0x7f7e6c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:33.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.178+0000 7f7e667fc700 1 -- 172.21.15.2:0/3415780220 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c00f890 con 0x7f7e700ff270 2023-11-16T18:23:33.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.178+0000 7f7e667fc700 1 -- 172.21.15.2:0/3415780220 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e6c010460 con 0x7f7e700ff270 2023-11-16T18:23:33.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.178+0000 7f7e7619d700 1 -- 172.21.15.2:0/3415780220 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7010f050 con 0x7f7e700ff270 2023-11-16T18:23:33.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.178+0000 7f7e667fc700 1 -- 172.21.15.2:0/3415780220 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c00e610 con 0x7f7e700ff270 2023-11-16T18:23:33.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.178+0000 7f7e7619d700 1 -- 172.21.15.2:0/3415780220 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7010eec0 con 0x7f7e700ff270 2023-11-16T18:23:33.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.179+0000 7f7e7619d700 1 -- 172.21.15.2:0/3415780220 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7e700fb7f0 con 0x7f7e700ff270 2023-11-16T18:23:33.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.180+0000 7f7e667fc700 1 -- 172.21.15.2:0/3415780220 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f7e6c0108d0 con 0x7f7e700ff270 2023-11-16T18:23:33.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.181+0000 7f7e667fc700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7e5c0a1260 0x7f7e5c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.181+0000 7f7e667fc700 1 -- 172.21.15.2:0/3415780220 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7e6c0cced0 con 0x7f7e700ff270 2023-11-16T18:23:33.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.181+0000 7f7e7499a700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7e5c0a1260 0x7f7e5c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.181+0000 7f7e667fc700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180] conn(0x7f7e5c0abdc0 0x7f7e5c0ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:33.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.181+0000 7f7e667fc700 1 -- 172.21.15.2:0/3415780220 --> [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e5c0ae8c0 con 0x7f7e5c0abdc0 2023-11-16T18:23:33.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.181+0000 7f7e667fc700 1 -- 172.21.15.2:0/3415780220 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e6c0cd2e0 con 0x7f7e700ff270 2023-11-16T18:23:33.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.181+0000 7f7e7599c700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180] conn(0x7f7e5c0abdc0 0x7f7e5c0ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:33.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.181+0000 7f7e7499a700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7e5c0a1260 0x7f7e5c0a3710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7e701978d0 tx=0x7f7e6000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:33.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.182+0000 7f7e7599c700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180] conn(0x7f7e5c0abdc0 0x7f7e5c0ae1f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:33.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.182+0000 7f7e667fc700 1 -- 172.21.15.2:0/3415780220 <== osd.9 v2:172.21.15.2:6826/1605296180 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f7e5c0ae8c0 con 0x7f7e5c0abdc0 2023-11-16T18:23:33.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.199+0000 7f7e7619d700 1 -- 172.21.15.2:0/3415780220 --> [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7e7010f1e0 con 0x7f7e5c0abdc0 2023-11-16T18:23:33.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.200+0000 7f7e667fc700 1 -- 172.21.15.2:0/3415780220 <== osd.9 v2:172.21.15.2:6826/1605296180 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7e7010f1e0 con 0x7f7e5c0abdc0 2023-11-16T18:23:33.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.200+0000 7f7e7619d700 1 -- 172.21.15.2:0/3415780220 >> [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180] conn(0x7f7e5c0abdc0 msgr2=0x7f7e5c0ae1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.200+0000 7f7e7619d700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180] conn(0x7f7e5c0abdc0 0x7f7e5c0ae1f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:33.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.200+0000 7f7e7619d700 1 -- 172.21.15.2:0/3415780220 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7e5c0a1260 msgr2=0x7f7e5c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.200+0000 7f7e7619d700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7e5c0a1260 0x7f7e5c0a3710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7e701978d0 tx=0x7f7e6000b410 comp rx=0 tx=0).stop 2023-11-16T18:23:33.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.200+0000 7f7e7619d700 1 -- 172.21.15.2:0/3415780220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e700ff270 msgr2=0x7f7e701962b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:33.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.200+0000 7f7e7619d700 1 --2- 172.21.15.2:0/3415780220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e700ff270 0x7f7e701962b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00d950 tx=0x7f7e6c00dc60 comp rx=0 tx=0).stop 2023-11-16T18:23:33.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.200+0000 7f7e7599c700 1 -- 172.21.15.2:0/3415780220 reap_dead start 2023-11-16T18:23:33.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.200+0000 7f7e7619d700 1 -- 172.21.15.2:0/3415780220 shutdown_connections 2023-11-16T18:23:33.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.200+0000 7f7e7619d700 1 -- 172.21.15.2:0/3415780220 >> 172.21.15.2:0/3415780220 conn(0x7f7e700fab10 msgr2=0x7f7e70102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:33.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.200+0000 7f7e7619d700 1 -- 172.21.15.2:0/3415780220 shutdown_connections 2023-11-16T18:23:33.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:33.200+0000 7f7e7619d700 1 -- 172.21.15.2:0/3415780220 wait complete. 2023-11-16T18:23:33.424 INFO:teuthology.orchestra.run.smithi002.stdout:210453397543 2023-11-16T18:23:33.425 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.7 2023-11-16T18:23:33.831 INFO:teuthology.orchestra.run.smithi002.stdout:158913789998 2023-11-16T18:23:33.831 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.5 2023-11-16T18:23:34.019 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:33 smithi002 ceph-mon[98807]: pgmap v284: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 5 op/s 2023-11-16T18:23:34.130 INFO:teuthology.orchestra.run.smithi002.stdout:274877906974 2023-11-16T18:23:34.130 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.9 2023-11-16T18:23:34.163 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:34.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:33 smithi040 ceph-mon[100251]: pgmap v284: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 5 op/s 2023-11-16T18:23:34.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:33 smithi188 ceph-mon[100639]: pgmap v284: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 5 op/s 2023-11-16T18:23:34.331 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:34.402 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:34.681 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:34.682 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:34.686 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:34.686 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:34.804 INFO:teuthology.orchestra.run.smithi002.stdout:77309411387 2023-11-16T18:23:34.805 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.2 2023-11-16T18:23:34.805 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:34.959 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:35.804 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:36.093 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:35 smithi002 ceph-mon[98807]: pgmap v285: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-16T18:23:36.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:35 smithi040 ceph-mon[100251]: pgmap v285: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-16T18:23:36.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:35 smithi188 ceph-mon[100639]: pgmap v285: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-16T18:23:37.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.218+0000 7f5adeb16700 1 -- 172.21.15.2:0/3647293391 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ad81049a0 msgr2=0x7f5ad8104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:37.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.219+0000 7f5adeb16700 1 --2- 172.21.15.2:0/3647293391 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ad81049a0 0x7f5ad8104d70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009990 tx=0x7f5acc009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:37.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.219+0000 7f5adeb16700 1 -- 172.21.15.2:0/3647293391 shutdown_connections 2023-11-16T18:23:37.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.219+0000 7f5adeb16700 1 --2- 172.21.15.2:0/3647293391 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ad80ff350 0x7f5ad810b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:37.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.219+0000 7f5adeb16700 1 --2- 172.21.15.2:0/3647293391 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ad81049a0 0x7f5ad8104d70 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:37.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.219+0000 7f5adeb16700 1 --2- 172.21.15.2:0/3647293391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad80fe9a0 0x7f5ad80fee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:37.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.219+0000 7f5adeb16700 1 -- 172.21.15.2:0/3647293391 >> 172.21.15.2:0/3647293391 conn(0x7f5ad80758a0 msgr2=0x7f5ad8075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:37.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.219+0000 7f5adeb16700 1 -- 172.21.15.2:0/3647293391 shutdown_connections 2023-11-16T18:23:37.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.219+0000 7f5adeb16700 1 -- 172.21.15.2:0/3647293391 wait complete. 2023-11-16T18:23:37.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.219+0000 7f5adeb16700 1 Processor -- start 2023-11-16T18:23:37.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adeb16700 1 -- start start 2023-11-16T18:23:37.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adeb16700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ad80fe9a0 0x7f5ad8195280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:37.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adeb16700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad80ff350 0x7f5ad81957c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:37.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adeb16700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ad81049a0 0x7f5ad8199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:37.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adeb16700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad819a240 con 0x7f5ad80fe9a0 2023-11-16T18:23:37.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adeb16700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad819a3b0 con 0x7f5ad80ff350 2023-11-16T18:23:37.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adeb16700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad819a570 con 0x7f5ad81049a0 2023-11-16T18:23:37.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adc8b2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ad80fe9a0 0x7f5ad8195280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:37.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adc8b2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ad80fe9a0 0x7f5ad8195280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48634/0 (socket says 172.21.15.2:48634) 2023-11-16T18:23:37.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adc8b2700 1 -- 172.21.15.2:0/1148284766 learned_addr learned my addr 172.21.15.2:0/1148284766 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:37.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5ad7fff700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad80ff350 0x7f5ad81957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5add0b3700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ad81049a0 0x7f5ad8199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adc8b2700 1 -- 172.21.15.2:0/1148284766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ad81049a0 msgr2=0x7f5ad8199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adc8b2700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ad81049a0 0x7f5ad8199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adc8b2700 1 -- 172.21.15.2:0/1148284766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad80ff350 msgr2=0x7f5ad81957c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adc8b2700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad80ff350 0x7f5ad81957c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5adc8b2700 1 -- 172.21.15.2:0/1148284766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc0095f0 con 0x7f5ad80fe9a0 2023-11-16T18:23:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5add0b3700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ad81049a0 0x7f5ad8199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.220+0000 7f5ad7fff700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad80ff350 0x7f5ad81957c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.221+0000 7f5adc8b2700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ad80fe9a0 0x7f5ad8195280 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5acc003730 tx=0x7f5acc00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.221+0000 7f5ad5ffb700 1 -- 172.21.15.2:0/1148284766 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc01d070 con 0x7f5ad80fe9a0 2023-11-16T18:23:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.221+0000 7f5adeb16700 1 -- 172.21.15.2:0/1148284766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad810e9d0 con 0x7f5ad80fe9a0 2023-11-16T18:23:37.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.221+0000 7f5ad5ffb700 1 -- 172.21.15.2:0/1148284766 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5acc00fc30 con 0x7f5ad80fe9a0 2023-11-16T18:23:37.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.221+0000 7f5ad5ffb700 1 -- 172.21.15.2:0/1148284766 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc0173f0 con 0x7f5ad80fe9a0 2023-11-16T18:23:37.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.221+0000 7f5adeb16700 1 -- 172.21.15.2:0/1148284766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad810e840 con 0x7f5ad80fe9a0 2023-11-16T18:23:37.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.223+0000 7f5adeb16700 1 -- 172.21.15.2:0/1148284766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad810eb60 con 0x7f5ad80fe9a0 2023-11-16T18:23:37.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.223+0000 7f5ad5ffb700 1 -- 172.21.15.2:0/1148284766 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f5acc022470 con 0x7f5ad80fe9a0 2023-11-16T18:23:37.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.224+0000 7f5ad5ffb700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5ac80a11e0 0x7f5ac80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:37.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.224+0000 7f5ad7fff700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5ac80a11e0 0x7f5ac80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:37.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.224+0000 7f5ad5ffb700 1 -- 172.21.15.2:0/1148284766 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5acc0cde80 con 0x7f5ad80fe9a0 2023-11-16T18:23:37.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.224+0000 7f5ad7fff700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5ac80a11e0 0x7f5ac80a3690 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4005950 tx=0x7f5ac40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:37.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.228+0000 7f5ad5ffb700 1 -- 172.21.15.2:0/1148284766 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5acc05c730 con 0x7f5ad80fe9a0 2023-11-16T18:23:37.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.439+0000 7f5adeb16700 1 -- 172.21.15.2:0/1148284766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f5ad8196680 con 0x7f5ad80fe9a0 2023-11-16T18:23:37.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.440+0000 7f5ad5ffb700 1 -- 172.21.15.2:0/1148284766 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5acc097310 con 0x7f5ad80fe9a0 2023-11-16T18:23:37.441 INFO:teuthology.orchestra.run.smithi002.stdout:244813135907 2023-11-16T18:23:37.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.442+0000 7f5adeb16700 1 -- 172.21.15.2:0/1148284766 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5ac80a11e0 msgr2=0x7f5ac80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:37.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.442+0000 7f5adeb16700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5ac80a11e0 0x7f5ac80a3690 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4005950 tx=0x7f5ac40058e0 comp rx=0 tx=0).stop 2023-11-16T18:23:37.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.442+0000 7f5adeb16700 1 -- 172.21.15.2:0/1148284766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ad80fe9a0 msgr2=0x7f5ad8195280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:37.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.442+0000 7f5adeb16700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ad80fe9a0 0x7f5ad8195280 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5acc003730 tx=0x7f5acc00f740 comp rx=0 tx=0).stop 2023-11-16T18:23:37.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.442+0000 7f5adeb16700 1 -- 172.21.15.2:0/1148284766 shutdown_connections 2023-11-16T18:23:37.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.442+0000 7f5adeb16700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5ac80a11e0 0x7f5ac80a3690 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:37.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.442+0000 7f5adeb16700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ad81049a0 0x7f5ad8199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:37.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.442+0000 7f5adeb16700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ad80fe9a0 0x7f5ad8195280 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:37.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.442+0000 7f5adeb16700 1 --2- 172.21.15.2:0/1148284766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad80ff350 0x7f5ad81957c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:37.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.442+0000 7f5adeb16700 1 -- 172.21.15.2:0/1148284766 >> 172.21.15.2:0/1148284766 conn(0x7f5ad80758a0 msgr2=0x7f5ad8102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:37.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.442+0000 7f5adeb16700 1 -- 172.21.15.2:0/1148284766 shutdown_connections 2023-11-16T18:23:37.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:37.442+0000 7f5adeb16700 1 -- 172.21.15.2:0/1148284766 wait complete. 2023-11-16T18:23:37.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:37 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1148284766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-11-16T18:23:37.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:37 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1148284766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-11-16T18:23:37.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:37 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1148284766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-11-16T18:23:38.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2023-11-16T18:23:38.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2023-11-16T18:23:38.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2023-11-16T18:23:38.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2023-11-16T18:23:38.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2023-11-16T18:23:38.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2023-11-16T18:23:38.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2023-11-16T18:23:38.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-11-16T18:23:38.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2023-11-16T18:23:38.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2023-11-16T18:23:38.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:38 smithi002 ceph-mon[98807]: pgmap v286: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-11-16T18:23:38.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2023-11-16T18:23:38.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2023-11-16T18:23:38.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2023-11-16T18:23:38.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2023-11-16T18:23:38.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2023-11-16T18:23:38.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2023-11-16T18:23:38.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2023-11-16T18:23:38.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-11-16T18:23:38.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2023-11-16T18:23:38.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2023-11-16T18:23:38.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:38 smithi188 ceph-mon[100639]: pgmap v286: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-11-16T18:23:38.805 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135905 got 244813135907 for osd.8 2023-11-16T18:23:38.805 DEBUG:teuthology.parallel:result is None 2023-11-16T18:23:38.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.876+0000 7fb02ca02700 1 -- 172.21.15.2:0/1103768187 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0280ff260 msgr2=0x7fb0280ff630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:38.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.876+0000 7fb02ca02700 1 --2- 172.21.15.2:0/1103768187 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0280ff260 0x7fb0280ff630 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb0180099c0 tx=0x7fb018009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:38.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.876+0000 7fb02ca02700 1 -- 172.21.15.2:0/1103768187 shutdown_connections 2023-11-16T18:23:38.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.876+0000 7fb02ca02700 1 --2- 172.21.15.2:0/1103768187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb028104300 0x7fb02810be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:38.878 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.876+0000 7fb02ca02700 1 --2- 172.21.15.2:0/1103768187 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0280ff260 0x7fb0280ff630 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:38.878 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.876+0000 7fb02ca02700 1 --2- 172.21.15.2:0/1103768187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0280ffc00 0x7fb028103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:38.878 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.876+0000 7fb02ca02700 1 -- 172.21.15.2:0/1103768187 >> 172.21.15.2:0/1103768187 conn(0x7fb0280fab20 msgr2=0x7fb0280fcf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:38.878 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.876+0000 7fb02ca02700 1 -- 172.21.15.2:0/1103768187 shutdown_connections 2023-11-16T18:23:38.878 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.876+0000 7fb02ca02700 1 -- 172.21.15.2:0/1103768187 wait complete. 2023-11-16T18:23:38.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.877+0000 7fb02ca02700 1 Processor -- start 2023-11-16T18:23:38.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.877+0000 7fb02ca02700 1 -- start start 2023-11-16T18:23:38.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.877+0000 7fb02ca02700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0280ff260 0x7fb028196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:38.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.877+0000 7fb02ca02700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0280ffc00 0x7fb028196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:38.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.877+0000 7fb02ca02700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb028104300 0x7fb02819aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:38.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.877+0000 7fb02ca02700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02819b0d0 con 0x7fb028104300 2023-11-16T18:23:38.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.877+0000 7fb02ca02700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02819b240 con 0x7fb0280ffc00 2023-11-16T18:23:38.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.877+0000 7fb02ca02700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02819b400 con 0x7fb0280ff260 2023-11-16T18:23:38.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.877+0000 7fb02759e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb028104300 0x7fb02819aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:38.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.877+0000 7fb02759e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb028104300 0x7fb02819aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48650/0 (socket says 172.21.15.2:48650) 2023-11-16T18:23:38.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.877+0000 7fb02759e700 1 -- 172.21.15.2:0/2695315630 learned_addr learned my addr 172.21.15.2:0/2695315630 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:38.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.877+0000 7fb02659c700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0280ffc00 0x7fb028196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:38.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.878+0000 7fb026d9d700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0280ff260 0x7fb028196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:38.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.878+0000 7fb02759e700 1 -- 172.21.15.2:0/2695315630 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0280ff260 msgr2=0x7fb028196110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:38.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.878+0000 7fb02759e700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0280ff260 0x7fb028196110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:38.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.878+0000 7fb02759e700 1 -- 172.21.15.2:0/2695315630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0280ffc00 msgr2=0x7fb028196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:38.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.878+0000 7fb02759e700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0280ffc00 0x7fb028196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:38.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.878+0000 7fb02759e700 1 -- 172.21.15.2:0/2695315630 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180095f0 con 0x7fb028104300 2023-11-16T18:23:38.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.878+0000 7fb02659c700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0280ffc00 0x7fb028196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:38.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.878+0000 7fb02759e700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb028104300 0x7fb02819aac0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb020007d70 tx=0x7fb02000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:38.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.878+0000 7fb00ffff700 1 -- 172.21.15.2:0/2695315630 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02000ad40 con 0x7fb028104300 2023-11-16T18:23:38.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.878+0000 7fb02ca02700 1 -- 172.21.15.2:0/2695315630 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02810f090 con 0x7fb028104300 2023-11-16T18:23:38.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.878+0000 7fb00ffff700 1 -- 172.21.15.2:0/2695315630 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0200128e0 con 0x7fb028104300 2023-11-16T18:23:38.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.878+0000 7fb00ffff700 1 -- 172.21.15.2:0/2695315630 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0200116b0 con 0x7fb028104300 2023-11-16T18:23:38.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.878+0000 7fb02ca02700 1 -- 172.21.15.2:0/2695315630 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02810ef00 con 0x7fb028104300 2023-11-16T18:23:38.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.879+0000 7fb02ca02700 1 -- 172.21.15.2:0/2695315630 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02810f220 con 0x7fb028104300 2023-11-16T18:23:38.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.881+0000 7fb00ffff700 1 -- 172.21.15.2:0/2695315630 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fb020012420 con 0x7fb028104300 2023-11-16T18:23:38.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.881+0000 7fb00ffff700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb0100a11e0 0x7fb0100a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:38.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.881+0000 7fb00ffff700 1 -- 172.21.15.2:0/2695315630 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb0200cced0 con 0x7fb028104300 2023-11-16T18:23:38.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.881+0000 7fb026d9d700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb0100a11e0 0x7fb0100a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:38.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.882+0000 7fb026d9d700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb0100a11e0 0x7fb0100a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb018006010 tx=0x7fb018009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:38.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.889+0000 7fb00ffff700 1 -- 172.21.15.2:0/2695315630 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb0200962c0 con 0x7fb028104300 2023-11-16T18:23:38.908 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.907+0000 7f4f8db55700 1 -- 172.21.15.2:0/2021329255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f88101e50 msgr2=0x7f4f8810a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.907+0000 7f4f8db55700 1 --2- 172.21.15.2:0/2021329255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f88101e50 0x7f4f8810a350 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c009990 tx=0x7f4f7c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.907+0000 7f4f8db55700 1 -- 172.21.15.2:0/2021329255 shutdown_connections 2023-11-16T18:23:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.907+0000 7f4f8db55700 1 --2- 172.21.15.2:0/2021329255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f8810a890 0x7f4f8810ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.907+0000 7f4f8db55700 1 --2- 172.21.15.2:0/2021329255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f88101540 0x7f4f88101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.907+0000 7f4f8db55700 1 --2- 172.21.15.2:0/2021329255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f88101e50 0x7f4f8810a350 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.907+0000 7f4f8db55700 1 -- 172.21.15.2:0/2021329255 >> 172.21.15.2:0/2021329255 conn(0x7f4f880770a0 msgr2=0x7f4f880774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.907+0000 7f4f8db55700 1 -- 172.21.15.2:0/2021329255 shutdown_connections 2023-11-16T18:23:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.907+0000 7f4f8db55700 1 -- 172.21.15.2:0/2021329255 wait complete. 2023-11-16T18:23:38.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.908+0000 7f4f8db55700 1 Processor -- start 2023-11-16T18:23:38.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.908+0000 7f4f8db55700 1 -- start start 2023-11-16T18:23:38.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f8db55700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f88101540 0x7f4f88196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:38.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f8db55700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f88101e50 0x7f4f881968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:38.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f8db55700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f8810a890 0x7f4f8819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:38.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f8db55700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8819b350 con 0x7f4f8810a890 2023-11-16T18:23:38.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f8db55700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8819b4c0 con 0x7f4f88101e50 2023-11-16T18:23:38.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f8db55700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8819b680 con 0x7f4f88101540 2023-11-16T18:23:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f87fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f8810a890 0x7f4f8819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f87fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f8810a890 0x7f4f8819ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48664/0 (socket says 172.21.15.2:48664) 2023-11-16T18:23:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f87fff700 1 -- 172.21.15.2:0/1497406467 learned_addr learned my addr 172.21.15.2:0/1497406467 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f877fe700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f88101540 0x7f4f88196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f86ffd700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f88101e50 0x7f4f881968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f87fff700 1 -- 172.21.15.2:0/1497406467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f88101540 msgr2=0x7f4f88196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f87fff700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f88101540 0x7f4f88196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f87fff700 1 -- 172.21.15.2:0/1497406467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f88101e50 msgr2=0x7f4f881968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f87fff700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f88101e50 0x7f4f881968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f87fff700 1 -- 172.21.15.2:0/1497406467 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c0095f0 con 0x7f4f8810a890 2023-11-16T18:23:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.909+0000 7f4f877fe700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f88101540 0x7f4f88196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:23:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.910+0000 7f4f87fff700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f8810a890 0x7f4f8819ad40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000ef30 tx=0x7f4f8000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.910+0000 7f4f84ff9700 1 -- 172.21.15.2:0/1497406467 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f80018070 con 0x7f4f8810a890 2023-11-16T18:23:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.910+0000 7f4f8db55700 1 -- 172.21.15.2:0/1497406467 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f88110310 con 0x7f4f8810a890 2023-11-16T18:23:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.910+0000 7f4f84ff9700 1 -- 172.21.15.2:0/1497406467 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f8000f040 con 0x7f4f8810a890 2023-11-16T18:23:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.910+0000 7f4f84ff9700 1 -- 172.21.15.2:0/1497406467 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8000cd00 con 0x7f4f8810a890 2023-11-16T18:23:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.910+0000 7f4f8db55700 1 -- 172.21.15.2:0/1497406467 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f88110180 con 0x7f4f8810a890 2023-11-16T18:23:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.911+0000 7f4f8db55700 1 -- 172.21.15.2:0/1497406467 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f74005320 con 0x7f4f8810a890 2023-11-16T18:23:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.912+0000 7f4f84ff9700 1 -- 172.21.15.2:0/1497406467 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f4f80004000 con 0x7f4f8810a890 2023-11-16T18:23:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.913+0000 7f4f84ff9700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4f700a1260 0x7f4f700a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.913+0000 7f4f877fe700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4f700a1260 0x7f4f700a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.913+0000 7f4f84ff9700 1 -- 172.21.15.2:0/1497406467 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4f800cc7d0 con 0x7f4f8810a890 2023-11-16T18:23:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.913+0000 7f4f877fe700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4f700a1260 0x7f4f700a3710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4f78011490 tx=0x7f4f78009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:38.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:38.917+0000 7f4f84ff9700 1 -- 172.21.15.2:0/1497406467 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4f80099570 con 0x7f4f8810a890 2023-11-16T18:23:38.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2023-11-16T18:23:38.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2023-11-16T18:23:38.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2023-11-16T18:23:38.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2023-11-16T18:23:38.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2023-11-16T18:23:38.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2023-11-16T18:23:38.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2023-11-16T18:23:38.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-11-16T18:23:38.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2023-11-16T18:23:38.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2023-11-16T18:23:38.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:38 smithi040 ceph-mon[100251]: pgmap v286: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-11-16T18:23:39.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.095+0000 7fb02ca02700 1 -- 172.21.15.2:0/2695315630 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fb02819b5c0 con 0x7fb028104300 2023-11-16T18:23:39.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.096+0000 7fb00ffff700 1 -- 172.21.15.2:0/2695315630 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fb020016090 con 0x7fb028104300 2023-11-16T18:23:39.097 INFO:teuthology.orchestra.run.smithi002.stdout:330712481814 2023-11-16T18:23:39.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.098+0000 7fb02ca02700 1 -- 172.21.15.2:0/2695315630 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb0100a11e0 msgr2=0x7fb0100a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:39.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.098+0000 7fb02ca02700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb0100a11e0 0x7fb0100a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb018006010 tx=0x7fb018009f10 comp rx=0 tx=0).stop 2023-11-16T18:23:39.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.098+0000 7fb02ca02700 1 -- 172.21.15.2:0/2695315630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb028104300 msgr2=0x7fb02819aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:39.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.098+0000 7fb02ca02700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb028104300 0x7fb02819aac0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb020007d70 tx=0x7fb02000a340 comp rx=0 tx=0).stop 2023-11-16T18:23:39.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.098+0000 7fb02ca02700 1 -- 172.21.15.2:0/2695315630 shutdown_connections 2023-11-16T18:23:39.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.098+0000 7fb02ca02700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb0100a11e0 0x7fb0100a3690 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:39.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.098+0000 7fb02ca02700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0280ff260 0x7fb028196110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:39.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.098+0000 7fb02ca02700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb028104300 0x7fb02819aac0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:39.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.098+0000 7fb02ca02700 1 --2- 172.21.15.2:0/2695315630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0280ffc00 0x7fb028196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:39.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.098+0000 7fb02ca02700 1 -- 172.21.15.2:0/2695315630 >> 172.21.15.2:0/2695315630 conn(0x7fb0280fab20 msgr2=0x7fb028069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:39.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.098+0000 7fb02ca02700 1 -- 172.21.15.2:0/2695315630 shutdown_connections 2023-11-16T18:23:39.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.098+0000 7fb02ca02700 1 -- 172.21.15.2:0/2695315630 wait complete. 2023-11-16T18:23:39.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.119+0000 7f4f8db55700 1 -- 172.21.15.2:0/1497406467 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f4f74005cc0 con 0x7f4f8810a890 2023-11-16T18:23:39.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.119+0000 7f4f84ff9700 1 -- 172.21.15.2:0/1497406467 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4f80099100 con 0x7f4f8810a890 2023-11-16T18:23:39.120 INFO:teuthology.orchestra.run.smithi002.stdout:107374182457 2023-11-16T18:23:39.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.123+0000 7f4f8db55700 1 -- 172.21.15.2:0/1497406467 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4f700a1260 msgr2=0x7f4f700a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:39.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.123+0000 7f4f8db55700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4f700a1260 0x7f4f700a3710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4f78011490 tx=0x7f4f78009260 comp rx=0 tx=0).stop 2023-11-16T18:23:39.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.123+0000 7f4f8db55700 1 -- 172.21.15.2:0/1497406467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f8810a890 msgr2=0x7f4f8819ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:39.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.123+0000 7f4f8db55700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f8810a890 0x7f4f8819ad40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000ef30 tx=0x7f4f8000c3a0 comp rx=0 tx=0).stop 2023-11-16T18:23:39.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.123+0000 7f4f8db55700 1 -- 172.21.15.2:0/1497406467 shutdown_connections 2023-11-16T18:23:39.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.123+0000 7f4f8db55700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4f700a1260 0x7f4f700a3710 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:39.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.123+0000 7f4f8db55700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f88101540 0x7f4f88196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:39.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.123+0000 7f4f8db55700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f8810a890 0x7f4f8819ad40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:39.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.123+0000 7f4f8db55700 1 --2- 172.21.15.2:0/1497406467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f88101e50 0x7f4f881968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:39.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.123+0000 7f4f8db55700 1 -- 172.21.15.2:0/1497406467 >> 172.21.15.2:0/1497406467 conn(0x7f4f880770a0 msgr2=0x7f4f880fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:39.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.123+0000 7f4f8db55700 1 -- 172.21.15.2:0/1497406467 shutdown_connections 2023-11-16T18:23:39.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:39.123+0000 7f4f8db55700 1 -- 172.21.15.2:0/1497406467 wait complete. 2023-11-16T18:23:39.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2023-11-16T18:23:39.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2023-11-16T18:23:39.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2023-11-16T18:23:39.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2023-11-16T18:23:39.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2023-11-16T18:23:39.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2023-11-16T18:23:39.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2023-11-16T18:23:39.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-11-16T18:23:39.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2023-11-16T18:23:39.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2023-11-16T18:23:39.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:39 smithi002 ceph-mon[98807]: osdmap e87: 12 total, 12 up, 12 in 2023-11-16T18:23:39.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:39 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2695315630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-11-16T18:23:39.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:39 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1497406467' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-16T18:23:39.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2023-11-16T18:23:39.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2023-11-16T18:23:39.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2023-11-16T18:23:39.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2023-11-16T18:23:39.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2023-11-16T18:23:39.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2023-11-16T18:23:39.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2023-11-16T18:23:39.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-11-16T18:23:39.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2023-11-16T18:23:39.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2023-11-16T18:23:39.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:39 smithi188 ceph-mon[100639]: osdmap e87: 12 total, 12 up, 12 in 2023-11-16T18:23:39.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:39 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2695315630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-11-16T18:23:39.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:39 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1497406467' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-16T18:23:39.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2023-11-16T18:23:39.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2023-11-16T18:23:39.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2023-11-16T18:23:39.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2023-11-16T18:23:39.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2023-11-16T18:23:39.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2023-11-16T18:23:39.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2023-11-16T18:23:39.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-11-16T18:23:39.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2023-11-16T18:23:39.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2023-11-16T18:23:39.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:39 smithi040 ceph-mon[100251]: osdmap e87: 12 total, 12 up, 12 in 2023-11-16T18:23:39.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:39 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2695315630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-11-16T18:23:39.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:39 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1497406467' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-16T18:23:40.199 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:40.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.279+0000 7f8c1fd39700 1 -- 172.21.15.2:0/2421616782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c180ff460 msgr2=0x7f8c180ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:40.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.279+0000 7f8c1fd39700 1 --2- 172.21.15.2:0/2421616782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c180ff460 0x7f8c180ff830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8c140099c0 tx=0x7f8c14009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:40.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.279+0000 7f8c1fd39700 1 -- 172.21.15.2:0/2421616782 shutdown_connections 2023-11-16T18:23:40.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.279+0000 7f8c1fd39700 1 --2- 172.21.15.2:0/2421616782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c18104500 0x7f8c1810c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:40.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.279+0000 7f8c1fd39700 1 --2- 172.21.15.2:0/2421616782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c180ff460 0x7f8c180ff830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:40.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.279+0000 7f8c1fd39700 1 --2- 172.21.15.2:0/2421616782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c180ffe00 0x7f8c18103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:40.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.279+0000 7f8c1fd39700 1 -- 172.21.15.2:0/2421616782 >> 172.21.15.2:0/2421616782 conn(0x7f8c180770a0 msgr2=0x7f8c180774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:40.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.279+0000 7f8c1fd39700 1 -- 172.21.15.2:0/2421616782 shutdown_connections 2023-11-16T18:23:40.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.279+0000 7f8c1fd39700 1 -- 172.21.15.2:0/2421616782 wait complete. 2023-11-16T18:23:40.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.279+0000 7f8c1fd39700 1 Processor -- start 2023-11-16T18:23:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.279+0000 7f8c1fd39700 1 -- start start 2023-11-16T18:23:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1fd39700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c180ff460 0x7f8c18197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1fd39700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c180ffe00 0x7f8c181977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1fd39700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c18104500 0x7f8c181a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1fd39700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c181980b0 con 0x7f8c180ffe00 2023-11-16T18:23:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1fd39700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c181a2820 con 0x7f8c180ff460 2023-11-16T18:23:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1fd39700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c181a2990 con 0x7f8c18104500 2023-11-16T18:23:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1d2d4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c180ffe00 0x7f8c181977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1d2d4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c180ffe00 0x7f8c181977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48674/0 (socket says 172.21.15.2:48674) 2023-11-16T18:23:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1d2d4700 1 -- 172.21.15.2:0/1088285204 learned_addr learned my addr 172.21.15.2:0/1088285204 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1e2d6700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c18104500 0x7f8c181a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1dad5700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c180ff460 0x7f8c18197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1d2d4700 1 -- 172.21.15.2:0/1088285204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c18104500 msgr2=0x7f8c181a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1d2d4700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c18104500 0x7f8c181a22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1d2d4700 1 -- 172.21.15.2:0/1088285204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c180ff460 msgr2=0x7f8c18197260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1d2d4700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c180ff460 0x7f8c18197260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1d2d4700 1 -- 172.21.15.2:0/1088285204 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c140095f0 con 0x7f8c180ffe00 2023-11-16T18:23:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1d2d4700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c180ffe00 0x7f8c181977a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c00d950 tx=0x7f8c0c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c0affd700 1 -- 172.21.15.2:0/1088285204 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c009b40 con 0x7f8c180ffe00 2023-11-16T18:23:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.280+0000 7f8c1fd39700 1 -- 172.21.15.2:0/1088285204 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1810f250 con 0x7f8c180ffe00 2023-11-16T18:23:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.281+0000 7f8c0affd700 1 -- 172.21.15.2:0/1088285204 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c0c00de30 con 0x7f8c180ffe00 2023-11-16T18:23:40.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.281+0000 7f8c0affd700 1 -- 172.21.15.2:0/1088285204 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c005580 con 0x7f8c180ffe00 2023-11-16T18:23:40.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.281+0000 7f8c1fd39700 1 -- 172.21.15.2:0/1088285204 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1810f0c0 con 0x7f8c180ffe00 2023-11-16T18:23:40.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.281+0000 7f8c1fd39700 1 -- 172.21.15.2:0/1088285204 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1810f3e0 con 0x7f8c180ffe00 2023-11-16T18:23:40.286 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.283+0000 7f8c0affd700 1 -- 172.21.15.2:0/1088285204 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f8c0c004750 con 0x7f8c180ffe00 2023-11-16T18:23:40.286 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.283+0000 7f8c0affd700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8c040a1260 0x7f8c040a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:40.286 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.283+0000 7f8c1dad5700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8c040a1260 0x7f8c040a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:40.286 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.283+0000 7f8c0affd700 1 -- 172.21.15.2:0/1088285204 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8c0c0cca90 con 0x7f8c180ffe00 2023-11-16T18:23:40.286 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.284+0000 7f8c1dad5700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8c040a1260 0x7f8c040a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8c14000c00 tx=0x7f8c14005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:40.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.287+0000 7f8c0affd700 1 -- 172.21.15.2:0/1088285204 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8c0c095df0 con 0x7f8c180ffe00 2023-11-16T18:23:40.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.500+0000 7f8c1fd39700 1 -- 172.21.15.2:0/1088285204 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f8c18068ba0 con 0x7f8c180ffe00 2023-11-16T18:23:40.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.500+0000 7f8c0affd700 1 -- 172.21.15.2:0/1088285204 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f8c0c017070 con 0x7f8c180ffe00 2023-11-16T18:23:40.502 INFO:teuthology.orchestra.run.smithi002.stdout:296352743450 2023-11-16T18:23:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.502+0000 7f8c1fd39700 1 -- 172.21.15.2:0/1088285204 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8c040a1260 msgr2=0x7f8c040a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.502+0000 7f8c1fd39700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8c040a1260 0x7f8c040a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8c14000c00 tx=0x7f8c14005fb0 comp rx=0 tx=0).stop 2023-11-16T18:23:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.502+0000 7f8c1fd39700 1 -- 172.21.15.2:0/1088285204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c180ffe00 msgr2=0x7f8c181977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.502+0000 7f8c1fd39700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c180ffe00 0x7f8c181977a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c00d950 tx=0x7f8c0c00dd10 comp rx=0 tx=0).stop 2023-11-16T18:23:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.502+0000 7f8c1fd39700 1 -- 172.21.15.2:0/1088285204 shutdown_connections 2023-11-16T18:23:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.502+0000 7f8c1fd39700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8c040a1260 0x7f8c040a3710 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.502+0000 7f8c1fd39700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c18104500 0x7f8c181a22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.502+0000 7f8c1fd39700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c180ff460 0x7f8c18197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.502+0000 7f8c1fd39700 1 --2- 172.21.15.2:0/1088285204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c180ffe00 0x7f8c181977a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.502+0000 7f8c1fd39700 1 -- 172.21.15.2:0/1088285204 >> 172.21.15.2:0/1088285204 conn(0x7f8c180770a0 msgr2=0x7f8c180fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.503+0000 7f8c1fd39700 1 -- 172.21.15.2:0/1088285204 shutdown_connections 2023-11-16T18:23:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.503+0000 7f8c1fd39700 1 -- 172.21.15.2:0/1088285204 wait complete. 2023-11-16T18:23:40.551 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182456 got 107374182457 for osd.3 2023-11-16T18:23:40.552 DEBUG:teuthology.parallel:result is None 2023-11-16T18:23:40.637 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:40 smithi002 ceph-mon[98807]: osdmap e88: 12 total, 12 up, 12 in 2023-11-16T18:23:40.637 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:40 smithi002 ceph-mon[98807]: pgmap v289: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2023-11-16T18:23:40.638 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:40 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1088285204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-11-16T18:23:40.730 INFO:tasks.cephadm.ceph_manager.ceph:need seq 330712481813 got 330712481814 for osd.11 2023-11-16T18:23:40.730 DEBUG:teuthology.parallel:result is None 2023-11-16T18:23:40.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:40 smithi188 ceph-mon[100639]: osdmap e88: 12 total, 12 up, 12 in 2023-11-16T18:23:40.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:40 smithi188 ceph-mon[100639]: pgmap v289: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2023-11-16T18:23:40.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:40 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1088285204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-11-16T18:23:40.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:40 smithi040 ceph-mon[100251]: osdmap e88: 12 total, 12 up, 12 in 2023-11-16T18:23:40.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:40 smithi040 ceph-mon[100251]: pgmap v289: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2023-11-16T18:23:40.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:40 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1088285204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-11-16T18:23:41.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.790+0000 7f15d7177700 1 -- 172.21.15.2:0/2712060982 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d01044e0 msgr2=0x7f15d010c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.790+0000 7f15d7177700 1 --2- 172.21.15.2:0/2712060982 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d01044e0 0x7f15d010c060 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f15c800b0d0 tx=0x7f15c800b3e0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.790+0000 7f15d7177700 1 -- 172.21.15.2:0/2712060982 shutdown_connections 2023-11-16T18:23:41.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.790+0000 7f15d7177700 1 --2- 172.21.15.2:0/2712060982 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d01044e0 0x7f15d010c060 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.790+0000 7f15d7177700 1 --2- 172.21.15.2:0/2712060982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d00ff440 0x7f15d00ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.790+0000 7f15d7177700 1 --2- 172.21.15.2:0/2712060982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f15d00ffde0 0x7f15d0103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.790+0000 7f15d7177700 1 -- 172.21.15.2:0/2712060982 >> 172.21.15.2:0/2712060982 conn(0x7f15d00facc0 msgr2=0x7f15d00fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:41.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.790+0000 7f15d7177700 1 -- 172.21.15.2:0/2712060982 shutdown_connections 2023-11-16T18:23:41.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.790+0000 7f15d7177700 1 -- 172.21.15.2:0/2712060982 wait complete. 2023-11-16T18:23:41.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.790+0000 7f15d7177700 1 Processor -- start 2023-11-16T18:23:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.790+0000 7f15d7177700 1 -- start start 2023-11-16T18:23:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d7177700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f15d00ff440 0x7f15d01964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d7177700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d00ffde0 0x7f15d0196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d7177700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d01044e0 0x7f15d019ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d7177700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d019b490 con 0x7f15d01044e0 2023-11-16T18:23:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d7177700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d019b600 con 0x7f15d00ffde0 2023-11-16T18:23:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d7177700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d019b7c0 con 0x7f15d00ff440 2023-11-16T18:23:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d5714700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d01044e0 0x7f15d019ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d5714700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d01044e0 0x7f15d019ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48690/0 (socket says 172.21.15.2:48690) 2023-11-16T18:23:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d5714700 1 -- 172.21.15.2:0/1825086750 learned_addr learned my addr 172.21.15.2:0/1825086750 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d4f13700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f15d00ff440 0x7f15d01964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15cffff700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d00ffde0 0x7f15d0196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d5714700 1 -- 172.21.15.2:0/1825086750 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f15d00ff440 msgr2=0x7f15d01964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d5714700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f15d00ff440 0x7f15d01964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d5714700 1 -- 172.21.15.2:0/1825086750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d00ffde0 msgr2=0x7f15d0196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.011 INFO:teuthology.orchestra.run.smithi002.stdout:34359738439 2023-11-16T18:23:41.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d5714700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d00ffde0 0x7f15d0196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d5714700 1 -- 172.21.15.2:0/1825086750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c8009d00 con 0x7f15d01044e0 2023-11-16T18:23:41.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.791+0000 7f15d5714700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d01044e0 0x7f15d019ae80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f15c80077f0 tx=0x7f15c8007820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:41.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.792+0000 7f15cdffb700 1 -- 172.21.15.2:0/1825086750 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c800e040 con 0x7f15d01044e0 2023-11-16T18:23:41.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.792+0000 7f15cdffb700 1 -- 172.21.15.2:0/1825086750 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15c8003e10 con 0x7f15d01044e0 2023-11-16T18:23:41.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.792+0000 7f15d7177700 1 -- 172.21.15.2:0/1825086750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d010f290 con 0x7f15d01044e0 2023-11-16T18:23:41.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.792+0000 7f15cdffb700 1 -- 172.21.15.2:0/1825086750 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c801b990 con 0x7f15d01044e0 2023-11-16T18:23:41.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.792+0000 7f15d7177700 1 -- 172.21.15.2:0/1825086750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d010f100 con 0x7f15d01044e0 2023-11-16T18:23:41.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.793+0000 7f15d7177700 1 -- 172.21.15.2:0/1825086750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d010f420 con 0x7f15d01044e0 2023-11-16T18:23:41.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.795+0000 7f15cdffb700 1 -- 172.21.15.2:0/1825086750 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f15c80055c0 con 0x7f15d01044e0 2023-11-16T18:23:41.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.795+0000 7f15cdffb700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f15c00a1330 0x7f15c00a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.795+0000 7f15cdffb700 1 -- 172.21.15.2:0/1825086750 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f15c80ce9b0 con 0x7f15d01044e0 2023-11-16T18:23:41.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.796+0000 7f15d4f13700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f15c00a1330 0x7f15c00a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.796+0000 7f15d4f13700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f15c00a1330 0x7f15c00a37e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f15c4005950 tx=0x7f15c400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:41.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.799+0000 7f15cdffb700 1 -- 172.21.15.2:0/1825086750 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f15c8097da0 con 0x7f15d01044e0 2023-11-16T18:23:41.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.003+0000 7f15d7177700 1 -- 172.21.15.2:0/1825086750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f15d01a3550 con 0x7f15d01044e0 2023-11-16T18:23:41.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.003+0000 7f15cdffb700 1 -- 172.21.15.2:0/1825086750 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f15c809b3c0 con 0x7f15d01044e0 2023-11-16T18:23:41.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.005+0000 7f15d7177700 1 -- 172.21.15.2:0/1825086750 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f15c00a1330 msgr2=0x7f15c00a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.005+0000 7f15d7177700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f15c00a1330 0x7f15c00a37e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f15c4005950 tx=0x7f15c400b410 comp rx=0 tx=0).stop 2023-11-16T18:23:41.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.005+0000 7f15d7177700 1 -- 172.21.15.2:0/1825086750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d01044e0 msgr2=0x7f15d019ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.005+0000 7f15d7177700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d01044e0 0x7f15d019ae80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f15c80077f0 tx=0x7f15c8007820 comp rx=0 tx=0).stop 2023-11-16T18:23:41.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.005+0000 7f15d7177700 1 -- 172.21.15.2:0/1825086750 shutdown_connections 2023-11-16T18:23:41.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.005+0000 7f15d7177700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f15c00a1330 0x7f15c00a37e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.005+0000 7f15d7177700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f15d00ff440 0x7f15d01964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.005+0000 7f15d7177700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d01044e0 0x7f15d019ae80 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.005+0000 7f15d7177700 1 --2- 172.21.15.2:0/1825086750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d00ffde0 0x7f15d0196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.005+0000 7f15d7177700 1 -- 172.21.15.2:0/1825086750 >> 172.21.15.2:0/1825086750 conn(0x7f15d00facc0 msgr2=0x7f15d0102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:41.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.005+0000 7f15d7177700 1 -- 172.21.15.2:0/1825086750 shutdown_connections 2023-11-16T18:23:41.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.005+0000 7f15d7177700 1 -- 172.21.15.2:0/1825086750 wait complete. 2023-11-16T18:23:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.249+0000 7f2b9e635700 1 -- 172.21.15.2:0/4269949198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b980ff460 msgr2=0x7f2b980ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.249+0000 7f2b9e635700 1 --2- 172.21.15.2:0/4269949198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b980ff460 0x7f2b980ff830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2b880099c0 tx=0x7f2b88009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.249+0000 7f2b9e635700 1 -- 172.21.15.2:0/4269949198 shutdown_connections 2023-11-16T18:23:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.250+0000 7f2b9e635700 1 --2- 172.21.15.2:0/4269949198 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b98104500 0x7f2b9810c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.250+0000 7f2b9e635700 1 --2- 172.21.15.2:0/4269949198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b980ff460 0x7f2b980ff830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.250+0000 7f2b9e635700 1 --2- 172.21.15.2:0/4269949198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b980ffe00 0x7f2b98103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.250+0000 7f2b9e635700 1 -- 172.21.15.2:0/4269949198 >> 172.21.15.2:0/4269949198 conn(0x7f2b980770a0 msgr2=0x7f2b980774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.250+0000 7f2b9e635700 1 -- 172.21.15.2:0/4269949198 shutdown_connections 2023-11-16T18:23:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.250+0000 7f2b9e635700 1 -- 172.21.15.2:0/4269949198 wait complete. 2023-11-16T18:23:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.250+0000 7f2b9e635700 1 Processor -- start 2023-11-16T18:23:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.250+0000 7f2b9e635700 1 -- start start 2023-11-16T18:23:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b9e635700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b980ff460 0x7f2b981964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b9e635700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b980ffe00 0x7f2b98196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b9e635700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b98104500 0x7f2b9819ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b9e635700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9819b4a0 con 0x7f2b980ff460 2023-11-16T18:23:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b9e635700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9819b610 con 0x7f2b98104500 2023-11-16T18:23:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b9e635700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9819b7d0 con 0x7f2b980ffe00 2023-11-16T18:23:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b97fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b980ff460 0x7f2b981964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b97fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b980ff460 0x7f2b981964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48714/0 (socket says 172.21.15.2:48714) 2023-11-16T18:23:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b97fff700 1 -- 172.21.15.2:0/524840209 learned_addr learned my addr 172.21.15.2:0/524840209 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b977fe700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b980ffe00 0x7f2b98196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b9cbd2700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b98104500 0x7f2b9819ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b97fff700 1 -- 172.21.15.2:0/524840209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b980ffe00 msgr2=0x7f2b98196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b97fff700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b980ffe00 0x7f2b98196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b97fff700 1 -- 172.21.15.2:0/524840209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b98104500 msgr2=0x7f2b9819ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b97fff700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b98104500 0x7f2b9819ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b97fff700 1 -- 172.21.15.2:0/524840209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b880095f0 con 0x7f2b980ff460 2023-11-16T18:23:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b9cbd2700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b98104500 0x7f2b9819ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.251+0000 7f2b97fff700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b980ff460 0x7f2b981964e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800b580 tx=0x7f2b8800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.252+0000 7f2b957fa700 1 -- 172.21.15.2:0/524840209 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8801d070 con 0x7f2b980ff460 2023-11-16T18:23:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.252+0000 7f2b9e635700 1 -- 172.21.15.2:0/524840209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9810f240 con 0x7f2b980ff460 2023-11-16T18:23:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.252+0000 7f2b957fa700 1 -- 172.21.15.2:0/524840209 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b88022470 con 0x7f2b980ff460 2023-11-16T18:23:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.252+0000 7f2b957fa700 1 -- 172.21.15.2:0/524840209 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b88017650 con 0x7f2b980ff460 2023-11-16T18:23:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.252+0000 7f2b9e635700 1 -- 172.21.15.2:0/524840209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9810f0b0 con 0x7f2b980ff460 2023-11-16T18:23:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.253+0000 7f2b9e635700 1 -- 172.21.15.2:0/524840209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9810f3d0 con 0x7f2b980ff460 2023-11-16T18:23:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.255+0000 7f2b957fa700 1 -- 172.21.15.2:0/524840209 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f2b880225e0 con 0x7f2b980ff460 2023-11-16T18:23:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.255+0000 7f2b957fa700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2b800a11e0 0x7f2b800a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.255+0000 7f2b957fa700 1 -- 172.21.15.2:0/524840209 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2b880cdf70 con 0x7f2b980ff460 2023-11-16T18:23:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.255+0000 7f2b977fe700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2b800a11e0 0x7f2b800a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.256+0000 7f2b977fe700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2b800a11e0 0x7f2b800a3690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2b98197b00 tx=0x7f2b8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.259+0000 7f2b957fa700 1 -- 172.21.15.2:0/524840209 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2b880972e0 con 0x7f2b980ff460 2023-11-16T18:23:41.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.820+0000 7ff81a22d700 1 -- 172.21.15.2:0/2650937873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff814101cf0 msgr2=0x7ff81410a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.820+0000 7ff81a22d700 1 --2- 172.21.15.2:0/2650937873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff814101cf0 0x7ff81410a3c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff808009990 tx=0x7ff808009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.820+0000 7ff81a22d700 1 -- 172.21.15.2:0/2650937873 shutdown_connections 2023-11-16T18:23:41.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.820+0000 7ff81a22d700 1 --2- 172.21.15.2:0/2650937873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8141013e0 0x7ff8141017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.820+0000 7ff81a22d700 1 --2- 172.21.15.2:0/2650937873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff81410a900 0x7ff81410cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.820+0000 7ff81a22d700 1 --2- 172.21.15.2:0/2650937873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff814101cf0 0x7ff81410a3c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.820+0000 7ff81a22d700 1 -- 172.21.15.2:0/2650937873 >> 172.21.15.2:0/2650937873 conn(0x7ff8140770d0 msgr2=0x7ff8140774d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:41.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.820+0000 7ff81a22d700 1 -- 172.21.15.2:0/2650937873 shutdown_connections 2023-11-16T18:23:41.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.820+0000 7ff81a22d700 1 -- 172.21.15.2:0/2650937873 wait complete. 2023-11-16T18:23:41.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.821+0000 7ff81a22d700 1 Processor -- start 2023-11-16T18:23:41.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff81a22d700 1 -- start start 2023-11-16T18:23:41.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff81a22d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8141013e0 0x7ff814196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff81a22d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff814101cf0 0x7ff8141969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff81a22d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81410a900 0x7ff81419ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff81a22d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81419b440 con 0x7ff81410a900 2023-11-16T18:23:41.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff81a22d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81419b5b0 con 0x7ff814101cf0 2023-11-16T18:23:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff81a22d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81419b770 con 0x7ff8141013e0 2023-11-16T18:23:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff813fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81410a900 0x7ff81419ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff813fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81410a900 0x7ff81419ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48704/0 (socket says 172.21.15.2:48704) 2023-11-16T18:23:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff813fff700 1 -- 172.21.15.2:0/195695582 learned_addr learned my addr 172.21.15.2:0/195695582 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff812ffd700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff814101cf0 0x7ff8141969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff8137fe700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8141013e0 0x7ff814196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff813fff700 1 -- 172.21.15.2:0/195695582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8141013e0 msgr2=0x7ff814196480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff813fff700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8141013e0 0x7ff814196480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff813fff700 1 -- 172.21.15.2:0/195695582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff814101cf0 msgr2=0x7ff8141969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff813fff700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff814101cf0 0x7ff8141969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.822+0000 7ff813fff700 1 -- 172.21.15.2:0/195695582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8080095f0 con 0x7ff81410a900 2023-11-16T18:23:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.823+0000 7ff813fff700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81410a900 0x7ff81419ae30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00ada0 tx=0x7ff80c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.823+0000 7ff810ff9700 1 -- 172.21.15.2:0/195695582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c00f070 con 0x7ff81410a900 2023-11-16T18:23:41.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.823+0000 7ff810ff9700 1 -- 172.21.15.2:0/195695582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff80c007ad0 con 0x7ff81410a900 2023-11-16T18:23:41.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.823+0000 7ff81a22d700 1 -- 172.21.15.2:0/195695582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff814110380 con 0x7ff81410a900 2023-11-16T18:23:41.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.823+0000 7ff810ff9700 1 -- 172.21.15.2:0/195695582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c005280 con 0x7ff81410a900 2023-11-16T18:23:41.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.823+0000 7ff81a22d700 1 -- 172.21.15.2:0/195695582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8141101f0 con 0x7ff81410a900 2023-11-16T18:23:41.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.824+0000 7ff81a22d700 1 -- 172.21.15.2:0/195695582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff814110510 con 0x7ff81410a900 2023-11-16T18:23:41.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.825+0000 7ff810ff9700 1 -- 172.21.15.2:0/195695582 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ff80c00c8d0 con 0x7ff81410a900 2023-11-16T18:23:41.296 INFO:teuthology.orchestra.run.smithi002.stdout:158913790000 2023-11-16T18:23:41.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.826+0000 7ff810ff9700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff7fc0a1260 0x7ff7fc0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.826+0000 7ff810ff9700 1 -- 172.21.15.2:0/195695582 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff80c0cc180 con 0x7ff81410a900 2023-11-16T18:23:41.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.826+0000 7ff8137fe700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff7fc0a1260 0x7ff7fc0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.826+0000 7ff8137fe700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff7fc0a1260 0x7ff7fc0a3710 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff804005950 tx=0x7ff80400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:41.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:40.830+0000 7ff810ff9700 1 -- 172.21.15.2:0/195695582 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff80c0954f0 con 0x7ff81410a900 2023-11-16T18:23:41.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.034+0000 7ff81a22d700 1 -- 172.21.15.2:0/195695582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ff814066ed0 con 0x7ff81410a900 2023-11-16T18:23:41.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.034+0000 7ff810ff9700 1 -- 172.21.15.2:0/195695582 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff80c098b10 con 0x7ff81410a900 2023-11-16T18:23:41.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.037+0000 7ff81a22d700 1 -- 172.21.15.2:0/195695582 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff7fc0a1260 msgr2=0x7ff7fc0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.037+0000 7ff81a22d700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff7fc0a1260 0x7ff7fc0a3710 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff804005950 tx=0x7ff80400b410 comp rx=0 tx=0).stop 2023-11-16T18:23:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.037+0000 7ff81a22d700 1 -- 172.21.15.2:0/195695582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81410a900 msgr2=0x7ff81419ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.037+0000 7ff81a22d700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81410a900 0x7ff81419ae30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00ada0 tx=0x7ff80c00c330 comp rx=0 tx=0).stop 2023-11-16T18:23:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.037+0000 7ff81a22d700 1 -- 172.21.15.2:0/195695582 shutdown_connections 2023-11-16T18:23:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.037+0000 7ff81a22d700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff7fc0a1260 0x7ff7fc0a3710 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.037+0000 7ff81a22d700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8141013e0 0x7ff814196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.037+0000 7ff81a22d700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81410a900 0x7ff81419ae30 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.037+0000 7ff81a22d700 1 --2- 172.21.15.2:0/195695582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff814101cf0 0x7ff8141969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.037+0000 7ff81a22d700 1 -- 172.21.15.2:0/195695582 >> 172.21.15.2:0/195695582 conn(0x7ff8140770d0 msgr2=0x7ff814102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.037+0000 7ff81a22d700 1 -- 172.21.15.2:0/195695582 shutdown_connections 2023-11-16T18:23:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.037+0000 7ff81a22d700 1 -- 172.21.15.2:0/195695582 wait complete. 2023-11-16T18:23:41.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.463+0000 7f2b9e635700 1 -- 172.21.15.2:0/524840209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2b98066ea0 con 0x7f2b980ff460 2023-11-16T18:23:41.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.464+0000 7f2b957fa700 1 -- 172.21.15.2:0/524840209 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2b8809a900 con 0x7f2b980ff460 2023-11-16T18:23:41.466 INFO:teuthology.orchestra.run.smithi002.stdout:55834574913 2023-11-16T18:23:41.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.466+0000 7f2b9e635700 1 -- 172.21.15.2:0/524840209 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2b800a11e0 msgr2=0x7f2b800a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.466+0000 7f2b9e635700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2b800a11e0 0x7f2b800a3690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2b98197b00 tx=0x7f2b8c009260 comp rx=0 tx=0).stop 2023-11-16T18:23:41.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.466+0000 7f2b9e635700 1 -- 172.21.15.2:0/524840209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b980ff460 msgr2=0x7f2b981964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.466+0000 7f2b9e635700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b980ff460 0x7f2b981964e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800b580 tx=0x7f2b8800bfd0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.466+0000 7f2b9e635700 1 -- 172.21.15.2:0/524840209 shutdown_connections 2023-11-16T18:23:41.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.466+0000 7f2b9e635700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2b800a11e0 0x7f2b800a3690 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.466+0000 7f2b9e635700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b98104500 0x7f2b9819ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.466+0000 7f2b9e635700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b980ff460 0x7f2b981964e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.466+0000 7f2b9e635700 1 --2- 172.21.15.2:0/524840209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b980ffe00 0x7f2b98196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.466+0000 7f2b9e635700 1 -- 172.21.15.2:0/524840209 >> 172.21.15.2:0/524840209 conn(0x7f2b980770a0 msgr2=0x7f2b980fc660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:41.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.466+0000 7f2b9e635700 1 -- 172.21.15.2:0/524840209 shutdown_connections 2023-11-16T18:23:41.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.466+0000 7f2b9e635700 1 -- 172.21.15.2:0/524840209 wait complete. 2023-11-16T18:23:41.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.556+0000 7feb7c61a700 1 -- 172.21.15.2:0/192617459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb740ff350 msgr2=0x7feb7410b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.556+0000 7feb7c61a700 1 --2- 172.21.15.2:0/192617459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb740ff350 0x7feb7410b7e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00b0a0 tx=0x7feb6c00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.556+0000 7feb7c61a700 1 -- 172.21.15.2:0/192617459 shutdown_connections 2023-11-16T18:23:41.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.556+0000 7feb7c61a700 1 --2- 172.21.15.2:0/192617459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb741049a0 0x7feb74104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.556+0000 7feb7c61a700 1 --2- 172.21.15.2:0/192617459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb740ff350 0x7feb7410b7e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.556+0000 7feb7c61a700 1 --2- 172.21.15.2:0/192617459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb740fe9a0 0x7feb740fee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.556+0000 7feb7c61a700 1 -- 172.21.15.2:0/192617459 >> 172.21.15.2:0/192617459 conn(0x7feb740758a0 msgr2=0x7feb74075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:41.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.556+0000 7feb7c61a700 1 -- 172.21.15.2:0/192617459 shutdown_connections 2023-11-16T18:23:41.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.556+0000 7feb7c61a700 1 -- 172.21.15.2:0/192617459 wait complete. 2023-11-16T18:23:41.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.557+0000 7feb7c61a700 1 Processor -- start 2023-11-16T18:23:41.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.557+0000 7feb7c61a700 1 -- start start 2023-11-16T18:23:41.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.557+0000 7feb7c61a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb740fe9a0 0x7feb74195300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.557+0000 7feb7c61a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb740ff350 0x7feb74195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.557+0000 7feb7c61a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb741049a0 0x7feb74199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.557+0000 7feb7c61a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7419a2c0 con 0x7feb740fe9a0 2023-11-16T18:23:41.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.557+0000 7feb7c61a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7419a430 con 0x7feb741049a0 2023-11-16T18:23:41.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.557+0000 7feb7c61a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7419a5f0 con 0x7feb740ff350 2023-11-16T18:23:41.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.557+0000 7feb7a3b6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb740fe9a0 0x7feb74195300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.557+0000 7feb7a3b6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb740fe9a0 0x7feb74195300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48726/0 (socket says 172.21.15.2:48726) 2023-11-16T18:23:41.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.557+0000 7feb7a3b6700 1 -- 172.21.15.2:0/1410623943 learned_addr learned my addr 172.21.15.2:0/1410623943 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.558+0000 7feb7abb7700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb741049a0 0x7feb74199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.558+0000 7feb7a3b6700 1 -- 172.21.15.2:0/1410623943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb740ff350 msgr2=0x7feb74195840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.558+0000 7feb7a3b6700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb740ff350 0x7feb74195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.558+0000 7feb7a3b6700 1 -- 172.21.15.2:0/1410623943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb741049a0 msgr2=0x7feb74199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.558+0000 7feb7a3b6700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb741049a0 0x7feb74199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.558+0000 7feb7a3b6700 1 -- 172.21.15.2:0/1410623943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c009d00 con 0x7feb740fe9a0 2023-11-16T18:23:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.558+0000 7feb7abb7700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb741049a0 0x7feb74199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.558+0000 7feb7a3b6700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb740fe9a0 0x7feb74195300 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7feb7000c990 tx=0x7feb7000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.558+0000 7feb6b7fe700 1 -- 172.21.15.2:0/1410623943 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70007b00 con 0x7feb740fe9a0 2023-11-16T18:23:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.558+0000 7feb7c61a700 1 -- 172.21.15.2:0/1410623943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7410ea20 con 0x7feb740fe9a0 2023-11-16T18:23:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.558+0000 7feb6b7fe700 1 -- 172.21.15.2:0/1410623943 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb70007c60 con 0x7feb740fe9a0 2023-11-16T18:23:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.558+0000 7feb7c61a700 1 -- 172.21.15.2:0/1410623943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7410e890 con 0x7feb740fe9a0 2023-11-16T18:23:41.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.558+0000 7feb6b7fe700 1 -- 172.21.15.2:0/1410623943 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70018660 con 0x7feb740fe9a0 2023-11-16T18:23:41.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.559+0000 7feb7c61a700 1 -- 172.21.15.2:0/1410623943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb58005320 con 0x7feb740fe9a0 2023-11-16T18:23:41.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.561+0000 7feb6b7fe700 1 -- 172.21.15.2:0/1410623943 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7feb7000f450 con 0x7feb740fe9a0 2023-11-16T18:23:41.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.562+0000 7feb6b7fe700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feb600a1260 0x7feb600a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:41.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.562+0000 7feb6b7fe700 1 -- 172.21.15.2:0/1410623943 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7feb700cccb0 con 0x7feb740fe9a0 2023-11-16T18:23:41.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.562+0000 7feb79bb5700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feb600a1260 0x7feb600a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:41.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.562+0000 7feb79bb5700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feb600a1260 0x7feb600a3710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feb64005fd0 tx=0x7feb64005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:41.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.567+0000 7feb6b7fe700 1 -- 172.21.15.2:0/1410623943 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7feb70095f30 con 0x7feb740fe9a0 2023-11-16T18:23:41.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.779+0000 7feb7c61a700 1 -- 172.21.15.2:0/1410623943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7feb58005cc0 con 0x7feb740fe9a0 2023-11-16T18:23:41.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.779+0000 7feb6b7fe700 1 -- 172.21.15.2:0/1410623943 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7feb70099550 con 0x7feb740fe9a0 2023-11-16T18:23:41.781 INFO:teuthology.orchestra.run.smithi002.stdout:188978561069 2023-11-16T18:23:41.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.781+0000 7feb7c61a700 1 -- 172.21.15.2:0/1410623943 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feb600a1260 msgr2=0x7feb600a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.781+0000 7feb7c61a700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feb600a1260 0x7feb600a3710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feb64005fd0 tx=0x7feb64005d00 comp rx=0 tx=0).stop 2023-11-16T18:23:41.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.781+0000 7feb7c61a700 1 -- 172.21.15.2:0/1410623943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb740fe9a0 msgr2=0x7feb74195300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:41.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.781+0000 7feb7c61a700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb740fe9a0 0x7feb74195300 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7feb7000c990 tx=0x7feb7000cd50 comp rx=0 tx=0).stop 2023-11-16T18:23:41.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.781+0000 7feb7c61a700 1 -- 172.21.15.2:0/1410623943 shutdown_connections 2023-11-16T18:23:41.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.781+0000 7feb7c61a700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feb600a1260 0x7feb600a3710 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.781+0000 7feb7c61a700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb741049a0 0x7feb74199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.781+0000 7feb7c61a700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb740fe9a0 0x7feb74195300 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.781+0000 7feb7c61a700 1 --2- 172.21.15.2:0/1410623943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb740ff350 0x7feb74195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:41.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.781+0000 7feb7c61a700 1 -- 172.21.15.2:0/1410623943 >> 172.21.15.2:0/1410623943 conn(0x7feb740758a0 msgr2=0x7feb74067d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:41.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.781+0000 7feb7c61a700 1 -- 172.21.15.2:0/1410623943 shutdown_connections 2023-11-16T18:23:41.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:41.781+0000 7feb7c61a700 1 -- 172.21.15.2:0/1410623943 wait complete. 2023-11-16T18:23:41.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:41 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1825086750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-16T18:23:41.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:41 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/195695582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-16T18:23:41.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:41 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/524840209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-16T18:23:41.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:41 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1825086750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-16T18:23:41.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:41 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/195695582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-16T18:23:41.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:41 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/524840209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-16T18:23:41.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:41 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1825086750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-16T18:23:41.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:41 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/195695582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-16T18:23:41.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:41 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/524840209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-16T18:23:42.758 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738437 got 34359738439 for osd.0 2023-11-16T18:23:42.758 DEBUG:teuthology.parallel:result is None 2023-11-16T18:23:42.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:42 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1410623943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-16T18:23:42.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:42 smithi002 ceph-mon[98807]: pgmap v290: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 5 op/s; 7.6 KiB/s, 0 keys/s, 1 objects/s recovering 2023-11-16T18:23:42.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:42 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1410623943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-16T18:23:42.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:42 smithi188 ceph-mon[100639]: pgmap v290: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 5 op/s; 7.6 KiB/s, 0 keys/s, 1 objects/s recovering 2023-11-16T18:23:42.906 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743449 got 296352743450 for osd.10 2023-11-16T18:23:42.906 DEBUG:teuthology.parallel:result is None 2023-11-16T18:23:42.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:42 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1410623943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-16T18:23:42.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:42 smithi040 ceph-mon[100251]: pgmap v290: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 5 op/s; 7.6 KiB/s, 0 keys/s, 1 objects/s recovering 2023-11-16T18:23:43.050 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574911 got 55834574913 for osd.1 2023-11-16T18:23:43.050 DEBUG:teuthology.parallel:result is None 2023-11-16T18:23:43.175 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561067 got 188978561069 for osd.6 2023-11-16T18:23:43.176 DEBUG:teuthology.parallel:result is None 2023-11-16T18:23:43.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.173+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/738865063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4a410cba0 msgr2=0x7fb4a410ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:43.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.173+0000 7fb4ab2e8700 1 --2- 172.21.15.2:0/738865063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4a410cba0 0x7fb4a410ef80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00b0a0 tx=0x7fb49c00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.173+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/738865063 shutdown_connections 2023-11-16T18:23:43.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.173+0000 7fb4ab2e8700 1 --2- 172.21.15.2:0/738865063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4a4072e30 0x7fb4a4073200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.173+0000 7fb4ab2e8700 1 --2- 172.21.15.2:0/738865063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4a410cba0 0x7fb4a410ef80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.173+0000 7fb4ab2e8700 1 --2- 172.21.15.2:0/738865063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4a4073740 0x7fb4a410c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.173+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/738865063 >> 172.21.15.2:0/738865063 conn(0x7fb4a40fbde0 msgr2=0x7fb4a40fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:43.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.173+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/738865063 shutdown_connections 2023-11-16T18:23:43.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.173+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/738865063 wait complete. 2023-11-16T18:23:43.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.174+0000 7fb4ab2e8700 1 Processor -- start 2023-11-16T18:23:43.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.174+0000 7fb4ab2e8700 1 -- start start 2023-11-16T18:23:43.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.174+0000 7fb4ab2e8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4a4072e30 0x7fb4a419e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:43.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.174+0000 7fb4ab2e8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4a4073740 0x7fb4a419eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:43.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.174+0000 7fb4ab2e8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4a410cba0 0x7fb4a4198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:43.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.174+0000 7fb4ab2e8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a4198d00 con 0x7fb4a410cba0 2023-11-16T18:23:43.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.174+0000 7fb4ab2e8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a4198e70 con 0x7fb4a4073740 2023-11-16T18:23:43.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.174+0000 7fb4ab2e8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a4199030 con 0x7fb4a4072e30 2023-11-16T18:23:43.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.174+0000 7fb4a9885700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4a410cba0 0x7fb4a4198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:43.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.174+0000 7fb4a9885700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4a410cba0 0x7fb4a4198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48744/0 (socket says 172.21.15.2:48744) 2023-11-16T18:23:43.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.174+0000 7fb4a9885700 1 -- 172.21.15.2:0/3072437716 learned_addr learned my addr 172.21.15.2:0/3072437716 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:43.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.175+0000 7fb4a8883700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4a4073740 0x7fb4a419eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:43.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.175+0000 7fb4a9084700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4a4072e30 0x7fb4a419e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:43.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.175+0000 7fb4a9885700 1 -- 172.21.15.2:0/3072437716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4a4072e30 msgr2=0x7fb4a419e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:43.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.175+0000 7fb4a9885700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4a4072e30 0x7fb4a419e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.175+0000 7fb4a9885700 1 -- 172.21.15.2:0/3072437716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4a4073740 msgr2=0x7fb4a419eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:43.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.175+0000 7fb4a9885700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4a4073740 0x7fb4a419eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.175+0000 7fb4a9885700 1 -- 172.21.15.2:0/3072437716 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c009d00 con 0x7fb4a410cba0 2023-11-16T18:23:43.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.175+0000 7fb4a9885700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4a410cba0 0x7fb4a4198550 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00b780 tx=0x7fb49c00bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:43.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.175+0000 7fb49a7fc700 1 -- 172.21.15.2:0/3072437716 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c00e040 con 0x7fb4a410cba0 2023-11-16T18:23:43.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.175+0000 7fb49a7fc700 1 -- 172.21.15.2:0/3072437716 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb49c0094a0 con 0x7fb4a410cba0 2023-11-16T18:23:43.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.175+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/3072437716 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a4112510 con 0x7fb4a410cba0 2023-11-16T18:23:43.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.175+0000 7fb49a7fc700 1 -- 172.21.15.2:0/3072437716 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c01b6c0 con 0x7fb4a410cba0 2023-11-16T18:23:43.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.175+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/3072437716 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a4112380 con 0x7fb4a410cba0 2023-11-16T18:23:43.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.176+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/3072437716 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a41126a0 con 0x7fb4a410cba0 2023-11-16T18:23:43.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.177+0000 7fb49a7fc700 1 -- 172.21.15.2:0/3072437716 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fb49c01d470 con 0x7fb4a410cba0 2023-11-16T18:23:43.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.178+0000 7fb49a7fc700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb4900a1260 0x7fb4900a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:43.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.178+0000 7fb4a9084700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb4900a1260 0x7fb4900a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:43.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.178+0000 7fb49a7fc700 1 -- 172.21.15.2:0/3072437716 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb49c0ce780 con 0x7fb4a410cba0 2023-11-16T18:23:43.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.179+0000 7fb4a9084700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb4900a1260 0x7fb4900a3710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0005950 tx=0x7fb4a000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.182+0000 7fb49a7fc700 1 -- 172.21.15.2:0/3072437716 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb49c097af0 con 0x7fb4a410cba0 2023-11-16T18:23:43.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.391+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/3072437716 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb4a41074c0 con 0x7fb4a410cba0 2023-11-16T18:23:43.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.391+0000 7fb49a7fc700 1 -- 172.21.15.2:0/3072437716 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb49c09b110 con 0x7fb4a410cba0 2023-11-16T18:23:43.393 INFO:teuthology.orchestra.run.smithi002.stdout:137438953526 2023-11-16T18:23:43.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.393+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/3072437716 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb4900a1260 msgr2=0x7fb4900a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:43.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.393+0000 7fb4ab2e8700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb4900a1260 0x7fb4900a3710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0005950 tx=0x7fb4a000b410 comp rx=0 tx=0).stop 2023-11-16T18:23:43.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.393+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/3072437716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4a410cba0 msgr2=0x7fb4a4198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:43.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.393+0000 7fb4ab2e8700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4a410cba0 0x7fb4a4198550 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00b780 tx=0x7fb49c00bd60 comp rx=0 tx=0).stop 2023-11-16T18:23:43.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.393+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/3072437716 shutdown_connections 2023-11-16T18:23:43.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.393+0000 7fb4ab2e8700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb4900a1260 0x7fb4900a3710 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.393+0000 7fb4ab2e8700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4a4072e30 0x7fb4a419e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.393+0000 7fb4ab2e8700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4a410cba0 0x7fb4a4198550 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.393+0000 7fb4ab2e8700 1 --2- 172.21.15.2:0/3072437716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4a4073740 0x7fb4a419eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.393+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/3072437716 >> 172.21.15.2:0/3072437716 conn(0x7fb4a40fbde0 msgr2=0x7fb4a4104600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:43.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.393+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/3072437716 shutdown_connections 2023-11-16T18:23:43.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.393+0000 7fb4ab2e8700 1 -- 172.21.15.2:0/3072437716 wait complete. 2023-11-16T18:23:43.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:43 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3072437716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-16T18:23:43.626 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:43.731 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789998 got 158913790000 for osd.5 2023-11-16T18:23:43.731 DEBUG:teuthology.parallel:result is None 2023-11-16T18:23:43.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:43 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3072437716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-16T18:23:43.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.969+0000 7f45eecb1700 1 -- 172.21.15.2:0/2861847810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80ffde0 msgr2=0x7f45e8103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:43.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.969+0000 7f45eecb1700 1 --2- 172.21.15.2:0/2861847810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80ffde0 0x7f45e8103db0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f45d00099c0 tx=0x7f45d0009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.969+0000 7f45eecb1700 1 -- 172.21.15.2:0/2861847810 shutdown_connections 2023-11-16T18:23:43.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.969+0000 7f45eecb1700 1 --2- 172.21.15.2:0/2861847810 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45e81044e0 0x7f45e810c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.969+0000 7f45eecb1700 1 --2- 172.21.15.2:0/2861847810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45e80ff440 0x7f45e80ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.969+0000 7f45eecb1700 1 --2- 172.21.15.2:0/2861847810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80ffde0 0x7f45e8103db0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.969+0000 7f45eecb1700 1 -- 172.21.15.2:0/2861847810 >> 172.21.15.2:0/2861847810 conn(0x7f45e80facc0 msgr2=0x7f45e80fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:43.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.969+0000 7f45eecb1700 1 -- 172.21.15.2:0/2861847810 shutdown_connections 2023-11-16T18:23:43.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.969+0000 7f45eecb1700 1 -- 172.21.15.2:0/2861847810 wait complete. 2023-11-16T18:23:43.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eecb1700 1 Processor -- start 2023-11-16T18:23:43.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eecb1700 1 -- start start 2023-11-16T18:23:43.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eecb1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80ff440 0x7f45e8197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:43.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eecb1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45e80ffde0 0x7f45e81976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:43.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eecb1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45e81044e0 0x7f45e81a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:43.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eecb1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e81a2720 con 0x7f45e80ff440 2023-11-16T18:23:43.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eecb1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e81a2860 con 0x7f45e80ffde0 2023-11-16T18:23:43.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eecb1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e81a29a0 con 0x7f45e81044e0 2023-11-16T18:23:43.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eca4d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80ff440 0x7f45e8197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:43.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eca4d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80ff440 0x7f45e8197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48754/0 (socket says 172.21.15.2:48754) 2023-11-16T18:23:43.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eca4d700 1 -- 172.21.15.2:0/3228435995 learned_addr learned my addr 172.21.15.2:0/3228435995 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:43.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45ed24e700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45e81044e0 0x7f45e81a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:43.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45e7fff700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45e80ffde0 0x7f45e81976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:43.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eca4d700 1 -- 172.21.15.2:0/3228435995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45e81044e0 msgr2=0x7f45e81a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:43.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eca4d700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45e81044e0 0x7f45e81a21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eca4d700 1 -- 172.21.15.2:0/3228435995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45e80ffde0 msgr2=0x7f45e81976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:43.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eca4d700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45e80ffde0 0x7f45e81976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:43.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.970+0000 7f45eca4d700 1 -- 172.21.15.2:0/3228435995 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d00095f0 con 0x7f45e80ff440 2023-11-16T18:23:43.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.971+0000 7f45eca4d700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80ff440 0x7f45e8197160 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f45d800e9a0 tx=0x7f45d800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:43.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.971+0000 7f45e5ffb700 1 -- 172.21.15.2:0/3228435995 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d8009790 con 0x7f45e80ff440 2023-11-16T18:23:43.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.971+0000 7f45eecb1700 1 -- 172.21.15.2:0/3228435995 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e810f210 con 0x7f45e80ff440 2023-11-16T18:23:43.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.971+0000 7f45e5ffb700 1 -- 172.21.15.2:0/3228435995 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45d800cbd0 con 0x7f45e80ff440 2023-11-16T18:23:43.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.971+0000 7f45e5ffb700 1 -- 172.21.15.2:0/3228435995 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d8018680 con 0x7f45e80ff440 2023-11-16T18:23:43.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.971+0000 7f45eecb1700 1 -- 172.21.15.2:0/3228435995 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e810f080 con 0x7f45e80ff440 2023-11-16T18:23:43.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.972+0000 7f45eecb1700 1 -- 172.21.15.2:0/3228435995 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e810f3a0 con 0x7f45e80ff440 2023-11-16T18:23:43.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.973+0000 7f45e5ffb700 1 -- 172.21.15.2:0/3228435995 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f45d800cd40 con 0x7f45e80ff440 2023-11-16T18:23:43.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.974+0000 7f45e5ffb700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45d40a11e0 0x7f45d40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:43.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.974+0000 7f45e5ffb700 1 -- 172.21.15.2:0/3228435995 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f45d8014070 con 0x7f45e80ff440 2023-11-16T18:23:43.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.974+0000 7f45e7fff700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45d40a11e0 0x7f45d40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:43.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.974+0000 7f45e7fff700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45d40a11e0 0x7f45d40a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009990 tx=0x7f45d001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:43.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:43.978+0000 7f45e5ffb700 1 -- 172.21.15.2:0/3228435995 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f45d8051730 con 0x7f45e80ff440 2023-11-16T18:23:43.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:43 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3072437716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-16T18:23:44.050 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953523 got 137438953526 for osd.4 2023-11-16T18:23:44.050 DEBUG:teuthology.parallel:result is None 2023-11-16T18:23:44.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.185+0000 7f45eecb1700 1 -- 172.21.15.2:0/3228435995 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f45e8066ea0 con 0x7f45e80ff440 2023-11-16T18:23:44.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.186+0000 7f45e5ffb700 1 -- 172.21.15.2:0/3228435995 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f45d80962e0 con 0x7f45e80ff440 2023-11-16T18:23:44.188 INFO:teuthology.orchestra.run.smithi002.stdout:210453397545 2023-11-16T18:23:44.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.188+0000 7f45eecb1700 1 -- 172.21.15.2:0/3228435995 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45d40a11e0 msgr2=0x7f45d40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:44.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.188+0000 7f45eecb1700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45d40a11e0 0x7f45d40a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009990 tx=0x7f45d001a040 comp rx=0 tx=0).stop 2023-11-16T18:23:44.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.188+0000 7f45eecb1700 1 -- 172.21.15.2:0/3228435995 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80ff440 msgr2=0x7f45e8197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:44.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.188+0000 7f45eecb1700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80ff440 0x7f45e8197160 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f45d800e9a0 tx=0x7f45d800ecb0 comp rx=0 tx=0).stop 2023-11-16T18:23:44.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.188+0000 7f45eecb1700 1 -- 172.21.15.2:0/3228435995 shutdown_connections 2023-11-16T18:23:44.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.188+0000 7f45eecb1700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45d40a11e0 0x7f45d40a3690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:44.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.188+0000 7f45eecb1700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45e81044e0 0x7f45e81a21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:44.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.188+0000 7f45eecb1700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80ff440 0x7f45e8197160 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:44.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.188+0000 7f45eecb1700 1 --2- 172.21.15.2:0/3228435995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45e80ffde0 0x7f45e81976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:44.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.188+0000 7f45eecb1700 1 -- 172.21.15.2:0/3228435995 >> 172.21.15.2:0/3228435995 conn(0x7f45e80facc0 msgr2=0x7f45e80fb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:44.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.188+0000 7f45eecb1700 1 -- 172.21.15.2:0/3228435995 shutdown_connections 2023-11-16T18:23:44.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.188+0000 7f45eecb1700 1 -- 172.21.15.2:0/3228435995 wait complete. 2023-11-16T18:23:44.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:44 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3228435995' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-16T18:23:44.731 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397543 got 210453397545 for osd.7 2023-11-16T18:23:44.731 DEBUG:teuthology.parallel:result is None 2023-11-16T18:23:44.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:44 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3228435995' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-16T18:23:44.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.824+0000 7f1c543d5700 1 -- 172.21.15.2:0/1114031041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c0ff460 msgr2=0x7f1c4c0ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:44.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.824+0000 7f1c543d5700 1 --2- 172.21.15.2:0/1114031041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c0ff460 0x7f1c4c0ff830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1c400099c0 tx=0x7f1c40009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.824+0000 7f1c543d5700 1 -- 172.21.15.2:0/1114031041 shutdown_connections 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.824+0000 7f1c543d5700 1 --2- 172.21.15.2:0/1114031041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4c104500 0x7f1c4c10c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.824+0000 7f1c543d5700 1 --2- 172.21.15.2:0/1114031041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c0ff460 0x7f1c4c0ff830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.824+0000 7f1c543d5700 1 --2- 172.21.15.2:0/1114031041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c4c0ffe00 0x7f1c4c103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.824+0000 7f1c543d5700 1 -- 172.21.15.2:0/1114031041 >> 172.21.15.2:0/1114031041 conn(0x7f1c4c0770a0 msgr2=0x7f1c4c0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.824+0000 7f1c543d5700 1 -- 172.21.15.2:0/1114031041 shutdown_connections 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.824+0000 7f1c543d5700 1 -- 172.21.15.2:0/1114031041 wait complete. 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c543d5700 1 Processor -- start 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c543d5700 1 -- start start 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c543d5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4c0ff460 0x7f1c4c10c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c543d5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c0ffe00 0x7f1c4c10ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c543d5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c4c104500 0x7f1c4c10d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c543d5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c1a4990 con 0x7f1c4c0ffe00 2023-11-16T18:23:44.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c543d5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c1a4b00 con 0x7f1c4c0ff460 2023-11-16T18:23:44.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c543d5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c1a4cc0 con 0x7f1c4c104500 2023-11-16T18:23:44.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c51970700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c0ffe00 0x7f1c4c10ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:44.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c51970700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c0ffe00 0x7f1c4c10ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48770/0 (socket says 172.21.15.2:48770) 2023-11-16T18:23:44.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c51970700 1 -- 172.21.15.2:0/3013472121 learned_addr learned my addr 172.21.15.2:0/3013472121 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:44.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c52972700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c4c104500 0x7f1c4c10d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:44.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c52171700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4c0ff460 0x7f1c4c10c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:44.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c51970700 1 -- 172.21.15.2:0/3013472121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c4c104500 msgr2=0x7f1c4c10d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:44.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c51970700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c4c104500 0x7f1c4c10d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:44.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c51970700 1 -- 172.21.15.2:0/3013472121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4c0ff460 msgr2=0x7f1c4c10c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:44.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c51970700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4c0ff460 0x7f1c4c10c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:44.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c51970700 1 -- 172.21.15.2:0/3013472121 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c400095f0 con 0x7f1c4c0ffe00 2023-11-16T18:23:44.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.825+0000 7f1c52171700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4c0ff460 0x7f1c4c10c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:44.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.826+0000 7f1c51970700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c0ffe00 0x7f1c4c10ced0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800c8f0 tx=0x7f1c4800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:44.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.826+0000 7f1c3f7fe700 1 -- 172.21.15.2:0/3013472121 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48007940 con 0x7f1c4c0ffe00 2023-11-16T18:23:44.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.826+0000 7f1c3f7fe700 1 -- 172.21.15.2:0/3013472121 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c4800f450 con 0x7f1c4c0ffe00 2023-11-16T18:23:44.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.826+0000 7f1c543d5700 1 -- 172.21.15.2:0/3013472121 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4c1103e0 con 0x7f1c4c0ffe00 2023-11-16T18:23:44.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.826+0000 7f1c3f7fe700 1 -- 172.21.15.2:0/3013472121 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48018660 con 0x7f1c4c0ffe00 2023-11-16T18:23:44.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.826+0000 7f1c543d5700 1 -- 172.21.15.2:0/3013472121 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4c110250 con 0x7f1c4c0ffe00 2023-11-16T18:23:44.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.827+0000 7f1c543d5700 1 -- 172.21.15.2:0/3013472121 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c4c110570 con 0x7f1c4c0ffe00 2023-11-16T18:23:44.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.828+0000 7f1c3f7fe700 1 -- 172.21.15.2:0/3013472121 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f1c4800f5c0 con 0x7f1c4c0ffe00 2023-11-16T18:23:44.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.829+0000 7f1c3f7fe700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1c380a1260 0x7f1c380a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:44.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.829+0000 7f1c3f7fe700 1 -- 172.21.15.2:0/3013472121 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1c480ccce0 con 0x7f1c4c0ffe00 2023-11-16T18:23:44.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.829+0000 7f1c52171700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1c380a1260 0x7f1c380a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:44.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.829+0000 7f1c52171700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1c380a1260 0x7f1c380a3710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1c40006010 tx=0x7f1c40000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:44.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:44.832+0000 7f1c3f7fe700 1 -- 172.21.15.2:0/3013472121 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1c480960d0 con 0x7f1c4c0ffe00 2023-11-16T18:23:44.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:44 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3228435995' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-16T18:23:45.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.032+0000 7f1c543d5700 1 -- 172.21.15.2:0/3013472121 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f1c4c066ea0 con 0x7f1c4c0ffe00 2023-11-16T18:23:45.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.032+0000 7f1c3f7fe700 1 -- 172.21.15.2:0/3013472121 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1c480996f0 con 0x7f1c4c0ffe00 2023-11-16T18:23:45.035 INFO:teuthology.orchestra.run.smithi002.stdout:274877906976 2023-11-16T18:23:45.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.034+0000 7f1c543d5700 1 -- 172.21.15.2:0/3013472121 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1c380a1260 msgr2=0x7f1c380a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.034+0000 7f1c543d5700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1c380a1260 0x7f1c380a3710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1c40006010 tx=0x7f1c40000bc0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.034+0000 7f1c543d5700 1 -- 172.21.15.2:0/3013472121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c0ffe00 msgr2=0x7f1c4c10ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.034+0000 7f1c543d5700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c0ffe00 0x7f1c4c10ced0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800c8f0 tx=0x7f1c4800cc00 comp rx=0 tx=0).stop 2023-11-16T18:23:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.034+0000 7f1c543d5700 1 -- 172.21.15.2:0/3013472121 shutdown_connections 2023-11-16T18:23:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.034+0000 7f1c543d5700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1c380a1260 0x7f1c380a3710 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.034+0000 7f1c543d5700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c4c104500 0x7f1c4c10d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.034+0000 7f1c543d5700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4c0ff460 0x7f1c4c10c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.034+0000 7f1c543d5700 1 --2- 172.21.15.2:0/3013472121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c0ffe00 0x7f1c4c10ced0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.035+0000 7f1c543d5700 1 -- 172.21.15.2:0/3013472121 >> 172.21.15.2:0/3013472121 conn(0x7f1c4c0770a0 msgr2=0x7f1c4c102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.035+0000 7f1c543d5700 1 -- 172.21.15.2:0/3013472121 shutdown_connections 2023-11-16T18:23:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.035+0000 7f1c543d5700 1 -- 172.21.15.2:0/3013472121 wait complete. 2023-11-16T18:23:45.474 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877906974 got 274877906976 for osd.9 2023-11-16T18:23:45.474 DEBUG:teuthology.parallel:result is None 2023-11-16T18:23:45.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.730+0000 7f8e295b4700 1 -- 172.21.15.2:0/2941408773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e24100680 msgr2=0x7f8e24100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:45.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.730+0000 7f8e295b4700 1 --2- 172.21.15.2:0/2941408773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e24100680 0x7f8e24100a50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8e140099c0 tx=0x7f8e14009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.730+0000 7f8e295b4700 1 -- 172.21.15.2:0/2941408773 shutdown_connections 2023-11-16T18:23:45.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.730+0000 7f8e295b4700 1 --2- 172.21.15.2:0/2941408773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e24109c30 0x7f8e2410c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.730+0000 7f8e295b4700 1 --2- 172.21.15.2:0/2941408773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e24100680 0x7f8e24100a50 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.730+0000 7f8e295b4700 1 --2- 172.21.15.2:0/2941408773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e24100f90 0x7f8e241096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.730+0000 7f8e295b4700 1 -- 172.21.15.2:0/2941408773 >> 172.21.15.2:0/2941408773 conn(0x7f8e24074b30 msgr2=0x7f8e24074f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:45.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.730+0000 7f8e295b4700 1 -- 172.21.15.2:0/2941408773 shutdown_connections 2023-11-16T18:23:45.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.730+0000 7f8e295b4700 1 -- 172.21.15.2:0/2941408773 wait complete. 2023-11-16T18:23:45.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.731+0000 7f8e295b4700 1 Processor -- start 2023-11-16T18:23:45.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.731+0000 7f8e295b4700 1 -- start start 2023-11-16T18:23:45.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.731+0000 7f8e295b4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e24100680 0x7f8e241972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:45.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.731+0000 7f8e295b4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e24100f90 0x7f8e24197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:45.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.731+0000 7f8e295b4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e24109c30 0x7f8e241a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:45.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.731+0000 7f8e295b4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e24198160 con 0x7f8e24100680 2023-11-16T18:23:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.731+0000 7f8e295b4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e241a28a0 con 0x7f8e24100f90 2023-11-16T18:23:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.731+0000 7f8e295b4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e241a2a60 con 0x7f8e24109c30 2023-11-16T18:23:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.731+0000 7f8e22ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e24100680 0x7f8e241972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.731+0000 7f8e22ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e24100680 0x7f8e241972e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48782/0 (socket says 172.21.15.2:48782) 2023-11-16T18:23:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.731+0000 7f8e22ffd700 1 -- 172.21.15.2:0/206868007 learned_addr learned my addr 172.21.15.2:0/206868007 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.731+0000 7f8e227fc700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e24100f90 0x7f8e24197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.731+0000 7f8e237fe700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e24109c30 0x7f8e241a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.732+0000 7f8e22ffd700 1 -- 172.21.15.2:0/206868007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e24109c30 msgr2=0x7f8e241a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.732+0000 7f8e22ffd700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e24109c30 0x7f8e241a2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.732+0000 7f8e22ffd700 1 -- 172.21.15.2:0/206868007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e24100f90 msgr2=0x7f8e24197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.732+0000 7f8e22ffd700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e24100f90 0x7f8e24197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.732+0000 7f8e22ffd700 1 -- 172.21.15.2:0/206868007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e140095f0 con 0x7f8e24100680 2023-11-16T18:23:45.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.732+0000 7f8e237fe700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e24109c30 0x7f8e241a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:45.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.732+0000 7f8e227fc700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e24100f90 0x7f8e24197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:45.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.732+0000 7f8e22ffd700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e24100680 0x7f8e241972e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8e14000c00 tx=0x7f8e1400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:45.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.732+0000 7f8e1bfff700 1 -- 172.21.15.2:0/206868007 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1401d070 con 0x7f8e24100680 2023-11-16T18:23:45.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.732+0000 7f8e295b4700 1 -- 172.21.15.2:0/206868007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e2410f380 con 0x7f8e24100680 2023-11-16T18:23:45.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.732+0000 7f8e1bfff700 1 -- 172.21.15.2:0/206868007 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e14005510 con 0x7f8e24100680 2023-11-16T18:23:45.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.732+0000 7f8e1bfff700 1 -- 172.21.15.2:0/206868007 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14004cb0 con 0x7f8e24100680 2023-11-16T18:23:45.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.732+0000 7f8e295b4700 1 -- 172.21.15.2:0/206868007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2410f1f0 con 0x7f8e24100680 2023-11-16T18:23:45.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.733+0000 7f8e295b4700 1 -- 172.21.15.2:0/206868007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2410f510 con 0x7f8e24100680 2023-11-16T18:23:45.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.735+0000 7f8e1bfff700 1 -- 172.21.15.2:0/206868007 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f8e1400fc30 con 0x7f8e24100680 2023-11-16T18:23:45.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.736+0000 7f8e1bfff700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8e100a11e0 0x7f8e100a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:45.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.736+0000 7f8e227fc700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8e100a11e0 0x7f8e100a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:45.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.736+0000 7f8e1bfff700 1 -- 172.21.15.2:0/206868007 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8e140ce0a0 con 0x7f8e24100680 2023-11-16T18:23:45.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.736+0000 7f8e227fc700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8e100a11e0 0x7f8e100a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8e240ffe40 tx=0x7f8e0c00b2e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:45.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.739+0000 7f8e1bfff700 1 -- 172.21.15.2:0/206868007 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8e140973b0 con 0x7f8e24100680 2023-11-16T18:23:45.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:45 smithi188 ceph-mon[100639]: pgmap v291: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 255 B/s wr, 3 op/s; 7.6 KiB/s, 0 keys/s, 2 objects/s recovering 2023-11-16T18:23:45.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:45 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3013472121' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-11-16T18:23:45.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:45 smithi002 ceph-mon[98807]: pgmap v291: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 255 B/s wr, 3 op/s; 7.6 KiB/s, 0 keys/s, 2 objects/s recovering 2023-11-16T18:23:45.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:45 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3013472121' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-11-16T18:23:45.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.940+0000 7f8e295b4700 1 -- 172.21.15.2:0/206868007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f8e241a3410 con 0x7f8e24100680 2023-11-16T18:23:45.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.940+0000 7f8e1bfff700 1 -- 172.21.15.2:0/206868007 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8e1409a9d0 con 0x7f8e24100680 2023-11-16T18:23:45.944 INFO:teuthology.orchestra.run.smithi002.stdout:77309411389 2023-11-16T18:23:45.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.942+0000 7f8e295b4700 1 -- 172.21.15.2:0/206868007 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8e100a11e0 msgr2=0x7f8e100a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:45.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.942+0000 7f8e295b4700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8e100a11e0 0x7f8e100a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8e240ffe40 tx=0x7f8e0c00b2e0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.942+0000 7f8e295b4700 1 -- 172.21.15.2:0/206868007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e24100680 msgr2=0x7f8e241972e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:45.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.942+0000 7f8e295b4700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e24100680 0x7f8e241972e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8e14000c00 tx=0x7f8e1400f740 comp rx=0 tx=0).stop 2023-11-16T18:23:45.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.942+0000 7f8e295b4700 1 -- 172.21.15.2:0/206868007 shutdown_connections 2023-11-16T18:23:45.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.942+0000 7f8e295b4700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8e100a11e0 0x7f8e100a3690 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.942+0000 7f8e295b4700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e24109c30 0x7f8e241a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.942+0000 7f8e295b4700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e24100680 0x7f8e241972e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.942+0000 7f8e295b4700 1 --2- 172.21.15.2:0/206868007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e24100f90 0x7f8e24197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:45.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.942+0000 7f8e295b4700 1 -- 172.21.15.2:0/206868007 >> 172.21.15.2:0/206868007 conn(0x7f8e24074b30 msgr2=0x7f8e240fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:45.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.942+0000 7f8e295b4700 1 -- 172.21.15.2:0/206868007 shutdown_connections 2023-11-16T18:23:45.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:45.942+0000 7f8e295b4700 1 -- 172.21.15.2:0/206868007 wait complete. 2023-11-16T18:23:45.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:45 smithi040 ceph-mon[100251]: pgmap v291: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 255 B/s wr, 3 op/s; 7.6 KiB/s, 0 keys/s, 2 objects/s recovering 2023-11-16T18:23:45.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:45 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3013472121' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-11-16T18:23:46.361 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411387 got 77309411389 for osd.2 2023-11-16T18:23:46.361 DEBUG:teuthology.parallel:result is None 2023-11-16T18:23:46.361 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-11-16T18:23:46.362 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph pg dump --format=json 2023-11-16T18:23:46.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:46 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/206868007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-16T18:23:46.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:46 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/206868007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-16T18:23:46.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:46 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/206868007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-16T18:23:47.017 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:47.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:47 smithi188 ceph-mon[100639]: pgmap v292: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s; 7.6 KiB/s, 0 keys/s, 2 objects/s recovering 2023-11-16T18:23:47.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:47 smithi002 ceph-mon[98807]: pgmap v292: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s; 7.6 KiB/s, 0 keys/s, 2 objects/s recovering 2023-11-16T18:23:47.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:47 smithi040 ceph-mon[100251]: pgmap v292: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s; 7.6 KiB/s, 0 keys/s, 2 objects/s recovering 2023-11-16T18:23:48.227 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.223+0000 7fc438cbb700 1 -- 172.21.15.2:0/107902683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4340ffe00 msgr2=0x7fc434103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:48.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.223+0000 7fc438cbb700 1 --2- 172.21.15.2:0/107902683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4340ffe00 0x7fc434103dd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc4240099c0 tx=0x7fc424009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:48.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.224+0000 7fc438cbb700 1 -- 172.21.15.2:0/107902683 shutdown_connections 2023-11-16T18:23:48.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.224+0000 7fc438cbb700 1 --2- 172.21.15.2:0/107902683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc434104500 0x7fc43410c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:48.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.224+0000 7fc438cbb700 1 --2- 172.21.15.2:0/107902683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4340ff460 0x7fc4340ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:48.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.224+0000 7fc438cbb700 1 --2- 172.21.15.2:0/107902683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4340ffe00 0x7fc434103dd0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:48.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.224+0000 7fc438cbb700 1 -- 172.21.15.2:0/107902683 >> 172.21.15.2:0/107902683 conn(0x7fc4340770a0 msgr2=0x7fc4340774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:48.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.224+0000 7fc438cbb700 1 -- 172.21.15.2:0/107902683 shutdown_connections 2023-11-16T18:23:48.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.224+0000 7fc438cbb700 1 -- 172.21.15.2:0/107902683 wait complete. 2023-11-16T18:23:48.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.224+0000 7fc438cbb700 1 Processor -- start 2023-11-16T18:23:48.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.224+0000 7fc438cbb700 1 -- start start 2023-11-16T18:23:48.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc438cbb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4340ff460 0x7fc434100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:48.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc438cbb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4340ffe00 0x7fc4341013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:48.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc438cbb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc434104500 0x7fc434101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:48.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc438cbb700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4341a48f0 con 0x7fc4340ffe00 2023-11-16T18:23:48.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc438cbb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4341a4a60 con 0x7fc434104500 2023-11-16T18:23:48.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc438cbb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4341a4c20 con 0x7fc4340ff460 2023-11-16T18:23:48.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc431d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4340ffe00 0x7fc4341013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:48.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc431d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4340ffe00 0x7fc4341013b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48798/0 (socket says 172.21.15.2:48798) 2023-11-16T18:23:48.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc431d9b700 1 -- 172.21.15.2:0/4290393833 learned_addr learned my addr 172.21.15.2:0/4290393833 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:48.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc432d9d700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc434104500 0x7fc434101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:48.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc43259c700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4340ff460 0x7fc434100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:48.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc431d9b700 1 -- 172.21.15.2:0/4290393833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4340ff460 msgr2=0x7fc434100e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:48.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc431d9b700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4340ff460 0x7fc434100e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:48.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc431d9b700 1 -- 172.21.15.2:0/4290393833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc434104500 msgr2=0x7fc434101930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:48.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc431d9b700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc434104500 0x7fc434101930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:48.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc431d9b700 1 -- 172.21.15.2:0/4290393833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4240095f0 con 0x7fc4340ffe00 2023-11-16T18:23:48.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc432d9d700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc434104500 0x7fc434101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:48.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc431d9b700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4340ffe00 0x7fc4341013b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc424009990 tx=0x7fc424005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:48.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.225+0000 7fc4237fe700 1 -- 172.21.15.2:0/4290393833 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42401d070 con 0x7fc4340ffe00 2023-11-16T18:23:48.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.226+0000 7fc438cbb700 1 -- 172.21.15.2:0/4290393833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43410f230 con 0x7fc4340ffe00 2023-11-16T18:23:48.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.226+0000 7fc4237fe700 1 -- 172.21.15.2:0/4290393833 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc424004090 con 0x7fc4340ffe00 2023-11-16T18:23:48.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.226+0000 7fc4237fe700 1 -- 172.21.15.2:0/4290393833 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc424022d20 con 0x7fc4340ffe00 2023-11-16T18:23:48.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.226+0000 7fc438cbb700 1 -- 172.21.15.2:0/4290393833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43410f0a0 con 0x7fc4340ffe00 2023-11-16T18:23:48.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.227+0000 7fc438cbb700 1 -- 172.21.15.2:0/4290393833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43410f3c0 con 0x7fc4340ffe00 2023-11-16T18:23:48.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.228+0000 7fc4237fe700 1 -- 172.21.15.2:0/4290393833 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fc424022470 con 0x7fc4340ffe00 2023-11-16T18:23:48.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.228+0000 7fc4237fe700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc41c0a11e0 0x7fc41c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:48.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.228+0000 7fc43259c700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc41c0a11e0 0x7fc41c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:48.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.228+0000 7fc4237fe700 1 -- 172.21.15.2:0/4290393833 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc4240cdf40 con 0x7fc4340ffe00 2023-11-16T18:23:48.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.229+0000 7fc43259c700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc41c0a11e0 0x7fc41c0a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc428005950 tx=0x7fc4280058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:48.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.233+0000 7fc4237fe700 1 -- 172.21.15.2:0/4290393833 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc42405c730 con 0x7fc4340ffe00 2023-11-16T18:23:48.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.449+0000 7fc438cbb700 1 -- 172.21.15.2:0/4290393833 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc434068ea0 con 0x7fc41c0a11e0 2023-11-16T18:23:48.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.464+0000 7fc4237fe700 1 -- 172.21.15.2:0/4290393833 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422229 (secure 0 0 0) 0x7fc434068ea0 con 0x7fc41c0a11e0 2023-11-16T18:23:48.479 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.468+0000 7fc438cbb700 1 -- 172.21.15.2:0/4290393833 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc41c0a11e0 msgr2=0x7fc41c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.468+0000 7fc438cbb700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc41c0a11e0 0x7fc41c0a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc428005950 tx=0x7fc4280058e0 comp rx=0 tx=0).stop 2023-11-16T18:23:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.468+0000 7fc438cbb700 1 -- 172.21.15.2:0/4290393833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4340ffe00 msgr2=0x7fc4341013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.468+0000 7fc438cbb700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4340ffe00 0x7fc4341013b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc424009990 tx=0x7fc424005e80 comp rx=0 tx=0).stop 2023-11-16T18:23:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.468+0000 7fc438cbb700 1 -- 172.21.15.2:0/4290393833 shutdown_connections 2023-11-16T18:23:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.468+0000 7fc438cbb700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc41c0a11e0 0x7fc41c0a3690 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.468+0000 7fc438cbb700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4340ff460 0x7fc434100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.468+0000 7fc438cbb700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc434104500 0x7fc434101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.468+0000 7fc438cbb700 1 --2- 172.21.15.2:0/4290393833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4340ffe00 0x7fc4341013b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.468+0000 7fc438cbb700 1 -- 172.21.15.2:0/4290393833 >> 172.21.15.2:0/4290393833 conn(0x7fc4340770a0 msgr2=0x7fc434068790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.468+0000 7fc438cbb700 1 -- 172.21.15.2:0/4290393833 shutdown_connections 2023-11-16T18:23:48.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:48.468+0000 7fc438cbb700 1 -- 172.21.15.2:0/4290393833 wait complete. 2023-11-16T18:23:48.485 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-16T18:23:48.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:48 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:23:48.852 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:48 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:23:48.876 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2023-11-16T18:23:47.818574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655745,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":378,"num_read_kb":307,"num_write":337,"num_write_kb":3050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1242772,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80796,"kb_used_data":11708,"kb_used_omap":0,"kb_used_meta":68992,"kb_avail":1124861028,"statfs":{"total":1151940427776,"available":1151857692672,"internally_reserved":0,"allocated":11988992,"data_stored":7770365,"data_compressed":329214,"data_compressed_allocated":442368,"data_compressed_original":2416640,"omap_allocated":0,"internal_metadata":70647808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1652,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":20,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":62036,"num_keys_recovered":7,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.320591"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506980+0000","last_change":"2023-11-16T18:22:46.829814+0000","last_active":"2023-11-16T18:23:39.506980+0000","last_peered":"2023-11-16T18:23:39.506980+0000","last_clean":"2023-11-16T18:23:39.506980+0000","last_became_active":"2023-11-16T18:22:46.829407+0000","last_became_peered":"2023-11-16T18:22:46.829407+0000","last_unstale":"2023-11-16T18:23:39.506980+0000","last_undegraded":"2023-11-16T18:23:39.506980+0000","last_fullsized":"2023-11-16T18:23:39.506980+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:02:05.303420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505209+0000","last_change":"2023-11-16T18:22:46.818700+0000","last_active":"2023-11-16T18:23:39.505209+0000","last_peered":"2023-11-16T18:23:39.505209+0000","last_clean":"2023-11-16T18:23:39.505209+0000","last_became_active":"2023-11-16T18:22:46.818388+0000","last_became_peered":"2023-11-16T18:22:46.818388+0000","last_unstale":"2023-11-16T18:23:39.505209+0000","last_undegraded":"2023-11-16T18:23:39.505209+0000","last_fullsized":"2023-11-16T18:23:39.505209+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:05:25.862565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504017+0000","last_change":"2023-11-16T18:22:46.827732+0000","last_active":"2023-11-16T18:23:39.504017+0000","last_peered":"2023-11-16T18:23:39.504017+0000","last_clean":"2023-11-16T18:23:39.504017+0000","last_became_active":"2023-11-16T18:22:46.827328+0000","last_became_peered":"2023-11-16T18:22:46.827328+0000","last_unstale":"2023-11-16T18:23:39.504017+0000","last_undegraded":"2023-11-16T18:23:39.504017+0000","last_fullsized":"2023-11-16T18:23:39.504017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:44:05.572178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504758+0000","last_change":"2023-11-16T18:22:46.826837+0000","last_active":"2023-11-16T18:23:39.504758+0000","last_peered":"2023-11-16T18:23:39.504758+0000","last_clean":"2023-11-16T18:23:39.504758+0000","last_became_active":"2023-11-16T18:22:46.826532+0000","last_became_peered":"2023-11-16T18:22:46.826532+0000","last_unstale":"2023-11-16T18:23:39.504758+0000","last_undegraded":"2023-11-16T18:23:39.504758+0000","last_fullsized":"2023-11-16T18:23:39.504758+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:29:24.407020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504437+0000","last_change":"2023-11-16T18:22:46.841164+0000","last_active":"2023-11-16T18:23:39.504437+0000","last_peered":"2023-11-16T18:23:39.504437+0000","last_clean":"2023-11-16T18:23:39.504437+0000","last_became_active":"2023-11-16T18:22:46.840855+0000","last_became_peered":"2023-11-16T18:22:46.840855+0000","last_unstale":"2023-11-16T18:23:39.504437+0000","last_undegraded":"2023-11-16T18:23:39.504437+0000","last_fullsized":"2023-11-16T18:23:39.504437+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:46.380315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507548+0000","last_change":"2023-11-16T18:22:46.822171+0000","last_active":"2023-11-16T18:23:39.507548+0000","last_peered":"2023-11-16T18:23:39.507548+0000","last_clean":"2023-11-16T18:23:39.507548+0000","last_became_active":"2023-11-16T18:22:46.821860+0000","last_became_peered":"2023-11-16T18:22:46.821860+0000","last_unstale":"2023-11-16T18:23:39.507548+0000","last_undegraded":"2023-11-16T18:23:39.507548+0000","last_fullsized":"2023-11-16T18:23:39.507548+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:28:18.589285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502347+0000","last_change":"2023-11-16T18:22:46.832463+0000","last_active":"2023-11-16T18:23:39.502347+0000","last_peered":"2023-11-16T18:23:39.502347+0000","last_clean":"2023-11-16T18:23:39.502347+0000","last_became_active":"2023-11-16T18:22:46.832141+0000","last_became_peered":"2023-11-16T18:22:46.832141+0000","last_unstale":"2023-11-16T18:23:39.502347+0000","last_undegraded":"2023-11-16T18:23:39.502347+0000","last_fullsized":"2023-11-16T18:23:39.502347+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:12:03.361742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504425+0000","last_change":"2023-11-16T18:22:46.844239+0000","last_active":"2023-11-16T18:23:39.504425+0000","last_peered":"2023-11-16T18:23:39.504425+0000","last_clean":"2023-11-16T18:23:39.504425+0000","last_became_active":"2023-11-16T18:22:46.843906+0000","last_became_peered":"2023-11-16T18:22:46.843906+0000","last_unstale":"2023-11-16T18:23:39.504425+0000","last_undegraded":"2023-11-16T18:23:39.504425+0000","last_fullsized":"2023-11-16T18:23:39.504425+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:16:15.202242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508804+0000","last_change":"2023-11-16T18:22:46.834625+0000","last_active":"2023-11-16T18:23:39.508804+0000","last_peered":"2023-11-16T18:23:39.508804+0000","last_clean":"2023-11-16T18:23:39.508804+0000","last_became_active":"2023-11-16T18:22:46.834145+0000","last_became_peered":"2023-11-16T18:22:46.834145+0000","last_unstale":"2023-11-16T18:23:39.508804+0000","last_undegraded":"2023-11-16T18:23:39.508804+0000","last_fullsized":"2023-11-16T18:23:39.508804+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:25:16.196016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509061+0000","last_change":"2023-11-16T18:22:46.827569+0000","last_active":"2023-11-16T18:23:39.509061+0000","last_peered":"2023-11-16T18:23:39.509061+0000","last_clean":"2023-11-16T18:23:39.509061+0000","last_became_active":"2023-11-16T18:22:46.827234+0000","last_became_peered":"2023-11-16T18:22:46.827234+0000","last_unstale":"2023-11-16T18:23:39.509061+0000","last_undegraded":"2023-11-16T18:23:39.509061+0000","last_fullsized":"2023-11-16T18:23:39.509061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:38:30.584402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503061+0000","last_change":"2023-11-16T18:22:46.833701+0000","last_active":"2023-11-16T18:23:39.503061+0000","last_peered":"2023-11-16T18:23:39.503061+0000","last_clean":"2023-11-16T18:23:39.503061+0000","last_became_active":"2023-11-16T18:22:46.833333+0000","last_became_peered":"2023-11-16T18:22:46.833333+0000","last_unstale":"2023-11-16T18:23:39.503061+0000","last_undegraded":"2023-11-16T18:23:39.503061+0000","last_fullsized":"2023-11-16T18:23:39.503061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:31:52.429141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504924+0000","last_change":"2023-11-16T18:22:46.823222+0000","last_active":"2023-11-16T18:23:39.504924+0000","last_peered":"2023-11-16T18:23:39.504924+0000","last_clean":"2023-11-16T18:23:39.504924+0000","last_became_active":"2023-11-16T18:22:46.822224+0000","last_became_peered":"2023-11-16T18:22:46.822224+0000","last_unstale":"2023-11-16T18:23:39.504924+0000","last_undegraded":"2023-11-16T18:23:39.504924+0000","last_fullsized":"2023-11-16T18:23:39.504924+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:23:38.463744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584444+0000","last_change":"2023-11-16T18:22:46.818631+0000","last_active":"2023-11-16T18:23:39.584444+0000","last_peered":"2023-11-16T18:23:39.584444+0000","last_clean":"2023-11-16T18:23:39.584444+0000","last_became_active":"2023-11-16T18:22:46.818305+0000","last_became_peered":"2023-11-16T18:22:46.818305+0000","last_unstale":"2023-11-16T18:23:39.584444+0000","last_undegraded":"2023-11-16T18:23:39.584444+0000","last_fullsized":"2023-11-16T18:23:39.584444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:06:53.083409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503715+0000","last_change":"2023-11-16T18:22:46.825439+0000","last_active":"2023-11-16T18:23:39.503715+0000","last_peered":"2023-11-16T18:23:39.503715+0000","last_clean":"2023-11-16T18:23:39.503715+0000","last_became_active":"2023-11-16T18:22:46.825062+0000","last_became_peered":"2023-11-16T18:22:46.825062+0000","last_unstale":"2023-11-16T18:23:39.503715+0000","last_undegraded":"2023-11-16T18:23:39.503715+0000","last_fullsized":"2023-11-16T18:23:39.503715+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:32:03.819033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503546+0000","last_change":"2023-11-16T18:22:46.827730+0000","last_active":"2023-11-16T18:23:39.503546+0000","last_peered":"2023-11-16T18:23:39.503546+0000","last_clean":"2023-11-16T18:23:39.503546+0000","last_became_active":"2023-11-16T18:22:46.827416+0000","last_became_peered":"2023-11-16T18:22:46.827416+0000","last_unstale":"2023-11-16T18:23:39.503546+0000","last_undegraded":"2023-11-16T18:23:39.503546+0000","last_fullsized":"2023-11-16T18:23:39.503546+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:18:08.691944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507323+0000","last_change":"2023-11-16T18:22:46.834259+0000","last_active":"2023-11-16T18:23:39.507323+0000","last_peered":"2023-11-16T18:23:39.507323+0000","last_clean":"2023-11-16T18:23:39.507323+0000","last_became_active":"2023-11-16T18:22:46.833775+0000","last_became_peered":"2023-11-16T18:22:46.833775+0000","last_unstale":"2023-11-16T18:23:39.507323+0000","last_undegraded":"2023-11-16T18:23:39.507323+0000","last_fullsized":"2023-11-16T18:23:39.507323+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:02:17.762712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506738+0000","last_change":"2023-11-16T18:22:46.829475+0000","last_active":"2023-11-16T18:23:39.506738+0000","last_peered":"2023-11-16T18:23:39.506738+0000","last_clean":"2023-11-16T18:23:39.506738+0000","last_became_active":"2023-11-16T18:22:46.829157+0000","last_became_peered":"2023-11-16T18:22:46.829157+0000","last_unstale":"2023-11-16T18:23:39.506738+0000","last_undegraded":"2023-11-16T18:23:39.506738+0000","last_fullsized":"2023-11-16T18:23:39.506738+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:29.122041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501933+0000","last_change":"2023-11-16T18:22:46.821716+0000","last_active":"2023-11-16T18:23:39.501933+0000","last_peered":"2023-11-16T18:23:39.501933+0000","last_clean":"2023-11-16T18:23:39.501933+0000","last_became_active":"2023-11-16T18:22:46.821393+0000","last_became_peered":"2023-11-16T18:22:46.821393+0000","last_unstale":"2023-11-16T18:23:39.501933+0000","last_undegraded":"2023-11-16T18:23:39.501933+0000","last_fullsized":"2023-11-16T18:23:39.501933+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:10.453727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510952+0000","last_change":"2023-11-16T18:22:46.837433+0000","last_active":"2023-11-16T18:23:39.510952+0000","last_peered":"2023-11-16T18:23:39.510952+0000","last_clean":"2023-11-16T18:23:39.510952+0000","last_became_active":"2023-11-16T18:22:46.837143+0000","last_became_peered":"2023-11-16T18:22:46.837143+0000","last_unstale":"2023-11-16T18:23:39.510952+0000","last_undegraded":"2023-11-16T18:23:39.510952+0000","last_fullsized":"2023-11-16T18:23:39.510952+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:45:40.959535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510180+0000","last_change":"2023-11-16T18:22:51.123749+0000","last_active":"2023-11-16T18:23:39.510180+0000","last_peered":"2023-11-16T18:23:39.510180+0000","last_clean":"2023-11-16T18:23:39.510180+0000","last_became_active":"2023-11-16T18:22:51.123185+0000","last_became_peered":"2023-11-16T18:22:51.123185+0000","last_unstale":"2023-11-16T18:23:39.510180+0000","last_undegraded":"2023-11-16T18:23:39.510180+0000","last_fullsized":"2023-11-16T18:23:39.510180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:11:31.023936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501272+0000","last_change":"2023-11-16T18:22:46.385440+0000","last_active":"2023-11-16T18:23:39.501272+0000","last_peered":"2023-11-16T18:23:39.501272+0000","last_clean":"2023-11-16T18:23:39.501272+0000","last_became_active":"2023-11-16T18:22:46.385098+0000","last_became_peered":"2023-11-16T18:22:46.385098+0000","last_unstale":"2023-11-16T18:23:39.501272+0000","last_undegraded":"2023-11-16T18:23:39.501272+0000","last_fullsized":"2023-11-16T18:23:39.501272+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:08:49.547450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504280+0000","last_change":"2023-11-16T18:22:46.842681+0000","last_active":"2023-11-16T18:23:39.504280+0000","last_peered":"2023-11-16T18:23:39.504280+0000","last_clean":"2023-11-16T18:23:39.504280+0000","last_became_active":"2023-11-16T18:22:46.842408+0000","last_became_peered":"2023-11-16T18:22:46.842408+0000","last_unstale":"2023-11-16T18:23:39.504280+0000","last_undegraded":"2023-11-16T18:23:39.504280+0000","last_fullsized":"2023-11-16T18:23:39.504280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:37:57.050474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503905+0000","last_change":"2023-11-16T18:22:51.123450+0000","last_active":"2023-11-16T18:23:39.503905+0000","last_peered":"2023-11-16T18:23:39.503905+0000","last_clean":"2023-11-16T18:23:39.503905+0000","last_became_active":"2023-11-16T18:22:51.123002+0000","last_became_peered":"2023-11-16T18:22:51.123002+0000","last_unstale":"2023-11-16T18:23:39.503905+0000","last_undegraded":"2023-11-16T18:23:39.503905+0000","last_fullsized":"2023-11-16T18:23:39.503905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:48:43.003189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504416+0000","last_change":"2023-11-16T18:22:46.384768+0000","last_active":"2023-11-16T18:23:39.504416+0000","last_peered":"2023-11-16T18:23:39.504416+0000","last_clean":"2023-11-16T18:23:39.504416+0000","last_became_active":"2023-11-16T18:22:46.384246+0000","last_became_peered":"2023-11-16T18:22:46.384246+0000","last_unstale":"2023-11-16T18:23:39.504416+0000","last_undegraded":"2023-11-16T18:23:39.504416+0000","last_fullsized":"2023-11-16T18:23:39.504416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:20:51.259854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504317+0000","last_change":"2023-11-16T18:22:46.826069+0000","last_active":"2023-11-16T18:23:39.504317+0000","last_peered":"2023-11-16T18:23:39.504317+0000","last_clean":"2023-11-16T18:23:39.504317+0000","last_became_active":"2023-11-16T18:22:46.824676+0000","last_became_peered":"2023-11-16T18:22:46.824676+0000","last_unstale":"2023-11-16T18:23:39.504317+0000","last_undegraded":"2023-11-16T18:23:39.504317+0000","last_fullsized":"2023-11-16T18:23:39.504317+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:36:58.519456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504272+0000","last_change":"2023-11-16T18:22:51.127529+0000","last_active":"2023-11-16T18:23:39.504272+0000","last_peered":"2023-11-16T18:23:39.504272+0000","last_clean":"2023-11-16T18:23:39.504272+0000","last_became_active":"2023-11-16T18:22:51.126748+0000","last_became_peered":"2023-11-16T18:22:51.126748+0000","last_unstale":"2023-11-16T18:23:39.504272+0000","last_undegraded":"2023-11-16T18:23:39.504272+0000","last_fullsized":"2023-11-16T18:23:39.504272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:46:02.931821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505594+0000","last_change":"2023-11-16T18:22:46.385890+0000","last_active":"2023-11-16T18:23:39.505594+0000","last_peered":"2023-11-16T18:23:39.505594+0000","last_clean":"2023-11-16T18:23:39.505594+0000","last_became_active":"2023-11-16T18:22:46.385568+0000","last_became_peered":"2023-11-16T18:22:46.385568+0000","last_unstale":"2023-11-16T18:23:39.505594+0000","last_undegraded":"2023-11-16T18:23:39.505594+0000","last_fullsized":"2023-11-16T18:23:39.505594+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:45:23.088120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508492+0000","last_change":"2023-11-16T18:22:46.824796+0000","last_active":"2023-11-16T18:23:39.508492+0000","last_peered":"2023-11-16T18:23:39.508492+0000","last_clean":"2023-11-16T18:23:39.508492+0000","last_became_active":"2023-11-16T18:22:46.823895+0000","last_became_peered":"2023-11-16T18:22:46.823895+0000","last_unstale":"2023-11-16T18:23:39.508492+0000","last_undegraded":"2023-11-16T18:23:39.508492+0000","last_fullsized":"2023-11-16T18:23:39.508492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:57:28.297178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504555+0000","last_change":"2023-11-16T18:22:51.118714+0000","last_active":"2023-11-16T18:23:39.504555+0000","last_peered":"2023-11-16T18:23:39.504555+0000","last_clean":"2023-11-16T18:23:39.504555+0000","last_became_active":"2023-11-16T18:22:51.118216+0000","last_became_peered":"2023-11-16T18:22:51.118216+0000","last_unstale":"2023-11-16T18:23:39.504555+0000","last_undegraded":"2023-11-16T18:23:39.504555+0000","last_fullsized":"2023-11-16T18:23:39.504555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:27:57.339401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505119+0000","last_change":"2023-11-16T18:22:46.376431+0000","last_active":"2023-11-16T18:23:39.505119+0000","last_peered":"2023-11-16T18:23:39.505119+0000","last_clean":"2023-11-16T18:23:39.505119+0000","last_became_active":"2023-11-16T18:22:46.375914+0000","last_became_peered":"2023-11-16T18:22:46.375914+0000","last_unstale":"2023-11-16T18:23:39.505119+0000","last_undegraded":"2023-11-16T18:23:39.505119+0000","last_fullsized":"2023-11-16T18:23:39.505119+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:31:34.447838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508396+0000","last_change":"2023-11-16T18:22:46.825002+0000","last_active":"2023-11-16T18:23:39.508396+0000","last_peered":"2023-11-16T18:23:39.508396+0000","last_clean":"2023-11-16T18:23:39.508396+0000","last_became_active":"2023-11-16T18:22:46.824012+0000","last_became_peered":"2023-11-16T18:22:46.824012+0000","last_unstale":"2023-11-16T18:23:39.508396+0000","last_undegraded":"2023-11-16T18:23:39.508396+0000","last_fullsized":"2023-11-16T18:23:39.508396+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:04:41.816487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510910+0000","last_change":"2023-11-16T18:22:51.117623+0000","last_active":"2023-11-16T18:23:39.510910+0000","last_peered":"2023-11-16T18:23:39.510910+0000","last_clean":"2023-11-16T18:23:39.510910+0000","last_became_active":"2023-11-16T18:22:51.117183+0000","last_became_peered":"2023-11-16T18:22:51.117183+0000","last_unstale":"2023-11-16T18:23:39.510910+0000","last_undegraded":"2023-11-16T18:23:39.510910+0000","last_fullsized":"2023-11-16T18:23:39.510910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:16:00.063607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":86,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:45.523791+0000","last_change":"2023-11-16T18:23:39.664514+0000","last_active":"2023-11-16T18:23:45.523791+0000","last_peered":"2023-11-16T18:23:45.523791+0000","last_clean":"2023-11-16T18:23:45.523791+0000","last_became_active":"2023-11-16T18:23:39.511648+0000","last_became_peered":"2023-11-16T18:23:39.511648+0000","last_unstale":"2023-11-16T18:23:45.523791+0000","last_undegraded":"2023-11-16T18:23:45.523791+0000","last_fullsized":"2023-11-16T18:23:45.523791+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:08:22.914647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505046+0000","last_change":"2023-11-16T18:22:46.820073+0000","last_active":"2023-11-16T18:23:39.505046+0000","last_peered":"2023-11-16T18:23:39.505046+0000","last_clean":"2023-11-16T18:23:39.505046+0000","last_became_active":"2023-11-16T18:22:46.819764+0000","last_became_peered":"2023-11-16T18:22:46.819764+0000","last_unstale":"2023-11-16T18:23:39.505046+0000","last_undegraded":"2023-11-16T18:23:39.505046+0000","last_fullsized":"2023-11-16T18:23:39.505046+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:58.766286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510964+0000","last_change":"2023-11-16T18:22:51.116619+0000","last_active":"2023-11-16T18:23:39.510964+0000","last_peered":"2023-11-16T18:23:39.510964+0000","last_clean":"2023-11-16T18:23:39.510964+0000","last_became_active":"2023-11-16T18:22:51.116264+0000","last_became_peered":"2023-11-16T18:22:51.116264+0000","last_unstale":"2023-11-16T18:23:39.510964+0000","last_undegraded":"2023-11-16T18:23:39.510964+0000","last_fullsized":"2023-11-16T18:23:39.510964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:32:46.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507700+0000","last_change":"2023-11-16T18:22:46.389590+0000","last_active":"2023-11-16T18:23:39.507700+0000","last_peered":"2023-11-16T18:23:39.507700+0000","last_clean":"2023-11-16T18:23:39.507700+0000","last_became_active":"2023-11-16T18:22:46.389354+0000","last_became_peered":"2023-11-16T18:22:46.389354+0000","last_unstale":"2023-11-16T18:23:39.507700+0000","last_undegraded":"2023-11-16T18:23:39.507700+0000","last_fullsized":"2023-11-16T18:23:39.507700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:58:58.086957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509671+0000","last_change":"2023-11-16T18:22:46.841106+0000","last_active":"2023-11-16T18:23:39.509671+0000","last_peered":"2023-11-16T18:23:39.509671+0000","last_clean":"2023-11-16T18:23:39.509671+0000","last_became_active":"2023-11-16T18:22:46.840876+0000","last_became_peered":"2023-11-16T18:22:46.840876+0000","last_unstale":"2023-11-16T18:23:39.509671+0000","last_undegraded":"2023-11-16T18:23:39.509671+0000","last_fullsized":"2023-11-16T18:23:39.509671+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:46:23.740026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506599+0000","last_change":"2023-11-16T18:22:51.125813+0000","last_active":"2023-11-16T18:23:39.506599+0000","last_peered":"2023-11-16T18:23:39.506599+0000","last_clean":"2023-11-16T18:23:39.506599+0000","last_became_active":"2023-11-16T18:22:51.125277+0000","last_became_peered":"2023-11-16T18:22:51.125277+0000","last_unstale":"2023-11-16T18:23:39.506599+0000","last_undegraded":"2023-11-16T18:23:39.506599+0000","last_fullsized":"2023-11-16T18:23:39.506599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:02:51.083984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584572+0000","last_change":"2023-11-16T18:22:46.381208+0000","last_active":"2023-11-16T18:23:39.584572+0000","last_peered":"2023-11-16T18:23:39.584572+0000","last_clean":"2023-11-16T18:23:39.584572+0000","last_became_active":"2023-11-16T18:22:46.380351+0000","last_became_peered":"2023-11-16T18:22:46.380351+0000","last_unstale":"2023-11-16T18:23:39.584572+0000","last_undegraded":"2023-11-16T18:23:39.584572+0000","last_fullsized":"2023-11-16T18:23:39.584572+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:27:31.749114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504765+0000","last_change":"2023-11-16T18:22:46.843043+0000","last_active":"2023-11-16T18:23:39.504765+0000","last_peered":"2023-11-16T18:23:39.504765+0000","last_clean":"2023-11-16T18:23:39.504765+0000","last_became_active":"2023-11-16T18:22:46.841522+0000","last_became_peered":"2023-11-16T18:22:46.841522+0000","last_unstale":"2023-11-16T18:23:39.504765+0000","last_undegraded":"2023-11-16T18:23:39.504765+0000","last_fullsized":"2023-11-16T18:23:39.504765+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:53:29.309989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505013+0000","last_change":"2023-11-16T18:22:51.127729+0000","last_active":"2023-11-16T18:23:39.505013+0000","last_peered":"2023-11-16T18:23:39.505013+0000","last_clean":"2023-11-16T18:23:39.505013+0000","last_became_active":"2023-11-16T18:22:51.127303+0000","last_became_peered":"2023-11-16T18:22:51.127303+0000","last_unstale":"2023-11-16T18:23:39.505013+0000","last_undegraded":"2023-11-16T18:23:39.505013+0000","last_fullsized":"2023-11-16T18:23:39.505013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:16:37.982862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508172+0000","last_change":"2023-11-16T18:22:46.377437+0000","last_active":"2023-11-16T18:23:39.508172+0000","last_peered":"2023-11-16T18:23:39.508172+0000","last_clean":"2023-11-16T18:23:39.508172+0000","last_became_active":"2023-11-16T18:22:46.377120+0000","last_became_peered":"2023-11-16T18:22:46.377120+0000","last_unstale":"2023-11-16T18:23:39.508172+0000","last_undegraded":"2023-11-16T18:23:39.508172+0000","last_fullsized":"2023-11-16T18:23:39.508172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:02:31.673863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500756+0000","last_change":"2023-11-16T18:22:46.837804+0000","last_active":"2023-11-16T18:23:39.500756+0000","last_peered":"2023-11-16T18:23:39.500756+0000","last_clean":"2023-11-16T18:23:39.500756+0000","last_became_active":"2023-11-16T18:22:46.837432+0000","last_became_peered":"2023-11-16T18:22:46.837432+0000","last_unstale":"2023-11-16T18:23:39.500756+0000","last_undegraded":"2023-11-16T18:23:39.500756+0000","last_fullsized":"2023-11-16T18:23:39.500756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:10:14.516300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503848+0000","last_change":"2023-11-16T18:22:51.127546+0000","last_active":"2023-11-16T18:23:39.503848+0000","last_peered":"2023-11-16T18:23:39.503848+0000","last_clean":"2023-11-16T18:23:39.503848+0000","last_became_active":"2023-11-16T18:22:51.127133+0000","last_became_peered":"2023-11-16T18:22:51.127133+0000","last_unstale":"2023-11-16T18:23:39.503848+0000","last_undegraded":"2023-11-16T18:23:39.503848+0000","last_fullsized":"2023-11-16T18:23:39.503848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:02:33.244659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:43.520432+0000","last_change":"2023-11-16T18:22:46.386309+0000","last_active":"2023-11-16T18:23:43.520432+0000","last_peered":"2023-11-16T18:23:43.520432+0000","last_clean":"2023-11-16T18:23:43.520432+0000","last_became_active":"2023-11-16T18:22:46.385960+0000","last_became_peered":"2023-11-16T18:22:46.385960+0000","last_unstale":"2023-11-16T18:23:43.520432+0000","last_undegraded":"2023-11-16T18:23:43.520432+0000","last_fullsized":"2023-11-16T18:23:43.520432+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:46:45.743666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508584+0000","last_change":"2023-11-16T18:22:46.827211+0000","last_active":"2023-11-16T18:23:39.508584+0000","last_peered":"2023-11-16T18:23:39.508584+0000","last_clean":"2023-11-16T18:23:39.508584+0000","last_became_active":"2023-11-16T18:22:46.826396+0000","last_became_peered":"2023-11-16T18:22:46.826396+0000","last_unstale":"2023-11-16T18:23:39.508584+0000","last_undegraded":"2023-11-16T18:23:39.508584+0000","last_fullsized":"2023-11-16T18:23:39.508584+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:58:36.202010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.513065+0000","last_change":"2023-11-16T18:23:39.512554+0000","last_active":"2023-11-16T18:23:39.513065+0000","last_peered":"2023-11-16T18:23:39.513065+0000","last_clean":"2023-11-16T18:23:39.513065+0000","last_became_active":"2023-11-16T18:23:39.512118+0000","last_became_peered":"2023-11-16T18:23:39.512118+0000","last_unstale":"2023-11-16T18:23:39.513065+0000","last_undegraded":"2023-11-16T18:23:39.513065+0000","last_fullsized":"2023-11-16T18:23:39.513065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:55:33.269875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503214+0000","last_change":"2023-11-16T18:22:46.384680+0000","last_active":"2023-11-16T18:23:39.503214+0000","last_peered":"2023-11-16T18:23:39.503214+0000","last_clean":"2023-11-16T18:23:39.503214+0000","last_became_active":"2023-11-16T18:22:46.384290+0000","last_became_peered":"2023-11-16T18:22:46.384290+0000","last_unstale":"2023-11-16T18:23:39.503214+0000","last_undegraded":"2023-11-16T18:23:39.503214+0000","last_fullsized":"2023-11-16T18:23:39.503214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:05:40.782246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504447+0000","last_change":"2023-11-16T18:22:46.834726+0000","last_active":"2023-11-16T18:23:39.504447+0000","last_peered":"2023-11-16T18:23:39.504447+0000","last_clean":"2023-11-16T18:23:39.504447+0000","last_became_active":"2023-11-16T18:22:46.834424+0000","last_became_peered":"2023-11-16T18:22:46.834424+0000","last_unstale":"2023-11-16T18:23:39.504447+0000","last_undegraded":"2023-11-16T18:23:39.504447+0000","last_fullsized":"2023-11-16T18:23:39.504447+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:52:11.738947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.513064+0000","last_change":"2023-11-16T18:23:39.513064+0000","last_active":"2023-11-16T18:23:39.513064+0000","last_peered":"2023-11-16T18:23:39.513064+0000","last_clean":"2023-11-16T18:23:39.513064+0000","last_became_active":"2023-11-16T18:23:39.512606+0000","last_became_peered":"2023-11-16T18:23:39.512606+0000","last_unstale":"2023-11-16T18:23:39.513064+0000","last_undegraded":"2023-11-16T18:23:39.513064+0000","last_fullsized":"2023-11-16T18:23:39.513064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:08:18.327854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508258+0000","last_change":"2023-11-16T18:22:46.380804+0000","last_active":"2023-11-16T18:23:39.508258+0000","last_peered":"2023-11-16T18:23:39.508258+0000","last_clean":"2023-11-16T18:23:39.508258+0000","last_became_active":"2023-11-16T18:22:46.380477+0000","last_became_peered":"2023-11-16T18:22:46.380477+0000","last_unstale":"2023-11-16T18:23:39.508258+0000","last_undegraded":"2023-11-16T18:23:39.508258+0000","last_fullsized":"2023-11-16T18:23:39.508258+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:17:45.128995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.586650+0000","last_change":"2023-11-16T18:23:39.586650+0000","last_active":"2023-11-16T18:23:39.586650+0000","last_peered":"2023-11-16T18:23:39.586650+0000","last_clean":"2023-11-16T18:23:39.586650+0000","last_became_active":"2023-11-16T18:23:39.586241+0000","last_became_peered":"2023-11-16T18:23:39.586241+0000","last_unstale":"2023-11-16T18:23:39.586650+0000","last_undegraded":"2023-11-16T18:23:39.586650+0000","last_fullsized":"2023-11-16T18:23:39.586650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:15:14.907282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500754+0000","last_change":"2023-11-16T18:22:51.119711+0000","last_active":"2023-11-16T18:23:39.500754+0000","last_peered":"2023-11-16T18:23:39.500754+0000","last_clean":"2023-11-16T18:23:39.500754+0000","last_became_active":"2023-11-16T18:22:51.119304+0000","last_became_peered":"2023-11-16T18:22:51.119304+0000","last_unstale":"2023-11-16T18:23:39.500754+0000","last_undegraded":"2023-11-16T18:23:39.500754+0000","last_fullsized":"2023-11-16T18:23:39.500754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:37:34.450684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"86'31","reported_seq":157,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:44.523169+0000","last_change":"2023-11-16T18:23:39.671486+0000","last_active":"2023-11-16T18:23:44.523169+0000","last_peered":"2023-11-16T18:23:44.523169+0000","last_clean":"2023-11-16T18:23:44.523169+0000","last_became_active":"2023-11-16T18:23:39.513413+0000","last_became_peered":"2023-11-16T18:23:39.513413+0000","last_unstale":"2023-11-16T18:23:44.523169+0000","last_undegraded":"2023-11-16T18:23:44.523169+0000","last_fullsized":"2023-11-16T18:23:44.523169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:07:37.199058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60837,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":77,"num_read_kb":96,"num_write":32,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60837,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504736+0000","last_change":"2023-11-16T18:22:46.833537+0000","last_active":"2023-11-16T18:23:39.504736+0000","last_peered":"2023-11-16T18:23:39.504736+0000","last_clean":"2023-11-16T18:23:39.504736+0000","last_became_active":"2023-11-16T18:22:46.833217+0000","last_became_peered":"2023-11-16T18:22:46.833217+0000","last_unstale":"2023-11-16T18:23:39.504736+0000","last_undegraded":"2023-11-16T18:23:39.504736+0000","last_fullsized":"2023-11-16T18:23:39.504736+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:54:32.987198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585712+0000","last_change":"2023-11-16T18:23:39.585247+0000","last_active":"2023-11-16T18:23:39.585712+0000","last_peered":"2023-11-16T18:23:39.585712+0000","last_clean":"2023-11-16T18:23:39.585712+0000","last_became_active":"2023-11-16T18:23:39.584802+0000","last_became_peered":"2023-11-16T18:23:39.584802+0000","last_unstale":"2023-11-16T18:23:39.585712+0000","last_undegraded":"2023-11-16T18:23:39.585712+0000","last_fullsized":"2023-11-16T18:23:39.585712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:45:43.860286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509671+0000","last_change":"2023-11-16T18:22:46.382812+0000","last_active":"2023-11-16T18:23:39.509671+0000","last_peered":"2023-11-16T18:23:39.509671+0000","last_clean":"2023-11-16T18:23:39.509671+0000","last_became_active":"2023-11-16T18:22:46.382499+0000","last_became_peered":"2023-11-16T18:22:46.382499+0000","last_unstale":"2023-11-16T18:23:39.509671+0000","last_undegraded":"2023-11-16T18:23:39.509671+0000","last_fullsized":"2023-11-16T18:23:39.509671+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:48:39.263999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584764+0000","last_change":"2023-11-16T18:22:46.840917+0000","last_active":"2023-11-16T18:23:39.584764+0000","last_peered":"2023-11-16T18:23:39.584764+0000","last_clean":"2023-11-16T18:23:39.584764+0000","last_became_active":"2023-11-16T18:22:46.840579+0000","last_became_peered":"2023-11-16T18:22:46.840579+0000","last_unstale":"2023-11-16T18:23:39.584764+0000","last_undegraded":"2023-11-16T18:23:39.584764+0000","last_fullsized":"2023-11-16T18:23:39.584764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:05:17.920819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506477+0000","last_change":"2023-11-16T18:22:51.117852+0000","last_active":"2023-11-16T18:23:39.506477+0000","last_peered":"2023-11-16T18:23:39.506477+0000","last_clean":"2023-11-16T18:23:39.506477+0000","last_became_active":"2023-11-16T18:22:51.117390+0000","last_became_peered":"2023-11-16T18:22:51.117390+0000","last_unstale":"2023-11-16T18:23:39.506477+0000","last_undegraded":"2023-11-16T18:23:39.506477+0000","last_fullsized":"2023-11-16T18:23:39.506477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:11:37.167591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508136+0000","last_change":"2023-11-16T18:22:46.385137+0000","last_active":"2023-11-16T18:23:39.508136+0000","last_peered":"2023-11-16T18:23:39.508136+0000","last_clean":"2023-11-16T18:23:39.508136+0000","last_became_active":"2023-11-16T18:22:46.384636+0000","last_became_peered":"2023-11-16T18:22:46.384636+0000","last_unstale":"2023-11-16T18:23:39.508136+0000","last_undegraded":"2023-11-16T18:23:39.508136+0000","last_fullsized":"2023-11-16T18:23:39.508136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:11:10.885960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502619+0000","last_change":"2023-11-16T18:22:46.834786+0000","last_active":"2023-11-16T18:23:39.502619+0000","last_peered":"2023-11-16T18:23:39.502619+0000","last_clean":"2023-11-16T18:23:39.502619+0000","last_became_active":"2023-11-16T18:22:46.834338+0000","last_became_peered":"2023-11-16T18:22:46.834338+0000","last_unstale":"2023-11-16T18:23:39.502619+0000","last_undegraded":"2023-11-16T18:23:39.502619+0000","last_fullsized":"2023-11-16T18:23:39.502619+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:52:01.046978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500921+0000","last_change":"2023-11-16T18:22:51.124667+0000","last_active":"2023-11-16T18:23:39.500921+0000","last_peered":"2023-11-16T18:23:39.500921+0000","last_clean":"2023-11-16T18:23:39.500921+0000","last_became_active":"2023-11-16T18:22:51.124211+0000","last_became_peered":"2023-11-16T18:22:51.124211+0000","last_unstale":"2023-11-16T18:23:39.500921+0000","last_undegraded":"2023-11-16T18:23:39.500921+0000","last_fullsized":"2023-11-16T18:23:39.500921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:22:02.360209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:43.976174+0000","last_change":"2023-11-16T18:22:46.380829+0000","last_active":"2023-11-16T18:23:43.976174+0000","last_peered":"2023-11-16T18:23:43.976174+0000","last_clean":"2023-11-16T18:23:43.976174+0000","last_became_active":"2023-11-16T18:22:46.380466+0000","last_became_peered":"2023-11-16T18:22:46.380466+0000","last_unstale":"2023-11-16T18:23:43.976174+0000","last_undegraded":"2023-11-16T18:23:43.976174+0000","last_fullsized":"2023-11-16T18:23:43.976174+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:22:52.401291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502257+0000","last_change":"2023-11-16T18:22:46.835900+0000","last_active":"2023-11-16T18:23:39.502257+0000","last_peered":"2023-11-16T18:23:39.502257+0000","last_clean":"2023-11-16T18:23:39.502257+0000","last_became_active":"2023-11-16T18:22:46.835438+0000","last_became_peered":"2023-11-16T18:22:46.835438+0000","last_unstale":"2023-11-16T18:23:39.502257+0000","last_undegraded":"2023-11-16T18:23:39.502257+0000","last_fullsized":"2023-11-16T18:23:39.502257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:36:23.733440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585779+0000","last_change":"2023-11-16T18:22:51.120005+0000","last_active":"2023-11-16T18:23:39.585779+0000","last_peered":"2023-11-16T18:23:39.585779+0000","last_clean":"2023-11-16T18:23:39.585779+0000","last_became_active":"2023-11-16T18:22:51.119540+0000","last_became_peered":"2023-11-16T18:22:51.119540+0000","last_unstale":"2023-11-16T18:23:39.585779+0000","last_undegraded":"2023-11-16T18:23:39.585779+0000","last_fullsized":"2023-11-16T18:23:39.585779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:33:29.490015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:41.897854+0000","last_change":"2023-11-16T18:23:41.897338+0000","last_active":"2023-11-16T18:23:41.897854+0000","last_peered":"2023-11-16T18:23:41.897854+0000","last_clean":"2023-11-16T18:23:41.897854+0000","last_became_active":"2023-11-16T18:23:39.512018+0000","last_became_peered":"2023-11-16T18:23:39.512018+0000","last_unstale":"2023-11-16T18:23:41.897854+0000","last_undegraded":"2023-11-16T18:23:41.897854+0000","last_fullsized":"2023-11-16T18:23:41.897854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:53:22.707297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507902+0000","last_change":"2023-11-16T18:22:46.823838+0000","last_active":"2023-11-16T18:23:39.507902+0000","last_peered":"2023-11-16T18:23:39.507902+0000","last_clean":"2023-11-16T18:23:39.507902+0000","last_became_active":"2023-11-16T18:22:46.823545+0000","last_became_peered":"2023-11-16T18:22:46.823545+0000","last_unstale":"2023-11-16T18:23:39.507902+0000","last_undegraded":"2023-11-16T18:23:39.507902+0000","last_fullsized":"2023-11-16T18:23:39.507902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:11:32.214804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507865+0000","last_change":"2023-11-16T18:22:51.127371+0000","last_active":"2023-11-16T18:23:39.507865+0000","last_peered":"2023-11-16T18:23:39.507865+0000","last_clean":"2023-11-16T18:23:39.507865+0000","last_became_active":"2023-11-16T18:22:51.126951+0000","last_became_peered":"2023-11-16T18:22:51.126951+0000","last_unstale":"2023-11-16T18:23:39.507865+0000","last_undegraded":"2023-11-16T18:23:39.507865+0000","last_fullsized":"2023-11-16T18:23:39.507865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:55.418916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504323+0000","last_change":"2023-11-16T18:22:46.376561+0000","last_active":"2023-11-16T18:23:39.504323+0000","last_peered":"2023-11-16T18:23:39.504323+0000","last_clean":"2023-11-16T18:23:39.504323+0000","last_became_active":"2023-11-16T18:22:46.375426+0000","last_became_peered":"2023-11-16T18:22:46.375426+0000","last_unstale":"2023-11-16T18:23:39.504323+0000","last_undegraded":"2023-11-16T18:23:39.504323+0000","last_fullsized":"2023-11-16T18:23:39.504323+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:29:00.383105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505100+0000","last_change":"2023-11-16T18:22:46.826657+0000","last_active":"2023-11-16T18:23:39.505100+0000","last_peered":"2023-11-16T18:23:39.505100+0000","last_clean":"2023-11-16T18:23:39.505100+0000","last_became_active":"2023-11-16T18:22:46.826126+0000","last_became_peered":"2023-11-16T18:22:46.826126+0000","last_unstale":"2023-11-16T18:23:39.505100+0000","last_undegraded":"2023-11-16T18:23:39.505100+0000","last_fullsized":"2023-11-16T18:23:39.505100+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:13:34.794628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504904+0000","last_change":"2023-11-16T18:22:51.127949+0000","last_active":"2023-11-16T18:23:39.504904+0000","last_peered":"2023-11-16T18:23:39.504904+0000","last_clean":"2023-11-16T18:23:39.504904+0000","last_became_active":"2023-11-16T18:22:51.127388+0000","last_became_peered":"2023-11-16T18:22:51.127388+0000","last_unstale":"2023-11-16T18:23:39.504904+0000","last_undegraded":"2023-11-16T18:23:39.504904+0000","last_fullsized":"2023-11-16T18:23:39.504904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:37:30.745978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500985+0000","last_change":"2023-11-16T18:22:46.378568+0000","last_active":"2023-11-16T18:23:39.500985+0000","last_peered":"2023-11-16T18:23:39.500985+0000","last_clean":"2023-11-16T18:23:39.500985+0000","last_became_active":"2023-11-16T18:22:46.378018+0000","last_became_peered":"2023-11-16T18:22:46.378018+0000","last_unstale":"2023-11-16T18:23:39.500985+0000","last_undegraded":"2023-11-16T18:23:39.500985+0000","last_fullsized":"2023-11-16T18:23:39.500985+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:05:59.328232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509189+0000","last_change":"2023-11-16T18:22:46.841129+0000","last_active":"2023-11-16T18:23:39.509189+0000","last_peered":"2023-11-16T18:23:39.509189+0000","last_clean":"2023-11-16T18:23:39.509189+0000","last_became_active":"2023-11-16T18:22:46.840780+0000","last_became_peered":"2023-11-16T18:22:46.840780+0000","last_unstale":"2023-11-16T18:23:39.509189+0000","last_undegraded":"2023-11-16T18:23:39.509189+0000","last_fullsized":"2023-11-16T18:23:39.509189+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:09:39.912158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585594+0000","last_change":"2023-11-16T18:22:51.126221+0000","last_active":"2023-11-16T18:23:39.585594+0000","last_peered":"2023-11-16T18:23:39.585594+0000","last_clean":"2023-11-16T18:23:39.585594+0000","last_became_active":"2023-11-16T18:22:51.125770+0000","last_became_peered":"2023-11-16T18:22:51.125770+0000","last_unstale":"2023-11-16T18:23:39.585594+0000","last_undegraded":"2023-11-16T18:23:39.585594+0000","last_fullsized":"2023-11-16T18:23:39.585594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:35:06.867124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502402+0000","last_change":"2023-11-16T18:22:46.384872+0000","last_active":"2023-11-16T18:23:39.502402+0000","last_peered":"2023-11-16T18:23:39.502402+0000","last_clean":"2023-11-16T18:23:39.502402+0000","last_became_active":"2023-11-16T18:22:46.384499+0000","last_became_peered":"2023-11-16T18:22:46.384499+0000","last_unstale":"2023-11-16T18:23:39.502402+0000","last_undegraded":"2023-11-16T18:23:39.502402+0000","last_fullsized":"2023-11-16T18:23:39.502402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:26:53.535471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507989+0000","last_change":"2023-11-16T18:22:46.822574+0000","last_active":"2023-11-16T18:23:39.507989+0000","last_peered":"2023-11-16T18:23:39.507989+0000","last_clean":"2023-11-16T18:23:39.507989+0000","last_became_active":"2023-11-16T18:22:46.822230+0000","last_became_peered":"2023-11-16T18:22:46.822230+0000","last_unstale":"2023-11-16T18:23:39.507989+0000","last_undegraded":"2023-11-16T18:23:39.507989+0000","last_fullsized":"2023-11-16T18:23:39.507989+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:16:44.677701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585706+0000","last_change":"2023-11-16T18:22:51.117838+0000","last_active":"2023-11-16T18:23:39.585706+0000","last_peered":"2023-11-16T18:23:39.585706+0000","last_clean":"2023-11-16T18:23:39.585706+0000","last_became_active":"2023-11-16T18:22:51.117275+0000","last_became_peered":"2023-11-16T18:22:51.117275+0000","last_unstale":"2023-11-16T18:23:39.585706+0000","last_undegraded":"2023-11-16T18:23:39.585706+0000","last_fullsized":"2023-11-16T18:23:39.585706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:42:14.482945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508453+0000","last_change":"2023-11-16T18:22:46.827499+0000","last_active":"2023-11-16T18:23:39.508453+0000","last_peered":"2023-11-16T18:23:39.508453+0000","last_clean":"2023-11-16T18:23:39.508453+0000","last_became_active":"2023-11-16T18:22:46.827166+0000","last_became_peered":"2023-11-16T18:22:46.827166+0000","last_unstale":"2023-11-16T18:23:39.508453+0000","last_undegraded":"2023-11-16T18:23:39.508453+0000","last_fullsized":"2023-11-16T18:23:39.508453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:23:43.219136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:41.972510+0000","last_change":"2023-11-16T18:22:46.381397+0000","last_active":"2023-11-16T18:23:41.972510+0000","last_peered":"2023-11-16T18:23:41.972510+0000","last_clean":"2023-11-16T18:23:41.972510+0000","last_became_active":"2023-11-16T18:22:46.380939+0000","last_became_peered":"2023-11-16T18:22:46.380939+0000","last_unstale":"2023-11-16T18:23:41.972510+0000","last_undegraded":"2023-11-16T18:23:41.972510+0000","last_fullsized":"2023-11-16T18:23:41.972510+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:44:30.574771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502915+0000","last_change":"2023-11-16T18:22:51.125044+0000","last_active":"2023-11-16T18:23:39.502915+0000","last_peered":"2023-11-16T18:23:39.502915+0000","last_clean":"2023-11-16T18:23:39.502915+0000","last_became_active":"2023-11-16T18:22:51.124422+0000","last_became_peered":"2023-11-16T18:22:51.124422+0000","last_unstale":"2023-11-16T18:23:39.502915+0000","last_undegraded":"2023-11-16T18:23:39.502915+0000","last_fullsized":"2023-11-16T18:23:39.502915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:11.651599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503647+0000","last_change":"2023-11-16T18:22:46.841500+0000","last_active":"2023-11-16T18:23:39.503647+0000","last_peered":"2023-11-16T18:23:39.503647+0000","last_clean":"2023-11-16T18:23:39.503647+0000","last_became_active":"2023-11-16T18:22:46.841186+0000","last_became_peered":"2023-11-16T18:22:46.841186+0000","last_unstale":"2023-11-16T18:23:39.503647+0000","last_undegraded":"2023-11-16T18:23:39.503647+0000","last_fullsized":"2023-11-16T18:23:39.503647+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:38:25.407837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504887+0000","last_change":"2023-11-16T18:22:46.380988+0000","last_active":"2023-11-16T18:23:39.504887+0000","last_peered":"2023-11-16T18:23:39.504887+0000","last_clean":"2023-11-16T18:23:39.504887+0000","last_became_active":"2023-11-16T18:22:46.380616+0000","last_became_peered":"2023-11-16T18:22:46.380616+0000","last_unstale":"2023-11-16T18:23:39.504887+0000","last_undegraded":"2023-11-16T18:23:39.504887+0000","last_fullsized":"2023-11-16T18:23:39.504887+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:24:26.911536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502037+0000","last_change":"2023-11-16T18:21:44.282706+0000","last_active":"2023-11-16T18:23:39.502037+0000","last_peered":"2023-11-16T18:23:39.502037+0000","last_clean":"2023-11-16T18:23:39.502037+0000","last_became_active":"2023-11-16T18:21:44.282201+0000","last_became_peered":"2023-11-16T18:21:44.282201+0000","last_unstale":"2023-11-16T18:23:39.502037+0000","last_undegraded":"2023-11-16T18:23:39.502037+0000","last_fullsized":"2023-11-16T18:23:39.502037+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_clean_scrub_stamp":"2023-11-16T18:18:46.711191+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:41:17.702365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508868+0000","last_change":"2023-11-16T18:22:51.123051+0000","last_active":"2023-11-16T18:23:39.508868+0000","last_peered":"2023-11-16T18:23:39.508868+0000","last_clean":"2023-11-16T18:23:39.508868+0000","last_became_active":"2023-11-16T18:22:51.122635+0000","last_became_peered":"2023-11-16T18:22:51.122635+0000","last_unstale":"2023-11-16T18:23:39.508868+0000","last_undegraded":"2023-11-16T18:23:39.508868+0000","last_fullsized":"2023-11-16T18:23:39.508868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:59:09.977278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584296+0000","last_change":"2023-11-16T18:22:46.822625+0000","last_active":"2023-11-16T18:23:39.584296+0000","last_peered":"2023-11-16T18:23:39.584296+0000","last_clean":"2023-11-16T18:23:39.584296+0000","last_became_active":"2023-11-16T18:22:46.822296+0000","last_became_peered":"2023-11-16T18:22:46.822296+0000","last_unstale":"2023-11-16T18:23:39.584296+0000","last_undegraded":"2023-11-16T18:23:39.584296+0000","last_fullsized":"2023-11-16T18:23:39.584296+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:43:40.832756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502155+0000","last_change":"2023-11-16T18:22:46.380445+0000","last_active":"2023-11-16T18:23:39.502155+0000","last_peered":"2023-11-16T18:23:39.502155+0000","last_clean":"2023-11-16T18:23:39.502155+0000","last_became_active":"2023-11-16T18:22:46.380113+0000","last_became_peered":"2023-11-16T18:22:46.380113+0000","last_unstale":"2023-11-16T18:23:39.502155+0000","last_undegraded":"2023-11-16T18:23:39.502155+0000","last_fullsized":"2023-11-16T18:23:39.502155+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:17:48.769928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504943+0000","last_change":"2023-11-16T18:22:51.118457+0000","last_active":"2023-11-16T18:23:39.504943+0000","last_peered":"2023-11-16T18:23:39.504943+0000","last_clean":"2023-11-16T18:23:39.504943+0000","last_became_active":"2023-11-16T18:22:51.118013+0000","last_became_peered":"2023-11-16T18:22:51.118013+0000","last_unstale":"2023-11-16T18:23:39.504943+0000","last_undegraded":"2023-11-16T18:23:39.504943+0000","last_fullsized":"2023-11-16T18:23:39.504943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:11.432801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506267+0000","last_change":"2023-11-16T18:22:46.384799+0000","last_active":"2023-11-16T18:23:39.506267+0000","last_peered":"2023-11-16T18:23:39.506267+0000","last_clean":"2023-11-16T18:23:39.506267+0000","last_became_active":"2023-11-16T18:22:46.384482+0000","last_became_peered":"2023-11-16T18:22:46.384482+0000","last_unstale":"2023-11-16T18:23:39.506267+0000","last_undegraded":"2023-11-16T18:23:39.506267+0000","last_fullsized":"2023-11-16T18:23:39.506267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:42:48.220989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585377+0000","last_change":"2023-11-16T18:22:46.829541+0000","last_active":"2023-11-16T18:23:39.585377+0000","last_peered":"2023-11-16T18:23:39.585377+0000","last_clean":"2023-11-16T18:23:39.585377+0000","last_became_active":"2023-11-16T18:22:46.829006+0000","last_became_peered":"2023-11-16T18:22:46.829006+0000","last_unstale":"2023-11-16T18:23:39.585377+0000","last_undegraded":"2023-11-16T18:23:39.585377+0000","last_fullsized":"2023-11-16T18:23:39.585377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:13:02.956336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504640+0000","last_change":"2023-11-16T18:22:51.122751+0000","last_active":"2023-11-16T18:23:39.504640+0000","last_peered":"2023-11-16T18:23:39.504640+0000","last_clean":"2023-11-16T18:23:39.504640+0000","last_became_active":"2023-11-16T18:22:51.122423+0000","last_became_peered":"2023-11-16T18:22:51.122423+0000","last_unstale":"2023-11-16T18:23:39.504640+0000","last_undegraded":"2023-11-16T18:23:39.504640+0000","last_fullsized":"2023-11-16T18:23:39.504640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:27:49.338555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'5","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.516672+0000","last_change":"2023-11-16T18:23:39.516672+0000","last_active":"2023-11-16T18:23:39.516672+0000","last_peered":"2023-11-16T18:23:39.516672+0000","last_clean":"2023-11-16T18:23:39.516672+0000","last_became_active":"2023-11-16T18:23:39.509087+0000","last_became_peered":"2023-11-16T18:23:39.509087+0000","last_unstale":"2023-11-16T18:23:39.516672+0000","last_undegraded":"2023-11-16T18:23:39.516672+0000","last_fullsized":"2023-11-16T18:23:39.516672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:57:28.479767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506041+0000","last_change":"2023-11-16T18:22:46.843802+0000","last_active":"2023-11-16T18:23:39.506041+0000","last_peered":"2023-11-16T18:23:39.506041+0000","last_clean":"2023-11-16T18:23:39.506041+0000","last_became_active":"2023-11-16T18:22:46.842210+0000","last_became_peered":"2023-11-16T18:22:46.842210+0000","last_unstale":"2023-11-16T18:23:39.506041+0000","last_undegraded":"2023-11-16T18:23:39.506041+0000","last_fullsized":"2023-11-16T18:23:39.506041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:20:02.893140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509951+0000","last_change":"2023-11-16T18:22:51.124632+0000","last_active":"2023-11-16T18:23:39.509951+0000","last_peered":"2023-11-16T18:23:39.509951+0000","last_clean":"2023-11-16T18:23:39.509951+0000","last_became_active":"2023-11-16T18:22:51.124220+0000","last_became_peered":"2023-11-16T18:22:51.124220+0000","last_unstale":"2023-11-16T18:23:39.509951+0000","last_undegraded":"2023-11-16T18:23:39.509951+0000","last_fullsized":"2023-11-16T18:23:39.509951+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:51:57.981627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505140+0000","last_change":"2023-11-16T18:22:46.381456+0000","last_active":"2023-11-16T18:23:39.505140+0000","last_peered":"2023-11-16T18:23:39.505140+0000","last_clean":"2023-11-16T18:23:39.505140+0000","last_became_active":"2023-11-16T18:22:46.381137+0000","last_became_peered":"2023-11-16T18:22:46.381137+0000","last_unstale":"2023-11-16T18:23:39.505140+0000","last_undegraded":"2023-11-16T18:23:39.505140+0000","last_fullsized":"2023-11-16T18:23:39.505140+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:34:36.759190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505363+0000","last_change":"2023-11-16T18:22:46.838249+0000","last_active":"2023-11-16T18:23:39.505363+0000","last_peered":"2023-11-16T18:23:39.505363+0000","last_clean":"2023-11-16T18:23:39.505363+0000","last_became_active":"2023-11-16T18:22:46.837925+0000","last_became_peered":"2023-11-16T18:22:46.837925+0000","last_unstale":"2023-11-16T18:23:39.505363+0000","last_undegraded":"2023-11-16T18:23:39.505363+0000","last_fullsized":"2023-11-16T18:23:39.505363+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:02:17.287183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504877+0000","last_change":"2023-11-16T18:22:51.123687+0000","last_active":"2023-11-16T18:23:39.504877+0000","last_peered":"2023-11-16T18:23:39.504877+0000","last_clean":"2023-11-16T18:23:39.504877+0000","last_became_active":"2023-11-16T18:22:51.123209+0000","last_became_peered":"2023-11-16T18:22:51.123209+0000","last_unstale":"2023-11-16T18:23:39.504877+0000","last_undegraded":"2023-11-16T18:23:39.504877+0000","last_fullsized":"2023-11-16T18:23:39.504877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:09:15.998485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508037+0000","last_change":"2023-11-16T18:22:46.378038+0000","last_active":"2023-11-16T18:23:39.508037+0000","last_peered":"2023-11-16T18:23:39.508037+0000","last_clean":"2023-11-16T18:23:39.508037+0000","last_became_active":"2023-11-16T18:22:46.377680+0000","last_became_peered":"2023-11-16T18:22:46.377680+0000","last_unstale":"2023-11-16T18:23:39.508037+0000","last_undegraded":"2023-11-16T18:23:39.508037+0000","last_fullsized":"2023-11-16T18:23:39.508037+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:31.194156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504247+0000","last_change":"2023-11-16T18:22:46.821888+0000","last_active":"2023-11-16T18:23:39.504247+0000","last_peered":"2023-11-16T18:23:39.504247+0000","last_clean":"2023-11-16T18:23:39.504247+0000","last_became_active":"2023-11-16T18:22:46.821446+0000","last_became_peered":"2023-11-16T18:22:46.821446+0000","last_unstale":"2023-11-16T18:23:39.504247+0000","last_undegraded":"2023-11-16T18:23:39.504247+0000","last_fullsized":"2023-11-16T18:23:39.504247+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:11:30.632655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501162+0000","last_change":"2023-11-16T18:22:51.119717+0000","last_active":"2023-11-16T18:23:39.501162+0000","last_peered":"2023-11-16T18:23:39.501162+0000","last_clean":"2023-11-16T18:23:39.501162+0000","last_became_active":"2023-11-16T18:22:51.119256+0000","last_became_peered":"2023-11-16T18:22:51.119256+0000","last_unstale":"2023-11-16T18:23:39.501162+0000","last_undegraded":"2023-11-16T18:23:39.501162+0000","last_fullsized":"2023-11-16T18:23:39.501162+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:25:42.785279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504001+0000","last_change":"2023-11-16T18:22:46.375387+0000","last_active":"2023-11-16T18:23:39.504001+0000","last_peered":"2023-11-16T18:23:39.504001+0000","last_clean":"2023-11-16T18:23:39.504001+0000","last_became_active":"2023-11-16T18:22:46.374886+0000","last_became_peered":"2023-11-16T18:22:46.374886+0000","last_unstale":"2023-11-16T18:23:39.504001+0000","last_undegraded":"2023-11-16T18:23:39.504001+0000","last_fullsized":"2023-11-16T18:23:39.504001+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:18:24.699496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504057+0000","last_change":"2023-11-16T18:22:46.820315+0000","last_active":"2023-11-16T18:23:39.504057+0000","last_peered":"2023-11-16T18:23:39.504057+0000","last_clean":"2023-11-16T18:23:39.504057+0000","last_became_active":"2023-11-16T18:22:46.819592+0000","last_became_peered":"2023-11-16T18:22:46.819592+0000","last_unstale":"2023-11-16T18:23:39.504057+0000","last_undegraded":"2023-11-16T18:23:39.504057+0000","last_fullsized":"2023-11-16T18:23:39.504057+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:43:26.336090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.518963+0000","last_change":"2023-11-16T18:23:39.518963+0000","last_active":"2023-11-16T18:23:39.518963+0000","last_peered":"2023-11-16T18:23:39.518963+0000","last_clean":"2023-11-16T18:23:39.518963+0000","last_became_active":"2023-11-16T18:23:39.513817+0000","last_became_peered":"2023-11-16T18:23:39.513817+0000","last_unstale":"2023-11-16T18:23:39.518963+0000","last_undegraded":"2023-11-16T18:23:39.518963+0000","last_fullsized":"2023-11-16T18:23:39.518963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:16:46.629599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510801+0000","last_change":"2023-11-16T18:22:46.378638+0000","last_active":"2023-11-16T18:23:39.510801+0000","last_peered":"2023-11-16T18:23:39.510801+0000","last_clean":"2023-11-16T18:23:39.510801+0000","last_became_active":"2023-11-16T18:22:46.378341+0000","last_became_peered":"2023-11-16T18:22:46.378341+0000","last_unstale":"2023-11-16T18:23:39.510801+0000","last_undegraded":"2023-11-16T18:23:39.510801+0000","last_fullsized":"2023-11-16T18:23:39.510801+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:36:28.625148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504347+0000","last_change":"2023-11-16T18:22:46.843431+0000","last_active":"2023-11-16T18:23:39.504347+0000","last_peered":"2023-11-16T18:23:39.504347+0000","last_clean":"2023-11-16T18:23:39.504347+0000","last_became_active":"2023-11-16T18:22:46.842992+0000","last_became_peered":"2023-11-16T18:22:46.842992+0000","last_unstale":"2023-11-16T18:23:39.504347+0000","last_undegraded":"2023-11-16T18:23:39.504347+0000","last_fullsized":"2023-11-16T18:23:39.504347+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:23:28.821023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503912+0000","last_change":"2023-11-16T18:22:51.122640+0000","last_active":"2023-11-16T18:23:39.503912+0000","last_peered":"2023-11-16T18:23:39.503912+0000","last_clean":"2023-11-16T18:23:39.503912+0000","last_became_active":"2023-11-16T18:22:51.122006+0000","last_became_peered":"2023-11-16T18:22:51.122006+0000","last_unstale":"2023-11-16T18:23:39.503912+0000","last_undegraded":"2023-11-16T18:23:39.503912+0000","last_fullsized":"2023-11-16T18:23:39.503912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:27:30.406233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504268+0000","last_change":"2023-11-16T18:22:46.384837+0000","last_active":"2023-11-16T18:23:39.504268+0000","last_peered":"2023-11-16T18:23:39.504268+0000","last_clean":"2023-11-16T18:23:39.504268+0000","last_became_active":"2023-11-16T18:22:46.384541+0000","last_became_peered":"2023-11-16T18:22:46.384541+0000","last_unstale":"2023-11-16T18:23:39.504268+0000","last_undegraded":"2023-11-16T18:23:39.504268+0000","last_fullsized":"2023-11-16T18:23:39.504268+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:06:22.863895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502460+0000","last_change":"2023-11-16T18:22:46.839967+0000","last_active":"2023-11-16T18:23:39.502460+0000","last_peered":"2023-11-16T18:23:39.502460+0000","last_clean":"2023-11-16T18:23:39.502460+0000","last_became_active":"2023-11-16T18:22:46.839645+0000","last_became_peered":"2023-11-16T18:22:46.839645+0000","last_unstale":"2023-11-16T18:23:39.502460+0000","last_undegraded":"2023-11-16T18:23:39.502460+0000","last_fullsized":"2023-11-16T18:23:39.502460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:04:34.616063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507240+0000","last_change":"2023-11-16T18:22:51.126548+0000","last_active":"2023-11-16T18:23:39.507240+0000","last_peered":"2023-11-16T18:23:39.507240+0000","last_clean":"2023-11-16T18:23:39.507240+0000","last_became_active":"2023-11-16T18:22:51.126039+0000","last_became_peered":"2023-11-16T18:22:51.126039+0000","last_unstale":"2023-11-16T18:23:39.507240+0000","last_undegraded":"2023-11-16T18:23:39.507240+0000","last_fullsized":"2023-11-16T18:23:39.507240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:22:07.650645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.511343+0000","last_change":"2023-11-16T18:22:46.383947+0000","last_active":"2023-11-16T18:23:39.511343+0000","last_peered":"2023-11-16T18:23:39.511343+0000","last_clean":"2023-11-16T18:23:39.511343+0000","last_became_active":"2023-11-16T18:22:46.383661+0000","last_became_peered":"2023-11-16T18:22:46.383661+0000","last_unstale":"2023-11-16T18:23:39.511343+0000","last_undegraded":"2023-11-16T18:23:39.511343+0000","last_fullsized":"2023-11-16T18:23:39.511343+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:34.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503182+0000","last_change":"2023-11-16T18:22:46.836877+0000","last_active":"2023-11-16T18:23:39.503182+0000","last_peered":"2023-11-16T18:23:39.503182+0000","last_clean":"2023-11-16T18:23:39.503182+0000","last_became_active":"2023-11-16T18:22:46.836477+0000","last_became_peered":"2023-11-16T18:22:46.836477+0000","last_unstale":"2023-11-16T18:23:39.503182+0000","last_undegraded":"2023-11-16T18:23:39.503182+0000","last_fullsized":"2023-11-16T18:23:39.503182+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:38:49.681616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505171+0000","last_change":"2023-11-16T18:22:51.126930+0000","last_active":"2023-11-16T18:23:39.505171+0000","last_peered":"2023-11-16T18:23:39.505171+0000","last_clean":"2023-11-16T18:23:39.505171+0000","last_became_active":"2023-11-16T18:22:51.126409+0000","last_became_peered":"2023-11-16T18:22:51.126409+0000","last_unstale":"2023-11-16T18:23:39.505171+0000","last_undegraded":"2023-11-16T18:23:39.505171+0000","last_fullsized":"2023-11-16T18:23:39.505171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:31:50.012322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508533+0000","last_change":"2023-11-16T18:22:46.380820+0000","last_active":"2023-11-16T18:23:39.508533+0000","last_peered":"2023-11-16T18:23:39.508533+0000","last_clean":"2023-11-16T18:23:39.508533+0000","last_became_active":"2023-11-16T18:22:46.380446+0000","last_became_peered":"2023-11-16T18:22:46.380446+0000","last_unstale":"2023-11-16T18:23:39.508533+0000","last_undegraded":"2023-11-16T18:23:39.508533+0000","last_fullsized":"2023-11-16T18:23:39.508533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:20:07.923022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502955+0000","last_change":"2023-11-16T18:22:46.838153+0000","last_active":"2023-11-16T18:23:39.502955+0000","last_peered":"2023-11-16T18:23:39.502955+0000","last_clean":"2023-11-16T18:23:39.502955+0000","last_became_active":"2023-11-16T18:22:46.837836+0000","last_became_peered":"2023-11-16T18:22:46.837836+0000","last_unstale":"2023-11-16T18:23:39.502955+0000","last_undegraded":"2023-11-16T18:23:39.502955+0000","last_fullsized":"2023-11-16T18:23:39.502955+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:24:42.477381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507608+0000","last_change":"2023-11-16T18:22:51.119993+0000","last_active":"2023-11-16T18:23:39.507608+0000","last_peered":"2023-11-16T18:23:39.507608+0000","last_clean":"2023-11-16T18:23:39.507608+0000","last_became_active":"2023-11-16T18:22:51.119401+0000","last_became_peered":"2023-11-16T18:22:51.119401+0000","last_unstale":"2023-11-16T18:23:39.507608+0000","last_undegraded":"2023-11-16T18:23:39.507608+0000","last_fullsized":"2023-11-16T18:23:39.507608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:19:31.451143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585944+0000","last_change":"2023-11-16T18:22:46.383399+0000","last_active":"2023-11-16T18:23:39.585944+0000","last_peered":"2023-11-16T18:23:39.585944+0000","last_clean":"2023-11-16T18:23:39.585944+0000","last_became_active":"2023-11-16T18:22:46.382938+0000","last_became_peered":"2023-11-16T18:22:46.382938+0000","last_unstale":"2023-11-16T18:23:39.585944+0000","last_undegraded":"2023-11-16T18:23:39.585944+0000","last_fullsized":"2023-11-16T18:23:39.585944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:29.725086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503688+0000","last_change":"2023-11-16T18:22:46.836035+0000","last_active":"2023-11-16T18:23:39.503688+0000","last_peered":"2023-11-16T18:23:39.503688+0000","last_clean":"2023-11-16T18:23:39.503688+0000","last_became_active":"2023-11-16T18:22:46.835709+0000","last_became_peered":"2023-11-16T18:22:46.835709+0000","last_unstale":"2023-11-16T18:23:39.503688+0000","last_undegraded":"2023-11-16T18:23:39.503688+0000","last_fullsized":"2023-11-16T18:23:39.503688+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:32:57.558106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504778+0000","last_change":"2023-11-16T18:22:46.820535+0000","last_active":"2023-11-16T18:23:39.504778+0000","last_peered":"2023-11-16T18:23:39.504778+0000","last_clean":"2023-11-16T18:23:39.504778+0000","last_became_active":"2023-11-16T18:22:46.820009+0000","last_became_peered":"2023-11-16T18:22:46.820009+0000","last_unstale":"2023-11-16T18:23:39.504778+0000","last_undegraded":"2023-11-16T18:23:39.504778+0000","last_fullsized":"2023-11-16T18:23:39.504778+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:50:39.196560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510673+0000","last_change":"2023-11-16T18:22:46.831762+0000","last_active":"2023-11-16T18:23:39.510673+0000","last_peered":"2023-11-16T18:23:39.510673+0000","last_clean":"2023-11-16T18:23:39.510673+0000","last_became_active":"2023-11-16T18:22:46.831429+0000","last_became_peered":"2023-11-16T18:22:46.831429+0000","last_unstale":"2023-11-16T18:23:39.510673+0000","last_undegraded":"2023-11-16T18:23:39.510673+0000","last_fullsized":"2023-11-16T18:23:39.510673+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:15:44.338906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510533+0000","last_change":"2023-11-16T18:22:46.838133+0000","last_active":"2023-11-16T18:23:39.510533+0000","last_peered":"2023-11-16T18:23:39.510533+0000","last_clean":"2023-11-16T18:23:39.510533+0000","last_became_active":"2023-11-16T18:22:46.837845+0000","last_became_peered":"2023-11-16T18:22:46.837845+0000","last_unstale":"2023-11-16T18:23:39.510533+0000","last_undegraded":"2023-11-16T18:23:39.510533+0000","last_fullsized":"2023-11-16T18:23:39.510533+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:13:06.102358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585337+0000","last_change":"2023-11-16T18:23:39.585337+0000","last_active":"2023-11-16T18:23:39.585337+0000","last_peered":"2023-11-16T18:23:39.585337+0000","last_clean":"2023-11-16T18:23:39.585337+0000","last_became_active":"2023-11-16T18:23:39.584874+0000","last_became_peered":"2023-11-16T18:23:39.584874+0000","last_unstale":"2023-11-16T18:23:39.585337+0000","last_undegraded":"2023-11-16T18:23:39.585337+0000","last_fullsized":"2023-11-16T18:23:39.585337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:24:54.928438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585342+0000","last_change":"2023-11-16T18:22:46.823192+0000","last_active":"2023-11-16T18:23:39.585342+0000","last_peered":"2023-11-16T18:23:39.585342+0000","last_clean":"2023-11-16T18:23:39.585342+0000","last_became_active":"2023-11-16T18:22:46.822930+0000","last_became_peered":"2023-11-16T18:22:46.822930+0000","last_unstale":"2023-11-16T18:23:39.585342+0000","last_undegraded":"2023-11-16T18:23:39.585342+0000","last_fullsized":"2023-11-16T18:23:39.585342+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:49:48.831997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505366+0000","last_change":"2023-11-16T18:22:46.825815+0000","last_active":"2023-11-16T18:23:39.505366+0000","last_peered":"2023-11-16T18:23:39.505366+0000","last_clean":"2023-11-16T18:23:39.505366+0000","last_became_active":"2023-11-16T18:22:46.825517+0000","last_became_peered":"2023-11-16T18:22:46.825517+0000","last_unstale":"2023-11-16T18:23:39.505366+0000","last_undegraded":"2023-11-16T18:23:39.505366+0000","last_fullsized":"2023-11-16T18:23:39.505366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:32:02.468288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.511101+0000","last_change":"2023-11-16T18:22:46.826825+0000","last_active":"2023-11-16T18:23:39.511101+0000","last_peered":"2023-11-16T18:23:39.511101+0000","last_clean":"2023-11-16T18:23:39.511101+0000","last_became_active":"2023-11-16T18:22:46.826495+0000","last_became_peered":"2023-11-16T18:22:46.826495+0000","last_unstale":"2023-11-16T18:23:39.511101+0000","last_undegraded":"2023-11-16T18:23:39.511101+0000","last_fullsized":"2023-11-16T18:23:39.511101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:07:42.411162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506174+0000","last_change":"2023-11-16T18:22:46.844090+0000","last_active":"2023-11-16T18:23:39.506174+0000","last_peered":"2023-11-16T18:23:39.506174+0000","last_clean":"2023-11-16T18:23:39.506174+0000","last_became_active":"2023-11-16T18:22:46.843641+0000","last_became_peered":"2023-11-16T18:22:46.843641+0000","last_unstale":"2023-11-16T18:23:39.506174+0000","last_undegraded":"2023-11-16T18:23:39.506174+0000","last_fullsized":"2023-11-16T18:23:39.506174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:00:12.291257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508321+0000","last_change":"2023-11-16T18:22:46.838340+0000","last_active":"2023-11-16T18:23:39.508321+0000","last_peered":"2023-11-16T18:23:39.508321+0000","last_clean":"2023-11-16T18:23:39.508321+0000","last_became_active":"2023-11-16T18:22:46.838039+0000","last_became_peered":"2023-11-16T18:22:46.838039+0000","last_unstale":"2023-11-16T18:23:39.508321+0000","last_undegraded":"2023-11-16T18:23:39.508321+0000","last_fullsized":"2023-11-16T18:23:39.508321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:09:28.607361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504176+0000","last_change":"2023-11-16T18:22:46.841739+0000","last_active":"2023-11-16T18:23:39.504176+0000","last_peered":"2023-11-16T18:23:39.504176+0000","last_clean":"2023-11-16T18:23:39.504176+0000","last_became_active":"2023-11-16T18:22:46.841450+0000","last_became_peered":"2023-11-16T18:22:46.841450+0000","last_unstale":"2023-11-16T18:23:39.504176+0000","last_undegraded":"2023-11-16T18:23:39.504176+0000","last_fullsized":"2023-11-16T18:23:39.504176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:42:53.482129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508013+0000","last_change":"2023-11-16T18:22:46.839060+0000","last_active":"2023-11-16T18:23:39.508013+0000","last_peered":"2023-11-16T18:23:39.508013+0000","last_clean":"2023-11-16T18:23:39.508013+0000","last_became_active":"2023-11-16T18:22:46.838725+0000","last_became_peered":"2023-11-16T18:22:46.838725+0000","last_unstale":"2023-11-16T18:23:39.508013+0000","last_undegraded":"2023-11-16T18:23:39.508013+0000","last_fullsized":"2023-11-16T18:23:39.508013+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:33:35.374697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584469+0000","last_change":"2023-11-16T18:22:46.833439+0000","last_active":"2023-11-16T18:23:39.584469+0000","last_peered":"2023-11-16T18:23:39.584469+0000","last_clean":"2023-11-16T18:23:39.584469+0000","last_became_active":"2023-11-16T18:22:46.833111+0000","last_became_peered":"2023-11-16T18:22:46.833111+0000","last_unstale":"2023-11-16T18:23:39.584469+0000","last_undegraded":"2023-11-16T18:23:39.584469+0000","last_fullsized":"2023-11-16T18:23:39.584469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:10:36.066370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506665+0000","last_change":"2023-11-16T18:22:46.825185+0000","last_active":"2023-11-16T18:23:39.506665+0000","last_peered":"2023-11-16T18:23:39.506665+0000","last_clean":"2023-11-16T18:23:39.506665+0000","last_became_active":"2023-11-16T18:22:46.824866+0000","last_became_peered":"2023-11-16T18:22:46.824866+0000","last_unstale":"2023-11-16T18:23:39.506665+0000","last_undegraded":"2023-11-16T18:23:39.506665+0000","last_fullsized":"2023-11-16T18:23:39.506665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:46:05.263494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":215374,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65131,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":242,"num_read_kb":188,"num_write":154,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":61913,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":195393,"data_compressed":9627,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481816,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6140,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739012,"statfs":{"total":95995035648,"available":95988748288,"internally_reserved":0,"allocated":970752,"data_stored":485806,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.50800000000000001}]},{"osd":2,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.42299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.33800000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.6539999999999999,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":0.437},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.7130000000000001,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.56100000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":9,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.495}]},{"osd":10,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.753}]}]},{"osd":10,"up_from":69,"seq":296352743452,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6596,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738556,"statfs":{"total":95995035648,"available":95988281344,"internally_reserved":0,"allocated":1110016,"data_stored":1138237,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":1,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.35299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.51800000000000002},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.379}]},{"osd":2,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.82899999999999996,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.624},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.85699999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.659,"5min":1.659,"15min":1.659},"last":1.659},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.72499999999999998,"15min":0.73299999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":1.554}]},{"osd":4,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.629,"15min":0.63},"min":{"1min":0.44,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76800000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.5},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.36499999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.85099999999999998,"5min":0.996,"15min":0.996},"last":0.42999999999999999}]},{"osd":5,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.99399999999999999,"5min":1.206,"15min":1.206},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.745},"min":{"1min":0.55200000000000005,"5min":0.502,"15min":0.502},"max":{"1min":0.97799999999999998,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.64800000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.77400000000000002,"15min":0.77700000000000002},"min":{"1min":0.499,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.581},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.57399999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.609}]},{"osd":7,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55000000000000004}]},{"osd":8,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.84199999999999997,"15min":0.84999999999999998},"min":{"1min":0.63100000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.98899999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0189999999999999,"5min":1.155,"15min":1.155},"last":0.67600000000000005}]},{"osd":9,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.65600000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.6459999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.85699999999999998,"15min":0.85999999999999999},"min":{"1min":0.59799999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":1.4870000000000001}]},{"osd":11,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.72499999999999998}]}]},{"osd":9,"up_from":64,"seq":274877906977,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6340,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":978944,"data_stored":485930,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53600000000000003,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.45400000000000001,"15min":0.46600000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.51900000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.371}]},{"osd":1,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.66100000000000003,"15min":0.67500000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.74099999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.623,"15min":0.63500000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81200000000000006,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.65300000000000002,"15min":0.66300000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82399999999999995,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.63100000000000001,"15min":0.64200000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.80900000000000005,"5min":1.024,"15min":1.024},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.54100000000000004,"15min":0.55600000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63700000000000001,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.438},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.53800000000000003,"15min":0.56100000000000005},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.60699999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.70299999999999996,"15min":0.72299999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.91500000000000004,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74199999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.67700000000000005}]},{"osd":5,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.71299999999999997,"15min":0.72799999999999998},"min":{"1min":0.437,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83199999999999996,"5min":1.357,"15min":1.357},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.754,"15min":0.77400000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82199999999999995,"5min":1.196,"15min":1.196},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.379},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.42699999999999999}]},{"osd":7,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.81599999999999995,"15min":0.84399999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.79300000000000004,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.77200000000000002,"15min":0.79800000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.84999999999999998,"5min":1.157,"15min":1.157},"last":0.754}]},{"osd":8,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.78700000000000003,"15min":0.81100000000000005},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.874,"5min":1.248,"15min":1.248},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.82399999999999995,"15min":0.84899999999999998},"min":{"1min":0.53100000000000003,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.85399999999999998,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.69399999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.70599999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":8,"up_from":57,"seq":244813135909,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6460,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738692,"statfs":{"total":95995035648,"available":95988420608,"internally_reserved":0,"allocated":970752,"data_stored":485738,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.58299999999999996,"15min":0.58799999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.81699999999999995,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.42499999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.443,"5min":0.316,"15min":0.316},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.72499999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.36599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.48599999999999999,"15min":0.47599999999999998},"min":{"1min":0.39000000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.65300000000000002}]},{"osd":3,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.41799999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.628,"15min":0.628},"min":{"1min":0.41299999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.81399999999999995}]},{"osd":5,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52500000000000002,"15min":0.51300000000000001},"min":{"1min":0.49399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73899999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.37,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.77400000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.78900000000000003}]},{"osd":6,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65900000000000003,"15min":0.65700000000000003},"min":{"1min":0.51600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.52700000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.84799999999999998,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.80100000000000005}]},{"osd":7,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.68000000000000005,"15min":0.66700000000000004},"min":{"1min":0.52000000000000002,"5min":0.44,"15min":0.44},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.54300000000000004,"5min":0.434,"15min":0.434},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.83299999999999996}]},{"osd":9,"last update":"Thu Nov 16 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.60599999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95999999999999996,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.72999999999999998,"15min":0.72099999999999997},"min":{"1min":0.501,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.93899999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.873}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.72499999999999998,"15min":0.71099999999999997},"min":{"1min":0.60499999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.036,"5min":1.276,"15min":1.276},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.76400000000000001,"15min":0.74399999999999999},"min":{"1min":0.74299999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.92200000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]}]},{"osd":7,"up_from":49,"seq":210453397546,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6580,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738572,"statfs":{"total":95995035648,"available":95988297728,"internally_reserved":0,"allocated":1028096,"data_stored":1082052,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.60699999999999998,"15min":0.59499999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81999999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81699999999999995,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.44500000000000001,"15min":0.42799999999999999},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61499999999999999,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.48099999999999998,"15min":0.46600000000000003},"min":{"1min":0.36299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71899999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63100000000000001,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65000000000000002,"15min":0.622},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.876,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.83199999999999996}]},{"osd":3,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85699999999999998,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.42299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.83599999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.57799999999999996,"15min":0.55200000000000005},"min":{"1min":0.43099999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.77900000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.372,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76300000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.58699999999999997}]},{"osd":5,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70099999999999996,"15min":0.67900000000000005},"min":{"1min":0.47099999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89500000000000002,"5min":1.075,"15min":1.075},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.75600000000000001,"15min":0.748},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.94999999999999996,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.92600000000000005}]},{"osd":6,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.71599999999999997,"15min":0.69999999999999996},"min":{"1min":0.52600000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.74399999999999999,"15min":0.73199999999999998},"min":{"1min":0.47799999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.878}]},{"osd":8,"last update":"Thu Nov 16 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.747,"15min":0.74099999999999999},"min":{"1min":0.501,"5min":0.433,"15min":0.433},"max":{"1min":1.0529999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.77900000000000003,"15min":0.77700000000000002},"min":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0409999999999999,"5min":1.107,"15min":1.107},"last":0.96699999999999997}]},{"osd":9,"last update":"Thu Nov 16 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.79800000000000004,"15min":0.80500000000000005},"min":{"1min":0.48799999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.96799999999999997,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.83299999999999996,"15min":0.81799999999999995},"min":{"1min":0.47799999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":3.4119999999999999,"5min":3.4119999999999999,"15min":3.4119999999999999},"last":1.0369999999999999}]},{"osd":10,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":1.071}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]}]},{"osd":6,"up_from":44,"seq":188978561070,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6384,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738768,"statfs":{"total":95995035648,"available":95988498432,"internally_reserved":0,"allocated":958464,"data_stored":481106,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.39400000000000002,"15min":0.36799999999999999},"min":{"1min":0.309,"5min":0.251,"15min":0.251},"max":{"1min":0.64500000000000002,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.371},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.40500000000000003,"15min":0.38100000000000001},"min":{"1min":0.28499999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58499999999999996,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.624,"15min":0.59599999999999997},"min":{"1min":0.47399999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.53100000000000003},"min":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.44400000000000001}]},{"osd":2,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54200000000000004,"15min":0.52200000000000002},"min":{"1min":0.371,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81899999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60999999999999999,"15min":0.57199999999999995},"min":{"1min":0.40300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.52400000000000002,"15min":0.498},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77500000000000002,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45600000000000002,"15min":0.438},"min":{"1min":0.309,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.66600000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.35499999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.626,"15min":0.61099999999999999},"min":{"1min":0.44900000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89000000000000001,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70299999999999996,"15min":0.66800000000000004},"min":{"1min":0.54900000000000004,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.56399999999999995}]},{"osd":5,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59899999999999998,"15min":0.57899999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64700000000000002,"15min":0.626},"min":{"1min":0.438,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93700000000000006,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.60199999999999998}]},{"osd":7,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75800000000000001,"15min":0.75},"min":{"1min":0.505,"5min":0.497,"15min":0.497},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.79100000000000004,"15min":0.77600000000000002},"min":{"1min":0.54600000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.67000000000000004}]},{"osd":8,"last update":"Thu Nov 16 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.82399999999999995,"15min":0.82599999999999996},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.70199999999999996,"15min":0.70699999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85799999999999998,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.63900000000000001}]},{"osd":9,"last update":"Thu Nov 16 18:23:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.72899999999999998,"15min":0.753},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.751,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.84799999999999998,"15min":0.874},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.629}]},{"osd":10,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.85999999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.71099999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7476,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737676,"statfs":{"total":95995035648,"available":95987380224,"internally_reserved":0,"allocated":1028096,"data_stored":542544,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51000000000000001,"15min":0.52800000000000002},"min":{"1min":0.315,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66900000000000004,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.34799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.71299999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.92200000000000004}]},{"osd":2,"last update":"Thu Nov 16 18:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55800000000000005,"15min":0.53300000000000003},"min":{"1min":0.41099999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78100000000000003,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56999999999999995,"15min":0.54600000000000004},"min":{"1min":0.36199999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.74399999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75900000000000001}]},{"osd":3,"last update":"Thu Nov 16 18:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58199999999999996,"15min":0.56599999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.878,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.58199999999999996,"15min":0.56899999999999995},"min":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.94099999999999995}]},{"osd":4,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.45200000000000001,"15min":0.40799999999999997},"min":{"1min":0.372,"5min":0.252,"15min":0.252},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.47399999999999998,"15min":0.41999999999999998},"min":{"1min":0.39300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80000000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Nov 16 18:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.63600000000000001,"15min":0.624},"min":{"1min":0.49099999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85299999999999998,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.71199999999999997}]},{"osd":6,"last update":"Thu Nov 16 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61899999999999999,"15min":0.60899999999999999},"min":{"1min":0.41699999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.89100000000000001,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61299999999999999,"15min":0.59899999999999998},"min":{"1min":0.433,"5min":0.371,"15min":0.371},"max":{"1min":0.82199999999999995,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":1.028}]},{"osd":7,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.56899999999999995,"15min":0.53900000000000003},"min":{"1min":0.42399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.88200000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.622,"15min":0.58199999999999996},"min":{"1min":0.46500000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.91200000000000003,"5min":0.998,"15min":0.998},"last":0.88500000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.66800000000000004,"15min":0.65800000000000003},"min":{"1min":0.51900000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67000000000000004,"15min":0.66000000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.96799999999999997}]},{"osd":9,"last update":"Thu Nov 16 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.72399999999999998,"15min":0.71899999999999997},"min":{"1min":0.496,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.98199999999999998,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.755,"15min":0.749},"min":{"1min":0.499,"5min":0.495,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":1.036,"15min":1.036},"last":1.0720000000000001}]},{"osd":10,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":1.012}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]}]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7300,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737852,"statfs":{"total":95995035648,"available":95987560448,"internally_reserved":0,"allocated":978944,"data_stored":482221,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 18:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.55300000000000005,"15min":0.5},"min":{"1min":0.36799999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.923,"5min":2.923,"15min":2.923},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.56699999999999995,"15min":0.503},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.2109999999999999,"5min":3.2109999999999999,"15min":3.2109999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Nov 16 18:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.64900000000000002,"15min":0.72899999999999998},"min":{"1min":0.379,"5min":0.371,"15min":0.371},"max":{"1min":0.81399999999999995,"5min":5.3650000000000002,"15min":5.3650000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.624,"15min":0.67000000000000004},"min":{"1min":0.374,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.872,"5min":3.4009999999999998,"15min":3.4009999999999998},"last":0.47799999999999998}]},{"osd":3,"last update":"Thu Nov 16 18:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55800000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.45100000000000001},"min":{"1min":0.32700000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.60599999999999998,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.0249999999999999,"5min":3.0249999999999999,"15min":3.0249999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.61299999999999999,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":3.242,"5min":3.242,"15min":3.242},"last":0.65500000000000003}]},{"osd":5,"last update":"Thu Nov 16 18:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58099999999999996,"15min":0.57499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82099999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.505},{"interface":"front","average":{"1min":0.63,"5min":0.59999999999999998,"15min":0.58599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84099999999999997,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.53300000000000003}]},{"osd":6,"last update":"Thu Nov 16 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.55300000000000005,"15min":0.51900000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.8170000000000002,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.54700000000000004,"15min":0.53100000000000003},"min":{"1min":0.30399999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.434}]},{"osd":7,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.95099999999999996,"5min":1.004,"15min":1.004},"last":0.745}]},{"osd":8,"last update":"Thu Nov 16 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89600000000000002,"5min":1.052,"15min":1.052},"last":0.63},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.96699999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.60599999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:23:27 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70299999999999996,"15min":0.69799999999999995},"min":{"1min":0.40000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.3180000000000001,"5min":3.3180000000000001,"15min":3.3180000000000001},"last":0.621},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.72999999999999998,"15min":0.73999999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":0.69499999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.78600000000000003,"15min":0.78100000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.0009999999999999,"5min":3.0009999999999999,"15min":3.0009999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.77000000000000002,"15min":0.75800000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":3.2679999999999998,"5min":3.2679999999999998,"15min":3.2679999999999998},"last":0.68400000000000005}]},{"osd":11,"last update":"Thu Nov 16 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.66700000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7484,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737668,"statfs":{"total":95995035648,"available":95987372032,"internally_reserved":0,"allocated":970752,"data_stored":481744,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.50700000000000001,"15min":0.46999999999999997},"min":{"1min":0.30399999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.49399999999999999,"15min":0.46100000000000002},"min":{"1min":0.35599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.42999999999999999}]},{"osd":1,"last update":"Thu Nov 16 18:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.53900000000000003,"15min":0.51300000000000001},"min":{"1min":0.33000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.56399999999999995,"15min":0.53900000000000003},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.74099999999999999}]},{"osd":3,"last update":"Thu Nov 16 18:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.53900000000000003},"min":{"1min":0.38700000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84499999999999997,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.58999999999999997,"15min":0.54500000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.50600000000000001}]},{"osd":4,"last update":"Thu Nov 16 18:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.54500000000000004,"15min":0.52700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.56299999999999994,"15min":0.54400000000000004},"min":{"1min":0.36799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.67000000000000004}]},{"osd":5,"last update":"Thu Nov 16 18:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.44,"15min":0.41299999999999998},"min":{"1min":0.34699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.497,"15min":0.47099999999999997},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.45700000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63700000000000001,"15min":0.61499999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91500000000000004,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.66300000000000003,"15min":0.629},"min":{"1min":0.53000000000000003,"5min":0.501,"15min":0.501},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.628}]},{"osd":7,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.61899999999999999,"15min":0.61399999999999999},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.65700000000000003}]},{"osd":8,"last update":"Thu Nov 16 18:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.58399999999999996,"15min":0.56999999999999995},"min":{"1min":0.38200000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.377,"5min":1.377,"15min":1.377},"last":0.53000000000000003}]},{"osd":9,"last update":"Thu Nov 16 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.71299999999999997,"15min":0.70799999999999996},"min":{"1min":0.498,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.51300000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.60799999999999998}]},{"osd":10,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.73899999999999999,"15min":0.71899999999999997},"min":{"1min":0.60399999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.77200000000000002,"15min":0.76200000000000001},"min":{"1min":0.58799999999999997,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.242,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.72399999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6712,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738440,"statfs":{"total":95995035648,"available":95988162560,"internally_reserved":0,"allocated":966656,"data_stored":481027,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:14 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38300000000000001,"15min":0.36599999999999999},"min":{"1min":0.25,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.621,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39800000000000002,"15min":0.38700000000000001},"min":{"1min":0.26800000000000002,"5min":0.185,"15min":0.185},"max":{"1min":0.59399999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.34699999999999998}]},{"osd":1,"last update":"Thu Nov 16 18:23:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.52200000000000002},"min":{"1min":0.36899999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80600000000000005,"5min":1.129,"15min":1.129},"last":0.499},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55700000000000005,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70199999999999996,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.69999999999999996}]},{"osd":2,"last update":"Thu Nov 16 18:23:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.59899999999999998,"15min":0.56899999999999995},"min":{"1min":0.40699999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.94099999999999995,"5min":1.038,"15min":1.038},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58699999999999997,"15min":0.57099999999999995},"min":{"1min":0.376,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71899999999999997,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.44900000000000001}]},{"osd":4,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63500000000000001,"15min":0.60299999999999998},"min":{"1min":0.498,"5min":0.378,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":0.628},"min":{"1min":0.439,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79800000000000004,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77400000000000002,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.35999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98799999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.52400000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:23:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53900000000000003,"15min":0.49299999999999999},"min":{"1min":0.29999999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93500000000000005,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55900000000000005,"15min":0.53900000000000003},"min":{"1min":0.39800000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.86599999999999999,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.30599999999999999}]},{"osd":7,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.44600000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81399999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69599999999999995,"15min":0.68600000000000005},"min":{"1min":0.46700000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81499999999999995,"5min":1.087,"15min":1.087},"last":0.55600000000000005}]},{"osd":8,"last update":"Thu Nov 16 18:22:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.60799999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79800000000000004,"15min":0.82499999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.002,"5min":1.135,"15min":1.135},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.73899999999999999,"15min":0.75900000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.95899999999999996,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.37}]},{"osd":10,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.77000000000000002,"15min":0.78800000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.77600000000000002,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.82699999999999996,"15min":0.85199999999999998},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.81799999999999995,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.58099999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953527,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6608,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738544,"statfs":{"total":95995035648,"available":95988269056,"internally_reserved":0,"allocated":991232,"data_stored":546662,"data_compressed":27217,"data_compressed_allocated":36864,"data_compressed_original":106496,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.504,"15min":0.48699999999999999},"min":{"1min":0.39900000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.752,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.48899999999999999,"15min":0.46400000000000002},"min":{"1min":0.41799999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.502}]},{"osd":1,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40600000000000003,"15min":0.377},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.56299999999999994,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.439,"15min":0.41299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.60699999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59199999999999997,"15min":0.56599999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85399999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.61699999999999999,"15min":0.57499999999999996},"min":{"1min":0.51500000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83599999999999997,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.82999999999999996}]},{"osd":3,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57199999999999995,"15min":0.54000000000000004},"min":{"1min":0.40100000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.03,"5min":0.64800000000000002,"15min":0.56200000000000006},"min":{"1min":0.40300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":10.048,"5min":10.048,"15min":10.048},"last":0.629}]},{"osd":5,"last update":"Thu Nov 16 18:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.63900000000000001,"15min":0.60999999999999999},"min":{"1min":0.46500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.98599999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.65000000000000002,"15min":0.63400000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.81299999999999994}]},{"osd":6,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60699999999999998,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77100000000000002,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66100000000000003,"15min":0.63800000000000001},"min":{"1min":0.47999999999999998,"5min":0.432,"15min":0.432},"max":{"1min":0.86899999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90000000000000002}]},{"osd":7,"last update":"Thu Nov 16 18:23:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.626,"15min":0.57599999999999996},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95199999999999996,"5min":1.077,"15min":1.077},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56899999999999995,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76900000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.77600000000000002}]},{"osd":8,"last update":"Thu Nov 16 18:23:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.84799999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.54500000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.91900000000000004,"5min":1.109,"15min":1.109},"last":0.95499999999999996}]},{"osd":9,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.752,"15min":0.76100000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.91800000000000004,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80100000000000005,"15min":0.80700000000000005},"min":{"1min":0.60599999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91300000000000003,"5min":1.117,"15min":1.117},"last":0.96999999999999997}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.63400000000000001,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":0.997,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78200000000000003,"15min":0.78500000000000003},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96799999999999997,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":1.0169999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790002,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6716,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738436,"statfs":{"total":95995035648,"available":95988158464,"internally_reserved":0,"allocated":1036288,"data_stored":1077298,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52400000000000002,"15min":0.51200000000000001},"min":{"1min":0.38500000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.751,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.59899999999999998,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64800000000000002,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.46600000000000003}]},{"osd":1,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53300000000000003,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.37,"15min":0.37},"max":{"1min":0.78600000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.52700000000000002,"15min":0.51400000000000001},"min":{"1min":0.36799999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.746,"5min":0.873,"15min":0.873},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39500000000000002,"15min":0.39200000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39600000000000002,"15min":0.38},"min":{"1min":0.28100000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.434}]},{"osd":3,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84699999999999998,"5min":0.88,"15min":0.88},"last":0.51000000000000001}]},{"osd":4,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.375,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56799999999999995,"15min":0.56299999999999994},"min":{"1min":0.41399999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82199999999999995,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.70899999999999996}]},{"osd":6,"last update":"Thu Nov 16 18:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59199999999999997,"15min":0.59499999999999997},"min":{"1min":0.46000000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.68600000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59299999999999997,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81999999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.55400000000000005}]},{"osd":7,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.64000000000000001,"15min":0.629},"min":{"1min":0.48699999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.63800000000000001,"15min":0.625},"min":{"1min":0.44800000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.89600000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.69199999999999995}]},{"osd":8,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.83599999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.753,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66100000000000003}]},{"osd":9,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.66200000000000003,"15min":0.66000000000000003},"min":{"1min":0.53700000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.84299999999999997,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.751},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70299999999999996,"15min":0.70199999999999996},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.83299999999999996,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.72099999999999997}]},{"osd":10,"last update":"Thu Nov 16 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.72699999999999998,"15min":0.71599999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.70499999999999996,"15min":0.69299999999999995},"min":{"1min":0.53800000000000003,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93799999999999994,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.79100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34529,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":61823,"data_compressed":9627,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T18:23:48.899 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph pg dump --format=json 2023-11-16T18:23:48.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:48 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:23:49.560 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:49.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:49 smithi188 ceph-mon[100639]: pgmap v293: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 219 B/s wr, 4 op/s; 6.5 KiB/s, 0 keys/s, 1 objects/s recovering 2023-11-16T18:23:49.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:49 smithi188 ceph-mon[100639]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:23:49.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:49 smithi002 ceph-mon[98807]: pgmap v293: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 219 B/s wr, 4 op/s; 6.5 KiB/s, 0 keys/s, 1 objects/s recovering 2023-11-16T18:23:49.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:49 smithi002 ceph-mon[98807]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:23:49.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:49 smithi040 ceph-mon[100251]: pgmap v293: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 219 B/s wr, 4 op/s; 6.5 KiB/s, 0 keys/s, 1 objects/s recovering 2023-11-16T18:23:49.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:49 smithi040 ceph-mon[100251]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:23:50.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.878+0000 7f9181b35700 1 -- 172.21.15.2:0/2919300997 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f917c0ffe20 msgr2=0x7f917c103df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:50.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.878+0000 7f9181b35700 1 --2- 172.21.15.2:0/2919300997 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f917c0ffe20 0x7f917c103df0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f91700099c0 tx=0x7f9170009cd0 comp rx=0 tx=0).stop 2023-11-16T18:23:50.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.878+0000 7f9181b35700 1 -- 172.21.15.2:0/2919300997 shutdown_connections 2023-11-16T18:23:50.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.878+0000 7f9181b35700 1 --2- 172.21.15.2:0/2919300997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f917c104520 0x7f917c10c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:50.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.878+0000 7f9181b35700 1 --2- 172.21.15.2:0/2919300997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c0ff480 0x7f917c0ff850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:50.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.878+0000 7f9181b35700 1 --2- 172.21.15.2:0/2919300997 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f917c0ffe20 0x7f917c103df0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:50.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.878+0000 7f9181b35700 1 -- 172.21.15.2:0/2919300997 >> 172.21.15.2:0/2919300997 conn(0x7f917c0770d0 msgr2=0x7f917c0774d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:50.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.878+0000 7f9181b35700 1 -- 172.21.15.2:0/2919300997 shutdown_connections 2023-11-16T18:23:50.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.878+0000 7f9181b35700 1 -- 172.21.15.2:0/2919300997 wait complete. 2023-11-16T18:23:50.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.879+0000 7f9181b35700 1 Processor -- start 2023-11-16T18:23:50.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.879+0000 7f9181b35700 1 -- start start 2023-11-16T18:23:50.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.879+0000 7f9181b35700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c0ff480 0x7f917c1971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:50.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.879+0000 7f9181b35700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f917c0ffe20 0x7f917c197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:50.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.879+0000 7f9181b35700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f917c104520 0x7f917c1a2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:50.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.879+0000 7f9181b35700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c198040 con 0x7f917c104520 2023-11-16T18:23:50.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.879+0000 7f9181b35700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c1a27b0 con 0x7f917c0ff480 2023-11-16T18:23:50.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.879+0000 7f9181b35700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c1a2920 con 0x7f917c0ffe20 2023-11-16T18:23:50.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.879+0000 7f917bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f917c104520 0x7f917c1a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:50.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.879+0000 7f917bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f917c104520 0x7f917c1a2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48812/0 (socket says 172.21.15.2:48812) 2023-11-16T18:23:50.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.879+0000 7f917b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c0ff480 0x7f917c1971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:50.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.879+0000 7f917bfff700 1 -- 172.21.15.2:0/334257476 learned_addr learned my addr 172.21.15.2:0/334257476 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:50.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.879+0000 7f917affd700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f917c0ffe20 0x7f917c197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:50.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.880+0000 7f917bfff700 1 -- 172.21.15.2:0/334257476 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f917c0ffe20 msgr2=0x7f917c197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:50.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.880+0000 7f917bfff700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f917c0ffe20 0x7f917c197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:50.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.880+0000 7f917bfff700 1 -- 172.21.15.2:0/334257476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c0ff480 msgr2=0x7f917c1971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:50.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.880+0000 7f917bfff700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c0ff480 0x7f917c1971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:50.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.880+0000 7f917bfff700 1 -- 172.21.15.2:0/334257476 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91700095f0 con 0x7f917c104520 2023-11-16T18:23:50.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.880+0000 7f917b7fe700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c0ff480 0x7f917c1971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:23:50.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.880+0000 7f917affd700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f917c0ffe20 0x7f917c197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:23:50.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.880+0000 7f917bfff700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f917c104520 0x7f917c1a2270 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f916c00ef30 tx=0x7f916c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:50.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.880+0000 7f9178ff9700 1 -- 172.21.15.2:0/334257476 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c018070 con 0x7f917c104520 2023-11-16T18:23:50.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.880+0000 7f9181b35700 1 -- 172.21.15.2:0/334257476 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917c10f250 con 0x7f917c104520 2023-11-16T18:23:50.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.880+0000 7f9178ff9700 1 -- 172.21.15.2:0/334257476 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f916c00f040 con 0x7f917c104520 2023-11-16T18:23:50.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.880+0000 7f9178ff9700 1 -- 172.21.15.2:0/334257476 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c00cd00 con 0x7f917c104520 2023-11-16T18:23:50.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.880+0000 7f9181b35700 1 -- 172.21.15.2:0/334257476 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917c10f0c0 con 0x7f917c104520 2023-11-16T18:23:50.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.881+0000 7f9181b35700 1 -- 172.21.15.2:0/334257476 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f917c10f3e0 con 0x7f917c104520 2023-11-16T18:23:50.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.882+0000 7f9178ff9700 1 -- 172.21.15.2:0/334257476 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f916c003a00 con 0x7f917c104520 2023-11-16T18:23:50.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.883+0000 7f9178ff9700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f91640a11e0 0x7f91640a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:50.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.883+0000 7f9178ff9700 1 -- 172.21.15.2:0/334257476 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f916c0cd1c0 con 0x7f917c104520 2023-11-16T18:23:50.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.883+0000 7f917b7fe700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f91640a11e0 0x7f91640a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:50.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.884+0000 7f917b7fe700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f91640a11e0 0x7f91640a3690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9174011490 tx=0x7f9174009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:50.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:50.886+0000 7f9178ff9700 1 -- 172.21.15.2:0/334257476 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f916c096530 con 0x7f917c104520 2023-11-16T18:23:51.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.085+0000 7f9181b35700 1 -- 172.21.15.2:0/334257476 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f917c066ea0 con 0x7f91640a11e0 2023-11-16T18:23:51.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.103+0000 7f9178ff9700 1 -- 172.21.15.2:0/334257476 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+423119 (secure 0 0 0) 0x7f917c066ea0 con 0x7f91640a11e0 2023-11-16T18:23:51.111 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:51.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.107+0000 7f9181b35700 1 -- 172.21.15.2:0/334257476 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f91640a11e0 msgr2=0x7f91640a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:51.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.107+0000 7f9181b35700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f91640a11e0 0x7f91640a3690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9174011490 tx=0x7f9174009260 comp rx=0 tx=0).stop 2023-11-16T18:23:51.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.107+0000 7f9181b35700 1 -- 172.21.15.2:0/334257476 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f917c104520 msgr2=0x7f917c1a2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:51.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.107+0000 7f9181b35700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f917c104520 0x7f917c1a2270 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f916c00ef30 tx=0x7f916c00c3a0 comp rx=0 tx=0).stop 2023-11-16T18:23:51.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.107+0000 7f9181b35700 1 -- 172.21.15.2:0/334257476 shutdown_connections 2023-11-16T18:23:51.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.107+0000 7f9181b35700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f91640a11e0 0x7f91640a3690 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:51.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.107+0000 7f9181b35700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f917c104520 0x7f917c1a2270 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:51.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.107+0000 7f9181b35700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c0ff480 0x7f917c1971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:51.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.107+0000 7f9181b35700 1 --2- 172.21.15.2:0/334257476 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f917c0ffe20 0x7f917c197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:51.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.107+0000 7f9181b35700 1 -- 172.21.15.2:0/334257476 >> 172.21.15.2:0/334257476 conn(0x7f917c0770d0 msgr2=0x7f917c0fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:51.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.107+0000 7f9181b35700 1 -- 172.21.15.2:0/334257476 shutdown_connections 2023-11-16T18:23:51.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:51.108+0000 7f9181b35700 1 -- 172.21.15.2:0/334257476 wait complete. 2023-11-16T18:23:51.115 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-16T18:23:51.523 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2023-11-16T18:23:49.819079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655745,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":383,"num_read_kb":312,"num_write":337,"num_write_kb":3050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1242772,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80796,"kb_used_data":11708,"kb_used_omap":0,"kb_used_meta":68992,"kb_avail":1124861028,"statfs":{"total":1151940427776,"available":1151857692672,"internally_reserved":0,"allocated":11988992,"data_stored":7770365,"data_compressed":329214,"data_compressed_allocated":442368,"data_compressed_original":2416640,"omap_allocated":0,"internal_metadata":70647808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1652,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":25,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":62036,"num_keys_recovered":7,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.319780"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506980+0000","last_change":"2023-11-16T18:22:46.829814+0000","last_active":"2023-11-16T18:23:39.506980+0000","last_peered":"2023-11-16T18:23:39.506980+0000","last_clean":"2023-11-16T18:23:39.506980+0000","last_became_active":"2023-11-16T18:22:46.829407+0000","last_became_peered":"2023-11-16T18:22:46.829407+0000","last_unstale":"2023-11-16T18:23:39.506980+0000","last_undegraded":"2023-11-16T18:23:39.506980+0000","last_fullsized":"2023-11-16T18:23:39.506980+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:02:05.303420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505209+0000","last_change":"2023-11-16T18:22:46.818700+0000","last_active":"2023-11-16T18:23:39.505209+0000","last_peered":"2023-11-16T18:23:39.505209+0000","last_clean":"2023-11-16T18:23:39.505209+0000","last_became_active":"2023-11-16T18:22:46.818388+0000","last_became_peered":"2023-11-16T18:22:46.818388+0000","last_unstale":"2023-11-16T18:23:39.505209+0000","last_undegraded":"2023-11-16T18:23:39.505209+0000","last_fullsized":"2023-11-16T18:23:39.505209+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:05:25.862565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504017+0000","last_change":"2023-11-16T18:22:46.827732+0000","last_active":"2023-11-16T18:23:39.504017+0000","last_peered":"2023-11-16T18:23:39.504017+0000","last_clean":"2023-11-16T18:23:39.504017+0000","last_became_active":"2023-11-16T18:22:46.827328+0000","last_became_peered":"2023-11-16T18:22:46.827328+0000","last_unstale":"2023-11-16T18:23:39.504017+0000","last_undegraded":"2023-11-16T18:23:39.504017+0000","last_fullsized":"2023-11-16T18:23:39.504017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:44:05.572178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504758+0000","last_change":"2023-11-16T18:22:46.826837+0000","last_active":"2023-11-16T18:23:39.504758+0000","last_peered":"2023-11-16T18:23:39.504758+0000","last_clean":"2023-11-16T18:23:39.504758+0000","last_became_active":"2023-11-16T18:22:46.826532+0000","last_became_peered":"2023-11-16T18:22:46.826532+0000","last_unstale":"2023-11-16T18:23:39.504758+0000","last_undegraded":"2023-11-16T18:23:39.504758+0000","last_fullsized":"2023-11-16T18:23:39.504758+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:29:24.407020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504437+0000","last_change":"2023-11-16T18:22:46.841164+0000","last_active":"2023-11-16T18:23:39.504437+0000","last_peered":"2023-11-16T18:23:39.504437+0000","last_clean":"2023-11-16T18:23:39.504437+0000","last_became_active":"2023-11-16T18:22:46.840855+0000","last_became_peered":"2023-11-16T18:22:46.840855+0000","last_unstale":"2023-11-16T18:23:39.504437+0000","last_undegraded":"2023-11-16T18:23:39.504437+0000","last_fullsized":"2023-11-16T18:23:39.504437+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:46.380315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507548+0000","last_change":"2023-11-16T18:22:46.822171+0000","last_active":"2023-11-16T18:23:39.507548+0000","last_peered":"2023-11-16T18:23:39.507548+0000","last_clean":"2023-11-16T18:23:39.507548+0000","last_became_active":"2023-11-16T18:22:46.821860+0000","last_became_peered":"2023-11-16T18:22:46.821860+0000","last_unstale":"2023-11-16T18:23:39.507548+0000","last_undegraded":"2023-11-16T18:23:39.507548+0000","last_fullsized":"2023-11-16T18:23:39.507548+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:28:18.589285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502347+0000","last_change":"2023-11-16T18:22:46.832463+0000","last_active":"2023-11-16T18:23:39.502347+0000","last_peered":"2023-11-16T18:23:39.502347+0000","last_clean":"2023-11-16T18:23:39.502347+0000","last_became_active":"2023-11-16T18:22:46.832141+0000","last_became_peered":"2023-11-16T18:22:46.832141+0000","last_unstale":"2023-11-16T18:23:39.502347+0000","last_undegraded":"2023-11-16T18:23:39.502347+0000","last_fullsized":"2023-11-16T18:23:39.502347+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:12:03.361742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504425+0000","last_change":"2023-11-16T18:22:46.844239+0000","last_active":"2023-11-16T18:23:39.504425+0000","last_peered":"2023-11-16T18:23:39.504425+0000","last_clean":"2023-11-16T18:23:39.504425+0000","last_became_active":"2023-11-16T18:22:46.843906+0000","last_became_peered":"2023-11-16T18:22:46.843906+0000","last_unstale":"2023-11-16T18:23:39.504425+0000","last_undegraded":"2023-11-16T18:23:39.504425+0000","last_fullsized":"2023-11-16T18:23:39.504425+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:16:15.202242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508804+0000","last_change":"2023-11-16T18:22:46.834625+0000","last_active":"2023-11-16T18:23:39.508804+0000","last_peered":"2023-11-16T18:23:39.508804+0000","last_clean":"2023-11-16T18:23:39.508804+0000","last_became_active":"2023-11-16T18:22:46.834145+0000","last_became_peered":"2023-11-16T18:22:46.834145+0000","last_unstale":"2023-11-16T18:23:39.508804+0000","last_undegraded":"2023-11-16T18:23:39.508804+0000","last_fullsized":"2023-11-16T18:23:39.508804+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:25:16.196016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509061+0000","last_change":"2023-11-16T18:22:46.827569+0000","last_active":"2023-11-16T18:23:39.509061+0000","last_peered":"2023-11-16T18:23:39.509061+0000","last_clean":"2023-11-16T18:23:39.509061+0000","last_became_active":"2023-11-16T18:22:46.827234+0000","last_became_peered":"2023-11-16T18:22:46.827234+0000","last_unstale":"2023-11-16T18:23:39.509061+0000","last_undegraded":"2023-11-16T18:23:39.509061+0000","last_fullsized":"2023-11-16T18:23:39.509061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:38:30.584402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503061+0000","last_change":"2023-11-16T18:22:46.833701+0000","last_active":"2023-11-16T18:23:39.503061+0000","last_peered":"2023-11-16T18:23:39.503061+0000","last_clean":"2023-11-16T18:23:39.503061+0000","last_became_active":"2023-11-16T18:22:46.833333+0000","last_became_peered":"2023-11-16T18:22:46.833333+0000","last_unstale":"2023-11-16T18:23:39.503061+0000","last_undegraded":"2023-11-16T18:23:39.503061+0000","last_fullsized":"2023-11-16T18:23:39.503061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:31:52.429141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504924+0000","last_change":"2023-11-16T18:22:46.823222+0000","last_active":"2023-11-16T18:23:39.504924+0000","last_peered":"2023-11-16T18:23:39.504924+0000","last_clean":"2023-11-16T18:23:39.504924+0000","last_became_active":"2023-11-16T18:22:46.822224+0000","last_became_peered":"2023-11-16T18:22:46.822224+0000","last_unstale":"2023-11-16T18:23:39.504924+0000","last_undegraded":"2023-11-16T18:23:39.504924+0000","last_fullsized":"2023-11-16T18:23:39.504924+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:23:38.463744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584444+0000","last_change":"2023-11-16T18:22:46.818631+0000","last_active":"2023-11-16T18:23:39.584444+0000","last_peered":"2023-11-16T18:23:39.584444+0000","last_clean":"2023-11-16T18:23:39.584444+0000","last_became_active":"2023-11-16T18:22:46.818305+0000","last_became_peered":"2023-11-16T18:22:46.818305+0000","last_unstale":"2023-11-16T18:23:39.584444+0000","last_undegraded":"2023-11-16T18:23:39.584444+0000","last_fullsized":"2023-11-16T18:23:39.584444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:06:53.083409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503715+0000","last_change":"2023-11-16T18:22:46.825439+0000","last_active":"2023-11-16T18:23:39.503715+0000","last_peered":"2023-11-16T18:23:39.503715+0000","last_clean":"2023-11-16T18:23:39.503715+0000","last_became_active":"2023-11-16T18:22:46.825062+0000","last_became_peered":"2023-11-16T18:22:46.825062+0000","last_unstale":"2023-11-16T18:23:39.503715+0000","last_undegraded":"2023-11-16T18:23:39.503715+0000","last_fullsized":"2023-11-16T18:23:39.503715+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:32:03.819033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503546+0000","last_change":"2023-11-16T18:22:46.827730+0000","last_active":"2023-11-16T18:23:39.503546+0000","last_peered":"2023-11-16T18:23:39.503546+0000","last_clean":"2023-11-16T18:23:39.503546+0000","last_became_active":"2023-11-16T18:22:46.827416+0000","last_became_peered":"2023-11-16T18:22:46.827416+0000","last_unstale":"2023-11-16T18:23:39.503546+0000","last_undegraded":"2023-11-16T18:23:39.503546+0000","last_fullsized":"2023-11-16T18:23:39.503546+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:18:08.691944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507323+0000","last_change":"2023-11-16T18:22:46.834259+0000","last_active":"2023-11-16T18:23:39.507323+0000","last_peered":"2023-11-16T18:23:39.507323+0000","last_clean":"2023-11-16T18:23:39.507323+0000","last_became_active":"2023-11-16T18:22:46.833775+0000","last_became_peered":"2023-11-16T18:22:46.833775+0000","last_unstale":"2023-11-16T18:23:39.507323+0000","last_undegraded":"2023-11-16T18:23:39.507323+0000","last_fullsized":"2023-11-16T18:23:39.507323+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:02:17.762712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506738+0000","last_change":"2023-11-16T18:22:46.829475+0000","last_active":"2023-11-16T18:23:39.506738+0000","last_peered":"2023-11-16T18:23:39.506738+0000","last_clean":"2023-11-16T18:23:39.506738+0000","last_became_active":"2023-11-16T18:22:46.829157+0000","last_became_peered":"2023-11-16T18:22:46.829157+0000","last_unstale":"2023-11-16T18:23:39.506738+0000","last_undegraded":"2023-11-16T18:23:39.506738+0000","last_fullsized":"2023-11-16T18:23:39.506738+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:29.122041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501933+0000","last_change":"2023-11-16T18:22:46.821716+0000","last_active":"2023-11-16T18:23:39.501933+0000","last_peered":"2023-11-16T18:23:39.501933+0000","last_clean":"2023-11-16T18:23:39.501933+0000","last_became_active":"2023-11-16T18:22:46.821393+0000","last_became_peered":"2023-11-16T18:22:46.821393+0000","last_unstale":"2023-11-16T18:23:39.501933+0000","last_undegraded":"2023-11-16T18:23:39.501933+0000","last_fullsized":"2023-11-16T18:23:39.501933+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:10.453727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510952+0000","last_change":"2023-11-16T18:22:46.837433+0000","last_active":"2023-11-16T18:23:39.510952+0000","last_peered":"2023-11-16T18:23:39.510952+0000","last_clean":"2023-11-16T18:23:39.510952+0000","last_became_active":"2023-11-16T18:22:46.837143+0000","last_became_peered":"2023-11-16T18:22:46.837143+0000","last_unstale":"2023-11-16T18:23:39.510952+0000","last_undegraded":"2023-11-16T18:23:39.510952+0000","last_fullsized":"2023-11-16T18:23:39.510952+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:45:40.959535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510180+0000","last_change":"2023-11-16T18:22:51.123749+0000","last_active":"2023-11-16T18:23:39.510180+0000","last_peered":"2023-11-16T18:23:39.510180+0000","last_clean":"2023-11-16T18:23:39.510180+0000","last_became_active":"2023-11-16T18:22:51.123185+0000","last_became_peered":"2023-11-16T18:22:51.123185+0000","last_unstale":"2023-11-16T18:23:39.510180+0000","last_undegraded":"2023-11-16T18:23:39.510180+0000","last_fullsized":"2023-11-16T18:23:39.510180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:11:31.023936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501272+0000","last_change":"2023-11-16T18:22:46.385440+0000","last_active":"2023-11-16T18:23:39.501272+0000","last_peered":"2023-11-16T18:23:39.501272+0000","last_clean":"2023-11-16T18:23:39.501272+0000","last_became_active":"2023-11-16T18:22:46.385098+0000","last_became_peered":"2023-11-16T18:22:46.385098+0000","last_unstale":"2023-11-16T18:23:39.501272+0000","last_undegraded":"2023-11-16T18:23:39.501272+0000","last_fullsized":"2023-11-16T18:23:39.501272+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:08:49.547450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504280+0000","last_change":"2023-11-16T18:22:46.842681+0000","last_active":"2023-11-16T18:23:39.504280+0000","last_peered":"2023-11-16T18:23:39.504280+0000","last_clean":"2023-11-16T18:23:39.504280+0000","last_became_active":"2023-11-16T18:22:46.842408+0000","last_became_peered":"2023-11-16T18:22:46.842408+0000","last_unstale":"2023-11-16T18:23:39.504280+0000","last_undegraded":"2023-11-16T18:23:39.504280+0000","last_fullsized":"2023-11-16T18:23:39.504280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:37:57.050474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503905+0000","last_change":"2023-11-16T18:22:51.123450+0000","last_active":"2023-11-16T18:23:39.503905+0000","last_peered":"2023-11-16T18:23:39.503905+0000","last_clean":"2023-11-16T18:23:39.503905+0000","last_became_active":"2023-11-16T18:22:51.123002+0000","last_became_peered":"2023-11-16T18:22:51.123002+0000","last_unstale":"2023-11-16T18:23:39.503905+0000","last_undegraded":"2023-11-16T18:23:39.503905+0000","last_fullsized":"2023-11-16T18:23:39.503905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:48:43.003189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504416+0000","last_change":"2023-11-16T18:22:46.384768+0000","last_active":"2023-11-16T18:23:39.504416+0000","last_peered":"2023-11-16T18:23:39.504416+0000","last_clean":"2023-11-16T18:23:39.504416+0000","last_became_active":"2023-11-16T18:22:46.384246+0000","last_became_peered":"2023-11-16T18:22:46.384246+0000","last_unstale":"2023-11-16T18:23:39.504416+0000","last_undegraded":"2023-11-16T18:23:39.504416+0000","last_fullsized":"2023-11-16T18:23:39.504416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:20:51.259854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504317+0000","last_change":"2023-11-16T18:22:46.826069+0000","last_active":"2023-11-16T18:23:39.504317+0000","last_peered":"2023-11-16T18:23:39.504317+0000","last_clean":"2023-11-16T18:23:39.504317+0000","last_became_active":"2023-11-16T18:22:46.824676+0000","last_became_peered":"2023-11-16T18:22:46.824676+0000","last_unstale":"2023-11-16T18:23:39.504317+0000","last_undegraded":"2023-11-16T18:23:39.504317+0000","last_fullsized":"2023-11-16T18:23:39.504317+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:36:58.519456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504272+0000","last_change":"2023-11-16T18:22:51.127529+0000","last_active":"2023-11-16T18:23:39.504272+0000","last_peered":"2023-11-16T18:23:39.504272+0000","last_clean":"2023-11-16T18:23:39.504272+0000","last_became_active":"2023-11-16T18:22:51.126748+0000","last_became_peered":"2023-11-16T18:22:51.126748+0000","last_unstale":"2023-11-16T18:23:39.504272+0000","last_undegraded":"2023-11-16T18:23:39.504272+0000","last_fullsized":"2023-11-16T18:23:39.504272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:46:02.931821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505594+0000","last_change":"2023-11-16T18:22:46.385890+0000","last_active":"2023-11-16T18:23:39.505594+0000","last_peered":"2023-11-16T18:23:39.505594+0000","last_clean":"2023-11-16T18:23:39.505594+0000","last_became_active":"2023-11-16T18:22:46.385568+0000","last_became_peered":"2023-11-16T18:22:46.385568+0000","last_unstale":"2023-11-16T18:23:39.505594+0000","last_undegraded":"2023-11-16T18:23:39.505594+0000","last_fullsized":"2023-11-16T18:23:39.505594+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:45:23.088120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508492+0000","last_change":"2023-11-16T18:22:46.824796+0000","last_active":"2023-11-16T18:23:39.508492+0000","last_peered":"2023-11-16T18:23:39.508492+0000","last_clean":"2023-11-16T18:23:39.508492+0000","last_became_active":"2023-11-16T18:22:46.823895+0000","last_became_peered":"2023-11-16T18:22:46.823895+0000","last_unstale":"2023-11-16T18:23:39.508492+0000","last_undegraded":"2023-11-16T18:23:39.508492+0000","last_fullsized":"2023-11-16T18:23:39.508492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:57:28.297178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504555+0000","last_change":"2023-11-16T18:22:51.118714+0000","last_active":"2023-11-16T18:23:39.504555+0000","last_peered":"2023-11-16T18:23:39.504555+0000","last_clean":"2023-11-16T18:23:39.504555+0000","last_became_active":"2023-11-16T18:22:51.118216+0000","last_became_peered":"2023-11-16T18:22:51.118216+0000","last_unstale":"2023-11-16T18:23:39.504555+0000","last_undegraded":"2023-11-16T18:23:39.504555+0000","last_fullsized":"2023-11-16T18:23:39.504555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:27:57.339401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505119+0000","last_change":"2023-11-16T18:22:46.376431+0000","last_active":"2023-11-16T18:23:39.505119+0000","last_peered":"2023-11-16T18:23:39.505119+0000","last_clean":"2023-11-16T18:23:39.505119+0000","last_became_active":"2023-11-16T18:22:46.375914+0000","last_became_peered":"2023-11-16T18:22:46.375914+0000","last_unstale":"2023-11-16T18:23:39.505119+0000","last_undegraded":"2023-11-16T18:23:39.505119+0000","last_fullsized":"2023-11-16T18:23:39.505119+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:31:34.447838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508396+0000","last_change":"2023-11-16T18:22:46.825002+0000","last_active":"2023-11-16T18:23:39.508396+0000","last_peered":"2023-11-16T18:23:39.508396+0000","last_clean":"2023-11-16T18:23:39.508396+0000","last_became_active":"2023-11-16T18:22:46.824012+0000","last_became_peered":"2023-11-16T18:22:46.824012+0000","last_unstale":"2023-11-16T18:23:39.508396+0000","last_undegraded":"2023-11-16T18:23:39.508396+0000","last_fullsized":"2023-11-16T18:23:39.508396+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:04:41.816487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510910+0000","last_change":"2023-11-16T18:22:51.117623+0000","last_active":"2023-11-16T18:23:39.510910+0000","last_peered":"2023-11-16T18:23:39.510910+0000","last_clean":"2023-11-16T18:23:39.510910+0000","last_became_active":"2023-11-16T18:22:51.117183+0000","last_became_peered":"2023-11-16T18:22:51.117183+0000","last_unstale":"2023-11-16T18:23:39.510910+0000","last_undegraded":"2023-11-16T18:23:39.510910+0000","last_fullsized":"2023-11-16T18:23:39.510910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:16:00.063607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":86,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:45.523791+0000","last_change":"2023-11-16T18:23:39.664514+0000","last_active":"2023-11-16T18:23:45.523791+0000","last_peered":"2023-11-16T18:23:45.523791+0000","last_clean":"2023-11-16T18:23:45.523791+0000","last_became_active":"2023-11-16T18:23:39.511648+0000","last_became_peered":"2023-11-16T18:23:39.511648+0000","last_unstale":"2023-11-16T18:23:45.523791+0000","last_undegraded":"2023-11-16T18:23:45.523791+0000","last_fullsized":"2023-11-16T18:23:45.523791+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:08:22.914647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505046+0000","last_change":"2023-11-16T18:22:46.820073+0000","last_active":"2023-11-16T18:23:39.505046+0000","last_peered":"2023-11-16T18:23:39.505046+0000","last_clean":"2023-11-16T18:23:39.505046+0000","last_became_active":"2023-11-16T18:22:46.819764+0000","last_became_peered":"2023-11-16T18:22:46.819764+0000","last_unstale":"2023-11-16T18:23:39.505046+0000","last_undegraded":"2023-11-16T18:23:39.505046+0000","last_fullsized":"2023-11-16T18:23:39.505046+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:58.766286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510964+0000","last_change":"2023-11-16T18:22:51.116619+0000","last_active":"2023-11-16T18:23:39.510964+0000","last_peered":"2023-11-16T18:23:39.510964+0000","last_clean":"2023-11-16T18:23:39.510964+0000","last_became_active":"2023-11-16T18:22:51.116264+0000","last_became_peered":"2023-11-16T18:22:51.116264+0000","last_unstale":"2023-11-16T18:23:39.510964+0000","last_undegraded":"2023-11-16T18:23:39.510964+0000","last_fullsized":"2023-11-16T18:23:39.510964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:32:46.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507700+0000","last_change":"2023-11-16T18:22:46.389590+0000","last_active":"2023-11-16T18:23:39.507700+0000","last_peered":"2023-11-16T18:23:39.507700+0000","last_clean":"2023-11-16T18:23:39.507700+0000","last_became_active":"2023-11-16T18:22:46.389354+0000","last_became_peered":"2023-11-16T18:22:46.389354+0000","last_unstale":"2023-11-16T18:23:39.507700+0000","last_undegraded":"2023-11-16T18:23:39.507700+0000","last_fullsized":"2023-11-16T18:23:39.507700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:58:58.086957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509671+0000","last_change":"2023-11-16T18:22:46.841106+0000","last_active":"2023-11-16T18:23:39.509671+0000","last_peered":"2023-11-16T18:23:39.509671+0000","last_clean":"2023-11-16T18:23:39.509671+0000","last_became_active":"2023-11-16T18:22:46.840876+0000","last_became_peered":"2023-11-16T18:22:46.840876+0000","last_unstale":"2023-11-16T18:23:39.509671+0000","last_undegraded":"2023-11-16T18:23:39.509671+0000","last_fullsized":"2023-11-16T18:23:39.509671+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:46:23.740026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506599+0000","last_change":"2023-11-16T18:22:51.125813+0000","last_active":"2023-11-16T18:23:39.506599+0000","last_peered":"2023-11-16T18:23:39.506599+0000","last_clean":"2023-11-16T18:23:39.506599+0000","last_became_active":"2023-11-16T18:22:51.125277+0000","last_became_peered":"2023-11-16T18:22:51.125277+0000","last_unstale":"2023-11-16T18:23:39.506599+0000","last_undegraded":"2023-11-16T18:23:39.506599+0000","last_fullsized":"2023-11-16T18:23:39.506599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:02:51.083984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584572+0000","last_change":"2023-11-16T18:22:46.381208+0000","last_active":"2023-11-16T18:23:39.584572+0000","last_peered":"2023-11-16T18:23:39.584572+0000","last_clean":"2023-11-16T18:23:39.584572+0000","last_became_active":"2023-11-16T18:22:46.380351+0000","last_became_peered":"2023-11-16T18:22:46.380351+0000","last_unstale":"2023-11-16T18:23:39.584572+0000","last_undegraded":"2023-11-16T18:23:39.584572+0000","last_fullsized":"2023-11-16T18:23:39.584572+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:27:31.749114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504765+0000","last_change":"2023-11-16T18:22:46.843043+0000","last_active":"2023-11-16T18:23:39.504765+0000","last_peered":"2023-11-16T18:23:39.504765+0000","last_clean":"2023-11-16T18:23:39.504765+0000","last_became_active":"2023-11-16T18:22:46.841522+0000","last_became_peered":"2023-11-16T18:22:46.841522+0000","last_unstale":"2023-11-16T18:23:39.504765+0000","last_undegraded":"2023-11-16T18:23:39.504765+0000","last_fullsized":"2023-11-16T18:23:39.504765+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:53:29.309989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505013+0000","last_change":"2023-11-16T18:22:51.127729+0000","last_active":"2023-11-16T18:23:39.505013+0000","last_peered":"2023-11-16T18:23:39.505013+0000","last_clean":"2023-11-16T18:23:39.505013+0000","last_became_active":"2023-11-16T18:22:51.127303+0000","last_became_peered":"2023-11-16T18:22:51.127303+0000","last_unstale":"2023-11-16T18:23:39.505013+0000","last_undegraded":"2023-11-16T18:23:39.505013+0000","last_fullsized":"2023-11-16T18:23:39.505013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:16:37.982862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508172+0000","last_change":"2023-11-16T18:22:46.377437+0000","last_active":"2023-11-16T18:23:39.508172+0000","last_peered":"2023-11-16T18:23:39.508172+0000","last_clean":"2023-11-16T18:23:39.508172+0000","last_became_active":"2023-11-16T18:22:46.377120+0000","last_became_peered":"2023-11-16T18:22:46.377120+0000","last_unstale":"2023-11-16T18:23:39.508172+0000","last_undegraded":"2023-11-16T18:23:39.508172+0000","last_fullsized":"2023-11-16T18:23:39.508172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:02:31.673863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500756+0000","last_change":"2023-11-16T18:22:46.837804+0000","last_active":"2023-11-16T18:23:39.500756+0000","last_peered":"2023-11-16T18:23:39.500756+0000","last_clean":"2023-11-16T18:23:39.500756+0000","last_became_active":"2023-11-16T18:22:46.837432+0000","last_became_peered":"2023-11-16T18:22:46.837432+0000","last_unstale":"2023-11-16T18:23:39.500756+0000","last_undegraded":"2023-11-16T18:23:39.500756+0000","last_fullsized":"2023-11-16T18:23:39.500756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:10:14.516300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503848+0000","last_change":"2023-11-16T18:22:51.127546+0000","last_active":"2023-11-16T18:23:39.503848+0000","last_peered":"2023-11-16T18:23:39.503848+0000","last_clean":"2023-11-16T18:23:39.503848+0000","last_became_active":"2023-11-16T18:22:51.127133+0000","last_became_peered":"2023-11-16T18:22:51.127133+0000","last_unstale":"2023-11-16T18:23:39.503848+0000","last_undegraded":"2023-11-16T18:23:39.503848+0000","last_fullsized":"2023-11-16T18:23:39.503848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:02:33.244659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:43.520432+0000","last_change":"2023-11-16T18:22:46.386309+0000","last_active":"2023-11-16T18:23:43.520432+0000","last_peered":"2023-11-16T18:23:43.520432+0000","last_clean":"2023-11-16T18:23:43.520432+0000","last_became_active":"2023-11-16T18:22:46.385960+0000","last_became_peered":"2023-11-16T18:22:46.385960+0000","last_unstale":"2023-11-16T18:23:43.520432+0000","last_undegraded":"2023-11-16T18:23:43.520432+0000","last_fullsized":"2023-11-16T18:23:43.520432+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:46:45.743666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508584+0000","last_change":"2023-11-16T18:22:46.827211+0000","last_active":"2023-11-16T18:23:39.508584+0000","last_peered":"2023-11-16T18:23:39.508584+0000","last_clean":"2023-11-16T18:23:39.508584+0000","last_became_active":"2023-11-16T18:22:46.826396+0000","last_became_peered":"2023-11-16T18:22:46.826396+0000","last_unstale":"2023-11-16T18:23:39.508584+0000","last_undegraded":"2023-11-16T18:23:39.508584+0000","last_fullsized":"2023-11-16T18:23:39.508584+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:58:36.202010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.513065+0000","last_change":"2023-11-16T18:23:39.512554+0000","last_active":"2023-11-16T18:23:39.513065+0000","last_peered":"2023-11-16T18:23:39.513065+0000","last_clean":"2023-11-16T18:23:39.513065+0000","last_became_active":"2023-11-16T18:23:39.512118+0000","last_became_peered":"2023-11-16T18:23:39.512118+0000","last_unstale":"2023-11-16T18:23:39.513065+0000","last_undegraded":"2023-11-16T18:23:39.513065+0000","last_fullsized":"2023-11-16T18:23:39.513065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:55:33.269875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503214+0000","last_change":"2023-11-16T18:22:46.384680+0000","last_active":"2023-11-16T18:23:39.503214+0000","last_peered":"2023-11-16T18:23:39.503214+0000","last_clean":"2023-11-16T18:23:39.503214+0000","last_became_active":"2023-11-16T18:22:46.384290+0000","last_became_peered":"2023-11-16T18:22:46.384290+0000","last_unstale":"2023-11-16T18:23:39.503214+0000","last_undegraded":"2023-11-16T18:23:39.503214+0000","last_fullsized":"2023-11-16T18:23:39.503214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:05:40.782246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504447+0000","last_change":"2023-11-16T18:22:46.834726+0000","last_active":"2023-11-16T18:23:39.504447+0000","last_peered":"2023-11-16T18:23:39.504447+0000","last_clean":"2023-11-16T18:23:39.504447+0000","last_became_active":"2023-11-16T18:22:46.834424+0000","last_became_peered":"2023-11-16T18:22:46.834424+0000","last_unstale":"2023-11-16T18:23:39.504447+0000","last_undegraded":"2023-11-16T18:23:39.504447+0000","last_fullsized":"2023-11-16T18:23:39.504447+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:52:11.738947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.513064+0000","last_change":"2023-11-16T18:23:39.513064+0000","last_active":"2023-11-16T18:23:39.513064+0000","last_peered":"2023-11-16T18:23:39.513064+0000","last_clean":"2023-11-16T18:23:39.513064+0000","last_became_active":"2023-11-16T18:23:39.512606+0000","last_became_peered":"2023-11-16T18:23:39.512606+0000","last_unstale":"2023-11-16T18:23:39.513064+0000","last_undegraded":"2023-11-16T18:23:39.513064+0000","last_fullsized":"2023-11-16T18:23:39.513064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:08:18.327854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508258+0000","last_change":"2023-11-16T18:22:46.380804+0000","last_active":"2023-11-16T18:23:39.508258+0000","last_peered":"2023-11-16T18:23:39.508258+0000","last_clean":"2023-11-16T18:23:39.508258+0000","last_became_active":"2023-11-16T18:22:46.380477+0000","last_became_peered":"2023-11-16T18:22:46.380477+0000","last_unstale":"2023-11-16T18:23:39.508258+0000","last_undegraded":"2023-11-16T18:23:39.508258+0000","last_fullsized":"2023-11-16T18:23:39.508258+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:17:45.128995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.586650+0000","last_change":"2023-11-16T18:23:39.586650+0000","last_active":"2023-11-16T18:23:39.586650+0000","last_peered":"2023-11-16T18:23:39.586650+0000","last_clean":"2023-11-16T18:23:39.586650+0000","last_became_active":"2023-11-16T18:23:39.586241+0000","last_became_peered":"2023-11-16T18:23:39.586241+0000","last_unstale":"2023-11-16T18:23:39.586650+0000","last_undegraded":"2023-11-16T18:23:39.586650+0000","last_fullsized":"2023-11-16T18:23:39.586650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:15:14.907282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500754+0000","last_change":"2023-11-16T18:22:51.119711+0000","last_active":"2023-11-16T18:23:39.500754+0000","last_peered":"2023-11-16T18:23:39.500754+0000","last_clean":"2023-11-16T18:23:39.500754+0000","last_became_active":"2023-11-16T18:22:51.119304+0000","last_became_peered":"2023-11-16T18:22:51.119304+0000","last_unstale":"2023-11-16T18:23:39.500754+0000","last_undegraded":"2023-11-16T18:23:39.500754+0000","last_fullsized":"2023-11-16T18:23:39.500754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:37:34.450684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"86'31","reported_seq":157,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:44.523169+0000","last_change":"2023-11-16T18:23:39.671486+0000","last_active":"2023-11-16T18:23:44.523169+0000","last_peered":"2023-11-16T18:23:44.523169+0000","last_clean":"2023-11-16T18:23:44.523169+0000","last_became_active":"2023-11-16T18:23:39.513413+0000","last_became_peered":"2023-11-16T18:23:39.513413+0000","last_unstale":"2023-11-16T18:23:44.523169+0000","last_undegraded":"2023-11-16T18:23:44.523169+0000","last_fullsized":"2023-11-16T18:23:44.523169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:07:37.199058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60837,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":77,"num_read_kb":96,"num_write":32,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60837,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504736+0000","last_change":"2023-11-16T18:22:46.833537+0000","last_active":"2023-11-16T18:23:39.504736+0000","last_peered":"2023-11-16T18:23:39.504736+0000","last_clean":"2023-11-16T18:23:39.504736+0000","last_became_active":"2023-11-16T18:22:46.833217+0000","last_became_peered":"2023-11-16T18:22:46.833217+0000","last_unstale":"2023-11-16T18:23:39.504736+0000","last_undegraded":"2023-11-16T18:23:39.504736+0000","last_fullsized":"2023-11-16T18:23:39.504736+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:54:32.987198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585712+0000","last_change":"2023-11-16T18:23:39.585247+0000","last_active":"2023-11-16T18:23:39.585712+0000","last_peered":"2023-11-16T18:23:39.585712+0000","last_clean":"2023-11-16T18:23:39.585712+0000","last_became_active":"2023-11-16T18:23:39.584802+0000","last_became_peered":"2023-11-16T18:23:39.584802+0000","last_unstale":"2023-11-16T18:23:39.585712+0000","last_undegraded":"2023-11-16T18:23:39.585712+0000","last_fullsized":"2023-11-16T18:23:39.585712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:45:43.860286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509671+0000","last_change":"2023-11-16T18:22:46.382812+0000","last_active":"2023-11-16T18:23:39.509671+0000","last_peered":"2023-11-16T18:23:39.509671+0000","last_clean":"2023-11-16T18:23:39.509671+0000","last_became_active":"2023-11-16T18:22:46.382499+0000","last_became_peered":"2023-11-16T18:22:46.382499+0000","last_unstale":"2023-11-16T18:23:39.509671+0000","last_undegraded":"2023-11-16T18:23:39.509671+0000","last_fullsized":"2023-11-16T18:23:39.509671+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:48:39.263999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584764+0000","last_change":"2023-11-16T18:22:46.840917+0000","last_active":"2023-11-16T18:23:39.584764+0000","last_peered":"2023-11-16T18:23:39.584764+0000","last_clean":"2023-11-16T18:23:39.584764+0000","last_became_active":"2023-11-16T18:22:46.840579+0000","last_became_peered":"2023-11-16T18:22:46.840579+0000","last_unstale":"2023-11-16T18:23:39.584764+0000","last_undegraded":"2023-11-16T18:23:39.584764+0000","last_fullsized":"2023-11-16T18:23:39.584764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:05:17.920819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506477+0000","last_change":"2023-11-16T18:22:51.117852+0000","last_active":"2023-11-16T18:23:39.506477+0000","last_peered":"2023-11-16T18:23:39.506477+0000","last_clean":"2023-11-16T18:23:39.506477+0000","last_became_active":"2023-11-16T18:22:51.117390+0000","last_became_peered":"2023-11-16T18:22:51.117390+0000","last_unstale":"2023-11-16T18:23:39.506477+0000","last_undegraded":"2023-11-16T18:23:39.506477+0000","last_fullsized":"2023-11-16T18:23:39.506477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:11:37.167591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508136+0000","last_change":"2023-11-16T18:22:46.385137+0000","last_active":"2023-11-16T18:23:39.508136+0000","last_peered":"2023-11-16T18:23:39.508136+0000","last_clean":"2023-11-16T18:23:39.508136+0000","last_became_active":"2023-11-16T18:22:46.384636+0000","last_became_peered":"2023-11-16T18:22:46.384636+0000","last_unstale":"2023-11-16T18:23:39.508136+0000","last_undegraded":"2023-11-16T18:23:39.508136+0000","last_fullsized":"2023-11-16T18:23:39.508136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:11:10.885960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502619+0000","last_change":"2023-11-16T18:22:46.834786+0000","last_active":"2023-11-16T18:23:39.502619+0000","last_peered":"2023-11-16T18:23:39.502619+0000","last_clean":"2023-11-16T18:23:39.502619+0000","last_became_active":"2023-11-16T18:22:46.834338+0000","last_became_peered":"2023-11-16T18:22:46.834338+0000","last_unstale":"2023-11-16T18:23:39.502619+0000","last_undegraded":"2023-11-16T18:23:39.502619+0000","last_fullsized":"2023-11-16T18:23:39.502619+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:52:01.046978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500921+0000","last_change":"2023-11-16T18:22:51.124667+0000","last_active":"2023-11-16T18:23:39.500921+0000","last_peered":"2023-11-16T18:23:39.500921+0000","last_clean":"2023-11-16T18:23:39.500921+0000","last_became_active":"2023-11-16T18:22:51.124211+0000","last_became_peered":"2023-11-16T18:22:51.124211+0000","last_unstale":"2023-11-16T18:23:39.500921+0000","last_undegraded":"2023-11-16T18:23:39.500921+0000","last_fullsized":"2023-11-16T18:23:39.500921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:22:02.360209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:43.976174+0000","last_change":"2023-11-16T18:22:46.380829+0000","last_active":"2023-11-16T18:23:43.976174+0000","last_peered":"2023-11-16T18:23:43.976174+0000","last_clean":"2023-11-16T18:23:43.976174+0000","last_became_active":"2023-11-16T18:22:46.380466+0000","last_became_peered":"2023-11-16T18:22:46.380466+0000","last_unstale":"2023-11-16T18:23:43.976174+0000","last_undegraded":"2023-11-16T18:23:43.976174+0000","last_fullsized":"2023-11-16T18:23:43.976174+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:22:52.401291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502257+0000","last_change":"2023-11-16T18:22:46.835900+0000","last_active":"2023-11-16T18:23:39.502257+0000","last_peered":"2023-11-16T18:23:39.502257+0000","last_clean":"2023-11-16T18:23:39.502257+0000","last_became_active":"2023-11-16T18:22:46.835438+0000","last_became_peered":"2023-11-16T18:22:46.835438+0000","last_unstale":"2023-11-16T18:23:39.502257+0000","last_undegraded":"2023-11-16T18:23:39.502257+0000","last_fullsized":"2023-11-16T18:23:39.502257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:36:23.733440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585779+0000","last_change":"2023-11-16T18:22:51.120005+0000","last_active":"2023-11-16T18:23:39.585779+0000","last_peered":"2023-11-16T18:23:39.585779+0000","last_clean":"2023-11-16T18:23:39.585779+0000","last_became_active":"2023-11-16T18:22:51.119540+0000","last_became_peered":"2023-11-16T18:22:51.119540+0000","last_unstale":"2023-11-16T18:23:39.585779+0000","last_undegraded":"2023-11-16T18:23:39.585779+0000","last_fullsized":"2023-11-16T18:23:39.585779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:33:29.490015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:41.897854+0000","last_change":"2023-11-16T18:23:41.897338+0000","last_active":"2023-11-16T18:23:41.897854+0000","last_peered":"2023-11-16T18:23:41.897854+0000","last_clean":"2023-11-16T18:23:41.897854+0000","last_became_active":"2023-11-16T18:23:39.512018+0000","last_became_peered":"2023-11-16T18:23:39.512018+0000","last_unstale":"2023-11-16T18:23:41.897854+0000","last_undegraded":"2023-11-16T18:23:41.897854+0000","last_fullsized":"2023-11-16T18:23:41.897854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:53:22.707297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507902+0000","last_change":"2023-11-16T18:22:46.823838+0000","last_active":"2023-11-16T18:23:39.507902+0000","last_peered":"2023-11-16T18:23:39.507902+0000","last_clean":"2023-11-16T18:23:39.507902+0000","last_became_active":"2023-11-16T18:22:46.823545+0000","last_became_peered":"2023-11-16T18:22:46.823545+0000","last_unstale":"2023-11-16T18:23:39.507902+0000","last_undegraded":"2023-11-16T18:23:39.507902+0000","last_fullsized":"2023-11-16T18:23:39.507902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:11:32.214804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507865+0000","last_change":"2023-11-16T18:22:51.127371+0000","last_active":"2023-11-16T18:23:39.507865+0000","last_peered":"2023-11-16T18:23:39.507865+0000","last_clean":"2023-11-16T18:23:39.507865+0000","last_became_active":"2023-11-16T18:22:51.126951+0000","last_became_peered":"2023-11-16T18:22:51.126951+0000","last_unstale":"2023-11-16T18:23:39.507865+0000","last_undegraded":"2023-11-16T18:23:39.507865+0000","last_fullsized":"2023-11-16T18:23:39.507865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:55.418916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504323+0000","last_change":"2023-11-16T18:22:46.376561+0000","last_active":"2023-11-16T18:23:39.504323+0000","last_peered":"2023-11-16T18:23:39.504323+0000","last_clean":"2023-11-16T18:23:39.504323+0000","last_became_active":"2023-11-16T18:22:46.375426+0000","last_became_peered":"2023-11-16T18:22:46.375426+0000","last_unstale":"2023-11-16T18:23:39.504323+0000","last_undegraded":"2023-11-16T18:23:39.504323+0000","last_fullsized":"2023-11-16T18:23:39.504323+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:29:00.383105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505100+0000","last_change":"2023-11-16T18:22:46.826657+0000","last_active":"2023-11-16T18:23:39.505100+0000","last_peered":"2023-11-16T18:23:39.505100+0000","last_clean":"2023-11-16T18:23:39.505100+0000","last_became_active":"2023-11-16T18:22:46.826126+0000","last_became_peered":"2023-11-16T18:22:46.826126+0000","last_unstale":"2023-11-16T18:23:39.505100+0000","last_undegraded":"2023-11-16T18:23:39.505100+0000","last_fullsized":"2023-11-16T18:23:39.505100+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:13:34.794628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504904+0000","last_change":"2023-11-16T18:22:51.127949+0000","last_active":"2023-11-16T18:23:39.504904+0000","last_peered":"2023-11-16T18:23:39.504904+0000","last_clean":"2023-11-16T18:23:39.504904+0000","last_became_active":"2023-11-16T18:22:51.127388+0000","last_became_peered":"2023-11-16T18:22:51.127388+0000","last_unstale":"2023-11-16T18:23:39.504904+0000","last_undegraded":"2023-11-16T18:23:39.504904+0000","last_fullsized":"2023-11-16T18:23:39.504904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:37:30.745978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500985+0000","last_change":"2023-11-16T18:22:46.378568+0000","last_active":"2023-11-16T18:23:39.500985+0000","last_peered":"2023-11-16T18:23:39.500985+0000","last_clean":"2023-11-16T18:23:39.500985+0000","last_became_active":"2023-11-16T18:22:46.378018+0000","last_became_peered":"2023-11-16T18:22:46.378018+0000","last_unstale":"2023-11-16T18:23:39.500985+0000","last_undegraded":"2023-11-16T18:23:39.500985+0000","last_fullsized":"2023-11-16T18:23:39.500985+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:05:59.328232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509189+0000","last_change":"2023-11-16T18:22:46.841129+0000","last_active":"2023-11-16T18:23:39.509189+0000","last_peered":"2023-11-16T18:23:39.509189+0000","last_clean":"2023-11-16T18:23:39.509189+0000","last_became_active":"2023-11-16T18:22:46.840780+0000","last_became_peered":"2023-11-16T18:22:46.840780+0000","last_unstale":"2023-11-16T18:23:39.509189+0000","last_undegraded":"2023-11-16T18:23:39.509189+0000","last_fullsized":"2023-11-16T18:23:39.509189+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:09:39.912158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585594+0000","last_change":"2023-11-16T18:22:51.126221+0000","last_active":"2023-11-16T18:23:39.585594+0000","last_peered":"2023-11-16T18:23:39.585594+0000","last_clean":"2023-11-16T18:23:39.585594+0000","last_became_active":"2023-11-16T18:22:51.125770+0000","last_became_peered":"2023-11-16T18:22:51.125770+0000","last_unstale":"2023-11-16T18:23:39.585594+0000","last_undegraded":"2023-11-16T18:23:39.585594+0000","last_fullsized":"2023-11-16T18:23:39.585594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:35:06.867124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502402+0000","last_change":"2023-11-16T18:22:46.384872+0000","last_active":"2023-11-16T18:23:39.502402+0000","last_peered":"2023-11-16T18:23:39.502402+0000","last_clean":"2023-11-16T18:23:39.502402+0000","last_became_active":"2023-11-16T18:22:46.384499+0000","last_became_peered":"2023-11-16T18:22:46.384499+0000","last_unstale":"2023-11-16T18:23:39.502402+0000","last_undegraded":"2023-11-16T18:23:39.502402+0000","last_fullsized":"2023-11-16T18:23:39.502402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:26:53.535471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507989+0000","last_change":"2023-11-16T18:22:46.822574+0000","last_active":"2023-11-16T18:23:39.507989+0000","last_peered":"2023-11-16T18:23:39.507989+0000","last_clean":"2023-11-16T18:23:39.507989+0000","last_became_active":"2023-11-16T18:22:46.822230+0000","last_became_peered":"2023-11-16T18:22:46.822230+0000","last_unstale":"2023-11-16T18:23:39.507989+0000","last_undegraded":"2023-11-16T18:23:39.507989+0000","last_fullsized":"2023-11-16T18:23:39.507989+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:16:44.677701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585706+0000","last_change":"2023-11-16T18:22:51.117838+0000","last_active":"2023-11-16T18:23:39.585706+0000","last_peered":"2023-11-16T18:23:39.585706+0000","last_clean":"2023-11-16T18:23:39.585706+0000","last_became_active":"2023-11-16T18:22:51.117275+0000","last_became_peered":"2023-11-16T18:22:51.117275+0000","last_unstale":"2023-11-16T18:23:39.585706+0000","last_undegraded":"2023-11-16T18:23:39.585706+0000","last_fullsized":"2023-11-16T18:23:39.585706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:42:14.482945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508453+0000","last_change":"2023-11-16T18:22:46.827499+0000","last_active":"2023-11-16T18:23:39.508453+0000","last_peered":"2023-11-16T18:23:39.508453+0000","last_clean":"2023-11-16T18:23:39.508453+0000","last_became_active":"2023-11-16T18:22:46.827166+0000","last_became_peered":"2023-11-16T18:22:46.827166+0000","last_unstale":"2023-11-16T18:23:39.508453+0000","last_undegraded":"2023-11-16T18:23:39.508453+0000","last_fullsized":"2023-11-16T18:23:39.508453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:23:43.219136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:46.980203+0000","last_change":"2023-11-16T18:22:46.381397+0000","last_active":"2023-11-16T18:23:46.980203+0000","last_peered":"2023-11-16T18:23:46.980203+0000","last_clean":"2023-11-16T18:23:46.980203+0000","last_became_active":"2023-11-16T18:22:46.380939+0000","last_became_peered":"2023-11-16T18:22:46.380939+0000","last_unstale":"2023-11-16T18:23:46.980203+0000","last_undegraded":"2023-11-16T18:23:46.980203+0000","last_fullsized":"2023-11-16T18:23:46.980203+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:44:30.574771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502915+0000","last_change":"2023-11-16T18:22:51.125044+0000","last_active":"2023-11-16T18:23:39.502915+0000","last_peered":"2023-11-16T18:23:39.502915+0000","last_clean":"2023-11-16T18:23:39.502915+0000","last_became_active":"2023-11-16T18:22:51.124422+0000","last_became_peered":"2023-11-16T18:22:51.124422+0000","last_unstale":"2023-11-16T18:23:39.502915+0000","last_undegraded":"2023-11-16T18:23:39.502915+0000","last_fullsized":"2023-11-16T18:23:39.502915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:11.651599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503647+0000","last_change":"2023-11-16T18:22:46.841500+0000","last_active":"2023-11-16T18:23:39.503647+0000","last_peered":"2023-11-16T18:23:39.503647+0000","last_clean":"2023-11-16T18:23:39.503647+0000","last_became_active":"2023-11-16T18:22:46.841186+0000","last_became_peered":"2023-11-16T18:22:46.841186+0000","last_unstale":"2023-11-16T18:23:39.503647+0000","last_undegraded":"2023-11-16T18:23:39.503647+0000","last_fullsized":"2023-11-16T18:23:39.503647+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:38:25.407837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504887+0000","last_change":"2023-11-16T18:22:46.380988+0000","last_active":"2023-11-16T18:23:39.504887+0000","last_peered":"2023-11-16T18:23:39.504887+0000","last_clean":"2023-11-16T18:23:39.504887+0000","last_became_active":"2023-11-16T18:22:46.380616+0000","last_became_peered":"2023-11-16T18:22:46.380616+0000","last_unstale":"2023-11-16T18:23:39.504887+0000","last_undegraded":"2023-11-16T18:23:39.504887+0000","last_fullsized":"2023-11-16T18:23:39.504887+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:24:26.911536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502037+0000","last_change":"2023-11-16T18:21:44.282706+0000","last_active":"2023-11-16T18:23:39.502037+0000","last_peered":"2023-11-16T18:23:39.502037+0000","last_clean":"2023-11-16T18:23:39.502037+0000","last_became_active":"2023-11-16T18:21:44.282201+0000","last_became_peered":"2023-11-16T18:21:44.282201+0000","last_unstale":"2023-11-16T18:23:39.502037+0000","last_undegraded":"2023-11-16T18:23:39.502037+0000","last_fullsized":"2023-11-16T18:23:39.502037+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_clean_scrub_stamp":"2023-11-16T18:18:46.711191+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:41:17.702365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508868+0000","last_change":"2023-11-16T18:22:51.123051+0000","last_active":"2023-11-16T18:23:39.508868+0000","last_peered":"2023-11-16T18:23:39.508868+0000","last_clean":"2023-11-16T18:23:39.508868+0000","last_became_active":"2023-11-16T18:22:51.122635+0000","last_became_peered":"2023-11-16T18:22:51.122635+0000","last_unstale":"2023-11-16T18:23:39.508868+0000","last_undegraded":"2023-11-16T18:23:39.508868+0000","last_fullsized":"2023-11-16T18:23:39.508868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:59:09.977278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584296+0000","last_change":"2023-11-16T18:22:46.822625+0000","last_active":"2023-11-16T18:23:39.584296+0000","last_peered":"2023-11-16T18:23:39.584296+0000","last_clean":"2023-11-16T18:23:39.584296+0000","last_became_active":"2023-11-16T18:22:46.822296+0000","last_became_peered":"2023-11-16T18:22:46.822296+0000","last_unstale":"2023-11-16T18:23:39.584296+0000","last_undegraded":"2023-11-16T18:23:39.584296+0000","last_fullsized":"2023-11-16T18:23:39.584296+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:43:40.832756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502155+0000","last_change":"2023-11-16T18:22:46.380445+0000","last_active":"2023-11-16T18:23:39.502155+0000","last_peered":"2023-11-16T18:23:39.502155+0000","last_clean":"2023-11-16T18:23:39.502155+0000","last_became_active":"2023-11-16T18:22:46.380113+0000","last_became_peered":"2023-11-16T18:22:46.380113+0000","last_unstale":"2023-11-16T18:23:39.502155+0000","last_undegraded":"2023-11-16T18:23:39.502155+0000","last_fullsized":"2023-11-16T18:23:39.502155+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:17:48.769928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504943+0000","last_change":"2023-11-16T18:22:51.118457+0000","last_active":"2023-11-16T18:23:39.504943+0000","last_peered":"2023-11-16T18:23:39.504943+0000","last_clean":"2023-11-16T18:23:39.504943+0000","last_became_active":"2023-11-16T18:22:51.118013+0000","last_became_peered":"2023-11-16T18:22:51.118013+0000","last_unstale":"2023-11-16T18:23:39.504943+0000","last_undegraded":"2023-11-16T18:23:39.504943+0000","last_fullsized":"2023-11-16T18:23:39.504943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:11.432801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506267+0000","last_change":"2023-11-16T18:22:46.384799+0000","last_active":"2023-11-16T18:23:39.506267+0000","last_peered":"2023-11-16T18:23:39.506267+0000","last_clean":"2023-11-16T18:23:39.506267+0000","last_became_active":"2023-11-16T18:22:46.384482+0000","last_became_peered":"2023-11-16T18:22:46.384482+0000","last_unstale":"2023-11-16T18:23:39.506267+0000","last_undegraded":"2023-11-16T18:23:39.506267+0000","last_fullsized":"2023-11-16T18:23:39.506267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:42:48.220989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585377+0000","last_change":"2023-11-16T18:22:46.829541+0000","last_active":"2023-11-16T18:23:39.585377+0000","last_peered":"2023-11-16T18:23:39.585377+0000","last_clean":"2023-11-16T18:23:39.585377+0000","last_became_active":"2023-11-16T18:22:46.829006+0000","last_became_peered":"2023-11-16T18:22:46.829006+0000","last_unstale":"2023-11-16T18:23:39.585377+0000","last_undegraded":"2023-11-16T18:23:39.585377+0000","last_fullsized":"2023-11-16T18:23:39.585377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:13:02.956336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504640+0000","last_change":"2023-11-16T18:22:51.122751+0000","last_active":"2023-11-16T18:23:39.504640+0000","last_peered":"2023-11-16T18:23:39.504640+0000","last_clean":"2023-11-16T18:23:39.504640+0000","last_became_active":"2023-11-16T18:22:51.122423+0000","last_became_peered":"2023-11-16T18:22:51.122423+0000","last_unstale":"2023-11-16T18:23:39.504640+0000","last_undegraded":"2023-11-16T18:23:39.504640+0000","last_fullsized":"2023-11-16T18:23:39.504640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:27:49.338555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'5","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.516672+0000","last_change":"2023-11-16T18:23:39.516672+0000","last_active":"2023-11-16T18:23:39.516672+0000","last_peered":"2023-11-16T18:23:39.516672+0000","last_clean":"2023-11-16T18:23:39.516672+0000","last_became_active":"2023-11-16T18:23:39.509087+0000","last_became_peered":"2023-11-16T18:23:39.509087+0000","last_unstale":"2023-11-16T18:23:39.516672+0000","last_undegraded":"2023-11-16T18:23:39.516672+0000","last_fullsized":"2023-11-16T18:23:39.516672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:57:28.479767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506041+0000","last_change":"2023-11-16T18:22:46.843802+0000","last_active":"2023-11-16T18:23:39.506041+0000","last_peered":"2023-11-16T18:23:39.506041+0000","last_clean":"2023-11-16T18:23:39.506041+0000","last_became_active":"2023-11-16T18:22:46.842210+0000","last_became_peered":"2023-11-16T18:22:46.842210+0000","last_unstale":"2023-11-16T18:23:39.506041+0000","last_undegraded":"2023-11-16T18:23:39.506041+0000","last_fullsized":"2023-11-16T18:23:39.506041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:20:02.893140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509951+0000","last_change":"2023-11-16T18:22:51.124632+0000","last_active":"2023-11-16T18:23:39.509951+0000","last_peered":"2023-11-16T18:23:39.509951+0000","last_clean":"2023-11-16T18:23:39.509951+0000","last_became_active":"2023-11-16T18:22:51.124220+0000","last_became_peered":"2023-11-16T18:22:51.124220+0000","last_unstale":"2023-11-16T18:23:39.509951+0000","last_undegraded":"2023-11-16T18:23:39.509951+0000","last_fullsized":"2023-11-16T18:23:39.509951+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:51:57.981627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505140+0000","last_change":"2023-11-16T18:22:46.381456+0000","last_active":"2023-11-16T18:23:39.505140+0000","last_peered":"2023-11-16T18:23:39.505140+0000","last_clean":"2023-11-16T18:23:39.505140+0000","last_became_active":"2023-11-16T18:22:46.381137+0000","last_became_peered":"2023-11-16T18:22:46.381137+0000","last_unstale":"2023-11-16T18:23:39.505140+0000","last_undegraded":"2023-11-16T18:23:39.505140+0000","last_fullsized":"2023-11-16T18:23:39.505140+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:34:36.759190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505363+0000","last_change":"2023-11-16T18:22:46.838249+0000","last_active":"2023-11-16T18:23:39.505363+0000","last_peered":"2023-11-16T18:23:39.505363+0000","last_clean":"2023-11-16T18:23:39.505363+0000","last_became_active":"2023-11-16T18:22:46.837925+0000","last_became_peered":"2023-11-16T18:22:46.837925+0000","last_unstale":"2023-11-16T18:23:39.505363+0000","last_undegraded":"2023-11-16T18:23:39.505363+0000","last_fullsized":"2023-11-16T18:23:39.505363+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:02:17.287183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504877+0000","last_change":"2023-11-16T18:22:51.123687+0000","last_active":"2023-11-16T18:23:39.504877+0000","last_peered":"2023-11-16T18:23:39.504877+0000","last_clean":"2023-11-16T18:23:39.504877+0000","last_became_active":"2023-11-16T18:22:51.123209+0000","last_became_peered":"2023-11-16T18:22:51.123209+0000","last_unstale":"2023-11-16T18:23:39.504877+0000","last_undegraded":"2023-11-16T18:23:39.504877+0000","last_fullsized":"2023-11-16T18:23:39.504877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:09:15.998485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508037+0000","last_change":"2023-11-16T18:22:46.378038+0000","last_active":"2023-11-16T18:23:39.508037+0000","last_peered":"2023-11-16T18:23:39.508037+0000","last_clean":"2023-11-16T18:23:39.508037+0000","last_became_active":"2023-11-16T18:22:46.377680+0000","last_became_peered":"2023-11-16T18:22:46.377680+0000","last_unstale":"2023-11-16T18:23:39.508037+0000","last_undegraded":"2023-11-16T18:23:39.508037+0000","last_fullsized":"2023-11-16T18:23:39.508037+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:31.194156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504247+0000","last_change":"2023-11-16T18:22:46.821888+0000","last_active":"2023-11-16T18:23:39.504247+0000","last_peered":"2023-11-16T18:23:39.504247+0000","last_clean":"2023-11-16T18:23:39.504247+0000","last_became_active":"2023-11-16T18:22:46.821446+0000","last_became_peered":"2023-11-16T18:22:46.821446+0000","last_unstale":"2023-11-16T18:23:39.504247+0000","last_undegraded":"2023-11-16T18:23:39.504247+0000","last_fullsized":"2023-11-16T18:23:39.504247+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:11:30.632655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501162+0000","last_change":"2023-11-16T18:22:51.119717+0000","last_active":"2023-11-16T18:23:39.501162+0000","last_peered":"2023-11-16T18:23:39.501162+0000","last_clean":"2023-11-16T18:23:39.501162+0000","last_became_active":"2023-11-16T18:22:51.119256+0000","last_became_peered":"2023-11-16T18:22:51.119256+0000","last_unstale":"2023-11-16T18:23:39.501162+0000","last_undegraded":"2023-11-16T18:23:39.501162+0000","last_fullsized":"2023-11-16T18:23:39.501162+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:25:42.785279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504001+0000","last_change":"2023-11-16T18:22:46.375387+0000","last_active":"2023-11-16T18:23:39.504001+0000","last_peered":"2023-11-16T18:23:39.504001+0000","last_clean":"2023-11-16T18:23:39.504001+0000","last_became_active":"2023-11-16T18:22:46.374886+0000","last_became_peered":"2023-11-16T18:22:46.374886+0000","last_unstale":"2023-11-16T18:23:39.504001+0000","last_undegraded":"2023-11-16T18:23:39.504001+0000","last_fullsized":"2023-11-16T18:23:39.504001+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:18:24.699496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504057+0000","last_change":"2023-11-16T18:22:46.820315+0000","last_active":"2023-11-16T18:23:39.504057+0000","last_peered":"2023-11-16T18:23:39.504057+0000","last_clean":"2023-11-16T18:23:39.504057+0000","last_became_active":"2023-11-16T18:22:46.819592+0000","last_became_peered":"2023-11-16T18:22:46.819592+0000","last_unstale":"2023-11-16T18:23:39.504057+0000","last_undegraded":"2023-11-16T18:23:39.504057+0000","last_fullsized":"2023-11-16T18:23:39.504057+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:43:26.336090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.518963+0000","last_change":"2023-11-16T18:23:39.518963+0000","last_active":"2023-11-16T18:23:39.518963+0000","last_peered":"2023-11-16T18:23:39.518963+0000","last_clean":"2023-11-16T18:23:39.518963+0000","last_became_active":"2023-11-16T18:23:39.513817+0000","last_became_peered":"2023-11-16T18:23:39.513817+0000","last_unstale":"2023-11-16T18:23:39.518963+0000","last_undegraded":"2023-11-16T18:23:39.518963+0000","last_fullsized":"2023-11-16T18:23:39.518963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:16:46.629599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510801+0000","last_change":"2023-11-16T18:22:46.378638+0000","last_active":"2023-11-16T18:23:39.510801+0000","last_peered":"2023-11-16T18:23:39.510801+0000","last_clean":"2023-11-16T18:23:39.510801+0000","last_became_active":"2023-11-16T18:22:46.378341+0000","last_became_peered":"2023-11-16T18:22:46.378341+0000","last_unstale":"2023-11-16T18:23:39.510801+0000","last_undegraded":"2023-11-16T18:23:39.510801+0000","last_fullsized":"2023-11-16T18:23:39.510801+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:36:28.625148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504347+0000","last_change":"2023-11-16T18:22:46.843431+0000","last_active":"2023-11-16T18:23:39.504347+0000","last_peered":"2023-11-16T18:23:39.504347+0000","last_clean":"2023-11-16T18:23:39.504347+0000","last_became_active":"2023-11-16T18:22:46.842992+0000","last_became_peered":"2023-11-16T18:22:46.842992+0000","last_unstale":"2023-11-16T18:23:39.504347+0000","last_undegraded":"2023-11-16T18:23:39.504347+0000","last_fullsized":"2023-11-16T18:23:39.504347+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:23:28.821023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503912+0000","last_change":"2023-11-16T18:22:51.122640+0000","last_active":"2023-11-16T18:23:39.503912+0000","last_peered":"2023-11-16T18:23:39.503912+0000","last_clean":"2023-11-16T18:23:39.503912+0000","last_became_active":"2023-11-16T18:22:51.122006+0000","last_became_peered":"2023-11-16T18:22:51.122006+0000","last_unstale":"2023-11-16T18:23:39.503912+0000","last_undegraded":"2023-11-16T18:23:39.503912+0000","last_fullsized":"2023-11-16T18:23:39.503912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:27:30.406233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504268+0000","last_change":"2023-11-16T18:22:46.384837+0000","last_active":"2023-11-16T18:23:39.504268+0000","last_peered":"2023-11-16T18:23:39.504268+0000","last_clean":"2023-11-16T18:23:39.504268+0000","last_became_active":"2023-11-16T18:22:46.384541+0000","last_became_peered":"2023-11-16T18:22:46.384541+0000","last_unstale":"2023-11-16T18:23:39.504268+0000","last_undegraded":"2023-11-16T18:23:39.504268+0000","last_fullsized":"2023-11-16T18:23:39.504268+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:06:22.863895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502460+0000","last_change":"2023-11-16T18:22:46.839967+0000","last_active":"2023-11-16T18:23:39.502460+0000","last_peered":"2023-11-16T18:23:39.502460+0000","last_clean":"2023-11-16T18:23:39.502460+0000","last_became_active":"2023-11-16T18:22:46.839645+0000","last_became_peered":"2023-11-16T18:22:46.839645+0000","last_unstale":"2023-11-16T18:23:39.502460+0000","last_undegraded":"2023-11-16T18:23:39.502460+0000","last_fullsized":"2023-11-16T18:23:39.502460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:04:34.616063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507240+0000","last_change":"2023-11-16T18:22:51.126548+0000","last_active":"2023-11-16T18:23:39.507240+0000","last_peered":"2023-11-16T18:23:39.507240+0000","last_clean":"2023-11-16T18:23:39.507240+0000","last_became_active":"2023-11-16T18:22:51.126039+0000","last_became_peered":"2023-11-16T18:22:51.126039+0000","last_unstale":"2023-11-16T18:23:39.507240+0000","last_undegraded":"2023-11-16T18:23:39.507240+0000","last_fullsized":"2023-11-16T18:23:39.507240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:22:07.650645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.511343+0000","last_change":"2023-11-16T18:22:46.383947+0000","last_active":"2023-11-16T18:23:39.511343+0000","last_peered":"2023-11-16T18:23:39.511343+0000","last_clean":"2023-11-16T18:23:39.511343+0000","last_became_active":"2023-11-16T18:22:46.383661+0000","last_became_peered":"2023-11-16T18:22:46.383661+0000","last_unstale":"2023-11-16T18:23:39.511343+0000","last_undegraded":"2023-11-16T18:23:39.511343+0000","last_fullsized":"2023-11-16T18:23:39.511343+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:34.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503182+0000","last_change":"2023-11-16T18:22:46.836877+0000","last_active":"2023-11-16T18:23:39.503182+0000","last_peered":"2023-11-16T18:23:39.503182+0000","last_clean":"2023-11-16T18:23:39.503182+0000","last_became_active":"2023-11-16T18:22:46.836477+0000","last_became_peered":"2023-11-16T18:22:46.836477+0000","last_unstale":"2023-11-16T18:23:39.503182+0000","last_undegraded":"2023-11-16T18:23:39.503182+0000","last_fullsized":"2023-11-16T18:23:39.503182+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:38:49.681616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505171+0000","last_change":"2023-11-16T18:22:51.126930+0000","last_active":"2023-11-16T18:23:39.505171+0000","last_peered":"2023-11-16T18:23:39.505171+0000","last_clean":"2023-11-16T18:23:39.505171+0000","last_became_active":"2023-11-16T18:22:51.126409+0000","last_became_peered":"2023-11-16T18:22:51.126409+0000","last_unstale":"2023-11-16T18:23:39.505171+0000","last_undegraded":"2023-11-16T18:23:39.505171+0000","last_fullsized":"2023-11-16T18:23:39.505171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:31:50.012322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508533+0000","last_change":"2023-11-16T18:22:46.380820+0000","last_active":"2023-11-16T18:23:39.508533+0000","last_peered":"2023-11-16T18:23:39.508533+0000","last_clean":"2023-11-16T18:23:39.508533+0000","last_became_active":"2023-11-16T18:22:46.380446+0000","last_became_peered":"2023-11-16T18:22:46.380446+0000","last_unstale":"2023-11-16T18:23:39.508533+0000","last_undegraded":"2023-11-16T18:23:39.508533+0000","last_fullsized":"2023-11-16T18:23:39.508533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:20:07.923022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502955+0000","last_change":"2023-11-16T18:22:46.838153+0000","last_active":"2023-11-16T18:23:39.502955+0000","last_peered":"2023-11-16T18:23:39.502955+0000","last_clean":"2023-11-16T18:23:39.502955+0000","last_became_active":"2023-11-16T18:22:46.837836+0000","last_became_peered":"2023-11-16T18:22:46.837836+0000","last_unstale":"2023-11-16T18:23:39.502955+0000","last_undegraded":"2023-11-16T18:23:39.502955+0000","last_fullsized":"2023-11-16T18:23:39.502955+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:24:42.477381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507608+0000","last_change":"2023-11-16T18:22:51.119993+0000","last_active":"2023-11-16T18:23:39.507608+0000","last_peered":"2023-11-16T18:23:39.507608+0000","last_clean":"2023-11-16T18:23:39.507608+0000","last_became_active":"2023-11-16T18:22:51.119401+0000","last_became_peered":"2023-11-16T18:22:51.119401+0000","last_unstale":"2023-11-16T18:23:39.507608+0000","last_undegraded":"2023-11-16T18:23:39.507608+0000","last_fullsized":"2023-11-16T18:23:39.507608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:19:31.451143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585944+0000","last_change":"2023-11-16T18:22:46.383399+0000","last_active":"2023-11-16T18:23:39.585944+0000","last_peered":"2023-11-16T18:23:39.585944+0000","last_clean":"2023-11-16T18:23:39.585944+0000","last_became_active":"2023-11-16T18:22:46.382938+0000","last_became_peered":"2023-11-16T18:22:46.382938+0000","last_unstale":"2023-11-16T18:23:39.585944+0000","last_undegraded":"2023-11-16T18:23:39.585944+0000","last_fullsized":"2023-11-16T18:23:39.585944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:29.725086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503688+0000","last_change":"2023-11-16T18:22:46.836035+0000","last_active":"2023-11-16T18:23:39.503688+0000","last_peered":"2023-11-16T18:23:39.503688+0000","last_clean":"2023-11-16T18:23:39.503688+0000","last_became_active":"2023-11-16T18:22:46.835709+0000","last_became_peered":"2023-11-16T18:22:46.835709+0000","last_unstale":"2023-11-16T18:23:39.503688+0000","last_undegraded":"2023-11-16T18:23:39.503688+0000","last_fullsized":"2023-11-16T18:23:39.503688+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:32:57.558106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504778+0000","last_change":"2023-11-16T18:22:46.820535+0000","last_active":"2023-11-16T18:23:39.504778+0000","last_peered":"2023-11-16T18:23:39.504778+0000","last_clean":"2023-11-16T18:23:39.504778+0000","last_became_active":"2023-11-16T18:22:46.820009+0000","last_became_peered":"2023-11-16T18:22:46.820009+0000","last_unstale":"2023-11-16T18:23:39.504778+0000","last_undegraded":"2023-11-16T18:23:39.504778+0000","last_fullsized":"2023-11-16T18:23:39.504778+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:50:39.196560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510673+0000","last_change":"2023-11-16T18:22:46.831762+0000","last_active":"2023-11-16T18:23:39.510673+0000","last_peered":"2023-11-16T18:23:39.510673+0000","last_clean":"2023-11-16T18:23:39.510673+0000","last_became_active":"2023-11-16T18:22:46.831429+0000","last_became_peered":"2023-11-16T18:22:46.831429+0000","last_unstale":"2023-11-16T18:23:39.510673+0000","last_undegraded":"2023-11-16T18:23:39.510673+0000","last_fullsized":"2023-11-16T18:23:39.510673+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:15:44.338906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510533+0000","last_change":"2023-11-16T18:22:46.838133+0000","last_active":"2023-11-16T18:23:39.510533+0000","last_peered":"2023-11-16T18:23:39.510533+0000","last_clean":"2023-11-16T18:23:39.510533+0000","last_became_active":"2023-11-16T18:22:46.837845+0000","last_became_peered":"2023-11-16T18:22:46.837845+0000","last_unstale":"2023-11-16T18:23:39.510533+0000","last_undegraded":"2023-11-16T18:23:39.510533+0000","last_fullsized":"2023-11-16T18:23:39.510533+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:13:06.102358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585337+0000","last_change":"2023-11-16T18:23:39.585337+0000","last_active":"2023-11-16T18:23:39.585337+0000","last_peered":"2023-11-16T18:23:39.585337+0000","last_clean":"2023-11-16T18:23:39.585337+0000","last_became_active":"2023-11-16T18:23:39.584874+0000","last_became_peered":"2023-11-16T18:23:39.584874+0000","last_unstale":"2023-11-16T18:23:39.585337+0000","last_undegraded":"2023-11-16T18:23:39.585337+0000","last_fullsized":"2023-11-16T18:23:39.585337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:24:54.928438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585342+0000","last_change":"2023-11-16T18:22:46.823192+0000","last_active":"2023-11-16T18:23:39.585342+0000","last_peered":"2023-11-16T18:23:39.585342+0000","last_clean":"2023-11-16T18:23:39.585342+0000","last_became_active":"2023-11-16T18:22:46.822930+0000","last_became_peered":"2023-11-16T18:22:46.822930+0000","last_unstale":"2023-11-16T18:23:39.585342+0000","last_undegraded":"2023-11-16T18:23:39.585342+0000","last_fullsized":"2023-11-16T18:23:39.585342+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:49:48.831997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505366+0000","last_change":"2023-11-16T18:22:46.825815+0000","last_active":"2023-11-16T18:23:39.505366+0000","last_peered":"2023-11-16T18:23:39.505366+0000","last_clean":"2023-11-16T18:23:39.505366+0000","last_became_active":"2023-11-16T18:22:46.825517+0000","last_became_peered":"2023-11-16T18:22:46.825517+0000","last_unstale":"2023-11-16T18:23:39.505366+0000","last_undegraded":"2023-11-16T18:23:39.505366+0000","last_fullsized":"2023-11-16T18:23:39.505366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:32:02.468288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.511101+0000","last_change":"2023-11-16T18:22:46.826825+0000","last_active":"2023-11-16T18:23:39.511101+0000","last_peered":"2023-11-16T18:23:39.511101+0000","last_clean":"2023-11-16T18:23:39.511101+0000","last_became_active":"2023-11-16T18:22:46.826495+0000","last_became_peered":"2023-11-16T18:22:46.826495+0000","last_unstale":"2023-11-16T18:23:39.511101+0000","last_undegraded":"2023-11-16T18:23:39.511101+0000","last_fullsized":"2023-11-16T18:23:39.511101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:07:42.411162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506174+0000","last_change":"2023-11-16T18:22:46.844090+0000","last_active":"2023-11-16T18:23:39.506174+0000","last_peered":"2023-11-16T18:23:39.506174+0000","last_clean":"2023-11-16T18:23:39.506174+0000","last_became_active":"2023-11-16T18:22:46.843641+0000","last_became_peered":"2023-11-16T18:22:46.843641+0000","last_unstale":"2023-11-16T18:23:39.506174+0000","last_undegraded":"2023-11-16T18:23:39.506174+0000","last_fullsized":"2023-11-16T18:23:39.506174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:00:12.291257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508321+0000","last_change":"2023-11-16T18:22:46.838340+0000","last_active":"2023-11-16T18:23:39.508321+0000","last_peered":"2023-11-16T18:23:39.508321+0000","last_clean":"2023-11-16T18:23:39.508321+0000","last_became_active":"2023-11-16T18:22:46.838039+0000","last_became_peered":"2023-11-16T18:22:46.838039+0000","last_unstale":"2023-11-16T18:23:39.508321+0000","last_undegraded":"2023-11-16T18:23:39.508321+0000","last_fullsized":"2023-11-16T18:23:39.508321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:09:28.607361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504176+0000","last_change":"2023-11-16T18:22:46.841739+0000","last_active":"2023-11-16T18:23:39.504176+0000","last_peered":"2023-11-16T18:23:39.504176+0000","last_clean":"2023-11-16T18:23:39.504176+0000","last_became_active":"2023-11-16T18:22:46.841450+0000","last_became_peered":"2023-11-16T18:22:46.841450+0000","last_unstale":"2023-11-16T18:23:39.504176+0000","last_undegraded":"2023-11-16T18:23:39.504176+0000","last_fullsized":"2023-11-16T18:23:39.504176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:42:53.482129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508013+0000","last_change":"2023-11-16T18:22:46.839060+0000","last_active":"2023-11-16T18:23:39.508013+0000","last_peered":"2023-11-16T18:23:39.508013+0000","last_clean":"2023-11-16T18:23:39.508013+0000","last_became_active":"2023-11-16T18:22:46.838725+0000","last_became_peered":"2023-11-16T18:22:46.838725+0000","last_unstale":"2023-11-16T18:23:39.508013+0000","last_undegraded":"2023-11-16T18:23:39.508013+0000","last_fullsized":"2023-11-16T18:23:39.508013+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:33:35.374697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584469+0000","last_change":"2023-11-16T18:22:46.833439+0000","last_active":"2023-11-16T18:23:39.584469+0000","last_peered":"2023-11-16T18:23:39.584469+0000","last_clean":"2023-11-16T18:23:39.584469+0000","last_became_active":"2023-11-16T18:22:46.833111+0000","last_became_peered":"2023-11-16T18:22:46.833111+0000","last_unstale":"2023-11-16T18:23:39.584469+0000","last_undegraded":"2023-11-16T18:23:39.584469+0000","last_fullsized":"2023-11-16T18:23:39.584469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:10:36.066370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506665+0000","last_change":"2023-11-16T18:22:46.825185+0000","last_active":"2023-11-16T18:23:39.506665+0000","last_peered":"2023-11-16T18:23:39.506665+0000","last_clean":"2023-11-16T18:23:39.506665+0000","last_became_active":"2023-11-16T18:22:46.824866+0000","last_became_peered":"2023-11-16T18:22:46.824866+0000","last_unstale":"2023-11-16T18:23:39.506665+0000","last_undegraded":"2023-11-16T18:23:39.506665+0000","last_fullsized":"2023-11-16T18:23:39.506665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:46:05.263494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":215374,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65131,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":247,"num_read_kb":193,"num_write":154,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":61913,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":195393,"data_compressed":9627,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481817,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6140,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739012,"statfs":{"total":95995035648,"available":95988748288,"internally_reserved":0,"allocated":970752,"data_stored":485806,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"last":0.44700000000000001}]},{"osd":1,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.438},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.64500000000000002}]},{"osd":2,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.503},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.52700000000000002}]},{"osd":5,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.41199999999999998}]},{"osd":6,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.6539999999999999,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.7130000000000001,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":0.621}]},{"osd":7,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.68100000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":9,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.56899999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.82399999999999995}]}]},{"osd":10,"up_from":69,"seq":296352743452,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6596,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738556,"statfs":{"total":95995035648,"available":95988281344,"internally_reserved":0,"allocated":1110016,"data_stored":1138237,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":1,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.35299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.51800000000000002},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.379}]},{"osd":2,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.82899999999999996,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.624},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.85699999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.659,"5min":1.659,"15min":1.659},"last":1.659},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.72499999999999998,"15min":0.73299999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":1.554}]},{"osd":4,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.629,"15min":0.63},"min":{"1min":0.44,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76800000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.5},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.36499999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.85099999999999998,"5min":0.996,"15min":0.996},"last":0.42999999999999999}]},{"osd":5,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.99399999999999999,"5min":1.206,"15min":1.206},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.745},"min":{"1min":0.55200000000000005,"5min":0.502,"15min":0.502},"max":{"1min":0.97799999999999998,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.64800000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.77400000000000002,"15min":0.77700000000000002},"min":{"1min":0.499,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.581},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.57399999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.609}]},{"osd":7,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55000000000000004}]},{"osd":8,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.84199999999999997,"15min":0.84999999999999998},"min":{"1min":0.63100000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.98899999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0189999999999999,"5min":1.155,"15min":1.155},"last":0.67600000000000005}]},{"osd":9,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.65600000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.6459999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.85699999999999998,"15min":0.85999999999999999},"min":{"1min":0.59799999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":1.4870000000000001}]},{"osd":11,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.72499999999999998}]}]},{"osd":9,"up_from":64,"seq":274877906977,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6340,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":978944,"data_stored":485930,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53600000000000003,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.45400000000000001,"15min":0.46600000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.51900000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.371}]},{"osd":1,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.66100000000000003,"15min":0.67500000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.74099999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.623,"15min":0.63500000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81200000000000006,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.65300000000000002,"15min":0.66300000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82399999999999995,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.63100000000000001,"15min":0.64200000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.80900000000000005,"5min":1.024,"15min":1.024},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.54100000000000004,"15min":0.55600000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63700000000000001,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.438},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.53800000000000003,"15min":0.56100000000000005},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.60699999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.70299999999999996,"15min":0.72299999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.91500000000000004,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74199999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.67700000000000005}]},{"osd":5,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.71299999999999997,"15min":0.72799999999999998},"min":{"1min":0.437,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83199999999999996,"5min":1.357,"15min":1.357},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.754,"15min":0.77400000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82199999999999995,"5min":1.196,"15min":1.196},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.379},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.42699999999999999}]},{"osd":7,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.81599999999999995,"15min":0.84399999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.79300000000000004,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.77200000000000002,"15min":0.79800000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.84999999999999998,"5min":1.157,"15min":1.157},"last":0.754}]},{"osd":8,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.78700000000000003,"15min":0.81100000000000005},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.874,"5min":1.248,"15min":1.248},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.82399999999999995,"15min":0.84899999999999998},"min":{"1min":0.53100000000000003,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.85399999999999998,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.69399999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.70599999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":8,"up_from":57,"seq":244813135910,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6460,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738692,"statfs":{"total":95995035648,"available":95988420608,"internally_reserved":0,"allocated":970752,"data_stored":485738,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.58299999999999996,"15min":0.58799999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.81699999999999995,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.623}]},{"osd":1,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.42499999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.443,"5min":0.316,"15min":0.316},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.78300000000000003}]},{"osd":2,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.36599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.48599999999999999,"15min":0.47599999999999998},"min":{"1min":0.39000000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.41799999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.628,"15min":0.628},"min":{"1min":0.41299999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.59899999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.746}]},{"osd":5,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52500000000000002,"15min":0.51300000000000001},"min":{"1min":0.49399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73899999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.37,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.77400000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.70299999999999996}]},{"osd":6,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65900000000000003,"15min":0.65700000000000003},"min":{"1min":0.51600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.52700000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.84799999999999998,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.82199999999999995}]},{"osd":7,"last update":"Thu Nov 16 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.68000000000000005,"15min":0.66700000000000004},"min":{"1min":0.52000000000000002,"5min":0.44,"15min":0.44},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.54300000000000004,"5min":0.434,"15min":0.434},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.89500000000000002}]},{"osd":9,"last update":"Thu Nov 16 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.60599999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95999999999999996,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.72999999999999998,"15min":0.72099999999999997},"min":{"1min":0.501,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.93899999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.85399999999999998}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.72499999999999998,"15min":0.71099999999999997},"min":{"1min":0.60499999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.036,"5min":1.276,"15min":1.276},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.76400000000000001,"15min":0.74399999999999999},"min":{"1min":0.74299999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.99099999999999999}]},{"osd":11,"last update":"Thu Nov 16 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.90900000000000003}]}]},{"osd":7,"up_from":49,"seq":210453397546,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6580,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738572,"statfs":{"total":95995035648,"available":95988297728,"internally_reserved":0,"allocated":1028096,"data_stored":1082052,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.60699999999999998,"15min":0.59499999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81999999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81699999999999995,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.44500000000000001,"15min":0.42799999999999999},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61499999999999999,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.48099999999999998,"15min":0.46600000000000003},"min":{"1min":0.36299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71899999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63100000000000001,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65000000000000002,"15min":0.622},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.876,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.83199999999999996}]},{"osd":3,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85699999999999998,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.42299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.83599999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.57799999999999996,"15min":0.55200000000000005},"min":{"1min":0.43099999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.77900000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.372,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76300000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.58699999999999997}]},{"osd":5,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70099999999999996,"15min":0.67900000000000005},"min":{"1min":0.47099999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89500000000000002,"5min":1.075,"15min":1.075},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.75600000000000001,"15min":0.748},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.94999999999999996,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.92600000000000005}]},{"osd":6,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.71599999999999997,"15min":0.69999999999999996},"min":{"1min":0.52600000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.74399999999999999,"15min":0.73199999999999998},"min":{"1min":0.47799999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.878}]},{"osd":8,"last update":"Thu Nov 16 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.747,"15min":0.74099999999999999},"min":{"1min":0.501,"5min":0.433,"15min":0.433},"max":{"1min":1.0529999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.77900000000000003,"15min":0.77700000000000002},"min":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0409999999999999,"5min":1.107,"15min":1.107},"last":0.96699999999999997}]},{"osd":9,"last update":"Thu Nov 16 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.79800000000000004,"15min":0.80500000000000005},"min":{"1min":0.48799999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.96799999999999997,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.83299999999999996,"15min":0.81799999999999995},"min":{"1min":0.47799999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":3.4119999999999999,"5min":3.4119999999999999,"15min":3.4119999999999999},"last":1.0369999999999999}]},{"osd":10,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":1.071}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]}]},{"osd":6,"up_from":44,"seq":188978561071,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6384,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738768,"statfs":{"total":95995035648,"available":95988498432,"internally_reserved":0,"allocated":958464,"data_stored":481106,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.39400000000000002,"15min":0.36799999999999999},"min":{"1min":0.309,"5min":0.251,"15min":0.251},"max":{"1min":0.64500000000000002,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.314},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.40500000000000003,"15min":0.38100000000000001},"min":{"1min":0.28499999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58499999999999996,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.33100000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.624,"15min":0.59599999999999997},"min":{"1min":0.47399999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.44},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.53100000000000003},"min":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.45400000000000001}]},{"osd":2,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54200000000000004,"15min":0.52200000000000002},"min":{"1min":0.371,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81899999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60999999999999999,"15min":0.57199999999999995},"min":{"1min":0.40300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.52400000000000002,"15min":0.498},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77500000000000002,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45600000000000002,"15min":0.438},"min":{"1min":0.309,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.66600000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.626,"15min":0.61099999999999999},"min":{"1min":0.44900000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89000000000000001,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70299999999999996,"15min":0.66800000000000004},"min":{"1min":0.54900000000000004,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.53600000000000003}]},{"osd":5,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59899999999999998,"15min":0.57899999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64700000000000002,"15min":0.626},"min":{"1min":0.438,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93700000000000006,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.47099999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75800000000000001,"15min":0.75},"min":{"1min":0.505,"5min":0.497,"15min":0.497},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.79100000000000004,"15min":0.77600000000000002},"min":{"1min":0.54600000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.62}]},{"osd":8,"last update":"Thu Nov 16 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.82399999999999995,"15min":0.82599999999999996},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.70199999999999996,"15min":0.70699999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85799999999999998,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.51100000000000001}]},{"osd":9,"last update":"Thu Nov 16 18:23:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.72899999999999998,"15min":0.753},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.751,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.84799999999999998,"15min":0.874},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.65500000000000003}]},{"osd":10,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.85999999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.68200000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7476,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737676,"statfs":{"total":95995035648,"available":95987380224,"internally_reserved":0,"allocated":1028096,"data_stored":542544,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51000000000000001,"15min":0.52800000000000002},"min":{"1min":0.315,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66900000000000004,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.34799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.71299999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.92200000000000004}]},{"osd":2,"last update":"Thu Nov 16 18:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55800000000000005,"15min":0.53300000000000003},"min":{"1min":0.41099999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78100000000000003,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56999999999999995,"15min":0.54600000000000004},"min":{"1min":0.36199999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.74399999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75900000000000001}]},{"osd":3,"last update":"Thu Nov 16 18:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58199999999999996,"15min":0.56599999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.878,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.58199999999999996,"15min":0.56899999999999995},"min":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.94099999999999995}]},{"osd":4,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.45200000000000001,"15min":0.40799999999999997},"min":{"1min":0.372,"5min":0.252,"15min":0.252},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.47399999999999998,"15min":0.41999999999999998},"min":{"1min":0.39300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80000000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Nov 16 18:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.63600000000000001,"15min":0.624},"min":{"1min":0.49099999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85299999999999998,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.71199999999999997}]},{"osd":6,"last update":"Thu Nov 16 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61899999999999999,"15min":0.60899999999999999},"min":{"1min":0.41699999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.89100000000000001,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61299999999999999,"15min":0.59899999999999998},"min":{"1min":0.433,"5min":0.371,"15min":0.371},"max":{"1min":0.82199999999999995,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":1.028}]},{"osd":7,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.56899999999999995,"15min":0.53900000000000003},"min":{"1min":0.42399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.88200000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.622,"15min":0.58199999999999996},"min":{"1min":0.46500000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.91200000000000003,"5min":0.998,"15min":0.998},"last":0.88500000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.66800000000000004,"15min":0.65800000000000003},"min":{"1min":0.51900000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67000000000000004,"15min":0.66000000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.96799999999999997}]},{"osd":9,"last update":"Thu Nov 16 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.72399999999999998,"15min":0.71899999999999997},"min":{"1min":0.496,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.98199999999999998,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.755,"15min":0.749},"min":{"1min":0.499,"5min":0.495,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":1.036,"15min":1.036},"last":1.0720000000000001}]},{"osd":10,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":1.012}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]}]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7300,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737852,"statfs":{"total":95995035648,"available":95987560448,"internally_reserved":0,"allocated":978944,"data_stored":482221,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 18:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.55300000000000005,"15min":0.5},"min":{"1min":0.36799999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.923,"5min":2.923,"15min":2.923},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.56699999999999995,"15min":0.503},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.2109999999999999,"5min":3.2109999999999999,"15min":3.2109999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Nov 16 18:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.64900000000000002,"15min":0.72899999999999998},"min":{"1min":0.379,"5min":0.371,"15min":0.371},"max":{"1min":0.81399999999999995,"5min":5.3650000000000002,"15min":5.3650000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.624,"15min":0.67000000000000004},"min":{"1min":0.374,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.872,"5min":3.4009999999999998,"15min":3.4009999999999998},"last":0.47799999999999998}]},{"osd":3,"last update":"Thu Nov 16 18:21:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55800000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.45100000000000001},"min":{"1min":0.32700000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.60599999999999998,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.0249999999999999,"5min":3.0249999999999999,"15min":3.0249999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.61299999999999999,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":3.242,"5min":3.242,"15min":3.242},"last":0.65500000000000003}]},{"osd":5,"last update":"Thu Nov 16 18:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58099999999999996,"15min":0.57499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82099999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.505},{"interface":"front","average":{"1min":0.63,"5min":0.59999999999999998,"15min":0.58599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84099999999999997,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.53300000000000003}]},{"osd":6,"last update":"Thu Nov 16 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.55300000000000005,"15min":0.51900000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.8170000000000002,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.54700000000000004,"15min":0.53100000000000003},"min":{"1min":0.30399999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.434}]},{"osd":7,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.95099999999999996,"5min":1.004,"15min":1.004},"last":0.745}]},{"osd":8,"last update":"Thu Nov 16 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89600000000000002,"5min":1.052,"15min":1.052},"last":0.63},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.96699999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.60599999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:23:27 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70299999999999996,"15min":0.69799999999999995},"min":{"1min":0.40000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.3180000000000001,"5min":3.3180000000000001,"15min":3.3180000000000001},"last":0.621},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.72999999999999998,"15min":0.73999999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":0.69499999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.78600000000000003,"15min":0.78100000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.0009999999999999,"5min":3.0009999999999999,"15min":3.0009999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.77000000000000002,"15min":0.75800000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":3.2679999999999998,"5min":3.2679999999999998,"15min":3.2679999999999998},"last":0.68400000000000005}]},{"osd":11,"last update":"Thu Nov 16 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.66700000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7484,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737668,"statfs":{"total":95995035648,"available":95987372032,"internally_reserved":0,"allocated":970752,"data_stored":481744,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.50700000000000001,"15min":0.46999999999999997},"min":{"1min":0.30399999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.49399999999999999,"15min":0.46100000000000002},"min":{"1min":0.35599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.42999999999999999}]},{"osd":1,"last update":"Thu Nov 16 18:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.53900000000000003,"15min":0.51300000000000001},"min":{"1min":0.33000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.56399999999999995,"15min":0.53900000000000003},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.74099999999999999}]},{"osd":3,"last update":"Thu Nov 16 18:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.53900000000000003},"min":{"1min":0.38700000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84499999999999997,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.58999999999999997,"15min":0.54500000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.50600000000000001}]},{"osd":4,"last update":"Thu Nov 16 18:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.54500000000000004,"15min":0.52700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.56299999999999994,"15min":0.54400000000000004},"min":{"1min":0.36799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.67000000000000004}]},{"osd":5,"last update":"Thu Nov 16 18:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.44,"15min":0.41299999999999998},"min":{"1min":0.34699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.497,"15min":0.47099999999999997},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.45700000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63700000000000001,"15min":0.61499999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91500000000000004,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.66300000000000003,"15min":0.629},"min":{"1min":0.53000000000000003,"5min":0.501,"15min":0.501},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.628}]},{"osd":7,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.61899999999999999,"15min":0.61399999999999999},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.65700000000000003}]},{"osd":8,"last update":"Thu Nov 16 18:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.58399999999999996,"15min":0.56999999999999995},"min":{"1min":0.38200000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.377,"5min":1.377,"15min":1.377},"last":0.53000000000000003}]},{"osd":9,"last update":"Thu Nov 16 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.71299999999999997,"15min":0.70799999999999996},"min":{"1min":0.498,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.51300000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.60799999999999998}]},{"osd":10,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.73899999999999999,"15min":0.71899999999999997},"min":{"1min":0.60399999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.77200000000000002,"15min":0.76200000000000001},"min":{"1min":0.58799999999999997,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.242,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.72399999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6712,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738440,"statfs":{"total":95995035648,"available":95988162560,"internally_reserved":0,"allocated":966656,"data_stored":481027,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:14 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38300000000000001,"15min":0.36599999999999999},"min":{"1min":0.25,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.621,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39800000000000002,"15min":0.38700000000000001},"min":{"1min":0.26800000000000002,"5min":0.185,"15min":0.185},"max":{"1min":0.59399999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.30599999999999999}]},{"osd":1,"last update":"Thu Nov 16 18:23:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.52200000000000002},"min":{"1min":0.36899999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80600000000000005,"5min":1.129,"15min":1.129},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55700000000000005,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70199999999999996,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Nov 16 18:23:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.59899999999999998,"15min":0.56899999999999995},"min":{"1min":0.40699999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.94099999999999995,"5min":1.038,"15min":1.038},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58699999999999997,"15min":0.57099999999999995},"min":{"1min":0.376,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71899999999999997,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.503}]},{"osd":4,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63500000000000001,"15min":0.60299999999999998},"min":{"1min":0.498,"5min":0.378,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":0.628},"min":{"1min":0.439,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79800000000000004,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.47199999999999998}]},{"osd":5,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77400000000000002,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.35999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98799999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.48699999999999999}]},{"osd":6,"last update":"Thu Nov 16 18:23:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53900000000000003,"15min":0.49299999999999999},"min":{"1min":0.29999999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93500000000000005,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55900000000000005,"15min":0.53900000000000003},"min":{"1min":0.39800000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.86599999999999999,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.44600000000000001}]},{"osd":7,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.44600000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81399999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69599999999999995,"15min":0.68600000000000005},"min":{"1min":0.46700000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81499999999999995,"5min":1.087,"15min":1.087},"last":0.64200000000000002}]},{"osd":8,"last update":"Thu Nov 16 18:22:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.60799999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79800000000000004,"15min":0.82499999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.002,"5min":1.135,"15min":1.135},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.73899999999999999,"15min":0.75900000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.95899999999999996,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.51400000000000001}]},{"osd":10,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.77000000000000002,"15min":0.78800000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.77600000000000002,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.82699999999999996,"15min":0.85199999999999998},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.81799999999999995,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.60599999999999998}]},{"osd":11,"last update":"Thu Nov 16 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.625}]}]},{"osd":4,"up_from":32,"seq":137438953527,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6608,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738544,"statfs":{"total":95995035648,"available":95988269056,"internally_reserved":0,"allocated":991232,"data_stored":546662,"data_compressed":27217,"data_compressed_allocated":36864,"data_compressed_original":106496,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.504,"15min":0.48699999999999999},"min":{"1min":0.39900000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.752,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.48899999999999999,"15min":0.46400000000000002},"min":{"1min":0.41799999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.502}]},{"osd":1,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40600000000000003,"15min":0.377},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.56299999999999994,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.439,"15min":0.41299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.60699999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59199999999999997,"15min":0.56599999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85399999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.61699999999999999,"15min":0.57499999999999996},"min":{"1min":0.51500000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83599999999999997,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.82999999999999996}]},{"osd":3,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57199999999999995,"15min":0.54000000000000004},"min":{"1min":0.40100000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.03,"5min":0.64800000000000002,"15min":0.56200000000000006},"min":{"1min":0.40300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":10.048,"5min":10.048,"15min":10.048},"last":0.629}]},{"osd":5,"last update":"Thu Nov 16 18:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.63900000000000001,"15min":0.60999999999999999},"min":{"1min":0.46500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.98599999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.65000000000000002,"15min":0.63400000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.81299999999999994}]},{"osd":6,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60699999999999998,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77100000000000002,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66100000000000003,"15min":0.63800000000000001},"min":{"1min":0.47999999999999998,"5min":0.432,"15min":0.432},"max":{"1min":0.86899999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90000000000000002}]},{"osd":7,"last update":"Thu Nov 16 18:23:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.626,"15min":0.57599999999999996},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95199999999999996,"5min":1.077,"15min":1.077},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56899999999999995,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76900000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.77600000000000002}]},{"osd":8,"last update":"Thu Nov 16 18:23:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.84799999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.54500000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.91900000000000004,"5min":1.109,"15min":1.109},"last":0.95499999999999996}]},{"osd":9,"last update":"Thu Nov 16 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.752,"15min":0.76100000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.91800000000000004,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80100000000000005,"15min":0.80700000000000005},"min":{"1min":0.60599999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91300000000000003,"5min":1.117,"15min":1.117},"last":0.96999999999999997}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.63400000000000001,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":0.997,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78200000000000003,"15min":0.78500000000000003},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96799999999999997,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":1.0169999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790002,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6716,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738436,"statfs":{"total":95995035648,"available":95988158464,"internally_reserved":0,"allocated":1036288,"data_stored":1077298,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52400000000000002,"15min":0.51200000000000001},"min":{"1min":0.38500000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.751,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.59899999999999998,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64800000000000002,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.46600000000000003}]},{"osd":1,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53300000000000003,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.37,"15min":0.37},"max":{"1min":0.78600000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.52700000000000002,"15min":0.51400000000000001},"min":{"1min":0.36799999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.746,"5min":0.873,"15min":0.873},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39500000000000002,"15min":0.39200000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39600000000000002,"15min":0.38},"min":{"1min":0.28100000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.434}]},{"osd":3,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84699999999999998,"5min":0.88,"15min":0.88},"last":0.51000000000000001}]},{"osd":4,"last update":"Thu Nov 16 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.375,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56799999999999995,"15min":0.56299999999999994},"min":{"1min":0.41399999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82199999999999995,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.70899999999999996}]},{"osd":6,"last update":"Thu Nov 16 18:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59199999999999997,"15min":0.59499999999999997},"min":{"1min":0.46000000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.68600000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59299999999999997,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81999999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.55400000000000005}]},{"osd":7,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.64000000000000001,"15min":0.629},"min":{"1min":0.48699999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.63800000000000001,"15min":0.625},"min":{"1min":0.44800000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.89600000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.69199999999999995}]},{"osd":8,"last update":"Thu Nov 16 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.83599999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.753,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66100000000000003}]},{"osd":9,"last update":"Thu Nov 16 18:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.66200000000000003,"15min":0.66000000000000003},"min":{"1min":0.53700000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.84299999999999997,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.751},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70299999999999996,"15min":0.70199999999999996},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.83299999999999996,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.72099999999999997}]},{"osd":10,"last update":"Thu Nov 16 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.72699999999999998,"15min":0.71599999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.70499999999999996,"15min":0.69299999999999995},"min":{"1min":0.53800000000000003,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93799999999999994,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.79100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34529,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":61823,"data_compressed":9627,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T18:23:51.531 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-11-16T18:23:51.531 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-16T18:23:51.531 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-11-16T18:23:51.531 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph health --format=json 2023-11-16T18:23:51.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:51 smithi188 ceph-mon[100639]: pgmap v294: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 198 B/s wr, 4 op/s; 5.9 KiB/s, 0 keys/s, 1 objects/s recovering 2023-11-16T18:23:51.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:51 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:51 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:51 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:51 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:51.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:51 smithi002 ceph-mon[98807]: pgmap v294: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 198 B/s wr, 4 op/s; 5.9 KiB/s, 0 keys/s, 1 objects/s recovering 2023-11-16T18:23:51.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:51 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:51.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:51 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:51.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:51 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:51.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:51 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:51.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:51 smithi040 ceph-mon[100251]: pgmap v294: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 198 B/s wr, 4 op/s; 5.9 KiB/s, 0 keys/s, 1 objects/s recovering 2023-11-16T18:23:51.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:51 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:51.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:51 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:51.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:51 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:51.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:51 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:52.197 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:23:52.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:52 smithi040 ceph-mon[100251]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:23:52.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:52.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:52.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:23:52.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:23:52.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:52 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:53.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:52 smithi188 ceph-mon[100639]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:23:53.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:53.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:53.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:23:53.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:23:53.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:52 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:53.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:52 smithi002 ceph-mon[98807]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:23:53.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:53.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:53.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:23:53.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:23:53.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:52 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' 2023-11-16T18:23:53.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.411+0000 7f4aacbbe700 1 -- 172.21.15.2:0/392242267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa8104500 msgr2=0x7f4aa810c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:53.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.411+0000 7f4aacbbe700 1 --2- 172.21.15.2:0/392242267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa8104500 0x7f4aa810c080 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000b0d0 tx=0x7f4aa000b3e0 comp rx=0 tx=0).stop 2023-11-16T18:23:53.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.411+0000 7f4aacbbe700 1 -- 172.21.15.2:0/392242267 shutdown_connections 2023-11-16T18:23:53.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.411+0000 7f4aacbbe700 1 --2- 172.21.15.2:0/392242267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4aa80ff460 0x7f4aa80ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:53.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.411+0000 7f4aacbbe700 1 --2- 172.21.15.2:0/392242267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa8104500 0x7f4aa810c080 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:53.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.411+0000 7f4aacbbe700 1 --2- 172.21.15.2:0/392242267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aa80ffe00 0x7f4aa8103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:53.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.411+0000 7f4aacbbe700 1 -- 172.21.15.2:0/392242267 >> 172.21.15.2:0/392242267 conn(0x7f4aa80770a0 msgr2=0x7f4aa80774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:53.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.411+0000 7f4aacbbe700 1 -- 172.21.15.2:0/392242267 shutdown_connections 2023-11-16T18:23:53.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.411+0000 7f4aacbbe700 1 -- 172.21.15.2:0/392242267 wait complete. 2023-11-16T18:23:53.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.412+0000 7f4aacbbe700 1 Processor -- start 2023-11-16T18:23:53.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.412+0000 7f4aacbbe700 1 -- start start 2023-11-16T18:23:53.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.412+0000 7f4aacbbe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4aa80ff460 0x7f4aa810c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:53.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.412+0000 7f4aacbbe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa80ffe00 0x7f4aa810ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:53.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.412+0000 7f4aacbbe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aa8104500 0x7f4aa810d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:53.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.412+0000 7f4aacbbe700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa81a48d0 con 0x7f4aa80ffe00 2023-11-16T18:23:53.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.412+0000 7f4aacbbe700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa81a4a40 con 0x7f4aa8104500 2023-11-16T18:23:53.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.412+0000 7f4aacbbe700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa81a4c00 con 0x7f4aa80ff460 2023-11-16T18:23:53.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aa5d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa80ffe00 0x7f4aa810ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:53.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aa5d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa80ffe00 0x7f4aa810ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48822/0 (socket says 172.21.15.2:48822) 2023-11-16T18:23:53.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aa5d9b700 1 -- 172.21.15.2:0/2054220021 learned_addr learned my addr 172.21.15.2:0/2054220021 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:53.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aa659c700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4aa80ff460 0x7f4aa810c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:53.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aa6d9d700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aa8104500 0x7f4aa810d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:53.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aa5d9b700 1 -- 172.21.15.2:0/2054220021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4aa80ff460 msgr2=0x7f4aa810c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:53.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aa5d9b700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4aa80ff460 0x7f4aa810c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:53.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aa5d9b700 1 -- 172.21.15.2:0/2054220021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aa8104500 msgr2=0x7f4aa810d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:53.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aa5d9b700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aa8104500 0x7f4aa810d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:53.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aa5d9b700 1 -- 172.21.15.2:0/2054220021 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa0009d00 con 0x7f4aa80ffe00 2023-11-16T18:23:53.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aa6d9d700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aa8104500 0x7f4aa810d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:53.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aa5d9b700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa80ffe00 0x7f4aa810ced0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000c9e0 tx=0x7f4a9000cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:53.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4a9f7fe700 1 -- 172.21.15.2:0/2054220021 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90007aa0 con 0x7f4aa80ffe00 2023-11-16T18:23:53.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4a9f7fe700 1 -- 172.21.15.2:0/2054220021 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a90007c00 con 0x7f4aa80ffe00 2023-11-16T18:23:53.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aacbbe700 1 -- 172.21.15.2:0/2054220021 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa81103e0 con 0x7f4aa80ffe00 2023-11-16T18:23:53.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4a9f7fe700 1 -- 172.21.15.2:0/2054220021 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90018660 con 0x7f4aa80ffe00 2023-11-16T18:23:53.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.413+0000 7f4aacbbe700 1 -- 172.21.15.2:0/2054220021 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa8110250 con 0x7f4aa80ffe00 2023-11-16T18:23:53.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.414+0000 7f4aacbbe700 1 -- 172.21.15.2:0/2054220021 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa8110570 con 0x7f4aa80ffe00 2023-11-16T18:23:53.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.416+0000 7f4a9f7fe700 1 -- 172.21.15.2:0/2054220021 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f4a9000f450 con 0x7f4aa80ffe00 2023-11-16T18:23:53.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.416+0000 7f4a9f7fe700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4a940a11e0 0x7f4a940a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:53.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.416+0000 7f4a9f7fe700 1 -- 172.21.15.2:0/2054220021 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4a900cce90 con 0x7f4aa80ffe00 2023-11-16T18:23:53.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.416+0000 7f4aa659c700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4a940a11e0 0x7f4a940a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:53.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.417+0000 7f4aa659c700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4a940a11e0 0x7f4a940a3690 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4a98005950 tx=0x7f4a980058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:53.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.420+0000 7f4a9f7fe700 1 -- 172.21.15.2:0/2054220021 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4a90096170 con 0x7f4aa80ffe00 2023-11-16T18:23:53.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.659+0000 7f4aacbbe700 1 -- 172.21.15.2:0/2054220021 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4aa8066ea0 con 0x7f4aa80ffe00 2023-11-16T18:23:53.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.660+0000 7f4a9f7fe700 1 -- 172.21.15.2:0/2054220021 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f4a90099790 con 0x7f4aa80ffe00 2023-11-16T18:23:53.662 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:53.662 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-16T18:23:53.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.661+0000 7f4aacbbe700 1 -- 172.21.15.2:0/2054220021 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4a940a11e0 msgr2=0x7f4a940a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:53.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.661+0000 7f4aacbbe700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4a940a11e0 0x7f4a940a3690 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4a98005950 tx=0x7f4a980058e0 comp rx=0 tx=0).stop 2023-11-16T18:23:53.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.662+0000 7f4aacbbe700 1 -- 172.21.15.2:0/2054220021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa80ffe00 msgr2=0x7f4aa810ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:53.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.662+0000 7f4aacbbe700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa80ffe00 0x7f4aa810ced0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000c9e0 tx=0x7f4a9000cda0 comp rx=0 tx=0).stop 2023-11-16T18:23:53.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.662+0000 7f4aacbbe700 1 -- 172.21.15.2:0/2054220021 shutdown_connections 2023-11-16T18:23:53.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.662+0000 7f4aacbbe700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f4a940a11e0 0x7f4a940a3690 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:53.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.662+0000 7f4aacbbe700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4aa80ff460 0x7f4aa810c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:53.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.662+0000 7f4aacbbe700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aa8104500 0x7f4aa810d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:53.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.662+0000 7f4aacbbe700 1 --2- 172.21.15.2:0/2054220021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa80ffe00 0x7f4aa810ced0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:53.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.662+0000 7f4aacbbe700 1 -- 172.21.15.2:0/2054220021 >> 172.21.15.2:0/2054220021 conn(0x7f4aa80770a0 msgr2=0x7f4aa80fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:53.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.662+0000 7f4aacbbe700 1 -- 172.21.15.2:0/2054220021 shutdown_connections 2023-11-16T18:23:53.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:53.662+0000 7f4aacbbe700 1 -- 172.21.15.2:0/2054220021 wait complete. 2023-11-16T18:23:53.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:53 smithi040 ceph-mon[100251]: pgmap v295: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 6 op/s; 5.0 KiB/s, 0 keys/s, 1 objects/s recovering 2023-11-16T18:23:53.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:53 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:23:53.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:53 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:23:53.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:53 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:23:53.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:53 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:23:53.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:53 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2054220021' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-16T18:23:54.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:53 smithi002 ceph-mon[98807]: pgmap v295: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 6 op/s; 5.0 KiB/s, 0 keys/s, 1 objects/s recovering 2023-11-16T18:23:54.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:53 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:23:54.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:53 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:23:54.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:53 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:23:54.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:53 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:23:54.043 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:53 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2054220021' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-16T18:23:54.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:53 smithi188 ceph-mon[100639]: pgmap v295: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 6 op/s; 5.0 KiB/s, 0 keys/s, 1 objects/s recovering 2023-11-16T18:23:54.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:53 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:23:54.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:53 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:23:54.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:53 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:23:54.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:53 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:23:54.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:53 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2054220021' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-16T18:23:54.053 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-11-16T18:23:54.053 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:23:54.184 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.180+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3279571847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5ac1003b0 msgr2=0x7fd5ac100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:54.184 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.180+0000 7fd5b3a6b700 1 --2- 172.21.15.2:0/3279571847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5ac1003b0 0x7fd5ac100800 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009990 tx=0x7fd5a8009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.180+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3279571847 shutdown_connections 2023-11-16T18:23:54.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.180+0000 7fd5b3a6b700 1 --2- 172.21.15.2:0/3279571847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5ac0ff1c0 0x7fd5ac0ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.180+0000 7fd5b3a6b700 1 --2- 172.21.15.2:0/3279571847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5ac100d40 0x7fd5ac107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.180+0000 7fd5b3a6b700 1 --2- 172.21.15.2:0/3279571847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5ac1003b0 0x7fd5ac100800 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.180+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3279571847 >> 172.21.15.2:0/3279571847 conn(0x7fd5ac0fa7c0 msgr2=0x7fd5ac0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:54.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.180+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3279571847 shutdown_connections 2023-11-16T18:23:54.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.180+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3279571847 wait complete. 2023-11-16T18:23:54.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.181+0000 7fd5b3a6b700 1 Processor -- start 2023-11-16T18:23:54.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.181+0000 7fd5b3a6b700 1 -- start start 2023-11-16T18:23:54.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.181+0000 7fd5b3a6b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5ac0ff1c0 0x7fd5ac1960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:54.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.181+0000 7fd5b3a6b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5ac1003b0 0x7fd5ac1965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:54.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.181+0000 7fd5b3a6b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5ac100d40 0x7fd5ac19d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:54.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.181+0000 7fd5b3a6b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac19dc30 con 0x7fd5ac1003b0 2023-11-16T18:23:54.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.181+0000 7fd5b3a6b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac19dd70 con 0x7fd5ac0ff1c0 2023-11-16T18:23:54.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.181+0000 7fd5b3a6b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac19df00 con 0x7fd5ac100d40 2023-11-16T18:23:54.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.181+0000 7fd5b1006700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5ac1003b0 0x7fd5ac1965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:54.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.181+0000 7fd5b1006700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5ac1003b0 0x7fd5ac1965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48836/0 (socket says 172.21.15.2:48836) 2023-11-16T18:23:54.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.181+0000 7fd5b1006700 1 -- 172.21.15.2:0/3346084461 learned_addr learned my addr 172.21.15.2:0/3346084461 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:54.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.181+0000 7fd5b1807700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5ac0ff1c0 0x7fd5ac1960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:54.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.181+0000 7fd5b2008700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5ac100d40 0x7fd5ac19d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:54.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.182+0000 7fd5b1006700 1 -- 172.21.15.2:0/3346084461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5ac100d40 msgr2=0x7fd5ac19d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:54.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.182+0000 7fd5b1006700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5ac100d40 0x7fd5ac19d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.182+0000 7fd5b1006700 1 -- 172.21.15.2:0/3346084461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5ac0ff1c0 msgr2=0x7fd5ac1960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:54.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.182+0000 7fd5b1006700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5ac0ff1c0 0x7fd5ac1960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.182+0000 7fd5b1006700 1 -- 172.21.15.2:0/3346084461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a80095f0 con 0x7fd5ac1003b0 2023-11-16T18:23:54.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.182+0000 7fd5b1006700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5ac1003b0 0x7fd5ac1965f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8006010 tx=0x7fd5a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:54.187 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.182+0000 7fd5a2ffd700 1 -- 172.21.15.2:0/3346084461 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a801d070 con 0x7fd5ac1003b0 2023-11-16T18:23:54.187 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.182+0000 7fd5a2ffd700 1 -- 172.21.15.2:0/3346084461 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5a80041d0 con 0x7fd5ac1003b0 2023-11-16T18:23:54.187 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.182+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3346084461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ac10ab30 con 0x7fd5ac1003b0 2023-11-16T18:23:54.187 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.182+0000 7fd5a2ffd700 1 -- 172.21.15.2:0/3346084461 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a800f460 con 0x7fd5ac1003b0 2023-11-16T18:23:54.187 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.182+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3346084461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ac10a9a0 con 0x7fd5ac1003b0 2023-11-16T18:23:54.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.183+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3346084461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ac107340 con 0x7fd5ac1003b0 2023-11-16T18:23:54.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.184+0000 7fd5a2ffd700 1 -- 172.21.15.2:0/3346084461 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fd5a8003d10 con 0x7fd5ac1003b0 2023-11-16T18:23:54.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.185+0000 7fd5a2ffd700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd5980a11e0 0x7fd5980a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:54.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.185+0000 7fd5b1807700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd5980a11e0 0x7fd5980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:54.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.185+0000 7fd5a2ffd700 1 -- 172.21.15.2:0/3346084461 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd5a80ce3c0 con 0x7fd5ac1003b0 2023-11-16T18:23:54.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.185+0000 7fd5b1807700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd5980a11e0 0x7fd5980a3690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac100210 tx=0x7fd59c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:54.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.189+0000 7fd5a2ffd700 1 -- 172.21.15.2:0/3346084461 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd5a8097730 con 0x7fd5ac1003b0 2023-11-16T18:23:54.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.388+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3346084461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5ac061700 con 0x7fd5ac1003b0 2023-11-16T18:23:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.390+0000 7fd5a2ffd700 1 -- 172.21.15.2:0/3346084461 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24383 (secure 0 0 0) 0x7fd5a809ad50 con 0x7fd5ac1003b0 2023-11-16T18:23:54.393 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:54.393 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":88,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:23:39.497670+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:23:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.392+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3346084461 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd5980a11e0 msgr2=0x7fd5980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.392+0000 7fd5b3a6b700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd5980a11e0 0x7fd5980a3690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac100210 tx=0x7fd59c00a380 comp rx=0 tx=0).stop 2023-11-16T18:23:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.392+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3346084461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5ac1003b0 msgr2=0x7fd5ac1965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.392+0000 7fd5b3a6b700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5ac1003b0 0x7fd5ac1965f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8006010 tx=0x7fd5a8004730 comp rx=0 tx=0).stop 2023-11-16T18:23:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.392+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3346084461 shutdown_connections 2023-11-16T18:23:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.392+0000 7fd5b3a6b700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd5980a11e0 0x7fd5980a3690 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.392+0000 7fd5b3a6b700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5ac100d40 0x7fd5ac19d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.392+0000 7fd5b3a6b700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5ac0ff1c0 0x7fd5ac1960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.392+0000 7fd5b3a6b700 1 --2- 172.21.15.2:0/3346084461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5ac1003b0 0x7fd5ac1965f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.392+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3346084461 >> 172.21.15.2:0/3346084461 conn(0x7fd5ac0fa7c0 msgr2=0x7fd5ac0fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:54.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.392+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3346084461 shutdown_connections 2023-11-16T18:23:54.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.392+0000 7fd5b3a6b700 1 -- 172.21.15.2:0/3346084461 wait complete. 2023-11-16T18:23:54.407 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T18:22:44.902889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T18:22:46.187275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T18:22:49.539148+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T18:23:54.408 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T18:23:54.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.535+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/2220521666 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f380734f0 msgr2=0x7f9f38073960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:54.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.535+0000 7f9f3dd0e700 1 --2- 172.21.15.2:0/2220521666 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f380734f0 0x7f9f38073960 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9f28009990 tx=0x7f9f28009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.535+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/2220521666 shutdown_connections 2023-11-16T18:23:54.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.535+0000 7f9f3dd0e700 1 --2- 172.21.15.2:0/2220521666 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f38074ad0 0x7f9f38072f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.535+0000 7f9f3dd0e700 1 --2- 172.21.15.2:0/2220521666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f381081a0 0x7f9f3810a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.535+0000 7f9f3dd0e700 1 --2- 172.21.15.2:0/2220521666 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f380734f0 0x7f9f38073960 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.535+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/2220521666 >> 172.21.15.2:0/2220521666 conn(0x7f9f380fb910 msgr2=0x7f9f380fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:54.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.535+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/2220521666 shutdown_connections 2023-11-16T18:23:54.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.535+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/2220521666 wait complete. 2023-11-16T18:23:54.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.536+0000 7f9f3dd0e700 1 Processor -- start 2023-11-16T18:23:54.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.536+0000 7f9f3dd0e700 1 -- start start 2023-11-16T18:23:54.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.536+0000 7f9f3dd0e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f380734f0 0x7f9f381981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:54.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.536+0000 7f9f3dd0e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f38074ad0 0x7f9f381986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:54.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.536+0000 7f9f3dd0e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f381081a0 0x7f9f3819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:54.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.536+0000 7f9f3dd0e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3819fd20 con 0x7f9f381081a0 2023-11-16T18:23:54.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.536+0000 7f9f3dd0e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3819fe60 con 0x7f9f380734f0 2023-11-16T18:23:54.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.536+0000 7f9f3dd0e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3819fff0 con 0x7f9f38074ad0 2023-11-16T18:23:54.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.536+0000 7f9f37fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f381081a0 0x7f9f3819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:54.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.536+0000 7f9f37fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f381081a0 0x7f9f3819f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48852/0 (socket says 172.21.15.2:48852) 2023-11-16T18:23:54.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.536+0000 7f9f37fff700 1 -- 172.21.15.2:0/1314616471 learned_addr learned my addr 172.21.15.2:0/1314616471 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:54.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.536+0000 7f9f36ffd700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f38074ad0 0x7f9f381986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:54.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.536+0000 7f9f377fe700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f380734f0 0x7f9f381981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:54.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.537+0000 7f9f37fff700 1 -- 172.21.15.2:0/1314616471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f38074ad0 msgr2=0x7f9f381986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:54.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.537+0000 7f9f37fff700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f38074ad0 0x7f9f381986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.537+0000 7f9f37fff700 1 -- 172.21.15.2:0/1314616471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f380734f0 msgr2=0x7f9f381981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:54.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.537+0000 7f9f37fff700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f380734f0 0x7f9f381981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.537+0000 7f9f37fff700 1 -- 172.21.15.2:0/1314616471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f280095f0 con 0x7f9f381081a0 2023-11-16T18:23:54.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.537+0000 7f9f37fff700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f381081a0 0x7f9f3819f750 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000ef30 tx=0x7f9f3000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:54.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.537+0000 7f9f34ff9700 1 -- 172.21.15.2:0/1314616471 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f30018070 con 0x7f9f381081a0 2023-11-16T18:23:54.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.537+0000 7f9f34ff9700 1 -- 172.21.15.2:0/1314616471 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f3000f040 con 0x7f9f381081a0 2023-11-16T18:23:54.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.537+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/1314616471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3810de30 con 0x7f9f381081a0 2023-11-16T18:23:54.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.537+0000 7f9f34ff9700 1 -- 172.21.15.2:0/1314616471 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3000cd00 con 0x7f9f381081a0 2023-11-16T18:23:54.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.537+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/1314616471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3810dca0 con 0x7f9f381081a0 2023-11-16T18:23:54.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.538+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/1314616471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f381068e0 con 0x7f9f381081a0 2023-11-16T18:23:54.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.539+0000 7f9f34ff9700 1 -- 172.21.15.2:0/1314616471 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f9f30003a00 con 0x7f9f381081a0 2023-11-16T18:23:54.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.540+0000 7f9f34ff9700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9f200a11e0 0x7f9f200a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:54.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.540+0000 7f9f377fe700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9f200a11e0 0x7f9f200a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:54.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.540+0000 7f9f34ff9700 1 -- 172.21.15.2:0/1314616471 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9f300cc960 con 0x7f9f381081a0 2023-11-16T18:23:54.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.540+0000 7f9f377fe700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9f200a11e0 0x7f9f200a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c009c60 tx=0x7f9f2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:54.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.544+0000 7f9f34ff9700 1 -- 172.21.15.2:0/1314616471 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9f30095c70 con 0x7f9f381081a0 2023-11-16T18:23:54.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.743+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/1314616471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9f38067ea0 con 0x7f9f381081a0 2023-11-16T18:23:54.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.744+0000 7f9f34ff9700 1 -- 172.21.15.2:0/1314616471 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9f30099290 con 0x7f9f381081a0 2023-11-16T18:23:54.746 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:23:54.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.746+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/1314616471 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9f200a11e0 msgr2=0x7f9f200a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:54.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.746+0000 7f9f3dd0e700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9f200a11e0 0x7f9f200a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c009c60 tx=0x7f9f2c009260 comp rx=0 tx=0).stop 2023-11-16T18:23:54.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.746+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/1314616471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f381081a0 msgr2=0x7f9f3819f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:54.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.746+0000 7f9f3dd0e700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f381081a0 0x7f9f3819f750 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000ef30 tx=0x7f9f3000c3a0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.746+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/1314616471 shutdown_connections 2023-11-16T18:23:54.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.746+0000 7f9f3dd0e700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f9f200a11e0 0x7f9f200a3690 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.746+0000 7f9f3dd0e700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f381081a0 0x7f9f3819f750 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.746+0000 7f9f3dd0e700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f380734f0 0x7f9f381981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.746+0000 7f9f3dd0e700 1 --2- 172.21.15.2:0/1314616471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f38074ad0 0x7f9f381986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.746+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/1314616471 >> 172.21.15.2:0/1314616471 conn(0x7f9f380fb910 msgr2=0x7f9f380fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:54.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.746+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/1314616471 shutdown_connections 2023-11-16T18:23:54.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.746+0000 7f9f3dd0e700 1 -- 172.21.15.2:0/1314616471 wait complete. 2023-11-16T18:23:54.762 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T18:23:54.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.889+0000 7efec5709700 1 -- 172.21.15.2:0/1103070577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec0100db0 msgr2=0x7efec0107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:54.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.889+0000 7efec5709700 1 --2- 172.21.15.2:0/1103070577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec0100db0 0x7efec0107650 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efeb800b0a0 tx=0x7efeb800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.889+0000 7efec5709700 1 -- 172.21.15.2:0/1103070577 shutdown_connections 2023-11-16T18:23:54.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.889+0000 7efec5709700 1 --2- 172.21.15.2:0/1103070577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec0100db0 0x7efec0107650 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.889+0000 7efec5709700 1 --2- 172.21.15.2:0/1103070577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efec00ff170 0x7efec00ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.889+0000 7efec5709700 1 --2- 172.21.15.2:0/1103070577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efec0100400 0x7efec0100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.889+0000 7efec5709700 1 -- 172.21.15.2:0/1103070577 >> 172.21.15.2:0/1103070577 conn(0x7efec00fa770 msgr2=0x7efec00fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:54.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.889+0000 7efec5709700 1 -- 172.21.15.2:0/1103070577 shutdown_connections 2023-11-16T18:23:54.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.889+0000 7efec5709700 1 -- 172.21.15.2:0/1103070577 wait complete. 2023-11-16T18:23:54.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.890+0000 7efec5709700 1 Processor -- start 2023-11-16T18:23:54.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.890+0000 7efec5709700 1 -- start start 2023-11-16T18:23:54.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.890+0000 7efec5709700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec00ff170 0x7efec0194f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:54.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.890+0000 7efec5709700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efec0100400 0x7efec01954c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.890+0000 7efec5709700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efec0100db0 0x7efec019c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.890+0000 7efec5709700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec019cb00 con 0x7efec00ff170 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.890+0000 7efec5709700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec019cc40 con 0x7efec0100db0 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.890+0000 7efec5709700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec019cdd0 con 0x7efec0100400 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.890+0000 7efebeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec00ff170 0x7efec0194f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.890+0000 7efebeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec00ff170 0x7efec0194f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48860/0 (socket says 172.21.15.2:48860) 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.890+0000 7efebeffd700 1 -- 172.21.15.2:0/4109001690 learned_addr learned my addr 172.21.15.2:0/4109001690 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efebf7fe700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efec0100db0 0x7efec019c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efebe7fc700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efec0100400 0x7efec01954c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efebeffd700 1 -- 172.21.15.2:0/4109001690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efec0100400 msgr2=0x7efec01954c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efebeffd700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efec0100400 0x7efec01954c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efebeffd700 1 -- 172.21.15.2:0/4109001690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efec0100db0 msgr2=0x7efec019c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efebeffd700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efec0100db0 0x7efec019c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efebeffd700 1 -- 172.21.15.2:0/4109001690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb8009d00 con 0x7efec00ff170 2023-11-16T18:23:54.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efebf7fe700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efec0100db0 0x7efec019c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:54.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efebeffd700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec00ff170 0x7efec0194f80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009970 tx=0x7efeb0009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:54.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efeb7fff700 1 -- 172.21.15.2:0/4109001690 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0012070 con 0x7efec00ff170 2023-11-16T18:23:54.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efec5709700 1 -- 172.21.15.2:0/4109001690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec010ab10 con 0x7efec00ff170 2023-11-16T18:23:54.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efeb7fff700 1 -- 172.21.15.2:0/4109001690 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efeb000d260 con 0x7efec00ff170 2023-11-16T18:23:54.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efeb7fff700 1 -- 172.21.15.2:0/4109001690 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0017a60 con 0x7efec00ff170 2023-11-16T18:23:54.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.891+0000 7efec5709700 1 -- 172.21.15.2:0/4109001690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec010a980 con 0x7efec00ff170 2023-11-16T18:23:54.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.892+0000 7efec5709700 1 -- 172.21.15.2:0/4109001690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec00695a0 con 0x7efec00ff170 2023-11-16T18:23:54.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.893+0000 7efeb7fff700 1 -- 172.21.15.2:0/4109001690 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7efeb0017bc0 con 0x7efec00ff170 2023-11-16T18:23:54.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.894+0000 7efeb7fff700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efeac0a11e0 0x7efeac0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:54.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.894+0000 7efeb7fff700 1 -- 172.21.15.2:0/4109001690 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efeb00ce180 con 0x7efec00ff170 2023-11-16T18:23:54.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.894+0000 7efebe7fc700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efeac0a11e0 0x7efeac0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:54.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.894+0000 7efebe7fc700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efeac0a11e0 0x7efeac0a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7efea8005950 tx=0x7efea80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:54.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:54.897+0000 7efeb7fff700 1 -- 172.21.15.2:0/4109001690 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efeb0097460 con 0x7efec00ff170 2023-11-16T18:23:54.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:54 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3346084461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:55.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:54 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3346084461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.098+0000 7efec5709700 1 -- 172.21.15.2:0/4109001690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efec0106140 con 0x7efec00ff170 2023-11-16T18:23:55.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.098+0000 7efeb7fff700 1 -- 172.21.15.2:0/4109001690 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7efeb009aa80 con 0x7efec00ff170 2023-11-16T18:23:55.101 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:23:55.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.101+0000 7efec5709700 1 -- 172.21.15.2:0/4109001690 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efeac0a11e0 msgr2=0x7efeac0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.101+0000 7efec5709700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efeac0a11e0 0x7efeac0a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7efea8005950 tx=0x7efea80058e0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.101+0000 7efec5709700 1 -- 172.21.15.2:0/4109001690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec00ff170 msgr2=0x7efec0194f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.101+0000 7efec5709700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec00ff170 0x7efec0194f80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009970 tx=0x7efeb0009d30 comp rx=0 tx=0).stop 2023-11-16T18:23:55.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.101+0000 7efec5709700 1 -- 172.21.15.2:0/4109001690 shutdown_connections 2023-11-16T18:23:55.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.101+0000 7efec5709700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efeac0a11e0 0x7efeac0a3690 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.101+0000 7efec5709700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efec0100db0 0x7efec019c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.101+0000 7efec5709700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec00ff170 0x7efec0194f80 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.101+0000 7efec5709700 1 --2- 172.21.15.2:0/4109001690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efec0100400 0x7efec01954c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.101+0000 7efec5709700 1 -- 172.21.15.2:0/4109001690 >> 172.21.15.2:0/4109001690 conn(0x7efec00fa770 msgr2=0x7efec00fca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:55.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.101+0000 7efec5709700 1 -- 172.21.15.2:0/4109001690 shutdown_connections 2023-11-16T18:23:55.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.101+0000 7efec5709700 1 -- 172.21.15.2:0/4109001690 wait complete. 2023-11-16T18:23:55.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:54 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3346084461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:55.118 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T18:23:55.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.246+0000 7f50b41a6700 1 -- 172.21.15.2:0/2698835470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50ac0ff170 msgr2=0x7f50ac0ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.246+0000 7f50b41a6700 1 --2- 172.21.15.2:0/2698835470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50ac0ff170 0x7f50ac0ff580 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f50a8009990 tx=0x7f50a8009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.246+0000 7f50b41a6700 1 -- 172.21.15.2:0/2698835470 shutdown_connections 2023-11-16T18:23:55.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.246+0000 7f50b41a6700 1 --2- 172.21.15.2:0/2698835470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50ac100db0 0x7f50ac107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.246+0000 7f50b41a6700 1 --2- 172.21.15.2:0/2698835470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50ac0ff170 0x7f50ac0ff580 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.246+0000 7f50b41a6700 1 --2- 172.21.15.2:0/2698835470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50ac100400 0x7f50ac100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.246+0000 7f50b41a6700 1 -- 172.21.15.2:0/2698835470 >> 172.21.15.2:0/2698835470 conn(0x7f50ac0fa750 msgr2=0x7f50ac0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:55.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.246+0000 7f50b41a6700 1 -- 172.21.15.2:0/2698835470 shutdown_connections 2023-11-16T18:23:55.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.246+0000 7f50b41a6700 1 -- 172.21.15.2:0/2698835470 wait complete. 2023-11-16T18:23:55.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.247+0000 7f50b41a6700 1 Processor -- start 2023-11-16T18:23:55.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.247+0000 7f50b41a6700 1 -- start start 2023-11-16T18:23:55.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.247+0000 7f50b41a6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50ac0ff170 0x7f50ac068950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:55.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.247+0000 7f50b41a6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50ac100400 0x7f50ac19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:55.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.247+0000 7f50b41a6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50ac100db0 0x7f50ac19d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:55.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.247+0000 7f50b41a6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac19daa0 con 0x7f50ac100db0 2023-11-16T18:23:55.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.247+0000 7f50b41a6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac19dc10 con 0x7f50ac100400 2023-11-16T18:23:55.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.247+0000 7f50b41a6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac19ddd0 con 0x7f50ac0ff170 2023-11-16T18:23:55.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.247+0000 7f50b2743700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50ac100db0 0x7f50ac19d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:55.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.247+0000 7f50b2743700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50ac100db0 0x7f50ac19d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48878/0 (socket says 172.21.15.2:48878) 2023-11-16T18:23:55.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.247+0000 7f50b2743700 1 -- 172.21.15.2:0/702446206 learned_addr learned my addr 172.21.15.2:0/702446206 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:55.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.248+0000 7f50b1741700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50ac100400 0x7f50ac19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:55.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.248+0000 7f50b1f42700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50ac0ff170 0x7f50ac068950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:55.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.248+0000 7f50b2743700 1 -- 172.21.15.2:0/702446206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50ac0ff170 msgr2=0x7f50ac068950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.248+0000 7f50b2743700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50ac0ff170 0x7f50ac068950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.248+0000 7f50b2743700 1 -- 172.21.15.2:0/702446206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50ac100400 msgr2=0x7f50ac19ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.248+0000 7f50b2743700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50ac100400 0x7f50ac19ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.248+0000 7f50b2743700 1 -- 172.21.15.2:0/702446206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a80095f0 con 0x7f50ac100db0 2023-11-16T18:23:55.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.248+0000 7f50b2743700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50ac100db0 0x7f50ac19d2d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f50a400ada0 tx=0x7f50a400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:55.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.248+0000 7f509effd700 1 -- 172.21.15.2:0/702446206 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a400f070 con 0x7f50ac100db0 2023-11-16T18:23:55.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.248+0000 7f509effd700 1 -- 172.21.15.2:0/702446206 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50a4007b30 con 0x7f50ac100db0 2023-11-16T18:23:55.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.248+0000 7f50b41a6700 1 -- 172.21.15.2:0/702446206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ac10ab60 con 0x7f50ac100db0 2023-11-16T18:23:55.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.248+0000 7f509effd700 1 -- 172.21.15.2:0/702446206 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4005280 con 0x7f50ac100db0 2023-11-16T18:23:55.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.248+0000 7f50b41a6700 1 -- 172.21.15.2:0/702446206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ac10a9d0 con 0x7f50ac100db0 2023-11-16T18:23:55.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.249+0000 7f50b41a6700 1 -- 172.21.15.2:0/702446206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ac106190 con 0x7f50ac100db0 2023-11-16T18:23:55.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.250+0000 7f509effd700 1 -- 172.21.15.2:0/702446206 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f50a400a090 con 0x7f50ac100db0 2023-11-16T18:23:55.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.251+0000 7f509effd700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f50980a11e0 0x7f50980a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:55.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.251+0000 7f509effd700 1 -- 172.21.15.2:0/702446206 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f50a40cc8e0 con 0x7f50ac100db0 2023-11-16T18:23:55.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.251+0000 7f50b1f42700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f50980a11e0 0x7f50980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:55.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.251+0000 7f50b1f42700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f50980a11e0 0x7f50980a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f50a800b5c0 tx=0x7f50a8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:55.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.254+0000 7f509effd700 1 -- 172.21.15.2:0/702446206 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f50a4099680 con 0x7f50ac100db0 2023-11-16T18:23:55.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.453+0000 7f50b41a6700 1 -- 172.21.15.2:0/702446206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f50ac061700 con 0x7f50ac100db0 2023-11-16T18:23:55.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.454+0000 7f509effd700 1 -- 172.21.15.2:0/702446206 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f50a4099210 con 0x7f50ac100db0 2023-11-16T18:23:55.456 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T18:23:55.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.456+0000 7f50b41a6700 1 -- 172.21.15.2:0/702446206 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f50980a11e0 msgr2=0x7f50980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.456+0000 7f50b41a6700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f50980a11e0 0x7f50980a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f50a800b5c0 tx=0x7f50a8005fb0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.456+0000 7f50b41a6700 1 -- 172.21.15.2:0/702446206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50ac100db0 msgr2=0x7f50ac19d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.456+0000 7f50b41a6700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50ac100db0 0x7f50ac19d2d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f50a400ada0 tx=0x7f50a400c330 comp rx=0 tx=0).stop 2023-11-16T18:23:55.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.456+0000 7f50b41a6700 1 -- 172.21.15.2:0/702446206 shutdown_connections 2023-11-16T18:23:55.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.456+0000 7f50b41a6700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f50980a11e0 0x7f50980a3690 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.456+0000 7f50b41a6700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50ac0ff170 0x7f50ac068950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.456+0000 7f50b41a6700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50ac100db0 0x7f50ac19d2d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.456+0000 7f50b41a6700 1 --2- 172.21.15.2:0/702446206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50ac100400 0x7f50ac19ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.456+0000 7f50b41a6700 1 -- 172.21.15.2:0/702446206 >> 172.21.15.2:0/702446206 conn(0x7f50ac0fa750 msgr2=0x7f50ac0fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:55.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.456+0000 7f50b41a6700 1 -- 172.21.15.2:0/702446206 shutdown_connections 2023-11-16T18:23:55.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.456+0000 7f50b41a6700 1 -- 172.21.15.2:0/702446206 wait complete. 2023-11-16T18:23:55.470 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T18:23:55.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.597+0000 7fa2e3d38700 1 -- 172.21.15.2:0/4196965931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2dc0ff170 msgr2=0x7fa2dc0ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.597+0000 7fa2e3d38700 1 --2- 172.21.15.2:0/4196965931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2dc0ff170 0x7fa2dc0ff580 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009990 tx=0x7fa2d8009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.598+0000 7fa2e3d38700 1 -- 172.21.15.2:0/4196965931 shutdown_connections 2023-11-16T18:23:55.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.598+0000 7fa2e3d38700 1 --2- 172.21.15.2:0/4196965931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2dc100db0 0x7fa2dc107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.598+0000 7fa2e3d38700 1 --2- 172.21.15.2:0/4196965931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2dc0ff170 0x7fa2dc0ff580 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.598+0000 7fa2e3d38700 1 --2- 172.21.15.2:0/4196965931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2dc100400 0x7fa2dc100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.598+0000 7fa2e3d38700 1 -- 172.21.15.2:0/4196965931 >> 172.21.15.2:0/4196965931 conn(0x7fa2dc0fa750 msgr2=0x7fa2dc0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:55.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.598+0000 7fa2e3d38700 1 -- 172.21.15.2:0/4196965931 shutdown_connections 2023-11-16T18:23:55.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.598+0000 7fa2e3d38700 1 -- 172.21.15.2:0/4196965931 wait complete. 2023-11-16T18:23:55.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.598+0000 7fa2e3d38700 1 Processor -- start 2023-11-16T18:23:55.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e3d38700 1 -- start start 2023-11-16T18:23:55.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e3d38700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2dc0ff170 0x7fa2dc194e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:55.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e3d38700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2dc100400 0x7fa2dc1953d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:55.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e3d38700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2dc100db0 0x7fa2dc19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:55.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e3d38700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc19ca10 con 0x7fa2dc100db0 2023-11-16T18:23:55.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e3d38700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc19cb50 con 0x7fa2dc100400 2023-11-16T18:23:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e3d38700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc19cce0 con 0x7fa2dc0ff170 2023-11-16T18:23:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e22d5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2dc100db0 0x7fa2dc19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e22d5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2dc100db0 0x7fa2dc19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48892/0 (socket says 172.21.15.2:48892) 2023-11-16T18:23:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e22d5700 1 -- 172.21.15.2:0/2578596920 learned_addr learned my addr 172.21.15.2:0/2578596920 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e1ad4700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2dc0ff170 0x7fa2dc194e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e12d3700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2dc100400 0x7fa2dc1953d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e22d5700 1 -- 172.21.15.2:0/2578596920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2dc0ff170 msgr2=0x7fa2dc194e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e22d5700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2dc0ff170 0x7fa2dc194e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e22d5700 1 -- 172.21.15.2:0/2578596920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2dc100400 msgr2=0x7fa2dc1953d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e22d5700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2dc100400 0x7fa2dc1953d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e22d5700 1 -- 172.21.15.2:0/2578596920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d80095f0 con 0x7fa2dc100db0 2023-11-16T18:23:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e12d3700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2dc100400 0x7fa2dc1953d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.599+0000 7fa2e22d5700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2dc100db0 0x7fa2dc19c440 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4007ee0 tx=0x7fa2d400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.600+0000 7fa2d2ffd700 1 -- 172.21.15.2:0/2578596920 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d4017070 con 0x7fa2dc100db0 2023-11-16T18:23:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.600+0000 7fa2e3d38700 1 -- 172.21.15.2:0/2578596920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2dc10ab10 con 0x7fa2dc100db0 2023-11-16T18:23:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.600+0000 7fa2d2ffd700 1 -- 172.21.15.2:0/2578596920 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa2d400f040 con 0x7fa2dc100db0 2023-11-16T18:23:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.600+0000 7fa2d2ffd700 1 -- 172.21.15.2:0/2578596920 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d4005740 con 0x7fa2dc100db0 2023-11-16T18:23:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.600+0000 7fa2e3d38700 1 -- 172.21.15.2:0/2578596920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2dc10a980 con 0x7fa2dc100db0 2023-11-16T18:23:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.601+0000 7fa2e3d38700 1 -- 172.21.15.2:0/2578596920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2dc1064f0 con 0x7fa2dc100db0 2023-11-16T18:23:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.602+0000 7fa2d2ffd700 1 -- 172.21.15.2:0/2578596920 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fa2d40058a0 con 0x7fa2dc100db0 2023-11-16T18:23:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.602+0000 7fa2d2ffd700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa2c80a11e0 0x7fa2c80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.603+0000 7fa2d2ffd700 1 -- 172.21.15.2:0/2578596920 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa2d40ccc70 con 0x7fa2dc100db0 2023-11-16T18:23:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.603+0000 7fa2e1ad4700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa2c80a11e0 0x7fa2c80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.603+0000 7fa2e1ad4700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa2c80a11e0 0x7fa2c80a3690 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8006010 tx=0x7fa2d800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.606+0000 7fa2d2ffd700 1 -- 172.21.15.2:0/2578596920 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa2d4095f50 con 0x7fa2dc100db0 2023-11-16T18:23:55.809 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.807+0000 7fa2e3d38700 1 -- 172.21.15.2:0/2578596920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa2dc067a50 con 0x7fa2dc100db0 2023-11-16T18:23:55.809 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.807+0000 7fa2d2ffd700 1 -- 172.21.15.2:0/2578596920 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa2d4099570 con 0x7fa2dc100db0 2023-11-16T18:23:55.810 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:23:55.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.810+0000 7fa2e3d38700 1 -- 172.21.15.2:0/2578596920 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa2c80a11e0 msgr2=0x7fa2c80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.810+0000 7fa2e3d38700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa2c80a11e0 0x7fa2c80a3690 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8006010 tx=0x7fa2d800b540 comp rx=0 tx=0).stop 2023-11-16T18:23:55.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.810+0000 7fa2e3d38700 1 -- 172.21.15.2:0/2578596920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2dc100db0 msgr2=0x7fa2dc19c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.810+0000 7fa2e3d38700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2dc100db0 0x7fa2dc19c440 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4007ee0 tx=0x7fa2d400d3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.810+0000 7fa2e3d38700 1 -- 172.21.15.2:0/2578596920 shutdown_connections 2023-11-16T18:23:55.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.810+0000 7fa2e3d38700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa2c80a11e0 0x7fa2c80a3690 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.810+0000 7fa2e3d38700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2dc0ff170 0x7fa2dc194e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.810+0000 7fa2e3d38700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2dc100db0 0x7fa2dc19c440 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.810+0000 7fa2e3d38700 1 --2- 172.21.15.2:0/2578596920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2dc100400 0x7fa2dc1953d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.810+0000 7fa2e3d38700 1 -- 172.21.15.2:0/2578596920 >> 172.21.15.2:0/2578596920 conn(0x7fa2dc0fa750 msgr2=0x7fa2dc103630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:55.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.810+0000 7fa2e3d38700 1 -- 172.21.15.2:0/2578596920 shutdown_connections 2023-11-16T18:23:55.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.810+0000 7fa2e3d38700 1 -- 172.21.15.2:0/2578596920 wait complete. 2023-11-16T18:23:55.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:55 smithi002 ceph-mon[98807]: pgmap v296: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2023-11-16T18:23:55.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:55 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1314616471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:23:55.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:55 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/4109001690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:23:55.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:55 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/702446206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:23:55.824 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:23:55.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.948+0000 7fc76b62c700 1 -- 172.21.15.2:0/1591446531 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc764100d40 msgr2=0x7fc764107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.948+0000 7fc76b62c700 1 --2- 172.21.15.2:0/1591446531 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc764100d40 0x7fc764107600 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc75c00b0a0 tx=0x7fc75c00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.948+0000 7fc76b62c700 1 -- 172.21.15.2:0/1591446531 shutdown_connections 2023-11-16T18:23:55.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.948+0000 7fc76b62c700 1 --2- 172.21.15.2:0/1591446531 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7640ff1c0 0x7fc7640ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.948+0000 7fc76b62c700 1 --2- 172.21.15.2:0/1591446531 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc764100d40 0x7fc764107600 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.948+0000 7fc76b62c700 1 --2- 172.21.15.2:0/1591446531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7641003b0 0x7fc764100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.948+0000 7fc76b62c700 1 -- 172.21.15.2:0/1591446531 >> 172.21.15.2:0/1591446531 conn(0x7fc7640fa7a0 msgr2=0x7fc7640fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:55.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.948+0000 7fc76b62c700 1 -- 172.21.15.2:0/1591446531 shutdown_connections 2023-11-16T18:23:55.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.948+0000 7fc76b62c700 1 -- 172.21.15.2:0/1591446531 wait complete. 2023-11-16T18:23:55.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.949+0000 7fc76b62c700 1 Processor -- start 2023-11-16T18:23:55.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.949+0000 7fc76b62c700 1 -- start start 2023-11-16T18:23:55.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.949+0000 7fc76b62c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7640ff1c0 0x7fc764196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:55.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.949+0000 7fc76b62c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7641003b0 0x7fc764196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:55.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.949+0000 7fc76b62c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc764100d40 0x7fc76419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:55.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.949+0000 7fc76b62c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76419dbd0 con 0x7fc764100d40 2023-11-16T18:23:55.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.949+0000 7fc76b62c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76419dd10 con 0x7fc7640ff1c0 2023-11-16T18:23:55.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.949+0000 7fc76b62c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76419dea0 con 0x7fc7641003b0 2023-11-16T18:23:55.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.949+0000 7fc769bc9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc764100d40 0x7fc76419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:55.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.949+0000 7fc769bc9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc764100d40 0x7fc76419d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48906/0 (socket says 172.21.15.2:48906) 2023-11-16T18:23:55.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.949+0000 7fc769bc9700 1 -- 172.21.15.2:0/1123532704 learned_addr learned my addr 172.21.15.2:0/1123532704 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:55.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.949+0000 7fc768bc7700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7641003b0 0x7fc764196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:55.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.949+0000 7fc7693c8700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7640ff1c0 0x7fc764196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:55.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.950+0000 7fc769bc9700 1 -- 172.21.15.2:0/1123532704 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7641003b0 msgr2=0x7fc764196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.950+0000 7fc769bc9700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7641003b0 0x7fc764196590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.950+0000 7fc769bc9700 1 -- 172.21.15.2:0/1123532704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7640ff1c0 msgr2=0x7fc764196050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.950+0000 7fc769bc9700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7640ff1c0 0x7fc764196050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.950+0000 7fc769bc9700 1 -- 172.21.15.2:0/1123532704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75c009d00 con 0x7fc764100d40 2023-11-16T18:23:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.950+0000 7fc768bc7700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7641003b0 0x7fc764196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.950+0000 7fc769bc9700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc764100d40 0x7fc76419d600 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc75c00b780 tx=0x7fc75c00bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.950+0000 7fc7567fc700 1 -- 172.21.15.2:0/1123532704 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c00e040 con 0x7fc764100d40 2023-11-16T18:23:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.950+0000 7fc7567fc700 1 -- 172.21.15.2:0/1123532704 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc75c004e60 con 0x7fc764100d40 2023-11-16T18:23:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.950+0000 7fc76b62c700 1 -- 172.21.15.2:0/1123532704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76410ab40 con 0x7fc764100d40 2023-11-16T18:23:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.950+0000 7fc7567fc700 1 -- 172.21.15.2:0/1123532704 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c01b870 con 0x7fc764100d40 2023-11-16T18:23:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.950+0000 7fc76b62c700 1 -- 172.21.15.2:0/1123532704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76410a9b0 con 0x7fc764100d40 2023-11-16T18:23:55.956 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.951+0000 7fc76b62c700 1 -- 172.21.15.2:0/1123532704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc76419e4f0 con 0x7fc764100d40 2023-11-16T18:23:55.956 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.952+0000 7fc7567fc700 1 -- 172.21.15.2:0/1123532704 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fc75c01b9d0 con 0x7fc764100d40 2023-11-16T18:23:55.956 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.952+0000 7fc7567fc700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc7500a1260 0x7fc7500a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:55.956 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.953+0000 7fc7693c8700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc7500a1260 0x7fc7500a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:55.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.953+0000 7fc7567fc700 1 -- 172.21.15.2:0/1123532704 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc75c0ce770 con 0x7fc764100d40 2023-11-16T18:23:55.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.953+0000 7fc7693c8700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc7500a1260 0x7fc7500a3710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc764100210 tx=0x7fc760005d70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:55.959 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:55.957+0000 7fc7567fc700 1 -- 172.21.15.2:0/1123532704 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc75c097ae0 con 0x7fc764100d40 2023-11-16T18:23:55.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:55 smithi040 ceph-mon[100251]: pgmap v296: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2023-11-16T18:23:55.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:55 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1314616471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:23:55.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:55 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/4109001690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:23:55.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:55 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/702446206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:23:56.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:55 smithi188 ceph-mon[100639]: pgmap v296: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2023-11-16T18:23:56.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:55 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1314616471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:23:56.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:55 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/4109001690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:23:56.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:55 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/702446206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:23:56.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.203+0000 7fc76b62c700 1 -- 172.21.15.2:0/1123532704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc76419eaa0 con 0x7fc764100d40 2023-11-16T18:23:56.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.204+0000 7fc7567fc700 1 -- 172.21.15.2:0/1123532704 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10774 (secure 0 0 0) 0x7fc75c09b100 con 0x7fc764100d40 2023-11-16T18:23:56.239 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:56.239 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:23:07.909835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:23:56.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.208+0000 7fc76b62c700 1 -- 172.21.15.2:0/1123532704 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc7500a1260 msgr2=0x7fc7500a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:56.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.208+0000 7fc76b62c700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc7500a1260 0x7fc7500a3710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc764100210 tx=0x7fc760005d70 comp rx=0 tx=0).stop 2023-11-16T18:23:56.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.208+0000 7fc76b62c700 1 -- 172.21.15.2:0/1123532704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc764100d40 msgr2=0x7fc76419d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:56.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.208+0000 7fc76b62c700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc764100d40 0x7fc76419d600 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc75c00b780 tx=0x7fc75c00bdb0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.208+0000 7fc76b62c700 1 -- 172.21.15.2:0/1123532704 shutdown_connections 2023-11-16T18:23:56.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.208+0000 7fc76b62c700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc7500a1260 0x7fc7500a3710 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.208+0000 7fc76b62c700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc764100d40 0x7fc76419d600 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.208+0000 7fc76b62c700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7640ff1c0 0x7fc764196050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.208+0000 7fc76b62c700 1 --2- 172.21.15.2:0/1123532704 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7641003b0 0x7fc764196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.208+0000 7fc76b62c700 1 -- 172.21.15.2:0/1123532704 >> 172.21.15.2:0/1123532704 conn(0x7fc7640fa7a0 msgr2=0x7fc764068e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:56.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.208+0000 7fc76b62c700 1 -- 172.21.15.2:0/1123532704 shutdown_connections 2023-11-16T18:23:56.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.208+0000 7fc76b62c700 1 -- 172.21.15.2:0/1123532704 wait complete. 2023-11-16T18:23:56.311 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T18:23:56.312 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:23:56.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.437+0000 7fd781eeb700 1 -- 172.21.15.2:0/3388035606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd77c0ffe50 msgr2=0x7fd77c107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:56.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.437+0000 7fd781eeb700 1 --2- 172.21.15.2:0/3388035606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd77c0ffe50 0x7fd77c107720 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd77400b0a0 tx=0x7fd77400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.437+0000 7fd781eeb700 1 -- 172.21.15.2:0/3388035606 shutdown_connections 2023-11-16T18:23:56.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.437+0000 7fd781eeb700 1 --2- 172.21.15.2:0/3388035606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd77c100790 0x7fd77c100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.437+0000 7fd781eeb700 1 --2- 172.21.15.2:0/3388035606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd77c0ffe50 0x7fd77c107720 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.437+0000 7fd781eeb700 1 --2- 172.21.15.2:0/3388035606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd77c0ff4c0 0x7fd77c0ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.437+0000 7fd781eeb700 1 -- 172.21.15.2:0/3388035606 >> 172.21.15.2:0/3388035606 conn(0x7fd77c0f9f80 msgr2=0x7fd77c0fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:56.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.437+0000 7fd781eeb700 1 -- 172.21.15.2:0/3388035606 shutdown_connections 2023-11-16T18:23:56.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.437+0000 7fd781eeb700 1 -- 172.21.15.2:0/3388035606 wait complete. 2023-11-16T18:23:56.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.438+0000 7fd781eeb700 1 Processor -- start 2023-11-16T18:23:56.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.438+0000 7fd781eeb700 1 -- start start 2023-11-16T18:23:56.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.438+0000 7fd781eeb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd77c0ff4c0 0x7fd77c196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:56.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.438+0000 7fd781eeb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd77c0ffe50 0x7fd77c196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:56.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.438+0000 7fd781eeb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd77c100790 0x7fd77c19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:56.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.438+0000 7fd781eeb700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c19dbd0 con 0x7fd77c0ff4c0 2023-11-16T18:23:56.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.438+0000 7fd781eeb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c19dd10 con 0x7fd77c100790 2023-11-16T18:23:56.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.438+0000 7fd781eeb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c19dea0 con 0x7fd77c0ffe50 2023-11-16T18:23:56.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd77b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd77c0ff4c0 0x7fd77c196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:56.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd77b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd77c0ff4c0 0x7fd77c196050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48916/0 (socket says 172.21.15.2:48916) 2023-11-16T18:23:56.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd77b7fe700 1 -- 172.21.15.2:0/3585176218 learned_addr learned my addr 172.21.15.2:0/3585176218 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:56.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd77bfff700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd77c100790 0x7fd77c19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:56.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd77affd700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd77c0ffe50 0x7fd77c196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd77b7fe700 1 -- 172.21.15.2:0/3585176218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd77c0ffe50 msgr2=0x7fd77c196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd77b7fe700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd77c0ffe50 0x7fd77c196590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd77b7fe700 1 -- 172.21.15.2:0/3585176218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd77c100790 msgr2=0x7fd77c19d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd77b7fe700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd77c100790 0x7fd77c19d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd77b7fe700 1 -- 172.21.15.2:0/3585176218 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd774009d00 con 0x7fd77c0ff4c0 2023-11-16T18:23:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd77affd700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd77c0ffe50 0x7fd77c196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd77bfff700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd77c100790 0x7fd77c19d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd77b7fe700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd77c0ff4c0 0x7fd77c196050 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00c990 tx=0x7fd76c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd778ff9700 1 -- 172.21.15.2:0/3585176218 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c007b00 con 0x7fd77c0ff4c0 2023-11-16T18:23:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd778ff9700 1 -- 172.21.15.2:0/3585176218 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd76c007c60 con 0x7fd77c0ff4c0 2023-11-16T18:23:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd781eeb700 1 -- 172.21.15.2:0/3585176218 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77c10ab60 con 0x7fd77c0ff4c0 2023-11-16T18:23:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd778ff9700 1 -- 172.21.15.2:0/3585176218 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c018660 con 0x7fd77c0ff4c0 2023-11-16T18:23:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.439+0000 7fd781eeb700 1 -- 172.21.15.2:0/3585176218 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77c10a9d0 con 0x7fd77c0ff4c0 2023-11-16T18:23:56.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.440+0000 7fd781eeb700 1 -- 172.21.15.2:0/3585176218 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77c19e640 con 0x7fd77c0ff4c0 2023-11-16T18:23:56.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.441+0000 7fd778ff9700 1 -- 172.21.15.2:0/3585176218 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fd76c00f450 con 0x7fd77c0ff4c0 2023-11-16T18:23:56.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.442+0000 7fd778ff9700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd7640a1260 0x7fd7640a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:56.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.442+0000 7fd77affd700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd7640a1260 0x7fd7640a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:56.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.442+0000 7fd778ff9700 1 -- 172.21.15.2:0/3585176218 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd76c0ccdb0 con 0x7fd77c0ff4c0 2023-11-16T18:23:56.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.443+0000 7fd77affd700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd7640a1260 0x7fd7640a3710 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd770005d90 tx=0x7fd770005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:56.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.446+0000 7fd778ff9700 1 -- 172.21.15.2:0/3585176218 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd76c0961a0 con 0x7fd77c0ff4c0 2023-11-16T18:23:56.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.645+0000 7fd781eeb700 1 -- 172.21.15.2:0/3585176218 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd77c19ebf0 con 0x7fd77c0ff4c0 2023-11-16T18:23:56.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.646+0000 7fd778ff9700 1 -- 172.21.15.2:0/3585176218 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24383 (secure 0 0 0) 0x7fd76c0997c0 con 0x7fd77c0ff4c0 2023-11-16T18:23:56.649 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:56.649 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":88,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:23:39.497670+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:23:56.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.648+0000 7fd781eeb700 1 -- 172.21.15.2:0/3585176218 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd7640a1260 msgr2=0x7fd7640a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:56.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.648+0000 7fd781eeb700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd7640a1260 0x7fd7640a3710 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd770005d90 tx=0x7fd770005d20 comp rx=0 tx=0).stop 2023-11-16T18:23:56.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.648+0000 7fd781eeb700 1 -- 172.21.15.2:0/3585176218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd77c0ff4c0 msgr2=0x7fd77c196050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:56.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.648+0000 7fd781eeb700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd77c0ff4c0 0x7fd77c196050 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00c990 tx=0x7fd76c00cd50 comp rx=0 tx=0).stop 2023-11-16T18:23:56.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.648+0000 7fd781eeb700 1 -- 172.21.15.2:0/3585176218 shutdown_connections 2023-11-16T18:23:56.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.648+0000 7fd781eeb700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd7640a1260 0x7fd7640a3710 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.648+0000 7fd781eeb700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd77c100790 0x7fd77c19d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.648+0000 7fd781eeb700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd77c0ff4c0 0x7fd77c196050 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.648+0000 7fd781eeb700 1 --2- 172.21.15.2:0/3585176218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd77c0ffe50 0x7fd77c196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.648+0000 7fd781eeb700 1 -- 172.21.15.2:0/3585176218 >> 172.21.15.2:0/3585176218 conn(0x7fd77c0f9f80 msgr2=0x7fd77c0fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:56.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.649+0000 7fd781eeb700 1 -- 172.21.15.2:0/3585176218 shutdown_connections 2023-11-16T18:23:56.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.649+0000 7fd781eeb700 1 -- 172.21.15.2:0/3585176218 wait complete. 2023-11-16T18:23:56.663 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T18:22:44.902889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T18:22:46.187275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T18:22:49.539148+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T18:23:56.664 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T18:23:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.792+0000 7ff94fbb0700 1 -- 172.21.15.2:0/1951567427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff948100d20 msgr2=0x7ff948103100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.792+0000 7ff94fbb0700 1 --2- 172.21.15.2:0/1951567427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff948100d20 0x7ff948103100 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff944009990 tx=0x7ff944009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.792+0000 7ff94fbb0700 1 -- 172.21.15.2:0/1951567427 shutdown_connections 2023-11-16T18:23:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.792+0000 7ff94fbb0700 1 --2- 172.21.15.2:0/1951567427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff948105f60 0x7ff9481083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.792+0000 7ff94fbb0700 1 --2- 172.21.15.2:0/1951567427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff948100d20 0x7ff948103100 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.792+0000 7ff94fbb0700 1 --2- 172.21.15.2:0/1951567427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff948103640 0x7ff948105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.792+0000 7ff94fbb0700 1 -- 172.21.15.2:0/1951567427 >> 172.21.15.2:0/1951567427 conn(0x7ff9480fa7a0 msgr2=0x7ff9480fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.792+0000 7ff94fbb0700 1 -- 172.21.15.2:0/1951567427 shutdown_connections 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.792+0000 7ff94fbb0700 1 -- 172.21.15.2:0/1951567427 wait complete. 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.792+0000 7ff94fbb0700 1 Processor -- start 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94fbb0700 1 -- start start 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94fbb0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff948100d20 0x7ff948195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94fbb0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff948103640 0x7ff9481964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94fbb0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff948105f60 0x7ff94819d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94fbb0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94819dae0 con 0x7ff948100d20 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94fbb0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94819dc20 con 0x7ff948105f60 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94fbb0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94819ddb0 con 0x7ff948103640 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94d94c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff948100d20 0x7ff948195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94d94c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff948100d20 0x7ff948195f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48930/0 (socket says 172.21.15.2:48930) 2023-11-16T18:23:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94d94c700 1 -- 172.21.15.2:0/4096569374 learned_addr learned my addr 172.21.15.2:0/4096569374 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94e14d700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff948105f60 0x7ff94819d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94d14b700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff948103640 0x7ff9481964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94d94c700 1 -- 172.21.15.2:0/4096569374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff948103640 msgr2=0x7ff9481964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94d94c700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff948103640 0x7ff9481964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94d94c700 1 -- 172.21.15.2:0/4096569374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff948105f60 msgr2=0x7ff94819d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94d94c700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff948105f60 0x7ff94819d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94d94c700 1 -- 172.21.15.2:0/4096569374 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9440095f0 con 0x7ff948100d20 2023-11-16T18:23:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94e14d700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff948105f60 0x7ff94819d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94d94c700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff948100d20 0x7ff948195f60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff944000c00 tx=0x7ff94400f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff93affd700 1 -- 172.21.15.2:0/4096569374 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94401d070 con 0x7ff948100d20 2023-11-16T18:23:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff93affd700 1 -- 172.21.15.2:0/4096569374 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff94400fc00 con 0x7ff948100d20 2023-11-16T18:23:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.793+0000 7ff94fbb0700 1 -- 172.21.15.2:0/4096569374 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94810bc20 con 0x7ff948100d20 2023-11-16T18:23:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.794+0000 7ff93affd700 1 -- 172.21.15.2:0/4096569374 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944017650 con 0x7ff948100d20 2023-11-16T18:23:56.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.794+0000 7ff94fbb0700 1 -- 172.21.15.2:0/4096569374 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94810ba90 con 0x7ff948100d20 2023-11-16T18:23:56.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.794+0000 7ff94fbb0700 1 -- 172.21.15.2:0/4096569374 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9480fc310 con 0x7ff948100d20 2023-11-16T18:23:56.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.795+0000 7ff93affd700 1 -- 172.21.15.2:0/4096569374 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ff944022470 con 0x7ff948100d20 2023-11-16T18:23:56.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.796+0000 7ff93affd700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff9340a11e0 0x7ff9340a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:56.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.796+0000 7ff94d14b700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff9340a11e0 0x7ff9340a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:56.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.796+0000 7ff93affd700 1 -- 172.21.15.2:0/4096569374 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff9440ce050 con 0x7ff948100d20 2023-11-16T18:23:56.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.797+0000 7ff94d14b700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff9340a11e0 0x7ff9340a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff93c005950 tx=0x7ff93c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:56.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:56.800+0000 7ff93affd700 1 -- 172.21.15.2:0/4096569374 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff944097360 con 0x7ff948100d20 2023-11-16T18:23:56.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:56 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2578596920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:23:56.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:56 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1123532704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:23:56.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:56 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3585176218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:57.002 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.000+0000 7ff94fbb0700 1 -- 172.21.15.2:0/4096569374 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff948061700 con 0x7ff948100d20 2023-11-16T18:23:57.002 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.000+0000 7ff93affd700 1 -- 172.21.15.2:0/4096569374 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff94409a980 con 0x7ff948100d20 2023-11-16T18:23:57.002 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:23:57.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.002+0000 7ff94fbb0700 1 -- 172.21.15.2:0/4096569374 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff9340a11e0 msgr2=0x7ff9340a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.002+0000 7ff94fbb0700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff9340a11e0 0x7ff9340a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff93c005950 tx=0x7ff93c0058e0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.002+0000 7ff94fbb0700 1 -- 172.21.15.2:0/4096569374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff948100d20 msgr2=0x7ff948195f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.002+0000 7ff94fbb0700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff948100d20 0x7ff948195f60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff944000c00 tx=0x7ff94400f710 comp rx=0 tx=0).stop 2023-11-16T18:23:57.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.002+0000 7ff94fbb0700 1 -- 172.21.15.2:0/4096569374 shutdown_connections 2023-11-16T18:23:57.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.002+0000 7ff94fbb0700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff9340a11e0 0x7ff9340a3690 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.002+0000 7ff94fbb0700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff948105f60 0x7ff94819d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.002+0000 7ff94fbb0700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff948100d20 0x7ff948195f60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.002+0000 7ff94fbb0700 1 --2- 172.21.15.2:0/4096569374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff948103640 0x7ff9481964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.002+0000 7ff94fbb0700 1 -- 172.21.15.2:0/4096569374 >> 172.21.15.2:0/4096569374 conn(0x7ff9480fa7a0 msgr2=0x7ff948101980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:57.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.002+0000 7ff94fbb0700 1 -- 172.21.15.2:0/4096569374 shutdown_connections 2023-11-16T18:23:57.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.002+0000 7ff94fbb0700 1 -- 172.21.15.2:0/4096569374 wait complete. 2023-11-16T18:23:57.015 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:56 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2578596920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:23:57.016 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:56 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1123532704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:23:57.016 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:56 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3585176218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:57.017 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T18:23:57.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:56 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2578596920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:23:57.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:56 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1123532704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:23:57.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:56 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3585176218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:57.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.143+0000 7efe75eaf700 1 -- 172.21.15.2:0/1947202836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe70100850 msgr2=0x7efe70100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.143+0000 7efe75eaf700 1 --2- 172.21.15.2:0/1947202836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe70100850 0x7efe70100c60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7efe60009990 tx=0x7efe60009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.143+0000 7efe75eaf700 1 -- 172.21.15.2:0/1947202836 shutdown_connections 2023-11-16T18:23:57.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.143+0000 7efe75eaf700 1 --2- 172.21.15.2:0/1947202836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe700ffdc0 0x7efe70107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.143+0000 7efe75eaf700 1 --2- 172.21.15.2:0/1947202836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe70100850 0x7efe70100c60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.143+0000 7efe75eaf700 1 --2- 172.21.15.2:0/1947202836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe700ff410 0x7efe700ff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.143+0000 7efe75eaf700 1 -- 172.21.15.2:0/1947202836 >> 172.21.15.2:0/1947202836 conn(0x7efe700f9f70 msgr2=0x7efe700fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:57.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.143+0000 7efe75eaf700 1 -- 172.21.15.2:0/1947202836 shutdown_connections 2023-11-16T18:23:57.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.143+0000 7efe75eaf700 1 -- 172.21.15.2:0/1947202836 wait complete. 2023-11-16T18:23:57.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.144+0000 7efe75eaf700 1 Processor -- start 2023-11-16T18:23:57.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.144+0000 7efe75eaf700 1 -- start start 2023-11-16T18:23:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.144+0000 7efe75eaf700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe700ff410 0x7efe70195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.144+0000 7efe75eaf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe700ffdc0 0x7efe70196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.144+0000 7efe75eaf700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe70100850 0x7efe7019d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.144+0000 7efe75eaf700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7019da90 con 0x7efe70100850 2023-11-16T18:23:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.144+0000 7efe75eaf700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7019dbd0 con 0x7efe700ffdc0 2023-11-16T18:23:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.144+0000 7efe75eaf700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7019dd60 con 0x7efe700ff410 2023-11-16T18:23:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.144+0000 7efe6ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe70100850 0x7efe7019d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe6ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe70100850 0x7efe7019d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48946/0 (socket says 172.21.15.2:48946) 2023-11-16T18:23:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe6ffff700 1 -- 172.21.15.2:0/1639929760 learned_addr learned my addr 172.21.15.2:0/1639929760 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe6effd700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe700ffdc0 0x7efe70196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe6f7fe700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe700ff410 0x7efe70195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe6ffff700 1 -- 172.21.15.2:0/1639929760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe700ff410 msgr2=0x7efe70195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe6ffff700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe700ff410 0x7efe70195f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe6ffff700 1 -- 172.21.15.2:0/1639929760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe700ffdc0 msgr2=0x7efe70196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe6ffff700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe700ffdc0 0x7efe70196450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe6ffff700 1 -- 172.21.15.2:0/1639929760 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe600095f0 con 0x7efe70100850 2023-11-16T18:23:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe6ffff700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe70100850 0x7efe7019d4c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efe6800ada0 tx=0x7efe6800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe6cff9700 1 -- 172.21.15.2:0/1639929760 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6800f070 con 0x7efe70100850 2023-11-16T18:23:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe75eaf700 1 -- 172.21.15.2:0/1639929760 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7010ab70 con 0x7efe70100850 2023-11-16T18:23:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe6cff9700 1 -- 172.21.15.2:0/1639929760 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe68007b30 con 0x7efe70100850 2023-11-16T18:23:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe6cff9700 1 -- 172.21.15.2:0/1639929760 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe680053e0 con 0x7efe70100850 2023-11-16T18:23:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.145+0000 7efe75eaf700 1 -- 172.21.15.2:0/1639929760 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7010a9e0 con 0x7efe70100850 2023-11-16T18:23:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.146+0000 7efe75eaf700 1 -- 172.21.15.2:0/1639929760 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7019e390 con 0x7efe70100850 2023-11-16T18:23:57.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.147+0000 7efe6cff9700 1 -- 172.21.15.2:0/1639929760 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7efe6800c980 con 0x7efe70100850 2023-11-16T18:23:57.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.148+0000 7efe6cff9700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe580a11e0 0x7efe580a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:57.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.148+0000 7efe6f7fe700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe580a11e0 0x7efe580a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:57.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.148+0000 7efe6cff9700 1 -- 172.21.15.2:0/1639929760 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efe680cc380 con 0x7efe70100850 2023-11-16T18:23:57.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.148+0000 7efe6f7fe700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe580a11e0 0x7efe580a3690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efe60006010 tx=0x7efe6000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:57.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.152+0000 7efe6cff9700 1 -- 172.21.15.2:0/1639929760 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efe68095690 con 0x7efe70100850 2023-11-16T18:23:57.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.350+0000 7efe75eaf700 1 -- 172.21.15.2:0/1639929760 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efe7019ea30 con 0x7efe70100850 2023-11-16T18:23:57.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.350+0000 7efe6cff9700 1 -- 172.21.15.2:0/1639929760 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7efe68098cb0 con 0x7efe70100850 2023-11-16T18:23:57.352 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:23:57.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.352+0000 7efe75eaf700 1 -- 172.21.15.2:0/1639929760 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe580a11e0 msgr2=0x7efe580a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.352+0000 7efe75eaf700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe580a11e0 0x7efe580a3690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efe60006010 tx=0x7efe6000b540 comp rx=0 tx=0).stop 2023-11-16T18:23:57.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.352+0000 7efe75eaf700 1 -- 172.21.15.2:0/1639929760 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe70100850 msgr2=0x7efe7019d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.352+0000 7efe75eaf700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe70100850 0x7efe7019d4c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efe6800ada0 tx=0x7efe6800c330 comp rx=0 tx=0).stop 2023-11-16T18:23:57.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.352+0000 7efe75eaf700 1 -- 172.21.15.2:0/1639929760 shutdown_connections 2023-11-16T18:23:57.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.352+0000 7efe75eaf700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe580a11e0 0x7efe580a3690 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.352+0000 7efe75eaf700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe700ff410 0x7efe70195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.352+0000 7efe75eaf700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe70100850 0x7efe7019d4c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.352+0000 7efe75eaf700 1 --2- 172.21.15.2:0/1639929760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe700ffdc0 0x7efe70196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.352+0000 7efe75eaf700 1 -- 172.21.15.2:0/1639929760 >> 172.21.15.2:0/1639929760 conn(0x7efe700f9f70 msgr2=0x7efe700fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:57.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.352+0000 7efe75eaf700 1 -- 172.21.15.2:0/1639929760 shutdown_connections 2023-11-16T18:23:57.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.352+0000 7efe75eaf700 1 -- 172.21.15.2:0/1639929760 wait complete. 2023-11-16T18:23:57.367 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T18:23:57.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.494+0000 7fcb47a16700 1 -- 172.21.15.2:0/1722778022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb400ff170 msgr2=0x7fcb400ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.494+0000 7fcb47a16700 1 --2- 172.21.15.2:0/1722778022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb400ff170 0x7fcb400ff580 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c009990 tx=0x7fcb3c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.494+0000 7fcb47a16700 1 -- 172.21.15.2:0/1722778022 shutdown_connections 2023-11-16T18:23:57.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.494+0000 7fcb47a16700 1 --2- 172.21.15.2:0/1722778022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb40100db0 0x7fcb40107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.494+0000 7fcb47a16700 1 --2- 172.21.15.2:0/1722778022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb400ff170 0x7fcb400ff580 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.494+0000 7fcb47a16700 1 --2- 172.21.15.2:0/1722778022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb40100400 0x7fcb40100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.494+0000 7fcb47a16700 1 -- 172.21.15.2:0/1722778022 >> 172.21.15.2:0/1722778022 conn(0x7fcb400fa750 msgr2=0x7fcb400fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:57.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.494+0000 7fcb47a16700 1 -- 172.21.15.2:0/1722778022 shutdown_connections 2023-11-16T18:23:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.494+0000 7fcb47a16700 1 -- 172.21.15.2:0/1722778022 wait complete. 2023-11-16T18:23:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.494+0000 7fcb47a16700 1 Processor -- start 2023-11-16T18:23:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.494+0000 7fcb47a16700 1 -- start start 2023-11-16T18:23:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb47a16700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb400ff170 0x7fcb40195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb47a16700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb40100400 0x7fcb401964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb47a16700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb40100db0 0x7fcb4019d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb47a16700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4019db10 con 0x7fcb40100db0 2023-11-16T18:23:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb47a16700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4019dc50 con 0x7fcb40100400 2023-11-16T18:23:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb47a16700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4019dde0 con 0x7fcb400ff170 2023-11-16T18:23:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb45fb3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb40100db0 0x7fcb4019d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb45fb3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb40100db0 0x7fcb4019d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48960/0 (socket says 172.21.15.2:48960) 2023-11-16T18:23:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb45fb3700 1 -- 172.21.15.2:0/3876975785 learned_addr learned my addr 172.21.15.2:0/3876975785 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb44fb1700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb40100400 0x7fcb401964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:57.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb457b2700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb400ff170 0x7fcb40195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:57.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb45fb3700 1 -- 172.21.15.2:0/3876975785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb400ff170 msgr2=0x7fcb40195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb45fb3700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb400ff170 0x7fcb40195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb45fb3700 1 -- 172.21.15.2:0/3876975785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb40100400 msgr2=0x7fcb401964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb45fb3700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb40100400 0x7fcb401964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb45fb3700 1 -- 172.21.15.2:0/3876975785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3c0095f0 con 0x7fcb40100db0 2023-11-16T18:23:57.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb44fb1700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb40100400 0x7fcb401964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:57.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb45fb3700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb40100db0 0x7fcb4019d540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800ef30 tx=0x7fcb3800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:57.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb367fc700 1 -- 172.21.15.2:0/3876975785 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38018070 con 0x7fcb40100db0 2023-11-16T18:23:57.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb47a16700 1 -- 172.21.15.2:0/3876975785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4010ab10 con 0x7fcb40100db0 2023-11-16T18:23:57.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb367fc700 1 -- 172.21.15.2:0/3876975785 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb3800f040 con 0x7fcb40100db0 2023-11-16T18:23:57.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb367fc700 1 -- 172.21.15.2:0/3876975785 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3800cdb0 con 0x7fcb40100db0 2023-11-16T18:23:57.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.495+0000 7fcb47a16700 1 -- 172.21.15.2:0/3876975785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4010a980 con 0x7fcb40100db0 2023-11-16T18:23:57.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.496+0000 7fcb47a16700 1 -- 172.21.15.2:0/3876975785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4019e470 con 0x7fcb40100db0 2023-11-16T18:23:57.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.498+0000 7fcb367fc700 1 -- 172.21.15.2:0/3876975785 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fcb38003a00 con 0x7fcb40100db0 2023-11-16T18:23:57.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.499+0000 7fcb367fc700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcb2c0a11e0 0x7fcb2c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:57.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.499+0000 7fcb457b2700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcb2c0a11e0 0x7fcb2c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:57.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.499+0000 7fcb367fc700 1 -- 172.21.15.2:0/3876975785 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcb380cd2b0 con 0x7fcb40100db0 2023-11-16T18:23:57.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.499+0000 7fcb457b2700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcb2c0a11e0 0x7fcb2c0a3690 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00b2e0 tx=0x7fcb3c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:57.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.502+0000 7fcb367fc700 1 -- 172.21.15.2:0/3876975785 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcb380966c0 con 0x7fcb40100db0 2023-11-16T18:23:57.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.703+0000 7fcb47a16700 1 -- 172.21.15.2:0/3876975785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb4019eb10 con 0x7fcb40100db0 2023-11-16T18:23:57.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.704+0000 7fcb367fc700 1 -- 172.21.15.2:0/3876975785 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcb38099ce0 con 0x7fcb40100db0 2023-11-16T18:23:57.706 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T18:23:57.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.706+0000 7fcb47a16700 1 -- 172.21.15.2:0/3876975785 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcb2c0a11e0 msgr2=0x7fcb2c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.706+0000 7fcb47a16700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcb2c0a11e0 0x7fcb2c0a3690 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00b2e0 tx=0x7fcb3c005fb0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.706+0000 7fcb47a16700 1 -- 172.21.15.2:0/3876975785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb40100db0 msgr2=0x7fcb4019d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.706+0000 7fcb47a16700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb40100db0 0x7fcb4019d540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800ef30 tx=0x7fcb3800c3a0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.706+0000 7fcb47a16700 1 -- 172.21.15.2:0/3876975785 shutdown_connections 2023-11-16T18:23:57.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.706+0000 7fcb47a16700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcb2c0a11e0 0x7fcb2c0a3690 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.706+0000 7fcb47a16700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb400ff170 0x7fcb40195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.706+0000 7fcb47a16700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb40100db0 0x7fcb4019d540 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.706+0000 7fcb47a16700 1 --2- 172.21.15.2:0/3876975785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb40100400 0x7fcb401964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.706+0000 7fcb47a16700 1 -- 172.21.15.2:0/3876975785 >> 172.21.15.2:0/3876975785 conn(0x7fcb400fa750 msgr2=0x7fcb40067dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:57.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.706+0000 7fcb47a16700 1 -- 172.21.15.2:0/3876975785 shutdown_connections 2023-11-16T18:23:57.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.706+0000 7fcb47a16700 1 -- 172.21.15.2:0/3876975785 wait complete. 2023-11-16T18:23:57.721 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T18:23:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.847+0000 7f3a31011700 1 -- 172.21.15.2:0/1026034553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a2c0ff1b0 msgr2=0x7f3a2c0ff5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.847+0000 7f3a31011700 1 --2- 172.21.15.2:0/1026034553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a2c0ff1b0 0x7f3a2c0ff5c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c009990 tx=0x7f3a1c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.847+0000 7f3a31011700 1 -- 172.21.15.2:0/1026034553 shutdown_connections 2023-11-16T18:23:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.847+0000 7f3a31011700 1 --2- 172.21.15.2:0/1026034553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a2c100d30 0x7f3a2c1075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.847+0000 7f3a31011700 1 --2- 172.21.15.2:0/1026034553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a2c0ff1b0 0x7f3a2c0ff5c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.847+0000 7f3a31011700 1 --2- 172.21.15.2:0/1026034553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a2c1003a0 0x7f3a2c1007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.847+0000 7f3a31011700 1 -- 172.21.15.2:0/1026034553 >> 172.21.15.2:0/1026034553 conn(0x7f3a2c0fa7b0 msgr2=0x7f3a2c0fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.847+0000 7f3a31011700 1 -- 172.21.15.2:0/1026034553 shutdown_connections 2023-11-16T18:23:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.848+0000 7f3a31011700 1 -- 172.21.15.2:0/1026034553 wait complete. 2023-11-16T18:23:57.852 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.848+0000 7f3a31011700 1 Processor -- start 2023-11-16T18:23:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.848+0000 7f3a31011700 1 -- start start 2023-11-16T18:23:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a31011700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a2c0ff1b0 0x7f3a2c1034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a31011700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a2c1003a0 0x7f3a2c101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a31011700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a2c100d30 0x7f3a2c102070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a31011700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2c102810 con 0x7f3a2c1003a0 2023-11-16T18:23:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a31011700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2c102980 con 0x7f3a2c0ff1b0 2023-11-16T18:23:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a31011700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2c1a0030 con 0x7f3a2c100d30 2023-11-16T18:23:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a2a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a2c1003a0 0x7f3a2c101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a2a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a2c1003a0 0x7f3a2c101b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48972/0 (socket says 172.21.15.2:48972) 2023-11-16T18:23:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a2a59c700 1 -- 172.21.15.2:0/1007534836 learned_addr learned my addr 172.21.15.2:0/1007534836 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a2ad9d700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a2c0ff1b0 0x7f3a2c1034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a2b59e700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a2c100d30 0x7f3a2c102070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a2a59c700 1 -- 172.21.15.2:0/1007534836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a2c100d30 msgr2=0x7f3a2c102070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a2a59c700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a2c100d30 0x7f3a2c102070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a2a59c700 1 -- 172.21.15.2:0/1007534836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a2c0ff1b0 msgr2=0x7f3a2c1034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a2a59c700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a2c0ff1b0 0x7f3a2c1034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a2a59c700 1 -- 172.21.15.2:0/1007534836 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c0095f0 con 0x7f3a2c1003a0 2023-11-16T18:23:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a2a59c700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a2c1003a0 0x7f3a2c101b30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000d980 tx=0x7f3a2000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a13fff700 1 -- 172.21.15.2:0/1007534836 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20004a30 con 0x7f3a2c1003a0 2023-11-16T18:23:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a31011700 1 -- 172.21.15.2:0/1007534836 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2c10ab30 con 0x7f3a2c1003a0 2023-11-16T18:23:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a13fff700 1 -- 172.21.15.2:0/1007534836 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a200054e0 con 0x7f3a2c1003a0 2023-11-16T18:23:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a13fff700 1 -- 172.21.15.2:0/1007534836 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20009c90 con 0x7f3a2c1003a0 2023-11-16T18:23:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.849+0000 7f3a31011700 1 -- 172.21.15.2:0/1007534836 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2c10a9a0 con 0x7f3a2c1003a0 2023-11-16T18:23:57.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.851+0000 7f3a31011700 1 -- 172.21.15.2:0/1007534836 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a2c107330 con 0x7f3a2c1003a0 2023-11-16T18:23:57.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.852+0000 7f3a13fff700 1 -- 172.21.15.2:0/1007534836 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f3a20004c20 con 0x7f3a2c1003a0 2023-11-16T18:23:57.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.853+0000 7f3a13fff700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3a140a11e0 0x7f3a140a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:57.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.853+0000 7f3a13fff700 1 -- 172.21.15.2:0/1007534836 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3a200ccd80 con 0x7f3a2c1003a0 2023-11-16T18:23:57.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.853+0000 7f3a2ad9d700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3a140a11e0 0x7f3a140a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:57.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.853+0000 7f3a2ad9d700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3a140a11e0 0x7f3a140a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00b5c0 tx=0x7f3a1c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:57.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:57.856+0000 7f3a13fff700 1 -- 172.21.15.2:0/1007534836 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3a20096060 con 0x7f3a2c1003a0 2023-11-16T18:23:57.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:57 smithi040 ceph-mon[100251]: pgmap v297: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T18:23:57.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:57 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/4096569374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:23:57.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:57 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1639929760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:23:57.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:57 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3876975785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:23:58.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:57 smithi188 ceph-mon[100639]: pgmap v297: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T18:23:58.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:57 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/4096569374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:23:58.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:57 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1639929760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:23:58.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:57 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3876975785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:23:58.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.052+0000 7f3a31011700 1 -- 172.21.15.2:0/1007534836 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3a2c067ea0 con 0x7f3a2c1003a0 2023-11-16T18:23:58.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.053+0000 7f3a13fff700 1 -- 172.21.15.2:0/1007534836 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3a200170a0 con 0x7f3a2c1003a0 2023-11-16T18:23:58.056 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:23:58.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.055+0000 7f3a31011700 1 -- 172.21.15.2:0/1007534836 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3a140a11e0 msgr2=0x7f3a140a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.055+0000 7f3a31011700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3a140a11e0 0x7f3a140a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00b5c0 tx=0x7f3a1c005960 comp rx=0 tx=0).stop 2023-11-16T18:23:58.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.055+0000 7f3a31011700 1 -- 172.21.15.2:0/1007534836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a2c1003a0 msgr2=0x7f3a2c101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.055+0000 7f3a31011700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a2c1003a0 0x7f3a2c101b30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000d980 tx=0x7f3a2000dc90 comp rx=0 tx=0).stop 2023-11-16T18:23:58.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.055+0000 7f3a31011700 1 -- 172.21.15.2:0/1007534836 shutdown_connections 2023-11-16T18:23:58.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.055+0000 7f3a31011700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f3a140a11e0 0x7f3a140a3690 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.055+0000 7f3a31011700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a2c100d30 0x7f3a2c102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.055+0000 7f3a31011700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a2c0ff1b0 0x7f3a2c1034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.055+0000 7f3a31011700 1 --2- 172.21.15.2:0/1007534836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a2c1003a0 0x7f3a2c101b30 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.055+0000 7f3a31011700 1 -- 172.21.15.2:0/1007534836 >> 172.21.15.2:0/1007534836 conn(0x7f3a2c0fa7b0 msgr2=0x7f3a2c0fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:58.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.055+0000 7f3a31011700 1 -- 172.21.15.2:0/1007534836 shutdown_connections 2023-11-16T18:23:58.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.055+0000 7f3a31011700 1 -- 172.21.15.2:0/1007534836 wait complete. 2023-11-16T18:23:58.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:57 smithi002 ceph-mon[98807]: pgmap v297: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T18:23:58.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:57 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/4096569374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:23:58.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:57 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1639929760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:23:58.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:57 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3876975785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:23:58.075 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:23:58.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.200+0000 7f135a1f8700 1 -- 172.21.15.2:0/1692345794 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13540ff1c0 msgr2=0x7f13540ff5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.200+0000 7f135a1f8700 1 --2- 172.21.15.2:0/1692345794 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13540ff1c0 0x7f13540ff5d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1344009990 tx=0x7f1344009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.200+0000 7f135a1f8700 1 -- 172.21.15.2:0/1692345794 shutdown_connections 2023-11-16T18:23:58.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.200+0000 7f135a1f8700 1 --2- 172.21.15.2:0/1692345794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1354100d40 0x7f1354107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.200+0000 7f135a1f8700 1 --2- 172.21.15.2:0/1692345794 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13540ff1c0 0x7f13540ff5d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.200+0000 7f135a1f8700 1 --2- 172.21.15.2:0/1692345794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13541003b0 0x7f1354100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.200+0000 7f135a1f8700 1 -- 172.21.15.2:0/1692345794 >> 172.21.15.2:0/1692345794 conn(0x7f13540fa7c0 msgr2=0x7f13540fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:58.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.200+0000 7f135a1f8700 1 -- 172.21.15.2:0/1692345794 shutdown_connections 2023-11-16T18:23:58.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.200+0000 7f135a1f8700 1 -- 172.21.15.2:0/1692345794 wait complete. 2023-11-16T18:23:58.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f135a1f8700 1 Processor -- start 2023-11-16T18:23:58.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f135a1f8700 1 -- start start 2023-11-16T18:23:58.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f135a1f8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13540ff1c0 0x7f13541960f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:58.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f135a1f8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13541003b0 0x7f1354196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:58.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f135a1f8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1354100d40 0x7f135419d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:58.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f135a1f8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135419dc70 con 0x7f1354100d40 2023-11-16T18:23:58.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f135a1f8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135419ddb0 con 0x7f13541003b0 2023-11-16T18:23:58.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f135a1f8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135419df40 con 0x7f13540ff1c0 2023-11-16T18:23:58.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f1353fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1354100d40 0x7f135419d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:58.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f1353fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1354100d40 0x7f135419d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48988/0 (socket says 172.21.15.2:48988) 2023-11-16T18:23:58.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f1353fff700 1 -- 172.21.15.2:0/2617526167 learned_addr learned my addr 172.21.15.2:0/2617526167 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:58.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f13537fe700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13540ff1c0 0x7f13541960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:58.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f1352ffd700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13541003b0 0x7f1354196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:58.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f1353fff700 1 -- 172.21.15.2:0/2617526167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13540ff1c0 msgr2=0x7f13541960f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f1353fff700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13540ff1c0 0x7f13541960f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f1353fff700 1 -- 172.21.15.2:0/2617526167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13541003b0 msgr2=0x7f1354196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f1353fff700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13541003b0 0x7f1354196630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f1353fff700 1 -- 172.21.15.2:0/2617526167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13440095f0 con 0x7f1354100d40 2023-11-16T18:23:58.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.201+0000 7f1352ffd700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13541003b0 0x7f1354196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:58.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.202+0000 7f1353fff700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1354100d40 0x7f135419d6a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f134c007ee0 tx=0x7f134c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:58.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.202+0000 7f1350ff9700 1 -- 172.21.15.2:0/2617526167 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c017070 con 0x7f1354100d40 2023-11-16T18:23:58.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.202+0000 7f1350ff9700 1 -- 172.21.15.2:0/2617526167 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f134c00f040 con 0x7f1354100d40 2023-11-16T18:23:58.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.202+0000 7f135a1f8700 1 -- 172.21.15.2:0/2617526167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135410ab80 con 0x7f1354100d40 2023-11-16T18:23:58.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.202+0000 7f1350ff9700 1 -- 172.21.15.2:0/2617526167 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c005740 con 0x7f1354100d40 2023-11-16T18:23:58.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.202+0000 7f135a1f8700 1 -- 172.21.15.2:0/2617526167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135410a9f0 con 0x7f1354100d40 2023-11-16T18:23:58.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.203+0000 7f135a1f8700 1 -- 172.21.15.2:0/2617526167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135419e5e0 con 0x7f1354100d40 2023-11-16T18:23:58.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.204+0000 7f1350ff9700 1 -- 172.21.15.2:0/2617526167 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f134c0058a0 con 0x7f1354100d40 2023-11-16T18:23:58.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.204+0000 7f1350ff9700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f133c0a11e0 0x7f133c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:58.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.205+0000 7f1350ff9700 1 -- 172.21.15.2:0/2617526167 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f134c0cccb0 con 0x7f1354100d40 2023-11-16T18:23:58.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.205+0000 7f13537fe700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f133c0a11e0 0x7f133c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:58.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.205+0000 7f13537fe700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f133c0a11e0 0x7f133c0a3690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f134400b5c0 tx=0x7f1344005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:58.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.208+0000 7f1350ff9700 1 -- 172.21.15.2:0/2617526167 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f134c095f30 con 0x7f1354100d40 2023-11-16T18:23:58.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.454+0000 7f135a1f8700 1 -- 172.21.15.2:0/2617526167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f135419eb90 con 0x7f1354100d40 2023-11-16T18:23:58.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.455+0000 7f1350ff9700 1 -- 172.21.15.2:0/2617526167 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10774 (secure 0 0 0) 0x7f134c099550 con 0x7f1354100d40 2023-11-16T18:23:58.457 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:58.457 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:23:07.909835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:23:58.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.457+0000 7f135a1f8700 1 -- 172.21.15.2:0/2617526167 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f133c0a11e0 msgr2=0x7f133c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.457+0000 7f135a1f8700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f133c0a11e0 0x7f133c0a3690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f134400b5c0 tx=0x7f1344005fb0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.457+0000 7f135a1f8700 1 -- 172.21.15.2:0/2617526167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1354100d40 msgr2=0x7f135419d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.457+0000 7f135a1f8700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1354100d40 0x7f135419d6a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f134c007ee0 tx=0x7f134c00d3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.457+0000 7f135a1f8700 1 -- 172.21.15.2:0/2617526167 shutdown_connections 2023-11-16T18:23:58.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.457+0000 7f135a1f8700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f133c0a11e0 0x7f133c0a3690 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.457+0000 7f135a1f8700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13540ff1c0 0x7f13541960f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.457+0000 7f135a1f8700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1354100d40 0x7f135419d6a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.457+0000 7f135a1f8700 1 --2- 172.21.15.2:0/2617526167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13541003b0 0x7f1354196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.457+0000 7f135a1f8700 1 -- 172.21.15.2:0/2617526167 >> 172.21.15.2:0/2617526167 conn(0x7f13540fa7c0 msgr2=0x7f1354067dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:58.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.457+0000 7f135a1f8700 1 -- 172.21.15.2:0/2617526167 shutdown_connections 2023-11-16T18:23:58.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.457+0000 7f135a1f8700 1 -- 172.21.15.2:0/2617526167 wait complete. 2023-11-16T18:23:58.460 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T18:23:58.472 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:23:58.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.596+0000 7f41340d1700 1 -- 172.21.15.2:0/2124106249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f412c0fe320 msgr2=0x7f412c0fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.596+0000 7f41340d1700 1 --2- 172.21.15.2:0/2124106249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f412c0fe320 0x7f412c0fe770 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4128009990 tx=0x7f4128009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.596+0000 7f41340d1700 1 -- 172.21.15.2:0/2124106249 shutdown_connections 2023-11-16T18:23:58.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.596+0000 7f41340d1700 1 --2- 172.21.15.2:0/2124106249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f412c1018b0 0x7f412c101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.596+0000 7f41340d1700 1 --2- 172.21.15.2:0/2124106249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f412c0fecb0 0x7f412c108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.596+0000 7f41340d1700 1 --2- 172.21.15.2:0/2124106249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f412c0fe320 0x7f412c0fe770 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.596+0000 7f41340d1700 1 -- 172.21.15.2:0/2124106249 >> 172.21.15.2:0/2124106249 conn(0x7f412c0f9f50 msgr2=0x7f412c0fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:58.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.596+0000 7f41340d1700 1 -- 172.21.15.2:0/2124106249 shutdown_connections 2023-11-16T18:23:58.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.596+0000 7f41340d1700 1 -- 172.21.15.2:0/2124106249 wait complete. 2023-11-16T18:23:58.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f41340d1700 1 Processor -- start 2023-11-16T18:23:58.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f41340d1700 1 -- start start 2023-11-16T18:23:58.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f41340d1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f412c0fe320 0x7f412c1960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:58.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f41340d1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f412c0fecb0 0x7f412c196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:58.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f41340d1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f412c1018b0 0x7f412c19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:58.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f41340d1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c19dc40 con 0x7f412c0fecb0 2023-11-16T18:23:58.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f41340d1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c19dd80 con 0x7f412c0fe320 2023-11-16T18:23:58.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f41340d1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c19df10 con 0x7f412c1018b0 2023-11-16T18:23:58.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f413166c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f412c0fecb0 0x7f412c196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:58.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f413166c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f412c0fecb0 0x7f412c196600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49000/0 (socket says 172.21.15.2:49000) 2023-11-16T18:23:58.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f413166c700 1 -- 172.21.15.2:0/827452324 learned_addr learned my addr 172.21.15.2:0/827452324 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:58.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f4131e6d700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f412c0fe320 0x7f412c1960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:58.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f413166c700 1 -- 172.21.15.2:0/827452324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f412c1018b0 msgr2=0x7f412c19d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f413266e700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f412c1018b0 0x7f412c19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:58.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f413166c700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f412c1018b0 0x7f412c19d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f413166c700 1 -- 172.21.15.2:0/827452324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f412c0fe320 msgr2=0x7f412c1960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f413166c700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f412c0fe320 0x7f412c1960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f413166c700 1 -- 172.21.15.2:0/827452324 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41280095f0 con 0x7f412c0fecb0 2023-11-16T18:23:58.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.597+0000 7f4131e6d700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f412c0fe320 0x7f412c1960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:58.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.598+0000 7f413166c700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f412c0fecb0 0x7f412c196600 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4128000c00 tx=0x7f412800f830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:58.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.598+0000 7f411effd700 1 -- 172.21.15.2:0/827452324 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412801d070 con 0x7f412c0fecb0 2023-11-16T18:23:58.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.598+0000 7f411effd700 1 -- 172.21.15.2:0/827452324 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f412800fce0 con 0x7f412c0fecb0 2023-11-16T18:23:58.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.598+0000 7f41340d1700 1 -- 172.21.15.2:0/827452324 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412c10bc30 con 0x7f412c0fecb0 2023-11-16T18:23:58.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.598+0000 7f411effd700 1 -- 172.21.15.2:0/827452324 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41280176e0 con 0x7f412c0fecb0 2023-11-16T18:23:58.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.598+0000 7f41340d1700 1 -- 172.21.15.2:0/827452324 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412c10baa0 con 0x7f412c0fecb0 2023-11-16T18:23:58.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.599+0000 7f41340d1700 1 -- 172.21.15.2:0/827452324 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412c107a30 con 0x7f412c0fecb0 2023-11-16T18:23:58.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.600+0000 7f411effd700 1 -- 172.21.15.2:0/827452324 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f4128022470 con 0x7f412c0fecb0 2023-11-16T18:23:58.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.600+0000 7f411effd700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41180a1260 0x7f41180a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:58.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.601+0000 7f4131e6d700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41180a1260 0x7f41180a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:58.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.601+0000 7f411effd700 1 -- 172.21.15.2:0/827452324 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f41280ce620 con 0x7f412c0fecb0 2023-11-16T18:23:58.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.601+0000 7f4131e6d700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41180a1260 0x7f41180a3710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f412c067ae0 tx=0x7f4120005f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:58.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.604+0000 7f411effd700 1 -- 172.21.15.2:0/827452324 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4128097930 con 0x7f412c0fecb0 2023-11-16T18:23:58.809 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.807+0000 7f41340d1700 1 -- 172.21.15.2:0/827452324 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f412c02cfe0 con 0x7f412c0fecb0 2023-11-16T18:23:58.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.809+0000 7f411effd700 1 -- 172.21.15.2:0/827452324 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24383 (secure 0 0 0) 0x7f412809af50 con 0x7f412c0fecb0 2023-11-16T18:23:58.811 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:58.812 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":88,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:23:39.497670+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:23:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.811+0000 7f41340d1700 1 -- 172.21.15.2:0/827452324 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41180a1260 msgr2=0x7f41180a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.811+0000 7f41340d1700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41180a1260 0x7f41180a3710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f412c067ae0 tx=0x7f4120005f70 comp rx=0 tx=0).stop 2023-11-16T18:23:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.811+0000 7f41340d1700 1 -- 172.21.15.2:0/827452324 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f412c0fecb0 msgr2=0x7f412c196600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.811+0000 7f41340d1700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f412c0fecb0 0x7f412c196600 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4128000c00 tx=0x7f412800f830 comp rx=0 tx=0).stop 2023-11-16T18:23:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.811+0000 7f41340d1700 1 -- 172.21.15.2:0/827452324 shutdown_connections 2023-11-16T18:23:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.811+0000 7f41340d1700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41180a1260 0x7f41180a3710 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.811+0000 7f41340d1700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f412c1018b0 0x7f412c19d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.811+0000 7f41340d1700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f412c0fe320 0x7f412c1960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.811+0000 7f41340d1700 1 --2- 172.21.15.2:0/827452324 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f412c0fecb0 0x7f412c196600 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.811+0000 7f41340d1700 1 -- 172.21.15.2:0/827452324 >> 172.21.15.2:0/827452324 conn(0x7f412c0f9f50 msgr2=0x7f412c104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:58.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.811+0000 7f41340d1700 1 -- 172.21.15.2:0/827452324 shutdown_connections 2023-11-16T18:23:58.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.811+0000 7f41340d1700 1 -- 172.21.15.2:0/827452324 wait complete. 2023-11-16T18:23:58.827 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:23:58.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.952+0000 7f0b6503a700 1 -- 172.21.15.2:0/1088844502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b60100d40 msgr2=0x7f0b60107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.952+0000 7f0b6503a700 1 --2- 172.21.15.2:0/1088844502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b60100d40 0x7f0b60107600 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800b0a0 tx=0x7f0b5800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.952+0000 7f0b6503a700 1 -- 172.21.15.2:0/1088844502 shutdown_connections 2023-11-16T18:23:58.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.952+0000 7f0b6503a700 1 --2- 172.21.15.2:0/1088844502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b60100d40 0x7f0b60107600 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.952+0000 7f0b6503a700 1 --2- 172.21.15.2:0/1088844502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b600ff1c0 0x7f0b600ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.956 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.952+0000 7f0b6503a700 1 --2- 172.21.15.2:0/1088844502 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b601003b0 0x7f0b60100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.956 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.952+0000 7f0b6503a700 1 -- 172.21.15.2:0/1088844502 >> 172.21.15.2:0/1088844502 conn(0x7f0b600fa7a0 msgr2=0x7f0b600fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:58.956 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.952+0000 7f0b6503a700 1 -- 172.21.15.2:0/1088844502 shutdown_connections 2023-11-16T18:23:58.956 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.952+0000 7f0b6503a700 1 -- 172.21.15.2:0/1088844502 wait complete. 2023-11-16T18:23:58.959 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.953+0000 7f0b6503a700 1 Processor -- start 2023-11-16T18:23:58.959 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.953+0000 7f0b6503a700 1 -- start start 2023-11-16T18:23:58.959 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.953+0000 7f0b6503a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b600ff1c0 0x7f0b60069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:58.959 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b6503a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b601003b0 0x7f0b6019ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:58.959 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b6503a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b60100d40 0x7f0b6019d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:58.959 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b6503a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6019dcb0 con 0x7f0b601003b0 2023-11-16T18:23:58.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b6503a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6019de20 con 0x7f0b60100d40 2023-11-16T18:23:58.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b6503a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6019dfe0 con 0x7f0b600ff1c0 2023-11-16T18:23:58.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b5e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b601003b0 0x7f0b6019ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:58.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b5e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b601003b0 0x7f0b6019ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49014/0 (socket says 172.21.15.2:49014) 2023-11-16T18:23:58.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b5e59c700 1 -- 172.21.15.2:0/3145555133 learned_addr learned my addr 172.21.15.2:0/3145555133 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:58.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b5f59e700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b60100d40 0x7f0b6019d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:58.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b5ed9d700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b600ff1c0 0x7f0b60069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:58.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b5e59c700 1 -- 172.21.15.2:0/3145555133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b600ff1c0 msgr2=0x7f0b60069980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b5e59c700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b600ff1c0 0x7f0b60069980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b5e59c700 1 -- 172.21.15.2:0/3145555133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b60100d40 msgr2=0x7f0b6019d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:58.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b5e59c700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b60100d40 0x7f0b6019d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:58.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b5e59c700 1 -- 172.21.15.2:0/3145555133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b58009d00 con 0x7f0b601003b0 2023-11-16T18:23:58.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b5f59e700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b60100d40 0x7f0b6019d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:58.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b5e59c700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b601003b0 0x7f0b6019ae90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800d950 tx=0x7f0b4800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:58.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b57fff700 1 -- 172.21.15.2:0/3145555133 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b48009b40 con 0x7f0b601003b0 2023-11-16T18:23:58.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b57fff700 1 -- 172.21.15.2:0/3145555133 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b48009ca0 con 0x7f0b601003b0 2023-11-16T18:23:58.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b6503a700 1 -- 172.21.15.2:0/3145555133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b6010ab40 con 0x7f0b601003b0 2023-11-16T18:23:58.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b57fff700 1 -- 172.21.15.2:0/3145555133 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4800f620 con 0x7f0b601003b0 2023-11-16T18:23:58.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.954+0000 7f0b6503a700 1 -- 172.21.15.2:0/3145555133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b6010a9b0 con 0x7f0b601003b0 2023-11-16T18:23:58.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.955+0000 7f0b6503a700 1 -- 172.21.15.2:0/3145555133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b601066b0 con 0x7f0b601003b0 2023-11-16T18:23:58.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.956+0000 7f0b57fff700 1 -- 172.21.15.2:0/3145555133 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f0b48010460 con 0x7f0b601003b0 2023-11-16T18:23:58.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.957+0000 7f0b57fff700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0b4c0a11e0 0x7f0b4c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:58.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.957+0000 7f0b57fff700 1 -- 172.21.15.2:0/3145555133 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0b480cd090 con 0x7f0b601003b0 2023-11-16T18:23:58.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.957+0000 7f0b5ed9d700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0b4c0a11e0 0x7f0b4c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:58.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.958+0000 7f0b5ed9d700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0b4c0a11e0 0x7f0b4c0a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0b60100210 tx=0x7f0b50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:58.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:58.961+0000 7f0b57fff700 1 -- 172.21.15.2:0/3145555133 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0b48096370 con 0x7f0b601003b0 2023-11-16T18:23:58.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:58 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1007534836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:23:58.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:58 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2617526167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:23:59.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:58 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1007534836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:23:59.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:58 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2617526167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:23:59.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:58 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1007534836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:23:59.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:58 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2617526167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:23:59.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.209+0000 7f0b6503a700 1 -- 172.21.15.2:0/3145555133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b60061700 con 0x7f0b601003b0 2023-11-16T18:23:59.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.209+0000 7f0b57fff700 1 -- 172.21.15.2:0/3145555133 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10774 (secure 0 0 0) 0x7f0b48020090 con 0x7f0b601003b0 2023-11-16T18:23:59.212 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:23:59.212 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:23:07.909835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:23:59.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.212+0000 7f0b6503a700 1 -- 172.21.15.2:0/3145555133 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0b4c0a11e0 msgr2=0x7f0b4c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.212+0000 7f0b6503a700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0b4c0a11e0 0x7f0b4c0a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0b60100210 tx=0x7f0b50009260 comp rx=0 tx=0).stop 2023-11-16T18:23:59.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.212+0000 7f0b6503a700 1 -- 172.21.15.2:0/3145555133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b601003b0 msgr2=0x7f0b6019ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.212+0000 7f0b6503a700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b601003b0 0x7f0b6019ae90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800d950 tx=0x7f0b4800dd10 comp rx=0 tx=0).stop 2023-11-16T18:23:59.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.212+0000 7f0b6503a700 1 -- 172.21.15.2:0/3145555133 shutdown_connections 2023-11-16T18:23:59.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.212+0000 7f0b6503a700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0b4c0a11e0 0x7f0b4c0a3690 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.212+0000 7f0b6503a700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b600ff1c0 0x7f0b60069980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.212+0000 7f0b6503a700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b60100d40 0x7f0b6019d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.212+0000 7f0b6503a700 1 --2- 172.21.15.2:0/3145555133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b601003b0 0x7f0b6019ae90 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.212+0000 7f0b6503a700 1 -- 172.21.15.2:0/3145555133 >> 172.21.15.2:0/3145555133 conn(0x7f0b600fa7a0 msgr2=0x7f0b601037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.212+0000 7f0b6503a700 1 -- 172.21.15.2:0/3145555133 shutdown_connections 2023-11-16T18:23:59.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.212+0000 7f0b6503a700 1 -- 172.21.15.2:0/3145555133 wait complete. 2023-11-16T18:23:59.215 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T18:23:59.227 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-11-16T18:22:46.806019+0000', 'modified': '2023-11-16T18:23:07.909835+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24473, 'mds_1': 24460, 'mds_2': 24446}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24422': {'gid': 24422, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6835/17956118', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 17956118}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 17956118}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24427': {'gid': 24427, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6835/3630447129', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6834', 'nonce': 3630447129}, {'type': 'v1', 'addr': '172.21.15.188:6835', 'nonce': 3630447129}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24439': {'gid': 24439, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6837/264232322', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6836', 'nonce': 264232322}, {'type': 'v1', 'addr': '172.21.15.188:6837', 'nonce': 264232322}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24446': {'gid': 24446, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.40:6837/248461642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6836', 'nonce': 248461642}, {'type': 'v1', 'addr': '172.21.15.40:6837', 'nonce': 248461642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.188:6839/1646912892', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6838', 'nonce': 1646912892}, {'type': 'v1', 'addr': '172.21.15.188:6839', 'nonce': 1646912892}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24473': {'gid': 24473, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6839/3162585010', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 3162585010}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 3162585010}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-16T18:23:59.227 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-11-16T18:23:59.227 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-11-16T18:23:59.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.352+0000 7f75fd7aa700 1 -- 172.21.15.2:0/2128856993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75f8100220 msgr2=0x7f75f8100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.352+0000 7f75fd7aa700 1 --2- 172.21.15.2:0/2128856993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75f8100220 0x7f75f8100630 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009990 tx=0x7f75e8009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.352+0000 7f75fd7aa700 1 -- 172.21.15.2:0/2128856993 shutdown_connections 2023-11-16T18:23:59.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.352+0000 7f75fd7aa700 1 --2- 172.21.15.2:0/2128856993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f8101da0 0x7f75f8108660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.352+0000 7f75fd7aa700 1 --2- 172.21.15.2:0/2128856993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75f8100220 0x7f75f8100630 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.352+0000 7f75fd7aa700 1 --2- 172.21.15.2:0/2128856993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75f8101410 0x7f75f8101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.352+0000 7f75fd7aa700 1 -- 172.21.15.2:0/2128856993 >> 172.21.15.2:0/2128856993 conn(0x7f75f80fb820 msgr2=0x7f75f80fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.353+0000 7f75fd7aa700 1 -- 172.21.15.2:0/2128856993 shutdown_connections 2023-11-16T18:23:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.353+0000 7f75fd7aa700 1 -- 172.21.15.2:0/2128856993 wait complete. 2023-11-16T18:23:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.353+0000 7f75fd7aa700 1 Processor -- start 2023-11-16T18:23:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75fd7aa700 1 -- start start 2023-11-16T18:23:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75fd7aa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f8100220 0x7f75f8195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75fd7aa700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75f8101410 0x7f75f8196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75fd7aa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75f8101da0 0x7f75f819d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75fd7aa700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f819da90 con 0x7f75f8101da0 2023-11-16T18:23:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75fd7aa700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f819dbd0 con 0x7f75f8100220 2023-11-16T18:23:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75fd7aa700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f819dd60 con 0x7f75f8101410 2023-11-16T18:23:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75f77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75f8101da0 0x7f75f819d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75f77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75f8101da0 0x7f75f819d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49030/0 (socket says 172.21.15.2:49030) 2023-11-16T18:23:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75f77fe700 1 -- 172.21.15.2:0/3370008866 learned_addr learned my addr 172.21.15.2:0/3370008866 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f8100220 0x7f75f8195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75f67fc700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75f8101410 0x7f75f8196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75f77fe700 1 -- 172.21.15.2:0/3370008866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75f8101410 msgr2=0x7f75f8196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75f77fe700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75f8101410 0x7f75f8196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75f77fe700 1 -- 172.21.15.2:0/3370008866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f8100220 msgr2=0x7f75f8195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75f77fe700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f8100220 0x7f75f8195f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75f77fe700 1 -- 172.21.15.2:0/3370008866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e80095f0 con 0x7f75f8101da0 2023-11-16T18:23:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.354+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f8100220 0x7f75f8195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.355+0000 7f75f77fe700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75f8101da0 0x7f75f819d4c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f75f0007ee0 tx=0x7f75f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.355+0000 7f75dffff700 1 -- 172.21.15.2:0/3370008866 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f000ddd0 con 0x7f75f8101da0 2023-11-16T18:23:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.355+0000 7f75dffff700 1 -- 172.21.15.2:0/3370008866 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75f000f040 con 0x7f75f8101da0 2023-11-16T18:23:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.355+0000 7f75fd7aa700 1 -- 172.21.15.2:0/3370008866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f810bb90 con 0x7f75f8101da0 2023-11-16T18:23:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.355+0000 7f75dffff700 1 -- 172.21.15.2:0/3370008866 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f0015400 con 0x7f75f8101da0 2023-11-16T18:23:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.355+0000 7f75fd7aa700 1 -- 172.21.15.2:0/3370008866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f810ba00 con 0x7f75f8101da0 2023-11-16T18:23:59.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.357+0000 7f75dffff700 1 -- 172.21.15.2:0/3370008866 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f75f0004020 con 0x7f75f8101da0 2023-11-16T18:23:59.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.358+0000 7f75dffff700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f75e00a11e0 0x7f75e00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.358+0000 7f75dffff700 1 -- 172.21.15.2:0/3370008866 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f75f00cc780 con 0x7f75f8101da0 2023-11-16T18:23:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.358+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f75e00a11e0 0x7f75e00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.358+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f75e00a11e0 0x7f75e00a3690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f75e8006010 tx=0x7f75e800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.358+0000 7f7609936b80 1 Processor -- start 2023-11-16T18:23:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f7609936b80 1 -- start start 2023-11-16T18:23:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f7609936b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 0x55751acbc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f7609936b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbcee0 0x55751acbd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f7609936b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbd890 0x55751acc4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f7609936b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751abbd010 con 0x55751acbc050 2023-11-16T18:23:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f7609936b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751ac0f850 con 0x55751acbd890 2023-11-16T18:23:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f7609936b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751ac08ce0 con 0x55751acbcee0 2023-11-16T18:23:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f75f6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 0x55751acbc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f75f6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 0x55751acbc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49034/0 (socket says 172.21.15.2:49034) 2023-11-16T18:23:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f75f6ffd700 1 -- 172.21.15.2:0/1720641752 learned_addr learned my addr 172.21.15.2:0/1720641752 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f75f67fc700 1 --2- 172.21.15.2:0/1720641752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbcee0 0x55751acbd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f75f77fe700 1 --2- 172.21.15.2:0/1720641752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbd890 0x55751acc4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f75f6ffd700 1 -- 172.21.15.2:0/1720641752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbcee0 msgr2=0x55751acbd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/1720641752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbcee0 0x55751acbd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f75f6ffd700 1 -- 172.21.15.2:0/1720641752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbd890 msgr2=0x55751acc4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/1720641752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbd890 0x55751acc4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f75f6ffd700 1 -- 172.21.15.2:0/1720641752 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55751aa8a5e0 con 0x55751acbc050 2023-11-16T18:23:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f75f77fe700 1 --2- 172.21.15.2:0/1720641752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbd890 0x55751acc4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.359+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/1720641752 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 0x55751acbc460 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f75e8005510 tx=0x7f75e801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c4ae1b8ff41715 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.360+0000 7f75dd7fa700 1 -- 172.21.15.2:0/1720641752 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e802b070 con 0x55751acbc050 2023-11-16T18:23:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.360+0000 7f75dd7fa700 1 -- 172.21.15.2:0/1720641752 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75e801e040 con 0x55751acbc050 2023-11-16T18:23:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.360+0000 7f75dd7fa700 1 -- 172.21.15.2:0/1720641752 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8004ca0 con 0x55751acbc050 2023-11-16T18:23:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.360+0000 7f7609936b80 1 -- 172.21.15.2:0/1720641752 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 msgr2=0x55751acbc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.360+0000 7f7609936b80 1 --2- 172.21.15.2:0/1720641752 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 0x55751acbc460 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f75e8005510 tx=0x7f75e801f000 comp rx=0 tx=0).stop 2023-11-16T18:23:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.360+0000 7f7609936b80 1 -- 172.21.15.2:0/1720641752 shutdown_connections 2023-11-16T18:23:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.360+0000 7f7609936b80 1 --2- 172.21.15.2:0/1720641752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbd890 0x55751acc4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.360+0000 7f7609936b80 1 --2- 172.21.15.2:0/1720641752 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 0x55751acbc460 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.360+0000 7f7609936b80 1 --2- 172.21.15.2:0/1720641752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbcee0 0x55751acbd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.360+0000 7f7609936b80 1 -- 172.21.15.2:0/1720641752 >> 172.21.15.2:0/1720641752 conn(0x55751acad310 msgr2=0x55751acb2620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.360+0000 7f7609936b80 1 -- 172.21.15.2:0/1720641752 shutdown_connections 2023-11-16T18:23:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.360+0000 7f7609936b80 1 -- 172.21.15.2:0/1720641752 wait complete. 2023-11-16T18:23:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f7609936b80 1 -- start start 2023-11-16T18:23:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f7609936b80 1 Processor -- start 2023-11-16T18:23:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f7609936b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 0x55751acc5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f7609936b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbcee0 0x55751acc5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f7609936b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbd890 0x55751acccec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f7609936b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751abaa890 con 0x55751acbc050 2023-11-16T18:23:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f7609936b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751abaa530 con 0x55751acbd890 2023-11-16T18:23:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f7609936b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751aa8c8d0 con 0x55751acbcee0 2023-11-16T18:23:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f7609936b80 10 client.0 _ll_register_callbacks cb 0x55751abc8730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:23:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f75f6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 0x55751acc5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f75f6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 0x55751acc5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49040/0 (socket says 172.21.15.2:49040) 2023-11-16T18:23:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f75f6ffd700 1 -- 172.21.15.2:0/1817057298 learned_addr learned my addr 172.21.15.2:0/1817057298 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f75f67fc700 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbcee0 0x55751acc5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f75f77fe700 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbd890 0x55751acccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f75f6ffd700 1 -- 172.21.15.2:0/1817057298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbcee0 msgr2=0x55751acc5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.361+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbcee0 0x55751acc5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f75f6ffd700 1 -- 172.21.15.2:0/1817057298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbd890 msgr2=0x55751acccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbd890 0x55751acccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f75f6ffd700 1 -- 172.21.15.2:0/1817057298 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f8108f90 con 0x55751acbc050 2023-11-16T18:23:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f75f77fe700 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbd890 0x55751acccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 0x55751acc5940 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f75e8005f50 tx=0x7f75e8005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ea0caf68262eb4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f75daff5700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:23:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f75daff5700 1 -- 172.21.15.2:0/1817057298 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e802b070 con 0x55751acbc050 2023-11-16T18:23:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f75daff5700 1 -- 172.21.15.2:0/1817057298 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f75e800bcc0 con 0x55751acbc050 2023-11-16T18:23:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f7609936b80 1 -- 172.21.15.2:0/1817057298 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55751acaaad0 con 0x55751acbc050 2023-11-16T18:23:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f75daff5700 1 -- 172.21.15.2:0/1817057298 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75e800be80 con 0x55751acbc050 2023-11-16T18:23:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f7609936b80 10 client.15189 fetch_fsmap learned FSMap version 24 2023-11-16T18:23:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f7609936b80 1 -- 172.21.15.2:0/1817057298 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55751abcf140 con 0x55751acbc050 2023-11-16T18:23:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f75daff5700 1 -- 172.21.15.2:0/1817057298 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55751acaaad0 con 0x55751acbc050 2023-11-16T18:23:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f75daff5700 1 -- 172.21.15.2:0/1817057298 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f75e800f580 con 0x55751acbc050 2023-11-16T18:23:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f7609936b80 10 client.15189 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:23:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f7609936b80 10 client.15189 resolve_mds: resolved name 'h' to daemon mds.h 2023-11-16T18:23:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.362+0000 7f7609936b80 20 client.15189 populate_metadata read hostname 'smithi002' 2023-11-16T18:23:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.363+0000 7f7609936b80 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55751acd0730 0x55751accfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.363+0000 7f7609936b80 4 client.15189 mds_command: new command op to 24446 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:23:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.363+0000 7f7609936b80 1 -- 172.21.15.2:0/1817057298 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55751abcb050 con 0x55751acd0730 2023-11-16T18:23:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.363+0000 7f75f67fc700 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55751acd0730 0x55751accfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.363+0000 7f75f67fc700 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55751acd0730 0x55751accfdf0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dcc067cf44c0a368 server_cookie=1ef14f97ed7c2b54 in_seq=0 out_seq=0 2023-11-16T18:23:59.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.363+0000 7f75daff5700 10 client.15189 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:23:59.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f75daff5700 1 -- 172.21.15.2:0/1817057298 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55751abcb050 con 0x55751acd0730 2023-11-16T18:23:59.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f75daff5700 10 client.15189 handle_command_reply: tid=0 2023-11-16T18:23:59.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f7609936b80 1 client.15189 shutdown 2023-11-16T18:23:59.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f7609936b80 1 -- 172.21.15.2:0/1817057298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 msgr2=0x55751acc5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f7609936b80 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 0x55751acc5940 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f75e8005f50 tx=0x7f75e8005c00 comp rx=0 tx=0).stop 2023-11-16T18:23:59.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f7609936b80 1 -- 172.21.15.2:0/1817057298 shutdown_connections 2023-11-16T18:23:59.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f7609936b80 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55751acd0730 0x55751accfdf0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f7609936b80 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbd890 0x55751acccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f75daff5700 0 client.15189 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:23:59.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f7609936b80 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbc050 0x55751acc5940 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f7609936b80 1 --2- 172.21.15.2:0/1817057298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbcee0 0x55751acc5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f7609936b80 1 -- 172.21.15.2:0/1817057298 >> 172.21.15.2:0/1817057298 conn(0x55751acad310 msgr2=0x55751acb1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f7609936b80 1 -- 172.21.15.2:0/1817057298 shutdown_connections 2023-11-16T18:23:59.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.364+0000 7f7609936b80 1 -- 172.21.15.2:0/1817057298 wait complete. 2023-11-16T18:23:59.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.365+0000 7f7609936b80 20 client.15189 trim_cache size 0 max 16384 2023-11-16T18:23:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.382+0000 7f7609936b80 1 Processor -- start 2023-11-16T18:23:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.382+0000 7f7609936b80 1 -- start start 2023-11-16T18:23:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.382+0000 7f7609936b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbc050 0x55751acc56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.382+0000 7f7609936b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbcee0 0x55751accce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.382+0000 7f7609936b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbd890 0x55751acc5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.382+0000 7f7609936b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751acd1a30 con 0x55751acbcee0 2023-11-16T18:23:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.382+0000 7f7609936b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751acb31f0 con 0x55751acbc050 2023-11-16T18:23:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.382+0000 7f7609936b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751abcb050 con 0x55751acbd890 2023-11-16T18:23:59.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.383+0000 7f75f67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbcee0 0x55751accce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.383+0000 7f75f6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbc050 0x55751acc56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.383+0000 7f75f67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbcee0 0x55751accce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49050/0 (socket says 172.21.15.2:49050) 2023-11-16T18:23:59.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.383+0000 7f75f67fc700 1 -- 172.21.15.2:0/1697517032 learned_addr learned my addr 172.21.15.2:0/1697517032 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.383+0000 7f75f67fc700 1 -- 172.21.15.2:0/1697517032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbd890 msgr2=0x55751acc5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.383+0000 7f75f67fc700 1 --2- 172.21.15.2:0/1697517032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbd890 0x55751acc5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.383+0000 7f75f67fc700 1 -- 172.21.15.2:0/1697517032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbc050 msgr2=0x55751acc56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.383+0000 7f75f67fc700 1 --2- 172.21.15.2:0/1697517032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbc050 0x55751acc56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.383+0000 7f75f67fc700 1 -- 172.21.15.2:0/1697517032 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55751abcee90 con 0x55751acbcee0 2023-11-16T18:23:59.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.383+0000 7f75f67fc700 1 --2- 172.21.15.2:0/1697517032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbcee0 0x55751accce80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x55751acc64d0 tx=0x7f75ec00f700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0cfc08e0bd39de server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.383+0000 7f75ddffb700 1 -- 172.21.15.2:0/1697517032 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec016040 con 0x55751acbcee0 2023-11-16T18:23:59.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.383+0000 7f75ddffb700 1 -- 172.21.15.2:0/1697517032 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75ec01b950 con 0x55751acbcee0 2023-11-16T18:23:59.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.383+0000 7f75ddffb700 1 -- 172.21.15.2:0/1697517032 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec00e970 con 0x55751acbcee0 2023-11-16T18:23:59.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 -- 172.21.15.2:0/1697517032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbcee0 msgr2=0x55751accce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 --2- 172.21.15.2:0/1697517032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbcee0 0x55751accce80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x55751acc64d0 tx=0x7f75ec00f700 comp rx=0 tx=0).stop 2023-11-16T18:23:59.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 -- 172.21.15.2:0/1697517032 shutdown_connections 2023-11-16T18:23:59.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 --2- 172.21.15.2:0/1697517032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbd890 0x55751acc5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 --2- 172.21.15.2:0/1697517032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbc050 0x55751acc56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 --2- 172.21.15.2:0/1697517032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbcee0 0x55751accce80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 -- 172.21.15.2:0/1697517032 >> 172.21.15.2:0/1697517032 conn(0x55751acad310 msgr2=0x55751acbc420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 -- 172.21.15.2:0/1697517032 shutdown_connections 2023-11-16T18:23:59.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 -- 172.21.15.2:0/1697517032 wait complete. 2023-11-16T18:23:59.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 -- start start 2023-11-16T18:23:59.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 Processor -- start 2023-11-16T18:23:59.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbc050 0x55751acc9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbcee0 0x55751acca180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbd890 0x55751ad245d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751acca750 con 0x55751acbd890 2023-11-16T18:23:59.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751acca8c0 con 0x55751acbcee0 2023-11-16T18:23:59.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.384+0000 7f7609936b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751ad24b90 con 0x55751acbc050 2023-11-16T18:23:59.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f7609936b80 10 client.0 _ll_register_callbacks cb 0x7f75ec00b2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:23:59.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75f77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbd890 0x55751ad245d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75f77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbd890 0x55751ad245d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49058/0 (socket says 172.21.15.2:49058) 2023-11-16T18:23:59.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75f77fe700 1 -- 172.21.15.2:0/156867970 learned_addr learned my addr 172.21.15.2:0/156867970 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75f67fc700 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbcee0 0x55751acca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbc050 0x55751acc9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75f77fe700 1 -- 172.21.15.2:0/156867970 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbc050 msgr2=0x55751acc9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75f77fe700 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbc050 0x55751acc9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75f77fe700 1 -- 172.21.15.2:0/156867970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbcee0 msgr2=0x55751acca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75f77fe700 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbcee0 0x55751acca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75f77fe700 1 -- 172.21.15.2:0/156867970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ec00ba50 con 0x55751acbd890 2023-11-16T18:23:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbc050 0x55751acc9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75f77fe700 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbd890 0x55751ad245d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f75f00996b0 tx=0x7f75f00992e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721c79858e6c2e62 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75daff5700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:23:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75daff5700 1 -- 172.21.15.2:0/156867970 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f0099b80 con 0x55751acbd890 2023-11-16T18:23:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75daff5700 1 -- 172.21.15.2:0/156867970 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f75f009d750 con 0x55751acbd890 2023-11-16T18:23:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f7609936b80 1 -- 172.21.15.2:0/156867970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75e801e040 con 0x55751acbd890 2023-11-16T18:23:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f75daff5700 1 -- 172.21.15.2:0/156867970 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75f00a1b70 con 0x55751acbd890 2023-11-16T18:23:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f7609936b80 10 client.15195 fetch_fsmap learned FSMap version 24 2023-11-16T18:23:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.385+0000 7f7609936b80 1 -- 172.21.15.2:0/156867970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55751aa1fc10 con 0x55751acbd890 2023-11-16T18:23:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.386+0000 7f75daff5700 1 -- 172.21.15.2:0/156867970 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75e801e040 con 0x55751acbd890 2023-11-16T18:23:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.386+0000 7f75daff5700 1 -- 172.21.15.2:0/156867970 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f75f00a16e0 con 0x55751acbd890 2023-11-16T18:23:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.386+0000 7f7609936b80 10 client.15195 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:23:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.386+0000 7f7609936b80 10 client.15195 resolve_mds: resolved name 'h' to daemon mds.h 2023-11-16T18:23:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.386+0000 7f7609936b80 20 client.15195 populate_metadata read hostname 'smithi002' 2023-11-16T18:23:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.386+0000 7f7609936b80 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55751ad29970 0x55751ad26320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.386+0000 7f7609936b80 4 client.15195 mds_command: new command op to 24446 tid=0[{"prefix": "status"}] 2023-11-16T18:23:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.386+0000 7f7609936b80 1 -- 172.21.15.2:0/156867970 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55751abd7610 con 0x55751ad29970 2023-11-16T18:23:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.386+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55751ad29970 0x55751ad26320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f75f6ffd700 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55751ad29970 0x55751ad26320 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=45a11b35917f8bfc server_cookie=896bd0bae4fc5f8f in_seq=0 out_seq=0 2023-11-16T18:23:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f75daff5700 10 client.15195 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:23:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f75daff5700 1 -- 172.21.15.2:0/156867970 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55751abd7610 con 0x55751ad29970 2023-11-16T18:23:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f75daff5700 10 client.15195 handle_command_reply: tid=0 2023-11-16T18:23:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f7609936b80 1 client.15195 shutdown 2023-11-16T18:23:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f7609936b80 1 -- 172.21.15.2:0/156867970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbd890 msgr2=0x55751ad245d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f7609936b80 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbd890 0x55751ad245d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f75f00996b0 tx=0x7f75f00992e0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f7609936b80 1 -- 172.21.15.2:0/156867970 shutdown_connections 2023-11-16T18:23:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f7609936b80 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55751ad29970 0x55751ad26320 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f7609936b80 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55751acbc050 0x55751acc9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f75daff5700 0 client.15195 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:23:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f7609936b80 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55751acbd890 0x55751ad245d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f7609936b80 1 --2- 172.21.15.2:0/156867970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55751acbcee0 0x55751acca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f7609936b80 1 -- 172.21.15.2:0/156867970 >> 172.21.15.2:0/156867970 conn(0x55751acad310 msgr2=0x55751acc56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f7609936b80 1 -- 172.21.15.2:0/156867970 shutdown_connections 2023-11-16T18:23:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f7609936b80 1 -- 172.21.15.2:0/156867970 wait complete. 2023-11-16T18:23:59.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.387+0000 7f7609936b80 20 client.15195 trim_cache size 0 max 16384 2023-11-16T18:23:59.397 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:23:59.397 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "08160fac-84ac-11ee-8dbe-212e2dc638e7", 2023-11-16T18:23:59.397 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 2, 2023-11-16T18:23:59.397 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24446, 2023-11-16T18:23:59.397 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T18:23:59.397 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T18:23:59.398 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T18:23:59.398 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 52.484233211000003, 2023-11-16T18:23:59.398 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 24, 2023-11-16T18:23:59.398 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 88, 2023-11-16T18:23:59.398 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 86, 2023-11-16T18:23:59.398 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 87.030545876999994 2023-11-16T18:23:59.398 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:23:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.388+0000 7f75fd7aa700 1 -- 172.21.15.2:0/3370008866 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f75e00a11e0 msgr2=0x7f75e00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.388+0000 7f75fd7aa700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f75e00a11e0 0x7f75e00a3690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f75e8006010 tx=0x7f75e800b540 comp rx=0 tx=0).stop 2023-11-16T18:23:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.388+0000 7f75fd7aa700 1 -- 172.21.15.2:0/3370008866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75f8101da0 msgr2=0x7f75f819d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.388+0000 7f75fd7aa700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75f8101da0 0x7f75f819d4c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f75f0007ee0 tx=0x7f75f000d3b0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.388+0000 7f75fd7aa700 1 -- 172.21.15.2:0/3370008866 shutdown_connections 2023-11-16T18:23:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.388+0000 7f75fd7aa700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f75e00a11e0 0x7f75e00a3690 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.388+0000 7f75fd7aa700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75f8101da0 0x7f75f819d4c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.388+0000 7f75fd7aa700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f8100220 0x7f75f8195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.388+0000 7f75fd7aa700 1 --2- 172.21.15.2:0/3370008866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75f8101410 0x7f75f8196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.388+0000 7f75fd7aa700 1 -- 172.21.15.2:0/3370008866 >> 172.21.15.2:0/3370008866 conn(0x7f75f80fb820 msgr2=0x7f75f80fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.388+0000 7f75fd7aa700 1 -- 172.21.15.2:0/3370008866 shutdown_connections 2023-11-16T18:23:59.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.388+0000 7f75fd7aa700 1 -- 172.21.15.2:0/3370008866 wait complete. 2023-11-16T18:23:59.401 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-11-16T18:23:59.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.527+0000 7f0e27ef5700 1 -- 172.21.15.2:0/1232409551 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e20100ce0 msgr2=0x7f0e201030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.527+0000 7f0e27ef5700 1 --2- 172.21.15.2:0/1232409551 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e20100ce0 0x7f0e201030c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009990 tx=0x7f0e1c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.527+0000 7f0e27ef5700 1 -- 172.21.15.2:0/1232409551 shutdown_connections 2023-11-16T18:23:59.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.527+0000 7f0e27ef5700 1 --2- 172.21.15.2:0/1232409551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e20105f20 0x7f0e20108380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.527+0000 7f0e27ef5700 1 --2- 172.21.15.2:0/1232409551 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e20100ce0 0x7f0e201030c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.527+0000 7f0e27ef5700 1 --2- 172.21.15.2:0/1232409551 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e20103600 0x7f0e201059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.527+0000 7f0e27ef5700 1 -- 172.21.15.2:0/1232409551 >> 172.21.15.2:0/1232409551 conn(0x7f0e200fa740 msgr2=0x7f0e200fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.527+0000 7f0e27ef5700 1 -- 172.21.15.2:0/1232409551 shutdown_connections 2023-11-16T18:23:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.527+0000 7f0e27ef5700 1 -- 172.21.15.2:0/1232409551 wait complete. 2023-11-16T18:23:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.527+0000 7f0e27ef5700 1 Processor -- start 2023-11-16T18:23:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.527+0000 7f0e27ef5700 1 -- start start 2023-11-16T18:23:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.527+0000 7f0e27ef5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e20100ce0 0x7f0e20100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.527+0000 7f0e27ef5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e20103600 0x7f0e200fec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e27ef5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e20105f20 0x7f0e200ff1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e27ef5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e200ff970 con 0x7f0e20100ce0 2023-11-16T18:23:59.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e27ef5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e200ffae0 con 0x7f0e20105f20 2023-11-16T18:23:59.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e27ef5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2019ff90 con 0x7f0e20103600 2023-11-16T18:23:59.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e25c91700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e20100ce0 0x7f0e20100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e25c91700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e20100ce0 0x7f0e20100600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49068/0 (socket says 172.21.15.2:49068) 2023-11-16T18:23:59.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e25c91700 1 -- 172.21.15.2:0/3372309495 learned_addr learned my addr 172.21.15.2:0/3372309495 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e25c91700 1 -- 172.21.15.2:0/3372309495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e20103600 msgr2=0x7f0e200fec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e25c91700 1 --2- 172.21.15.2:0/3372309495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e20103600 0x7f0e200fec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e25c91700 1 -- 172.21.15.2:0/3372309495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e20105f20 msgr2=0x7f0e200ff1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e25c91700 1 --2- 172.21.15.2:0/3372309495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e20105f20 0x7f0e200ff1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e25c91700 1 -- 172.21.15.2:0/3372309495 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c0095f0 con 0x7f0e20100ce0 2023-11-16T18:23:59.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e25c91700 1 --2- 172.21.15.2:0/3372309495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e20100ce0 0x7f0e20100600 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c000c00 tx=0x7f0e1c00f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e16ffd700 1 -- 172.21.15.2:0/3372309495 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c01d070 con 0x7f0e20100ce0 2023-11-16T18:23:59.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e16ffd700 1 -- 172.21.15.2:0/3372309495 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e1c00fc00 con 0x7f0e20100ce0 2023-11-16T18:23:59.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e27ef5700 1 -- 172.21.15.2:0/3372309495 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2010bbc0 con 0x7f0e20100ce0 2023-11-16T18:23:59.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e16ffd700 1 -- 172.21.15.2:0/3372309495 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c017650 con 0x7f0e20100ce0 2023-11-16T18:23:59.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.528+0000 7f0e27ef5700 1 -- 172.21.15.2:0/3372309495 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2010ba30 con 0x7f0e20100ce0 2023-11-16T18:23:59.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.530+0000 7f0e16ffd700 1 -- 172.21.15.2:0/3372309495 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f0e1c022470 con 0x7f0e20100ce0 2023-11-16T18:23:59.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.531+0000 7f0e16ffd700 1 --2- 172.21.15.2:0/3372309495 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0e0c0a11e0 0x7f0e0c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.531+0000 7f0e25490700 1 --2- 172.21.15.2:0/3372309495 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0e0c0a11e0 0x7f0e0c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.531+0000 7f0e16ffd700 1 -- 172.21.15.2:0/3372309495 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0e1c0ce0c0 con 0x7f0e20100ce0 2023-11-16T18:23:59.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.532+0000 7f0e25490700 1 --2- 172.21.15.2:0/3372309495 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0e0c0a11e0 0x7f0e0c0a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0e10005950 tx=0x7f0e100058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.532+0000 7f0e34081b80 1 Processor -- start 2023-11-16T18:23:59.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.532+0000 7f0e34081b80 1 -- start start 2023-11-16T18:23:59.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.532+0000 7f0e34081b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb1fbb0 0x55cdebb1ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.532+0000 7f0e34081b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb20500 0x55cdebb20ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.532+0000 7f0e34081b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 0x55cdebb21530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.532+0000 7f0e34081b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdeba25010 con 0x55cdebb210c0 2023-11-16T18:23:59.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.532+0000 7f0e34081b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdeba77850 con 0x55cdebb1fbb0 2023-11-16T18:23:59.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.532+0000 7f0e34081b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdeba70ce0 con 0x55cdebb20500 2023-11-16T18:23:59.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.532+0000 7f0e26492700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 0x55cdebb21530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.532+0000 7f0e25c91700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb1fbb0 0x55cdebb1ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.532+0000 7f0e26492700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 0x55cdebb21530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49080/0 (socket says 172.21.15.2:49080) 2023-11-16T18:23:59.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.533+0000 7f0e26492700 1 -- 172.21.15.2:0/1958180036 learned_addr learned my addr 172.21.15.2:0/1958180036 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.533+0000 7f0e25490700 1 --2- 172.21.15.2:0/1958180036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb20500 0x55cdebb20ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.533+0000 7f0e25c91700 1 -- 172.21.15.2:0/1958180036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb20500 msgr2=0x55cdebb20ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.533+0000 7f0e25c91700 1 --2- 172.21.15.2:0/1958180036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb20500 0x55cdebb20ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.533+0000 7f0e25c91700 1 -- 172.21.15.2:0/1958180036 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 msgr2=0x55cdebb21530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.533+0000 7f0e25c91700 1 --2- 172.21.15.2:0/1958180036 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 0x55cdebb21530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.533+0000 7f0e25c91700 1 -- 172.21.15.2:0/1958180036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cdeb8f25e0 con 0x55cdebb1fbb0 2023-11-16T18:23:59.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.533+0000 7f0e26492700 1 --2- 172.21.15.2:0/1958180036 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 0x55cdebb21530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:23:59.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.533+0000 7f0e25c91700 1 --2- 172.21.15.2:0/1958180036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb1fbb0 0x55cdebb1ffc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0cb700 tx=0x7f0e1c009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b9df29d6ecf6a11 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.533+0000 7f0e0bfff700 1 -- 172.21.15.2:0/1958180036 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c0a3450 con 0x55cdebb1fbb0 2023-11-16T18:23:59.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.533+0000 7f0e0bfff700 1 -- 172.21.15.2:0/1958180036 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e1c099430 con 0x55cdebb1fbb0 2023-11-16T18:23:59.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.533+0000 7f0e0bfff700 1 -- 172.21.15.2:0/1958180036 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c0a2fc0 con 0x55cdebb1fbb0 2023-11-16T18:23:59.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.534+0000 7f0e34081b80 1 -- 172.21.15.2:0/1958180036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb1fbb0 msgr2=0x55cdebb1ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.534+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1958180036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb1fbb0 0x55cdebb1ffc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0cb700 tx=0x7f0e1c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.534+0000 7f0e34081b80 1 -- 172.21.15.2:0/1958180036 shutdown_connections 2023-11-16T18:23:59.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.534+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1958180036 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 0x55cdebb21530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.534+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1958180036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb1fbb0 0x55cdebb1ffc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.534+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1958180036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb20500 0x55cdebb20ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.534+0000 7f0e34081b80 1 -- 172.21.15.2:0/1958180036 >> 172.21.15.2:0/1958180036 conn(0x55cdebb15310 msgr2=0x55cdebb1a620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.534+0000 7f0e34081b80 1 -- 172.21.15.2:0/1958180036 shutdown_connections 2023-11-16T18:23:59.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.534+0000 7f0e34081b80 1 -- 172.21.15.2:0/1958180036 wait complete. 2023-11-16T18:23:59.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.534+0000 7f0e34081b80 1 -- start start 2023-11-16T18:23:59.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.535+0000 7f0e34081b80 1 Processor -- start 2023-11-16T18:23:59.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.535+0000 7f0e34081b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb1fbb0 0x55cdebb292a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.535+0000 7f0e34081b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb20500 0x55cdebb297e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.535+0000 7f0e34081b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb210c0 0x55cdebb30820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.535+0000 7f0e34081b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdeba12890 con 0x55cdebb1fbb0 2023-11-16T18:23:59.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.535+0000 7f0e34081b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdeba12530 con 0x55cdebb210c0 2023-11-16T18:23:59.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.535+0000 7f0e34081b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdeb8f48d0 con 0x55cdebb20500 2023-11-16T18:23:59.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.535+0000 7f0e34081b80 10 client.0 _ll_register_callbacks cb 0x55cdeba30730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:23:59.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.535+0000 7f0e25c91700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb1fbb0 0x55cdebb292a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.535+0000 7f0e25c91700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb1fbb0 0x55cdebb292a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49082/0 (socket says 172.21.15.2:49082) 2023-11-16T18:23:59.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.535+0000 7f0e25c91700 1 -- 172.21.15.2:0/1087595502 learned_addr learned my addr 172.21.15.2:0/1087595502 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.535+0000 7f0e25490700 1 --2- 172.21.15.2:0/1087595502 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb20500 0x55cdebb297e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.535+0000 7f0e26492700 1 --2- 172.21.15.2:0/1087595502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb210c0 0x55cdebb30820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e25c91700 1 -- 172.21.15.2:0/1087595502 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb20500 msgr2=0x55cdebb297e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e25c91700 1 --2- 172.21.15.2:0/1087595502 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb20500 0x55cdebb297e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e25c91700 1 -- 172.21.15.2:0/1087595502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb210c0 msgr2=0x55cdebb30820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e25c91700 1 --2- 172.21.15.2:0/1087595502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb210c0 0x55cdebb30820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e25c91700 1 -- 172.21.15.2:0/1087595502 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2010ba30 con 0x55cdebb1fbb0 2023-11-16T18:23:59.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e25c91700 1 --2- 172.21.15.2:0/1087595502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb1fbb0 0x55cdebb292a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c09ad90 tx=0x7f0e1c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e34467d432eb3a89 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e097fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:23:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e097fa700 1 -- 172.21.15.2:0/1087595502 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c0a3450 con 0x55cdebb1fbb0 2023-11-16T18:23:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e097fa700 1 -- 172.21.15.2:0/1087595502 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0e1c022720 con 0x55cdebb1fbb0 2023-11-16T18:23:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e34081b80 1 -- 172.21.15.2:0/1087595502 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cdebb12ad0 con 0x55cdebb1fbb0 2023-11-16T18:23:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e097fa700 1 -- 172.21.15.2:0/1087595502 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e1c0228e0 con 0x55cdebb1fbb0 2023-11-16T18:23:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e34081b80 10 client.15207 fetch_fsmap learned FSMap version 24 2023-11-16T18:23:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e34081b80 1 -- 172.21.15.2:0/1087595502 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55cdeba37140 con 0x55cdebb1fbb0 2023-11-16T18:23:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e097fa700 1 -- 172.21.15.2:0/1087595502 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cdebb12ad0 con 0x55cdebb1fbb0 2023-11-16T18:23:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e097fa700 1 -- 172.21.15.2:0/1087595502 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0e1c09a250 con 0x55cdebb1fbb0 2023-11-16T18:23:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e34081b80 10 client.15207 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:23:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e34081b80 10 client.15207 resolve_mds: resolved name 'i' to daemon mds.i 2023-11-16T18:23:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e34081b80 20 client.15207 populate_metadata read hostname 'smithi002' 2023-11-16T18:23:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1087595502 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55cdebb345f0 0x55cdebb33750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e34081b80 4 client.15207 mds_command: new command op to 24460 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:23:59.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.536+0000 7f0e34081b80 1 -- 172.21.15.2:0/1087595502 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cdeba33050 con 0x55cdebb345f0 2023-11-16T18:23:59.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.537+0000 7f0e25490700 1 --2- 172.21.15.2:0/1087595502 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55cdebb345f0 0x55cdebb33750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.537+0000 7f0e25490700 1 --2- 172.21.15.2:0/1087595502 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55cdebb345f0 0x55cdebb33750 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=adff3eb7e8e308db server_cookie=e165c4b43b16b15e in_seq=0 out_seq=0 2023-11-16T18:23:59.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.537+0000 7f0e097fa700 10 client.15207 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:23:59.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e097fa700 1 -- 172.21.15.2:0/1087595502 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55cdeba33050 con 0x55cdebb345f0 2023-11-16T18:23:59.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e097fa700 10 client.15207 handle_command_reply: tid=0 2023-11-16T18:23:59.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e34081b80 1 client.15207 shutdown 2023-11-16T18:23:59.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e34081b80 1 -- 172.21.15.2:0/1087595502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb1fbb0 msgr2=0x55cdebb292a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1087595502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb1fbb0 0x55cdebb292a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c09ad90 tx=0x7f0e1c009310 comp rx=0 tx=0).stop 2023-11-16T18:23:59.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e34081b80 1 -- 172.21.15.2:0/1087595502 shutdown_connections 2023-11-16T18:23:59.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1087595502 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55cdebb345f0 0x55cdebb33750 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1087595502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb210c0 0x55cdebb30820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1087595502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb1fbb0 0x55cdebb292a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1087595502 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb20500 0x55cdebb297e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e34081b80 1 -- 172.21.15.2:0/1087595502 >> 172.21.15.2:0/1087595502 conn(0x55cdebb15310 msgr2=0x55cdebb19d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e097fa700 0 client.15207 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:23:59.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e34081b80 1 -- 172.21.15.2:0/1087595502 shutdown_connections 2023-11-16T18:23:59.554 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.538+0000 7f0e34081b80 1 -- 172.21.15.2:0/1087595502 wait complete. 2023-11-16T18:23:59.554 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.539+0000 7f0e34081b80 20 client.15207 trim_cache size 0 max 16384 2023-11-16T18:23:59.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.556+0000 7f0e34081b80 1 Processor -- start 2023-11-16T18:23:59.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.556+0000 7f0e34081b80 1 -- start start 2023-11-16T18:23:59.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.556+0000 7f0e34081b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb1fbb0 0x55cdebb274a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.556+0000 7f0e34081b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb20500 0x55cdebb307e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.556+0000 7f0e34081b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 0x55cdebb23fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.556+0000 7f0e34081b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdebb28e70 con 0x55cdebb210c0 2023-11-16T18:23:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.556+0000 7f0e34081b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdeba70ce0 con 0x55cdebb20500 2023-11-16T18:23:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.556+0000 7f0e34081b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdeba25010 con 0x55cdebb1fbb0 2023-11-16T18:23:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.556+0000 7f0e26492700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 0x55cdebb23fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.557+0000 7f0e26492700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 0x55cdebb23fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49094/0 (socket says 172.21.15.2:49094) 2023-11-16T18:23:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.557+0000 7f0e26492700 1 -- 172.21.15.2:0/1994050873 learned_addr learned my addr 172.21.15.2:0/1994050873 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.557+0000 7f0e25490700 1 --2- 172.21.15.2:0/1994050873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb20500 0x55cdebb307e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.557+0000 7f0e25c91700 1 --2- 172.21.15.2:0/1994050873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb1fbb0 0x55cdebb274a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.557+0000 7f0e26492700 1 -- 172.21.15.2:0/1994050873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb1fbb0 msgr2=0x55cdebb274a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.557+0000 7f0e26492700 1 --2- 172.21.15.2:0/1994050873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb1fbb0 0x55cdebb274a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.557+0000 7f0e26492700 1 -- 172.21.15.2:0/1994050873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb20500 msgr2=0x55cdebb307e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.557+0000 7f0e26492700 1 --2- 172.21.15.2:0/1994050873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb20500 0x55cdebb307e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.557+0000 7f0e26492700 1 -- 172.21.15.2:0/1994050873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cdeba36e90 con 0x55cdebb210c0 2023-11-16T18:23:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.557+0000 7f0e26492700 1 --2- 172.21.15.2:0/1994050873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 0x55cdebb23fb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0e18012380 tx=0x7f0e18012690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6267092d2f7d58 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.557+0000 7f0e14ff9700 1 -- 172.21.15.2:0/1994050873 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18008d90 con 0x55cdebb210c0 2023-11-16T18:23:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.557+0000 7f0e14ff9700 1 -- 172.21.15.2:0/1994050873 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e18012ed0 con 0x55cdebb210c0 2023-11-16T18:23:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.557+0000 7f0e14ff9700 1 -- 172.21.15.2:0/1994050873 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18016430 con 0x55cdebb210c0 2023-11-16T18:23:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.558+0000 7f0e34081b80 1 -- 172.21.15.2:0/1994050873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 msgr2=0x55cdebb23fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.558+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1994050873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 0x55cdebb23fb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0e18012380 tx=0x7f0e18012690 comp rx=0 tx=0).stop 2023-11-16T18:23:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.558+0000 7f0e34081b80 1 -- 172.21.15.2:0/1994050873 shutdown_connections 2023-11-16T18:23:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.558+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1994050873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb1fbb0 0x55cdebb274a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.558+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1994050873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb210c0 0x55cdebb23fb0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.558+0000 7f0e34081b80 1 --2- 172.21.15.2:0/1994050873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb20500 0x55cdebb307e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.558+0000 7f0e34081b80 1 -- 172.21.15.2:0/1994050873 >> 172.21.15.2:0/1994050873 conn(0x55cdebb15310 msgr2=0x55cdebb33750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.558+0000 7f0e34081b80 1 -- 172.21.15.2:0/1994050873 shutdown_connections 2023-11-16T18:23:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.558+0000 7f0e34081b80 1 -- 172.21.15.2:0/1994050873 wait complete. 2023-11-16T18:23:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.558+0000 7f0e34081b80 1 -- start start 2023-11-16T18:23:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.558+0000 7f0e34081b80 1 Processor -- start 2023-11-16T18:23:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e34081b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb1fbb0 0x55cdebb2ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e34081b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb20500 0x55cdebb2b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e34081b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb210c0 0x55cdebb30270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e34081b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdebb2a030 con 0x55cdebb20500 2023-11-16T18:23:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e34081b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdebb307b0 con 0x55cdebb210c0 2023-11-16T18:23:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e34081b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdebb30940 con 0x55cdebb1fbb0 2023-11-16T18:23:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e34081b80 10 client.0 _ll_register_callbacks cb 0x7f0e18008860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:23:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e25490700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb20500 0x55cdebb2b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e25490700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb20500 0x55cdebb2b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49098/0 (socket says 172.21.15.2:49098) 2023-11-16T18:23:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e25490700 1 -- 172.21.15.2:0/165818018 learned_addr learned my addr 172.21.15.2:0/165818018 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e26492700 1 --2- 172.21.15.2:0/165818018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb210c0 0x55cdebb30270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e25c91700 1 --2- 172.21.15.2:0/165818018 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb1fbb0 0x55cdebb2ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e25490700 1 -- 172.21.15.2:0/165818018 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb1fbb0 msgr2=0x55cdebb2ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e25490700 1 --2- 172.21.15.2:0/165818018 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb1fbb0 0x55cdebb2ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e25490700 1 -- 172.21.15.2:0/165818018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb210c0 msgr2=0x55cdebb30270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e25490700 1 --2- 172.21.15.2:0/165818018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb210c0 0x55cdebb30270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e25490700 1 -- 172.21.15.2:0/165818018 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e18010040 con 0x55cdebb20500 2023-11-16T18:23:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e25490700 1 --2- 172.21.15.2:0/165818018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb20500 0x55cdebb2b340 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0e10005d90 tx=0x7f0e10005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95641f136b06f9c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e097fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:23:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e097fa700 1 -- 172.21.15.2:0/165818018 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e10024050 con 0x55cdebb20500 2023-11-16T18:23:59.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e097fa700 1 -- 172.21.15.2:0/165818018 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0e10017040 con 0x55cdebb20500 2023-11-16T18:23:59.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.559+0000 7f0e34081b80 1 -- 172.21.15.2:0/165818018 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e1c099430 con 0x55cdebb20500 2023-11-16T18:23:59.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.560+0000 7f0e097fa700 1 -- 172.21.15.2:0/165818018 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e1000db70 con 0x55cdebb20500 2023-11-16T18:23:59.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.560+0000 7f0e34081b80 10 client.15213 fetch_fsmap learned FSMap version 24 2023-11-16T18:23:59.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.560+0000 7f0e34081b80 1 -- 172.21.15.2:0/165818018 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55cdeb887c10 con 0x55cdebb20500 2023-11-16T18:23:59.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.560+0000 7f0e097fa700 1 -- 172.21.15.2:0/165818018 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e1c099430 con 0x55cdebb20500 2023-11-16T18:23:59.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.560+0000 7f0e097fa700 1 -- 172.21.15.2:0/165818018 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0e10010460 con 0x55cdebb20500 2023-11-16T18:23:59.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.560+0000 7f0e34081b80 10 client.15213 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:23:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.560+0000 7f0e34081b80 10 client.15213 resolve_mds: resolved name 'i' to daemon mds.i 2023-11-16T18:23:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.560+0000 7f0e34081b80 20 client.15213 populate_metadata read hostname 'smithi002' 2023-11-16T18:23:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.560+0000 7f0e34081b80 1 --2- 172.21.15.2:0/165818018 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55cdebb8c7d0 0x55cdebb89120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.560+0000 7f0e34081b80 4 client.15213 mds_command: new command op to 24460 tid=0[{"prefix": "status"}] 2023-11-16T18:23:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.560+0000 7f0e34081b80 1 -- 172.21.15.2:0/165818018 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cdeba3f610 con 0x55cdebb8c7d0 2023-11-16T18:23:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.560+0000 7f0e25c91700 1 --2- 172.21.15.2:0/165818018 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55cdebb8c7d0 0x55cdebb89120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e097fa700 10 client.15213 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:23:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e25c91700 1 --2- 172.21.15.2:0/165818018 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55cdebb8c7d0 0x55cdebb89120 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aeaf541cb6bba61c server_cookie=ca5aa6a1617c2f3d in_seq=0 out_seq=0 2023-11-16T18:23:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e097fa700 1 -- 172.21.15.2:0/165818018 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55cdeba3f610 con 0x55cdebb8c7d0 2023-11-16T18:23:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e097fa700 10 client.15213 handle_command_reply: tid=0 2023-11-16T18:23:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e34081b80 1 client.15213 shutdown 2023-11-16T18:23:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e34081b80 1 -- 172.21.15.2:0/165818018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb20500 msgr2=0x55cdebb2b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e34081b80 1 --2- 172.21.15.2:0/165818018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb20500 0x55cdebb2b340 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0e10005d90 tx=0x7f0e10005d20 comp rx=0 tx=0).stop 2023-11-16T18:23:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e34081b80 1 -- 172.21.15.2:0/165818018 shutdown_connections 2023-11-16T18:23:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e34081b80 1 --2- 172.21.15.2:0/165818018 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55cdebb8c7d0 0x55cdebb89120 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e34081b80 1 --2- 172.21.15.2:0/165818018 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cdebb1fbb0 0x55cdebb2ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e097fa700 0 client.15213 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:23:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e34081b80 1 --2- 172.21.15.2:0/165818018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdebb210c0 0x55cdebb30270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e34081b80 1 --2- 172.21.15.2:0/165818018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cdebb20500 0x55cdebb2b340 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e34081b80 1 -- 172.21.15.2:0/165818018 >> 172.21.15.2:0/165818018 conn(0x55cdebb15310 msgr2=0x55cdebb274a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e34081b80 1 -- 172.21.15.2:0/165818018 shutdown_connections 2023-11-16T18:23:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e34081b80 1 -- 172.21.15.2:0/165818018 wait complete. 2023-11-16T18:23:59.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.561+0000 7f0e34081b80 20 client.15213 trim_cache size 0 max 16384 2023-11-16T18:23:59.580 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:23:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "08160fac-84ac-11ee-8dbe-212e2dc638e7", 2023-11-16T18:23:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 1, 2023-11-16T18:23:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24460, 2023-11-16T18:23:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T18:23:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T18:23:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T18:23:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 53.666150133999999, 2023-11-16T18:23:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 24, 2023-11-16T18:23:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 88, 2023-11-16T18:23:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 86, 2023-11-16T18:23:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 84.936138575000001 2023-11-16T18:23:59.581 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:23:59.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.562+0000 7f0e27ef5700 1 -- 172.21.15.2:0/3372309495 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0e0c0a11e0 msgr2=0x7f0e0c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.562+0000 7f0e27ef5700 1 --2- 172.21.15.2:0/3372309495 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0e0c0a11e0 0x7f0e0c0a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0e10005950 tx=0x7f0e100058e0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.562+0000 7f0e27ef5700 1 -- 172.21.15.2:0/3372309495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e20100ce0 msgr2=0x7f0e20100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.562+0000 7f0e27ef5700 1 --2- 172.21.15.2:0/3372309495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e20100ce0 0x7f0e20100600 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c000c00 tx=0x7f0e1c00f710 comp rx=0 tx=0).stop 2023-11-16T18:23:59.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.562+0000 7f0e27ef5700 1 -- 172.21.15.2:0/3372309495 shutdown_connections 2023-11-16T18:23:59.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.562+0000 7f0e27ef5700 1 --2- 172.21.15.2:0/3372309495 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f0e0c0a11e0 0x7f0e0c0a3690 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.562+0000 7f0e27ef5700 1 --2- 172.21.15.2:0/3372309495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e20105f20 0x7f0e200ff1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.562+0000 7f0e27ef5700 1 --2- 172.21.15.2:0/3372309495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e20100ce0 0x7f0e20100600 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.562+0000 7f0e27ef5700 1 --2- 172.21.15.2:0/3372309495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e20103600 0x7f0e200fec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.562+0000 7f0e27ef5700 1 -- 172.21.15.2:0/3372309495 >> 172.21.15.2:0/3372309495 conn(0x7f0e200fa740 msgr2=0x7f0e20101940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.562+0000 7f0e27ef5700 1 -- 172.21.15.2:0/3372309495 shutdown_connections 2023-11-16T18:23:59.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.562+0000 7f0e27ef5700 1 -- 172.21.15.2:0/3372309495 wait complete. 2023-11-16T18:23:59.584 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-16T18:23:59.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.709+0000 7f7cad2fa700 1 -- 172.21.15.2:0/2677648966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca80ff150 msgr2=0x7f7ca80ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.709+0000 7f7cad2fa700 1 --2- 172.21.15.2:0/2677648966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca80ff150 0x7f7ca80ff560 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009990 tx=0x7f7c98009ca0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.709+0000 7f7cad2fa700 1 -- 172.21.15.2:0/2677648966 shutdown_connections 2023-11-16T18:23:59.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.709+0000 7f7cad2fa700 1 --2- 172.21.15.2:0/2677648966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ca8100cd0 0x7f7ca8107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.709+0000 7f7cad2fa700 1 --2- 172.21.15.2:0/2677648966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca80ff150 0x7f7ca80ff560 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.709+0000 7f7cad2fa700 1 --2- 172.21.15.2:0/2677648966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca8100340 0x7f7ca8100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.709+0000 7f7cad2fa700 1 -- 172.21.15.2:0/2677648966 >> 172.21.15.2:0/2677648966 conn(0x7f7ca80fa750 msgr2=0x7f7ca80fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.709+0000 7f7cad2fa700 1 -- 172.21.15.2:0/2677648966 shutdown_connections 2023-11-16T18:23:59.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.709+0000 7f7cad2fa700 1 -- 172.21.15.2:0/2677648966 wait complete. 2023-11-16T18:23:59.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.710+0000 7f7cad2fa700 1 Processor -- start 2023-11-16T18:23:59.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.710+0000 7f7cad2fa700 1 -- start start 2023-11-16T18:23:59.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.710+0000 7f7cad2fa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca80ff150 0x7f7ca81030e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.710+0000 7f7cad2fa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca8100340 0x7f7ca8101720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.710+0000 7f7cad2fa700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ca8100cd0 0x7f7ca8101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.710+0000 7f7cad2fa700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca8102400 con 0x7f7ca8100340 2023-11-16T18:23:59.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.710+0000 7f7cad2fa700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca8102570 con 0x7f7ca80ff150 2023-11-16T18:23:59.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.710+0000 7f7cad2fa700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca81a0010 con 0x7f7ca8100cd0 2023-11-16T18:23:59.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.710+0000 7f7ca67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca8100340 0x7f7ca8101720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.710+0000 7f7ca67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca8100340 0x7f7ca8101720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49112/0 (socket says 172.21.15.2:49112) 2023-11-16T18:23:59.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.710+0000 7f7ca67fc700 1 -- 172.21.15.2:0/2985840477 learned_addr learned my addr 172.21.15.2:0/2985840477 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.711+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/2985840477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca80ff150 0x7f7ca81030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.711+0000 7f7ca67fc700 1 -- 172.21.15.2:0/2985840477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ca8100cd0 msgr2=0x7f7ca8101c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.711+0000 7f7ca67fc700 1 --2- 172.21.15.2:0/2985840477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ca8100cd0 0x7f7ca8101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.711+0000 7f7ca67fc700 1 -- 172.21.15.2:0/2985840477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca80ff150 msgr2=0x7f7ca81030e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.711+0000 7f7ca67fc700 1 --2- 172.21.15.2:0/2985840477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca80ff150 0x7f7ca81030e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.711+0000 7f7ca67fc700 1 -- 172.21.15.2:0/2985840477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c980095f0 con 0x7f7ca8100340 2023-11-16T18:23:59.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.711+0000 7f7ca67fc700 1 --2- 172.21.15.2:0/2985840477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca8100340 0x7f7ca8101720 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7c9000d980 tx=0x7f7c9000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.711+0000 7f7c9ffff700 1 -- 172.21.15.2:0/2985840477 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c90004220 con 0x7f7ca8100340 2023-11-16T18:23:59.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.711+0000 7f7c9ffff700 1 -- 172.21.15.2:0/2985840477 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c90007640 con 0x7f7ca8100340 2023-11-16T18:23:59.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.711+0000 7f7cad2fa700 1 -- 172.21.15.2:0/2985840477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca810aac0 con 0x7f7ca8100340 2023-11-16T18:23:59.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.711+0000 7f7c9ffff700 1 -- 172.21.15.2:0/2985840477 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c90005610 con 0x7f7ca8100340 2023-11-16T18:23:59.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.711+0000 7f7cad2fa700 1 -- 172.21.15.2:0/2985840477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca810a930 con 0x7f7ca8100340 2023-11-16T18:23:59.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.713+0000 7f7c9ffff700 1 -- 172.21.15.2:0/2985840477 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f7c90004750 con 0x7f7ca8100340 2023-11-16T18:23:59.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.714+0000 7f7c9ffff700 1 --2- 172.21.15.2:0/2985840477 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7c940a11e0 0x7f7c940a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.714+0000 7f7c9ffff700 1 -- 172.21.15.2:0/2985840477 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7c900cccc0 con 0x7f7ca8100340 2023-11-16T18:23:59.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.714+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/2985840477 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7c940a11e0 0x7f7c940a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.714+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/2985840477 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7c940a11e0 0x7f7c940a3690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800b5c0 tx=0x7f7c98005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7cb9486b80 1 Processor -- start 2023-11-16T18:23:59.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7cb9486b80 1 -- start start 2023-11-16T18:23:59.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7cb9486b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e863bb0 0x56079e863fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7cb9486b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e864ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7cb9486b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 0x56079e865530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7cb9486b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079e769010 con 0x56079e864500 2023-11-16T18:23:59.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7cb9486b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079e7bb850 con 0x56079e863bb0 2023-11-16T18:23:59.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7cb9486b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079e7b4ce0 con 0x56079e8650c0 2023-11-16T18:23:59.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7ca67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e864ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7ca67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e864ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49120/0 (socket says 172.21.15.2:49120) 2023-11-16T18:23:59.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7ca67fc700 1 -- 172.21.15.2:0/2426059079 learned_addr learned my addr 172.21.15.2:0/2426059079 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/2426059079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e863bb0 0x56079e863fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7ca77fe700 1 --2- 172.21.15.2:0/2426059079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 0x56079e865530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7ca67fc700 1 -- 172.21.15.2:0/2426059079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 msgr2=0x56079e865530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7ca67fc700 1 --2- 172.21.15.2:0/2426059079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 0x56079e865530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7ca67fc700 1 -- 172.21.15.2:0/2426059079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e863bb0 msgr2=0x56079e863fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7ca67fc700 1 --2- 172.21.15.2:0/2426059079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e863bb0 0x56079e863fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.715+0000 7f7ca67fc700 1 -- 172.21.15.2:0/2426059079 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56079e6365e0 con 0x56079e864500 2023-11-16T18:23:59.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.716+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/2426059079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e863bb0 0x56079e863fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:23:59.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.716+0000 7f7ca77fe700 1 --2- 172.21.15.2:0/2426059079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 0x56079e865530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:59.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.716+0000 7f7ca67fc700 1 --2- 172.21.15.2:0/2426059079 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e864ae0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7c90005060 tx=0x7f7c900996d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fdd49a932c477c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.716+0000 7f7c9d7fa700 1 -- 172.21.15.2:0/2426059079 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c900a20b0 con 0x56079e864500 2023-11-16T18:23:59.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.716+0000 7f7c9d7fa700 1 -- 172.21.15.2:0/2426059079 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c90004380 con 0x56079e864500 2023-11-16T18:23:59.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.716+0000 7f7c9d7fa700 1 -- 172.21.15.2:0/2426059079 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c90099390 con 0x56079e864500 2023-11-16T18:23:59.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 -- 172.21.15.2:0/2426059079 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 msgr2=0x56079e864ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/2426059079 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e864ae0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7c90005060 tx=0x7f7c900996d0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 -- 172.21.15.2:0/2426059079 shutdown_connections 2023-11-16T18:23:59.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/2426059079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 0x56079e865530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/2426059079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e863bb0 0x56079e863fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/2426059079 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e864ae0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 -- 172.21.15.2:0/2426059079 >> 172.21.15.2:0/2426059079 conn(0x56079e859310 msgr2=0x56079e85e620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 -- 172.21.15.2:0/2426059079 shutdown_connections 2023-11-16T18:23:59.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 -- 172.21.15.2:0/2426059079 wait complete. 2023-11-16T18:23:59.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 -- start start 2023-11-16T18:23:59.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 Processor -- start 2023-11-16T18:23:59.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e863bb0 0x56079e86d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e86db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 0x56079e874bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.726 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079e756890 con 0x56079e864500 2023-11-16T18:23:59.726 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079e756530 con 0x56079e863bb0 2023-11-16T18:23:59.726 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079e6388d0 con 0x56079e8650c0 2023-11-16T18:23:59.726 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.717+0000 7f7cb9486b80 10 client.0 _ll_register_callbacks cb 0x56079e774730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:23:59.726 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7ca67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e86db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.726 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7ca67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e86db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49126/0 (socket says 172.21.15.2:49126) 2023-11-16T18:23:59.726 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7ca67fc700 1 -- 172.21.15.2:0/3689163742 learned_addr learned my addr 172.21.15.2:0/3689163742 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.726 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7ca77fe700 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 0x56079e874bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.726 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e863bb0 0x56079e86d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.726 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7ca67fc700 1 -- 172.21.15.2:0/3689163742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 msgr2=0x56079e874bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.726 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7ca67fc700 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 0x56079e874bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.726 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7ca67fc700 1 -- 172.21.15.2:0/3689163742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e863bb0 msgr2=0x56079e86d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7ca67fc700 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e863bb0 0x56079e86d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7ca67fc700 1 -- 172.21.15.2:0/3689163742 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca810a930 con 0x56079e864500 2023-11-16T18:23:59.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e863bb0 0x56079e86d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:23:59.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7ca67fc700 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e86db90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7c900966d0 tx=0x7f7c900050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274fc0272c622be5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7c8e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:23:59.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7c8e7fc700 1 -- 172.21.15.2:0/3689163742 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c900a20b0 con 0x56079e864500 2023-11-16T18:23:59.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7c8e7fc700 1 -- 172.21.15.2:0/3689163742 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7c90099360 con 0x56079e864500 2023-11-16T18:23:59.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7cb9486b80 1 -- 172.21.15.2:0/3689163742 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56079e856ad0 con 0x56079e864500 2023-11-16T18:23:59.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7c8e7fc700 1 -- 172.21.15.2:0/3689163742 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c900a1c20 con 0x56079e864500 2023-11-16T18:23:59.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7cb9486b80 10 client.15225 fetch_fsmap learned FSMap version 24 2023-11-16T18:23:59.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7cb9486b80 1 -- 172.21.15.2:0/3689163742 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56079e77b140 con 0x56079e864500 2023-11-16T18:23:59.727 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.718+0000 7f7c8e7fc700 1 -- 172.21.15.2:0/3689163742 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56079e856ad0 con 0x56079e864500 2023-11-16T18:23:59.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.719+0000 7f7c8e7fc700 1 -- 172.21.15.2:0/3689163742 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7c90098bc0 con 0x56079e864500 2023-11-16T18:23:59.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.719+0000 7f7cb9486b80 10 client.15225 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:23:59.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.719+0000 7f7cb9486b80 10 client.15225 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T18:23:59.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.719+0000 7f7cb9486b80 20 client.15225 populate_metadata read hostname 'smithi002' 2023-11-16T18:23:59.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.719+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56079e878350 0x56079e877a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.719+0000 7f7cb9486b80 4 client.15225 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:23:59.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.719+0000 7f7cb9486b80 1 -- 172.21.15.2:0/3689163742 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56079e777050 con 0x56079e878350 2023-11-16T18:23:59.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.719+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56079e878350 0x56079e877a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56079e878350 0x56079e877a70 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4a2042dfefb755c server_cookie=368d2f2783f1932c in_seq=0 out_seq=0 2023-11-16T18:23:59.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7c8e7fc700 10 client.15225 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:23:59.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7c8e7fc700 1 -- 172.21.15.2:0/3689163742 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56079e777050 con 0x56079e878350 2023-11-16T18:23:59.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7c8e7fc700 10 client.15225 handle_command_reply: tid=0 2023-11-16T18:23:59.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7cb9486b80 1 client.15225 shutdown 2023-11-16T18:23:59.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7cb9486b80 1 -- 172.21.15.2:0/3689163742 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 msgr2=0x56079e86db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e86db90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7c900966d0 tx=0x7f7c900050b0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7cb9486b80 1 -- 172.21.15.2:0/3689163742 shutdown_connections 2023-11-16T18:23:59.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56079e878350 0x56079e877a70 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 0x56079e874bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7c8e7fc700 0 client.15225 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:23:59.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e863bb0 0x56079e86d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/3689163742 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e86db90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.720+0000 7f7cb9486b80 1 -- 172.21.15.2:0/3689163742 >> 172.21.15.2:0/3689163742 conn(0x56079e859310 msgr2=0x56079e85dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.721+0000 7f7cb9486b80 1 -- 172.21.15.2:0/3689163742 shutdown_connections 2023-11-16T18:23:59.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.721+0000 7f7cb9486b80 1 -- 172.21.15.2:0/3689163742 wait complete. 2023-11-16T18:23:59.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.721+0000 7f7cb9486b80 20 client.15225 trim_cache size 0 max 16384 2023-11-16T18:23:59.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.739+0000 7f7cb9486b80 1 Processor -- start 2023-11-16T18:23:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.739+0000 7f7cb9486b80 1 -- start start 2023-11-16T18:23:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.739+0000 7f7cb9486b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e863bb0 0x56079e86c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.739+0000 7f7cb9486b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e864500 0x56079e874b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.739+0000 7f7cb9486b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 0x56079e868050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.739+0000 7f7cb9486b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079e756890 con 0x56079e863bb0 2023-11-16T18:23:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.739+0000 7f7cb9486b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079e777050 con 0x56079e864500 2023-11-16T18:23:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.739+0000 7f7cb9486b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079e86e840 con 0x56079e8650c0 2023-11-16T18:23:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.739+0000 7f7ca6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e863bb0 0x56079e86c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.739+0000 7f7ca6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e863bb0 0x56079e86c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49132/0 (socket says 172.21.15.2:49132) 2023-11-16T18:23:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.739+0000 7f7ca6ffd700 1 -- 172.21.15.2:0/3516136115 learned_addr learned my addr 172.21.15.2:0/3516136115 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.740+0000 7f7ca67fc700 1 --2- 172.21.15.2:0/3516136115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e864500 0x56079e874b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.740+0000 7f7ca77fe700 1 --2- 172.21.15.2:0/3516136115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 0x56079e868050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.740+0000 7f7ca6ffd700 1 -- 172.21.15.2:0/3516136115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 msgr2=0x56079e868050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.740+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/3516136115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 0x56079e868050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.740+0000 7f7ca6ffd700 1 -- 172.21.15.2:0/3516136115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e864500 msgr2=0x56079e874b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.740+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/3516136115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e864500 0x56079e874b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.740+0000 7f7ca6ffd700 1 -- 172.21.15.2:0/3516136115 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56079e77ae90 con 0x56079e863bb0 2023-11-16T18:23:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.740+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/3516136115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e863bb0 0x56079e86c2a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7ca81001a0 tx=0x7f7c98020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5e1cf5f2079504 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.740+0000 7f7c9dffb700 1 -- 172.21.15.2:0/3516136115 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98005380 con 0x56079e863bb0 2023-11-16T18:23:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.740+0000 7f7c9dffb700 1 -- 172.21.15.2:0/3516136115 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c9801e040 con 0x56079e863bb0 2023-11-16T18:23:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.740+0000 7f7c9dffb700 1 -- 172.21.15.2:0/3516136115 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98003db0 con 0x56079e863bb0 2023-11-16T18:23:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.741+0000 7f7cb9486b80 1 -- 172.21.15.2:0/3516136115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e863bb0 msgr2=0x56079e86c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.741+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/3516136115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e863bb0 0x56079e86c2a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7ca81001a0 tx=0x7f7c98020000 comp rx=0 tx=0).stop 2023-11-16T18:23:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.741+0000 7f7cb9486b80 1 -- 172.21.15.2:0/3516136115 shutdown_connections 2023-11-16T18:23:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.741+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/3516136115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e8650c0 0x56079e868050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.741+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/3516136115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e863bb0 0x56079e86c2a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.741+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/3516136115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e864500 0x56079e874b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.741+0000 7f7cb9486b80 1 -- 172.21.15.2:0/3516136115 >> 172.21.15.2:0/3516136115 conn(0x56079e859310 msgr2=0x56079e877a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.741+0000 7f7cb9486b80 1 -- 172.21.15.2:0/3516136115 shutdown_connections 2023-11-16T18:23:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.741+0000 7f7cb9486b80 1 -- 172.21.15.2:0/3516136115 wait complete. 2023-11-16T18:23:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.741+0000 7f7cb9486b80 1 -- start start 2023-11-16T18:23:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.741+0000 7f7cb9486b80 1 Processor -- start 2023-11-16T18:23:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7cb9486b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e863bb0 0x56079e874570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7cb9486b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e874ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7cb9486b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e8650c0 0x56079e871010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7cb9486b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079e8716f0 con 0x56079e864500 2023-11-16T18:23:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7cb9486b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079e871830 con 0x56079e8650c0 2023-11-16T18:23:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7cb9486b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079e8719f0 con 0x56079e863bb0 2023-11-16T18:23:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7cb9486b80 10 client.0 _ll_register_callbacks cb 0x7f7c9000f3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:23:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7ca67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e874ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7ca67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e874ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49140/0 (socket says 172.21.15.2:49140) 2023-11-16T18:23:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7ca6ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e863bb0 0x56079e874570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7ca67fc700 1 -- 172.21.15.2:0/1122136381 learned_addr learned my addr 172.21.15.2:0/1122136381 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:23:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7ca77fe700 1 --2- 172.21.15.2:0/1122136381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e8650c0 0x56079e871010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7ca67fc700 1 -- 172.21.15.2:0/1122136381 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e863bb0 msgr2=0x56079e874570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7ca67fc700 1 --2- 172.21.15.2:0/1122136381 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e863bb0 0x56079e874570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7ca67fc700 1 -- 172.21.15.2:0/1122136381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e8650c0 msgr2=0x56079e871010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:23:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7ca67fc700 1 --2- 172.21.15.2:0/1122136381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e8650c0 0x56079e871010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7ca67fc700 1 -- 172.21.15.2:0/1122136381 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9802c040 con 0x56079e864500 2023-11-16T18:23:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7ca67fc700 1 --2- 172.21.15.2:0/1122136381 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e874ab0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7c900ca2a0 tx=0x7f7c90004fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf54fdabcc9a8d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:23:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7c8e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:23:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.742+0000 7f7c8e7fc700 1 -- 172.21.15.2:0/1122136381 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c90005770 con 0x56079e864500 2023-11-16T18:23:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7c8e7fc700 1 -- 172.21.15.2:0/1122136381 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7c900a20b0 con 0x56079e864500 2023-11-16T18:23:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7cb9486b80 1 -- 172.21.15.2:0/1122136381 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c90004380 con 0x56079e864500 2023-11-16T18:23:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7c8e7fc700 1 -- 172.21.15.2:0/1122136381 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c900b2210 con 0x56079e864500 2023-11-16T18:23:59.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7cb9486b80 10 client.15231 fetch_fsmap learned FSMap version 24 2023-11-16T18:23:59.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7cb9486b80 1 -- 172.21.15.2:0/1122136381 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56079e5cbc10 con 0x56079e864500 2023-11-16T18:23:59.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7c8e7fc700 1 -- 172.21.15.2:0/1122136381 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c90004380 con 0x56079e864500 2023-11-16T18:23:59.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7c8e7fc700 1 -- 172.21.15.2:0/1122136381 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7c90004b40 con 0x56079e864500 2023-11-16T18:23:59.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7cb9486b80 10 client.15231 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:23:59.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7cb9486b80 10 client.15231 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T18:23:59.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7cb9486b80 20 client.15231 populate_metadata read hostname 'smithi002' 2023-11-16T18:23:59.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/1122136381 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56079e8d0530 0x56079e8cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:23:59.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7cb9486b80 4 client.15231 mds_command: new command op to 24473 tid=0[{"prefix": "status"}] 2023-11-16T18:23:59.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7cb9486b80 1 -- 172.21.15.2:0/1122136381 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56079e783610 con 0x56079e8d0530 2023-11-16T18:23:59.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.743+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/1122136381 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56079e8d0530 0x56079e8cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:23:59.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7c8e7fc700 10 client.15231 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:23:59.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7ca6ffd700 1 --2- 172.21.15.2:0/1122136381 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56079e8d0530 0x56079e8cd020 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a482714ccdb19b8c server_cookie=6887da33bbef20da in_seq=0 out_seq=0 2023-11-16T18:23:59.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7c8e7fc700 1 -- 172.21.15.2:0/1122136381 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x56079e783610 con 0x56079e8d0530 2023-11-16T18:23:59.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7c8e7fc700 10 client.15231 handle_command_reply: tid=0 2023-11-16T18:23:59.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7cb9486b80 1 client.15231 shutdown 2023-11-16T18:23:59.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7cb9486b80 1 -- 172.21.15.2:0/1122136381 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 msgr2=0x56079e874ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/1122136381 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e874ab0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7c900ca2a0 tx=0x7f7c90004fd0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7cb9486b80 1 -- 172.21.15.2:0/1122136381 shutdown_connections 2023-11-16T18:23:59.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/1122136381 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56079e8d0530 0x56079e8cd020 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/1122136381 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56079e863bb0 0x56079e874570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/1122136381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56079e8650c0 0x56079e871010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7cb9486b80 1 --2- 172.21.15.2:0/1122136381 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56079e864500 0x56079e874ab0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7c8e7fc700 0 client.15231 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:23:59.754 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7cb9486b80 1 -- 172.21.15.2:0/1122136381 >> 172.21.15.2:0/1122136381 conn(0x56079e859310 msgr2=0x56079e86c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.754 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7cb9486b80 1 -- 172.21.15.2:0/1122136381 shutdown_connections 2023-11-16T18:23:59.754 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7cb9486b80 1 -- 172.21.15.2:0/1122136381 wait complete. 2023-11-16T18:23:59.754 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.744+0000 7f7cb9486b80 20 client.15231 trim_cache size 0 max 16384 2023-11-16T18:23:59.754 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:23:59.754 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "08160fac-84ac-11ee-8dbe-212e2dc638e7", 2023-11-16T18:23:59.754 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2023-11-16T18:23:59.754 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24473, 2023-11-16T18:23:59.754 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T18:23:59.755 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T18:23:59.755 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T18:23:59.755 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 72.935960949999995, 2023-11-16T18:23:59.755 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 24, 2023-11-16T18:23:59.755 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 88, 2023-11-16T18:23:59.755 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 86, 2023-11-16T18:23:59.755 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 78.536137685 2023-11-16T18:23:59.755 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:23:59.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.745+0000 7f7cad2fa700 1 -- 172.21.15.2:0/2985840477 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7c940a11e0 msgr2=0x7f7c940a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.745+0000 7f7cad2fa700 1 --2- 172.21.15.2:0/2985840477 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7c940a11e0 0x7f7c940a3690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800b5c0 tx=0x7f7c98005960 comp rx=0 tx=0).stop 2023-11-16T18:23:59.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.745+0000 7f7cad2fa700 1 -- 172.21.15.2:0/2985840477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca8100340 msgr2=0x7f7ca8101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:23:59.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.745+0000 7f7cad2fa700 1 --2- 172.21.15.2:0/2985840477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca8100340 0x7f7ca8101720 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7c9000d980 tx=0x7f7c9000dc90 comp rx=0 tx=0).stop 2023-11-16T18:23:59.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.745+0000 7f7cad2fa700 1 -- 172.21.15.2:0/2985840477 shutdown_connections 2023-11-16T18:23:59.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.745+0000 7f7cad2fa700 1 --2- 172.21.15.2:0/2985840477 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7c940a11e0 0x7f7c940a3690 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.745+0000 7f7cad2fa700 1 --2- 172.21.15.2:0/2985840477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ca8100cd0 0x7f7ca8101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.745+0000 7f7cad2fa700 1 --2- 172.21.15.2:0/2985840477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca80ff150 0x7f7ca81030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.745+0000 7f7cad2fa700 1 --2- 172.21.15.2:0/2985840477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca8100340 0x7f7ca8101720 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:23:59.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.745+0000 7f7cad2fa700 1 -- 172.21.15.2:0/2985840477 >> 172.21.15.2:0/2985840477 conn(0x7f7ca80fa750 msgr2=0x7f7ca8069100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:23:59.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.745+0000 7f7cad2fa700 1 -- 172.21.15.2:0/2985840477 shutdown_connections 2023-11-16T18:23:59.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:23:59.745+0000 7f7cad2fa700 1 -- 172.21.15.2:0/2985840477 wait complete. 2023-11-16T18:23:59.759 INFO:tasks.cephadm:Setup complete, yielding 2023-11-16T18:23:59.759 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-16T18:23:59.772 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T18:23:59.773 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch status' 2023-11-16T18:23:59.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:59 smithi002 ceph-mon[98807]: pgmap v298: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:23:59.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:59 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/827452324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:59.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:23:59 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3145555133' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:23:59.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:59 smithi040 ceph-mon[100251]: pgmap v298: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:23:59.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:59 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/827452324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:23:59.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:23:59 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3145555133' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:00.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:59 smithi188 ceph-mon[100639]: pgmap v298: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:24:00.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:59 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/827452324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:00.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:23:59 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3145555133' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:00.426 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:24:01.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.739+0000 7fca33f01700 1 -- 172.21.15.2:0/1318658016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca2c101fd0 msgr2=0x7fca2c108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:01.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.739+0000 7fca33f01700 1 --2- 172.21.15.2:0/1318658016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca2c101fd0 0x7fca2c108890 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fca2400b0d0 tx=0x7fca2400b3e0 comp rx=0 tx=0).stop 2023-11-16T18:24:01.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.739+0000 7fca33f01700 1 -- 172.21.15.2:0/1318658016 shutdown_connections 2023-11-16T18:24:01.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.739+0000 7fca33f01700 1 --2- 172.21.15.2:0/1318658016 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca2c0fe5c0 0x7fca2c0fe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:01.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.739+0000 7fca33f01700 1 --2- 172.21.15.2:0/1318658016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca2c101fd0 0x7fca2c108890 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:01.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.739+0000 7fca33f01700 1 --2- 172.21.15.2:0/1318658016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca2c101640 0x7fca2c101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:01.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.739+0000 7fca33f01700 1 -- 172.21.15.2:0/1318658016 >> 172.21.15.2:0/1318658016 conn(0x7fca2c0f9f70 msgr2=0x7fca2c0fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:01.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.739+0000 7fca33f01700 1 -- 172.21.15.2:0/1318658016 shutdown_connections 2023-11-16T18:24:01.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.739+0000 7fca33f01700 1 -- 172.21.15.2:0/1318658016 wait complete. 2023-11-16T18:24:01.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca33f01700 1 Processor -- start 2023-11-16T18:24:01.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca33f01700 1 -- start start 2023-11-16T18:24:01.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca33f01700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca2c0fe5c0 0x7fca2c196180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:01.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca33f01700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca2c101640 0x7fca2c1966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:01.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca33f01700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca2c101fd0 0x7fca2c19d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:01.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca33f01700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c19dd10 con 0x7fca2c101fd0 2023-11-16T18:24:01.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca33f01700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c19de50 con 0x7fca2c0fe5c0 2023-11-16T18:24:01.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca33f01700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c19dfe0 con 0x7fca2c101640 2023-11-16T18:24:01.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca3249e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca2c101fd0 0x7fca2c19d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:01.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca3249e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca2c101fd0 0x7fca2c19d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49156/0 (socket says 172.21.15.2:49156) 2023-11-16T18:24:01.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca3249e700 1 -- 172.21.15.2:0/169927407 learned_addr learned my addr 172.21.15.2:0/169927407 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:01.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca3149c700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca2c101640 0x7fca2c1966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:01.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca3249e700 1 -- 172.21.15.2:0/169927407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca2c101640 msgr2=0x7fca2c1966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:01.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca31c9d700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca2c0fe5c0 0x7fca2c196180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:01.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca3249e700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca2c101640 0x7fca2c1966c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:01.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca3249e700 1 -- 172.21.15.2:0/169927407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca2c0fe5c0 msgr2=0x7fca2c196180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:01.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca3249e700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca2c0fe5c0 0x7fca2c196180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:01.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.740+0000 7fca3249e700 1 -- 172.21.15.2:0/169927407 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca24009d00 con 0x7fca2c101fd0 2023-11-16T18:24:01.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.741+0000 7fca3249e700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca2c101fd0 0x7fca2c19d740 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fca24009440 tx=0x7fca24009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:01.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.741+0000 7fca22ffd700 1 -- 172.21.15.2:0/169927407 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2400e040 con 0x7fca2c101fd0 2023-11-16T18:24:01.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.741+0000 7fca33f01700 1 -- 172.21.15.2:0/169927407 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2c10bcb0 con 0x7fca2c101fd0 2023-11-16T18:24:01.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.741+0000 7fca22ffd700 1 -- 172.21.15.2:0/169927407 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca24012df0 con 0x7fca2c101fd0 2023-11-16T18:24:01.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.741+0000 7fca22ffd700 1 -- 172.21.15.2:0/169927407 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2401c400 con 0x7fca2c101fd0 2023-11-16T18:24:01.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.741+0000 7fca33f01700 1 -- 172.21.15.2:0/169927407 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c10bb20 con 0x7fca2c101fd0 2023-11-16T18:24:01.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.742+0000 7fca33f01700 1 -- 172.21.15.2:0/169927407 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c1079b0 con 0x7fca2c101fd0 2023-11-16T18:24:01.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.743+0000 7fca22ffd700 1 -- 172.21.15.2:0/169927407 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fca24012930 con 0x7fca2c101fd0 2023-11-16T18:24:01.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.743+0000 7fca22ffd700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fca180a11e0 0x7fca180a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:01.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.744+0000 7fca31c9d700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fca180a11e0 0x7fca180a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:01.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.744+0000 7fca22ffd700 1 -- 172.21.15.2:0/169927407 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fca240ce780 con 0x7fca2c101fd0 2023-11-16T18:24:01.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.744+0000 7fca31c9d700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fca180a11e0 0x7fca180a3690 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fca1c005950 tx=0x7fca1c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:01.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.748+0000 7fca22ffd700 1 -- 172.21.15.2:0/169927407 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fca24097af0 con 0x7fca2c101fd0 2023-11-16T18:24:01.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:01 smithi002 ceph-mon[98807]: pgmap v299: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:01.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.960+0000 7fca33f01700 1 -- 172.21.15.2:0/169927407 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fca2c061a00 con 0x7fca180a11e0 2023-11-16T18:24:01.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.961+0000 7fca22ffd700 1 -- 172.21.15.2:0/169927407 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fca2c061a00 con 0x7fca180a11e0 2023-11-16T18:24:01.963 INFO:teuthology.orchestra.run.smithi002.stdout:Backend: cephadm 2023-11-16T18:24:01.963 INFO:teuthology.orchestra.run.smithi002.stdout:Available: Yes 2023-11-16T18:24:01.963 INFO:teuthology.orchestra.run.smithi002.stdout:Paused: No 2023-11-16T18:24:01.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.963+0000 7fca33f01700 1 -- 172.21.15.2:0/169927407 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fca180a11e0 msgr2=0x7fca180a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.963+0000 7fca33f01700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fca180a11e0 0x7fca180a3690 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fca1c005950 tx=0x7fca1c0058e0 comp rx=0 tx=0).stop 2023-11-16T18:24:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.963+0000 7fca33f01700 1 -- 172.21.15.2:0/169927407 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca2c101fd0 msgr2=0x7fca2c19d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.963+0000 7fca33f01700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca2c101fd0 0x7fca2c19d740 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fca24009440 tx=0x7fca24009520 comp rx=0 tx=0).stop 2023-11-16T18:24:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.963+0000 7fca33f01700 1 -- 172.21.15.2:0/169927407 shutdown_connections 2023-11-16T18:24:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.963+0000 7fca33f01700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fca180a11e0 0x7fca180a3690 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.963+0000 7fca33f01700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca2c101fd0 0x7fca2c19d740 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:01.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.963+0000 7fca33f01700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca2c0fe5c0 0x7fca2c196180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:01.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.963+0000 7fca33f01700 1 --2- 172.21.15.2:0/169927407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca2c101640 0x7fca2c1966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:01.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.963+0000 7fca33f01700 1 -- 172.21.15.2:0/169927407 >> 172.21.15.2:0/169927407 conn(0x7fca2c0f9f70 msgr2=0x7fca2c104af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:01.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.963+0000 7fca33f01700 1 -- 172.21.15.2:0/169927407 shutdown_connections 2023-11-16T18:24:01.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:01.963+0000 7fca33f01700 1 -- 172.21.15.2:0/169927407 wait complete. 2023-11-16T18:24:01.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:01 smithi040 ceph-mon[100251]: pgmap v299: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:02.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:01 smithi188 ceph-mon[100639]: pgmap v299: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:02.390 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch ps' 2023-11-16T18:24:03.044 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:24:03.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:03 smithi002 ceph-mon[98807]: pgmap v300: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T18:24:03.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:03 smithi002 ceph-mon[98807]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:03.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:03 smithi040 ceph-mon[100251]: pgmap v300: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T18:24:03.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:03 smithi040 ceph-mon[100251]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:04.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:03 smithi188 ceph-mon[100639]: pgmap v300: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T18:24:04.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:03 smithi188 ceph-mon[100639]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:04.274 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.266+0000 7efeb2090700 1 -- 172.21.15.2:0/3192533255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0ff430 msgr2=0x7efeac0ff8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:04.274 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.266+0000 7efeb2090700 1 --2- 172.21.15.2:0/3192533255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0ff430 0x7efeac0ff8a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efea0009990 tx=0x7efea0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:04.274 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.266+0000 7efeb2090700 1 -- 172.21.15.2:0/3192533255 shutdown_connections 2023-11-16T18:24:04.274 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.266+0000 7efeb2090700 1 --2- 172.21.15.2:0/3192533255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efeac0ffde0 0x7efeac107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:04.274 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.266+0000 7efeb2090700 1 --2- 172.21.15.2:0/3192533255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeac100860 0x7efeac100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:04.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.266+0000 7efeb2090700 1 --2- 172.21.15.2:0/3192533255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0ff430 0x7efeac0ff8a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:04.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.266+0000 7efeb2090700 1 -- 172.21.15.2:0/3192533255 >> 172.21.15.2:0/3192533255 conn(0x7efeac0f9f70 msgr2=0x7efeac0fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:04.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.266+0000 7efeb2090700 1 -- 172.21.15.2:0/3192533255 shutdown_connections 2023-11-16T18:24:04.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.266+0000 7efeb2090700 1 -- 172.21.15.2:0/3192533255 wait complete. 2023-11-16T18:24:04.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.266+0000 7efeb2090700 1 Processor -- start 2023-11-16T18:24:04.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeb2090700 1 -- start start 2023-11-16T18:24:04.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeb2090700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeac0ff430 0x7efeac0feea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:04.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeb2090700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0ffde0 0x7efeac19ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:04.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeb2090700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efeac100860 0x7efeac19d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:04.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeb2090700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac19db70 con 0x7efeac0ffde0 2023-11-16T18:24:04.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeb2090700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac19dce0 con 0x7efeac0ff430 2023-11-16T18:24:04.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeb2090700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac19dea0 con 0x7efeac100860 2023-11-16T18:24:04.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeaaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0ffde0 0x7efeac19ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:04.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeaaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0ffde0 0x7efeac19ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49168/0 (socket says 172.21.15.2:49168) 2023-11-16T18:24:04.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeaaffd700 1 -- 172.21.15.2:0/3212243002 learned_addr learned my addr 172.21.15.2:0/3212243002 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:04.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeab7fe700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeac0ff430 0x7efeac0feea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:04.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeaaffd700 1 -- 172.21.15.2:0/3212243002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efeac100860 msgr2=0x7efeac19d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:04.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeabfff700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efeac100860 0x7efeac19d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:04.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeaaffd700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efeac100860 0x7efeac19d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:04.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeaaffd700 1 -- 172.21.15.2:0/3212243002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeac0ff430 msgr2=0x7efeac0feea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:04.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeaaffd700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeac0ff430 0x7efeac0feea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:04.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeaaffd700 1 -- 172.21.15.2:0/3212243002 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea00095f0 con 0x7efeac0ffde0 2023-11-16T18:24:04.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeab7fe700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeac0ff430 0x7efeac0feea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:04.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.267+0000 7efeaaffd700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0ffde0 0x7efeac19ad80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efea0000c00 tx=0x7efea000f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:04.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.268+0000 7efea8ff9700 1 -- 172.21.15.2:0/3212243002 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea001d070 con 0x7efeac0ffde0 2023-11-16T18:24:04.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.268+0000 7efea8ff9700 1 -- 172.21.15.2:0/3212243002 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efea000fc00 con 0x7efeac0ffde0 2023-11-16T18:24:04.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.268+0000 7efeb2090700 1 -- 172.21.15.2:0/3212243002 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeac10abf0 con 0x7efeac0ffde0 2023-11-16T18:24:04.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.268+0000 7efea8ff9700 1 -- 172.21.15.2:0/3212243002 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea0017650 con 0x7efeac0ffde0 2023-11-16T18:24:04.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.268+0000 7efeb2090700 1 -- 172.21.15.2:0/3212243002 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeac10aa60 con 0x7efeac0ffde0 2023-11-16T18:24:04.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.269+0000 7efeb2090700 1 -- 172.21.15.2:0/3212243002 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeac19e510 con 0x7efeac0ffde0 2023-11-16T18:24:04.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.270+0000 7efea8ff9700 1 -- 172.21.15.2:0/3212243002 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7efea0022470 con 0x7efeac0ffde0 2023-11-16T18:24:04.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.270+0000 7efea8ff9700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe940a1260 0x7efe940a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:04.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.271+0000 7efea8ff9700 1 -- 172.21.15.2:0/3212243002 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efea00ce280 con 0x7efeac0ffde0 2023-11-16T18:24:04.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.271+0000 7efeab7fe700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe940a1260 0x7efe940a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:04.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.271+0000 7efeab7fe700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe940a1260 0x7efe940a3710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efe9c005fd0 tx=0x7efe9c005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:04.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.274+0000 7efea8ff9700 1 -- 172.21.15.2:0/3212243002 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efea00975f0 con 0x7efeac0ffde0 2023-11-16T18:24:04.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.483+0000 7efeb2090700 1 -- 172.21.15.2:0/3212243002 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efeac060f30 con 0x7efe940a1260 2023-11-16T18:24:04.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.503+0000 7efea8ff9700 1 -- 172.21.15.2:0/3212243002 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7efeac060f30 con 0x7efe940a1260 2023-11-16T18:24:04.505 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-11-16T18:24:04.505 INFO:teuthology.orchestra.run.smithi002.stdout:mds.a smithi002 running (85s) 12s ago 85s 15.0M - 18.0.0-3905-g5e4a743a d5b1f852d682 f106bfdbe3e1 2023-11-16T18:24:04.505 INFO:teuthology.orchestra.run.smithi002.stdout:mds.b smithi040 running (83s) 13s ago 83s 29.9M - 18.0.0-3905-g5e4a743a d5b1f852d682 5cc16666f3a0 2023-11-16T18:24:04.506 INFO:teuthology.orchestra.run.smithi002.stdout:mds.c smithi188 running (94s) 13s ago 94s 18.8M - 18.0.0-3905-g5e4a743a d5b1f852d682 dd8f0aa2fa96 2023-11-16T18:24:04.506 INFO:teuthology.orchestra.run.smithi002.stdout:mds.d smithi002 running (101s) 12s ago 101s 17.3M - 18.0.0-3905-g5e4a743a d5b1f852d682 2fcf1e10ec4c 2023-11-16T18:24:04.506 INFO:teuthology.orchestra.run.smithi002.stdout:mds.e smithi040 running (99s) 13s ago 98s 19.7M - 18.0.0-3905-g5e4a743a d5b1f852d682 cfd9deb2856d 2023-11-16T18:24:04.506 INFO:teuthology.orchestra.run.smithi002.stdout:mds.f smithi188 running (96s) 13s ago 96s 17.3M - 18.0.0-3905-g5e4a743a d5b1f852d682 076cd0c592f7 2023-11-16T18:24:04.506 INFO:teuthology.orchestra.run.smithi002.stdout:mds.g smithi002 running (88s) 12s ago 87s 16.4M - 18.0.0-3905-g5e4a743a d5b1f852d682 73ab18e3bb5b 2023-11-16T18:24:04.506 INFO:teuthology.orchestra.run.smithi002.stdout:mds.h smithi040 running (92s) 13s ago 92s 18.9M - 18.0.0-3905-g5e4a743a d5b1f852d682 cbae0c4cdcb0 2023-11-16T18:24:04.507 INFO:teuthology.orchestra.run.smithi002.stdout:mds.i smithi188 running (90s) 13s ago 89s 20.1M - 18.0.0-3905-g5e4a743a d5b1f852d682 3741ae60c0b5 2023-11-16T18:24:04.507 INFO:teuthology.orchestra.run.smithi002.stdout:mds.j smithi002 running (103s) 12s ago 103s 16.6M - 18.0.0-3905-g5e4a743a d5b1f852d682 243d11b97b23 2023-11-16T18:24:04.507 INFO:teuthology.orchestra.run.smithi002.stdout:mds.k smithi040 running (105s) 13s ago 105s 16.4M - 18.0.0-3905-g5e4a743a d5b1f852d682 965126a20bd6 2023-11-16T18:24:04.507 INFO:teuthology.orchestra.run.smithi002.stdout:mds.l smithi188 running (108s) 13s ago 107s 17.2M - 18.0.0-3905-g5e4a743a d5b1f852d682 c8f325dfda19 2023-11-16T18:24:04.507 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi002 *:9283,8765 running (8m) 12s ago 8m 516M - 18.0.0-3905-g5e4a743a d5b1f852d682 a85ff49ab4d1 2023-11-16T18:24:04.507 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi040 *:8443,8765 running (6m) 13s ago 6m 446M - 18.0.0-3905-g5e4a743a d5b1f852d682 2112007917f6 2023-11-16T18:24:04.508 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.z smithi188 *:8443,8765 running (6m) 13s ago 6m 447M - 18.0.0-3905-g5e4a743a d5b1f852d682 74ea28b621ec 2023-11-16T18:24:04.508 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (8m) 12s ago 8m 94.7M 2048M 18.0.0-3905-g5e4a743a d5b1f852d682 a0ce6cc9456c 2023-11-16T18:24:04.508 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi040 running (6m) 13s ago 6m 62.4M 2048M 18.0.0-3905-g5e4a743a d5b1f852d682 aa5c16469767 2023-11-16T18:24:04.508 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi188 running (6m) 13s ago 6m 61.3M 2048M 18.0.0-3905-g5e4a743a d5b1f852d682 c96831932d00 2023-11-16T18:24:04.508 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (6m) 12s ago 6m 71.5M 4293M 18.0.0-3905-g5e4a743a d5b1f852d682 ec271b9cb533 2023-11-16T18:24:04.509 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi040 running (5m) 13s ago 5m 77.0M 4293M 18.0.0-3905-g5e4a743a d5b1f852d682 33d24e719b21 2023-11-16T18:24:04.509 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi188 running (5m) 13s ago 5m 76.3M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 f7da6084f4d2 2023-11-16T18:24:04.509 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (5m) 12s ago 5m 65.2M 4293M 18.0.0-3905-g5e4a743a d5b1f852d682 a89b73e47bee 2023-11-16T18:24:04.509 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi040 running (4m) 13s ago 4m 64.3M 4293M 18.0.0-3905-g5e4a743a d5b1f852d682 19f6e2595785 2023-11-16T18:24:04.509 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi188 running (4m) 13s ago 4m 69.6M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 2bc0605458e3 2023-11-16T18:24:04.509 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi002 running (3m) 12s ago 3m 63.0M 4293M 18.0.0-3905-g5e4a743a d5b1f852d682 3e4763c71748 2023-11-16T18:24:04.509 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi040 running (3m) 13s ago 3m 67.4M 4293M 18.0.0-3905-g5e4a743a d5b1f852d682 1cb77517740f 2023-11-16T18:24:04.510 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8 smithi188 running (3m) 13s ago 3m 60.7M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 4ab80f188174 2023-11-16T18:24:04.510 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9 smithi002 running (2m) 12s ago 2m 62.2M 4293M 18.0.0-3905-g5e4a743a d5b1f852d682 c1819de40b00 2023-11-16T18:24:04.510 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10 smithi040 running (2m) 13s ago 2m 63.9M 4293M 18.0.0-3905-g5e4a743a d5b1f852d682 bfb6bb025ca8 2023-11-16T18:24:04.511 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11 smithi188 running (2m) 13s ago 2m 58.8M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 d19ce099bdbf 2023-11-16T18:24:04.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.505+0000 7efeb2090700 1 -- 172.21.15.2:0/3212243002 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe940a1260 msgr2=0x7efe940a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:04.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.505+0000 7efeb2090700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe940a1260 0x7efe940a3710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efe9c005fd0 tx=0x7efe9c005ee0 comp rx=0 tx=0).stop 2023-11-16T18:24:04.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.505+0000 7efeb2090700 1 -- 172.21.15.2:0/3212243002 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0ffde0 msgr2=0x7efeac19ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:04.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.505+0000 7efeb2090700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0ffde0 0x7efeac19ad80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efea0000c00 tx=0x7efea000f710 comp rx=0 tx=0).stop 2023-11-16T18:24:04.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.505+0000 7efeb2090700 1 -- 172.21.15.2:0/3212243002 shutdown_connections 2023-11-16T18:24:04.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.506+0000 7efeb2090700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7efe940a1260 0x7efe940a3710 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:04.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.506+0000 7efeb2090700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efeac100860 0x7efeac19d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:04.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.506+0000 7efeb2090700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeac0ff430 0x7efeac0feea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:04.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.506+0000 7efeb2090700 1 --2- 172.21.15.2:0/3212243002 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0ffde0 0x7efeac19ad80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:04.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.506+0000 7efeb2090700 1 -- 172.21.15.2:0/3212243002 >> 172.21.15.2:0/3212243002 conn(0x7efeac0f9f70 msgr2=0x7efeac0fc1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:04.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.506+0000 7efeb2090700 1 -- 172.21.15.2:0/3212243002 shutdown_connections 2023-11-16T18:24:04.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:04.506+0000 7efeb2090700 1 -- 172.21.15.2:0/3212243002 wait complete. 2023-11-16T18:24:04.910 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch ls' 2023-11-16T18:24:05.565 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:24:05.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:05 smithi040 ceph-mon[100251]: pgmap v301: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:05.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:05 smithi040 ceph-mon[100251]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:06.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:05 smithi002 ceph-mon[98807]: pgmap v301: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:06.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:05 smithi002 ceph-mon[98807]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:06.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:05 smithi188 ceph-mon[100639]: pgmap v301: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:06.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:05 smithi188 ceph-mon[100639]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:06.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.765+0000 7f683c931700 1 -- 172.21.15.2:0/2989780348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6838100d70 msgr2=0x7f6838107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:06.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.765+0000 7f683c931700 1 --2- 172.21.15.2:0/2989780348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6838100d70 0x7f6838107630 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f683000b0a0 tx=0x7f683000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:06.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.766+0000 7f683c931700 1 -- 172.21.15.2:0/2989780348 shutdown_connections 2023-11-16T18:24:06.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.766+0000 7f683c931700 1 --2- 172.21.15.2:0/2989780348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6838100d70 0x7f6838107630 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:06.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.766+0000 7f683c931700 1 --2- 172.21.15.2:0/2989780348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68380ff1e0 0x7f68380ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:06.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.766+0000 7f683c931700 1 --2- 172.21.15.2:0/2989780348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68381003e0 0x7f6838100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:06.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.766+0000 7f683c931700 1 -- 172.21.15.2:0/2989780348 >> 172.21.15.2:0/2989780348 conn(0x7f68380fa7f0 msgr2=0x7f68380fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:06.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.766+0000 7f683c931700 1 -- 172.21.15.2:0/2989780348 shutdown_connections 2023-11-16T18:24:06.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.766+0000 7f683c931700 1 -- 172.21.15.2:0/2989780348 wait complete. 2023-11-16T18:24:06.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.767+0000 7f683c931700 1 Processor -- start 2023-11-16T18:24:06.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.767+0000 7f683c931700 1 -- start start 2023-11-16T18:24:06.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.767+0000 7f683c931700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68380ff1e0 0x7f6838194ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:06.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.767+0000 7f683c931700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68381003e0 0x7f6838195530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:06.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.767+0000 7f683c931700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6838100d70 0x7f683819c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:06.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.767+0000 7f683c931700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683819cb80 con 0x7f6838100d70 2023-11-16T18:24:06.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.767+0000 7f683c931700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683819ccc0 con 0x7f68380ff1e0 2023-11-16T18:24:06.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.767+0000 7f683c931700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683819ce50 con 0x7f68381003e0 2023-11-16T18:24:06.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.767+0000 7f6836d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6838100d70 0x7f683819c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:06.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.767+0000 7f6835d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68381003e0 0x7f6838195530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:06.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.767+0000 7f6836d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6838100d70 0x7f683819c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49182/0 (socket says 172.21.15.2:49182) 2023-11-16T18:24:06.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.767+0000 7f6836d9d700 1 -- 172.21.15.2:0/162467235 learned_addr learned my addr 172.21.15.2:0/162467235 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:06.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.767+0000 7f683659c700 1 --2- 172.21.15.2:0/162467235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68380ff1e0 0x7f6838194ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:06.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.768+0000 7f6836d9d700 1 -- 172.21.15.2:0/162467235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68381003e0 msgr2=0x7f6838195530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:06.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.768+0000 7f6836d9d700 1 --2- 172.21.15.2:0/162467235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68381003e0 0x7f6838195530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:06.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.768+0000 7f6836d9d700 1 -- 172.21.15.2:0/162467235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68380ff1e0 msgr2=0x7f6838194ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:06.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.768+0000 7f6836d9d700 1 --2- 172.21.15.2:0/162467235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68380ff1e0 0x7f6838194ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:06.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.768+0000 7f6836d9d700 1 -- 172.21.15.2:0/162467235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6830009d00 con 0x7f6838100d70 2023-11-16T18:24:06.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.768+0000 7f6836d9d700 1 --2- 172.21.15.2:0/162467235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6838100d70 0x7f683819c5b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6830007ab0 tx=0x7f6830004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:06.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.768+0000 7f682f7fe700 1 -- 172.21.15.2:0/162467235 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683000e040 con 0x7f6838100d70 2023-11-16T18:24:06.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.768+0000 7f682f7fe700 1 -- 172.21.15.2:0/162467235 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f683000cec0 con 0x7f6838100d70 2023-11-16T18:24:06.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.768+0000 7f683c931700 1 -- 172.21.15.2:0/162467235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683810ab30 con 0x7f6838100d70 2023-11-16T18:24:06.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.768+0000 7f682f7fe700 1 -- 172.21.15.2:0/162467235 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683001bb30 con 0x7f6838100d70 2023-11-16T18:24:06.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.768+0000 7f683c931700 1 -- 172.21.15.2:0/162467235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683810a9a0 con 0x7f6838100d70 2023-11-16T18:24:06.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.769+0000 7f683c931700 1 -- 172.21.15.2:0/162467235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6838066ea0 con 0x7f6838100d70 2023-11-16T18:24:06.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.771+0000 7f682f7fe700 1 -- 172.21.15.2:0/162467235 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f683001bc90 con 0x7f6838100d70 2023-11-16T18:24:06.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.771+0000 7f682f7fe700 1 --2- 172.21.15.2:0/162467235 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f68240a11e0 0x7f68240a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:06.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.771+0000 7f683659c700 1 --2- 172.21.15.2:0/162467235 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f68240a11e0 0x7f68240a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:06.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.771+0000 7f682f7fe700 1 -- 172.21.15.2:0/162467235 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f68300ceed0 con 0x7f6838100d70 2023-11-16T18:24:06.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.772+0000 7f683659c700 1 --2- 172.21.15.2:0/162467235 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f68240a11e0 0x7f68240a3690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6838100240 tx=0x7f682000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:06.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.775+0000 7f682f7fe700 1 -- 172.21.15.2:0/162467235 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6830098240 con 0x7f6838100d70 2023-11-16T18:24:06.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.985+0000 7f683c931700 1 -- 172.21.15.2:0/162467235 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f68381066d0 con 0x7f68240a11e0 2023-11-16T18:24:06.993 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.992+0000 7f682f7fe700 1 -- 172.21.15.2:0/162467235 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+985 (secure 0 0 0) 0x7f68381066d0 con 0x7f68240a11e0 2023-11-16T18:24:06.994 INFO:teuthology.orchestra.run.smithi002.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-11-16T18:24:06.994 INFO:teuthology.orchestra.run.smithi002.stdout:mds.all 12/12 15s ago 112s smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2023-11-16T18:24:06.994 INFO:teuthology.orchestra.run.smithi002.stdout:mgr 3/3 15s ago 6m smithi002=x;smithi040=y;smithi188=z;count:3 2023-11-16T18:24:06.994 INFO:teuthology.orchestra.run.smithi002.stdout:mon 3/3 15s ago 6m smithi002:172.21.15.2=a;smithi040:172.21.15.40=b;smithi188:172.21.15.188=c;count:3 2023-11-16T18:24:06.994 INFO:teuthology.orchestra.run.smithi002.stdout:osd 12 15s ago - 2023-11-16T18:24:06.996 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.994+0000 7f683c931700 1 -- 172.21.15.2:0/162467235 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f68240a11e0 msgr2=0x7f68240a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:06.996 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.994+0000 7f683c931700 1 --2- 172.21.15.2:0/162467235 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f68240a11e0 0x7f68240a3690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6838100240 tx=0x7f682000a380 comp rx=0 tx=0).stop 2023-11-16T18:24:06.996 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.994+0000 7f683c931700 1 -- 172.21.15.2:0/162467235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6838100d70 msgr2=0x7f683819c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:06.996 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.994+0000 7f683c931700 1 --2- 172.21.15.2:0/162467235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6838100d70 0x7f683819c5b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6830007ab0 tx=0x7f6830004d00 comp rx=0 tx=0).stop 2023-11-16T18:24:06.996 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.994+0000 7f683c931700 1 -- 172.21.15.2:0/162467235 shutdown_connections 2023-11-16T18:24:06.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.994+0000 7f683c931700 1 --2- 172.21.15.2:0/162467235 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f68240a11e0 0x7f68240a3690 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:06.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.994+0000 7f683c931700 1 --2- 172.21.15.2:0/162467235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6838100d70 0x7f683819c5b0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:06.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.994+0000 7f683c931700 1 --2- 172.21.15.2:0/162467235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68380ff1e0 0x7f6838194ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:06.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.994+0000 7f683c931700 1 --2- 172.21.15.2:0/162467235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68381003e0 0x7f6838195530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:06.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.994+0000 7f683c931700 1 -- 172.21.15.2:0/162467235 >> 172.21.15.2:0/162467235 conn(0x7f68380fa7f0 msgr2=0x7f6838103880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:06.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.995+0000 7f683c931700 1 -- 172.21.15.2:0/162467235 shutdown_connections 2023-11-16T18:24:06.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:06.995+0000 7f683c931700 1 -- 172.21.15.2:0/162467235 wait complete. 2023-11-16T18:24:07.422 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch host ls' 2023-11-16T18:24:08.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:07 smithi188 ceph-mon[100639]: pgmap v302: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T18:24:08.077 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:24:08.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:07 smithi002 ceph-mon[98807]: pgmap v302: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T18:24:08.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:07 smithi040 ceph-mon[100251]: pgmap v302: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T18:24:08.835 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:08 smithi002 ceph-mon[98807]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:09.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:08 smithi188 ceph-mon[100639]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:09.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:08 smithi040 ceph-mon[100251]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:09.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.275+0000 7f2b5156c700 1 -- 172.21.15.2:0/1908317062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c100fe0 msgr2=0x7f2b4c1033c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:09.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.275+0000 7f2b5156c700 1 --2- 172.21.15.2:0/1908317062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c100fe0 0x7f2b4c1033c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c009990 tx=0x7f2b3c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:09.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.275+0000 7f2b5156c700 1 -- 172.21.15.2:0/1908317062 shutdown_connections 2023-11-16T18:24:09.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.275+0000 7f2b5156c700 1 --2- 172.21.15.2:0/1908317062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b4c106220 0x7f2b4c108620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:09.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.275+0000 7f2b5156c700 1 --2- 172.21.15.2:0/1908317062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c100fe0 0x7f2b4c1033c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:09.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.275+0000 7f2b5156c700 1 --2- 172.21.15.2:0/1908317062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b4c103900 0x7f2b4c105ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:09.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.275+0000 7f2b5156c700 1 -- 172.21.15.2:0/1908317062 >> 172.21.15.2:0/1908317062 conn(0x7f2b4c0faa20 msgr2=0x7f2b4c0fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:09.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.275+0000 7f2b5156c700 1 -- 172.21.15.2:0/1908317062 shutdown_connections 2023-11-16T18:24:09.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.275+0000 7f2b5156c700 1 -- 172.21.15.2:0/1908317062 wait complete. 2023-11-16T18:24:09.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.275+0000 7f2b5156c700 1 Processor -- start 2023-11-16T18:24:09.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b5156c700 1 -- start start 2023-11-16T18:24:09.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b5156c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b4c100fe0 0x7f2b4c196200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:09.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b5156c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b4c103900 0x7f2b4c196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:09.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b5156c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c106220 0x7f2b4c19d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:09.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b5156c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c19dd90 con 0x7f2b4c106220 2023-11-16T18:24:09.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b5156c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c19ded0 con 0x7f2b4c100fe0 2023-11-16T18:24:09.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b5156c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c19e060 con 0x7f2b4c103900 2023-11-16T18:24:09.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b4b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c106220 0x7f2b4c19d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:09.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b4b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c106220 0x7f2b4c19d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49196/0 (socket says 172.21.15.2:49196) 2023-11-16T18:24:09.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b4b7fe700 1 -- 172.21.15.2:0/2445384953 learned_addr learned my addr 172.21.15.2:0/2445384953 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:09.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b4affd700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b4c100fe0 0x7f2b4c196200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:09.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b4a7fc700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b4c103900 0x7f2b4c196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:09.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b4b7fe700 1 -- 172.21.15.2:0/2445384953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b4c103900 msgr2=0x7f2b4c196740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:09.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b4b7fe700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b4c103900 0x7f2b4c196740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:09.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b4b7fe700 1 -- 172.21.15.2:0/2445384953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b4c100fe0 msgr2=0x7f2b4c196200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:09.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b4b7fe700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b4c100fe0 0x7f2b4c196200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:09.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b4b7fe700 1 -- 172.21.15.2:0/2445384953 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c0095f0 con 0x7f2b4c106220 2023-11-16T18:24:09.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b4affd700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b4c100fe0 0x7f2b4c196200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:09.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b4a7fc700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b4c103900 0x7f2b4c196740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:09.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.276+0000 7f2b4b7fe700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c106220 0x7f2b4c19d7c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2b44007ee0 tx=0x7f2b4400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:09.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.277+0000 7f2b33fff700 1 -- 172.21.15.2:0/2445384953 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b44004db0 con 0x7f2b4c106220 2023-11-16T18:24:09.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.277+0000 7f2b5156c700 1 -- 172.21.15.2:0/2445384953 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b4c10be80 con 0x7f2b4c106220 2023-11-16T18:24:09.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.277+0000 7f2b33fff700 1 -- 172.21.15.2:0/2445384953 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b4400f040 con 0x7f2b4c106220 2023-11-16T18:24:09.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.277+0000 7f2b33fff700 1 -- 172.21.15.2:0/2445384953 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b44005970 con 0x7f2b4c106220 2023-11-16T18:24:09.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.277+0000 7f2b5156c700 1 -- 172.21.15.2:0/2445384953 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b4c10bcf0 con 0x7f2b4c106220 2023-11-16T18:24:09.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.278+0000 7f2b5156c700 1 -- 172.21.15.2:0/2445384953 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4c0fc590 con 0x7f2b4c106220 2023-11-16T18:24:09.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.279+0000 7f2b33fff700 1 -- 172.21.15.2:0/2445384953 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f2b4400dbd0 con 0x7f2b4c106220 2023-11-16T18:24:09.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.279+0000 7f2b33fff700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2b340a11e0 0x7f2b340a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:09.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.279+0000 7f2b4affd700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2b340a11e0 0x7f2b340a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:09.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.279+0000 7f2b33fff700 1 -- 172.21.15.2:0/2445384953 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2b440cc7a0 con 0x7f2b4c106220 2023-11-16T18:24:09.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.280+0000 7f2b4affd700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2b340a11e0 0x7f2b340a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c006010 tx=0x7f2b3c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:09.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.284+0000 7f2b33fff700 1 -- 172.21.15.2:0/2445384953 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2b4404e730 con 0x7f2b4c106220 2023-11-16T18:24:09.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.488+0000 7f2b5156c700 1 -- 172.21.15.2:0/2445384953 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b4c060f30 con 0x7f2b340a11e0 2023-11-16T18:24:09.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.490+0000 7f2b33fff700 1 -- 172.21.15.2:0/2445384953 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f2b4c060f30 con 0x7f2b340a11e0 2023-11-16T18:24:09.491 INFO:teuthology.orchestra.run.smithi002.stdout:HOST ADDR LABELS STATUS 2023-11-16T18:24:09.492 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 172.21.15.2 2023-11-16T18:24:09.492 INFO:teuthology.orchestra.run.smithi002.stdout:smithi040 172.21.15.40 2023-11-16T18:24:09.492 INFO:teuthology.orchestra.run.smithi002.stdout:smithi188 172.21.15.188 2023-11-16T18:24:09.492 INFO:teuthology.orchestra.run.smithi002.stdout:3 hosts in cluster 2023-11-16T18:24:09.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.492+0000 7f2b5156c700 1 -- 172.21.15.2:0/2445384953 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2b340a11e0 msgr2=0x7f2b340a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:09.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.492+0000 7f2b5156c700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2b340a11e0 0x7f2b340a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c006010 tx=0x7f2b3c00b540 comp rx=0 tx=0).stop 2023-11-16T18:24:09.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.492+0000 7f2b5156c700 1 -- 172.21.15.2:0/2445384953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c106220 msgr2=0x7f2b4c19d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:09.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.492+0000 7f2b5156c700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c106220 0x7f2b4c19d7c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2b44007ee0 tx=0x7f2b4400d3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:09.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.492+0000 7f2b5156c700 1 -- 172.21.15.2:0/2445384953 shutdown_connections 2023-11-16T18:24:09.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.492+0000 7f2b5156c700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2b340a11e0 0x7f2b340a3690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:09.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.492+0000 7f2b5156c700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c106220 0x7f2b4c19d7c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:09.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.492+0000 7f2b5156c700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b4c100fe0 0x7f2b4c196200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:09.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.492+0000 7f2b5156c700 1 --2- 172.21.15.2:0/2445384953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b4c103900 0x7f2b4c196740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:09.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.492+0000 7f2b5156c700 1 -- 172.21.15.2:0/2445384953 >> 172.21.15.2:0/2445384953 conn(0x7f2b4c0faa20 msgr2=0x7f2b4c101ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:09.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.492+0000 7f2b5156c700 1 -- 172.21.15.2:0/2445384953 shutdown_connections 2023-11-16T18:24:09.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:09.492+0000 7f2b5156c700 1 -- 172.21.15.2:0/2445384953 wait complete. 2023-11-16T18:24:09.847 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:09 smithi002 ceph-mon[98807]: pgmap v303: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:24:09.932 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch device ls' 2023-11-16T18:24:10.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:09 smithi188 ceph-mon[100639]: pgmap v303: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:24:10.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:09 smithi040 ceph-mon[100251]: pgmap v303: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:24:10.588 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:24:10.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:10 smithi002 ceph-mon[98807]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:11.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:10 smithi188 ceph-mon[100639]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:11.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:10 smithi040 ceph-mon[100251]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:11.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.792+0000 7f14d3510700 1 -- 172.21.15.2:0/2226404922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14cc100d70 msgr2=0x7f14cc107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:11.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.792+0000 7f14d3510700 1 --2- 172.21.15.2:0/2226404922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14cc100d70 0x7f14cc107630 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f14c400b0a0 tx=0x7f14c400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:11.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.792+0000 7f14d3510700 1 -- 172.21.15.2:0/2226404922 shutdown_connections 2023-11-16T18:24:11.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.792+0000 7f14d3510700 1 --2- 172.21.15.2:0/2226404922 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14cc0ff1e0 0x7f14cc0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:11.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.792+0000 7f14d3510700 1 --2- 172.21.15.2:0/2226404922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14cc100d70 0x7f14cc107630 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:11.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.792+0000 7f14d3510700 1 --2- 172.21.15.2:0/2226404922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14cc1003e0 0x7f14cc100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:11.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.792+0000 7f14d3510700 1 -- 172.21.15.2:0/2226404922 >> 172.21.15.2:0/2226404922 conn(0x7f14cc0fa7f0 msgr2=0x7f14cc0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:11.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.792+0000 7f14d3510700 1 -- 172.21.15.2:0/2226404922 shutdown_connections 2023-11-16T18:24:11.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.793+0000 7f14d3510700 1 -- 172.21.15.2:0/2226404922 wait complete. 2023-11-16T18:24:11.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.793+0000 7f14d3510700 1 Processor -- start 2023-11-16T18:24:11.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.793+0000 7f14d3510700 1 -- start start 2023-11-16T18:24:11.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d3510700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14cc0ff1e0 0x7f14cc195230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:11.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d3510700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14cc1003e0 0x7f14cc195770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:11.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d3510700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14cc100d70 0x7f14cc19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:11.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d3510700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc19cdc0 con 0x7f14cc0ff1e0 2023-11-16T18:24:11.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d3510700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc19cf00 con 0x7f14cc1003e0 2023-11-16T18:24:11.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d3510700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc19d090 con 0x7f14cc100d70 2023-11-16T18:24:11.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d12ac700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14cc0ff1e0 0x7f14cc195230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:11.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d12ac700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14cc0ff1e0 0x7f14cc195230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49202/0 (socket says 172.21.15.2:49202) 2023-11-16T18:24:11.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d12ac700 1 -- 172.21.15.2:0/1864798782 learned_addr learned my addr 172.21.15.2:0/1864798782 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:11.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d0aab700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14cc1003e0 0x7f14cc195770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:11.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d1aad700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14cc100d70 0x7f14cc19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:11.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d12ac700 1 -- 172.21.15.2:0/1864798782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14cc100d70 msgr2=0x7f14cc19c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:11.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d12ac700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14cc100d70 0x7f14cc19c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:11.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d12ac700 1 -- 172.21.15.2:0/1864798782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14cc1003e0 msgr2=0x7f14cc195770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:11.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d12ac700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14cc1003e0 0x7f14cc195770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:11.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d12ac700 1 -- 172.21.15.2:0/1864798782 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c4009d00 con 0x7f14cc0ff1e0 2023-11-16T18:24:11.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d0aab700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14cc1003e0 0x7f14cc195770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:11.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14d12ac700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14cc0ff1e0 0x7f14cc195230 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00b530 tx=0x7f14bc00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:11.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.794+0000 7f14c27fc700 1 -- 172.21.15.2:0/1864798782 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc011830 con 0x7f14cc0ff1e0 2023-11-16T18:24:11.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.795+0000 7f14d3510700 1 -- 172.21.15.2:0/1864798782 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14cc10ac50 con 0x7f14cc0ff1e0 2023-11-16T18:24:11.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.795+0000 7f14c27fc700 1 -- 172.21.15.2:0/1864798782 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14bc011e50 con 0x7f14cc0ff1e0 2023-11-16T18:24:11.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.795+0000 7f14c27fc700 1 -- 172.21.15.2:0/1864798782 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc00f2e0 con 0x7f14cc0ff1e0 2023-11-16T18:24:11.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.795+0000 7f14d3510700 1 -- 172.21.15.2:0/1864798782 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14cc10aac0 con 0x7f14cc0ff1e0 2023-11-16T18:24:11.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.795+0000 7f14d3510700 1 -- 172.21.15.2:0/1864798782 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14cc1061f0 con 0x7f14cc0ff1e0 2023-11-16T18:24:11.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.797+0000 7f14c27fc700 1 -- 172.21.15.2:0/1864798782 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f14bc010580 con 0x7f14cc0ff1e0 2023-11-16T18:24:11.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.797+0000 7f14c27fc700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f14b80a11e0 0x7f14b80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:11.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.798+0000 7f14d0aab700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f14b80a11e0 0x7f14b80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:11.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.798+0000 7f14c27fc700 1 -- 172.21.15.2:0/1864798782 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f14bc0ccd70 con 0x7f14cc0ff1e0 2023-11-16T18:24:11.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.798+0000 7f14d0aab700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f14b80a11e0 0x7f14b80a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f14c80099d0 tx=0x7f14c8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:11.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:11.801+0000 7f14c27fc700 1 -- 172.21.15.2:0/1864798782 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f14bc096100 con 0x7f14cc0ff1e0 2023-11-16T18:24:11.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:11 smithi002 ceph-mon[98807]: pgmap v304: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:12.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.007+0000 7f14d3510700 1 -- 172.21.15.2:0/1864798782 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f14cc069130 con 0x7f14b80a11e0 2023-11-16T18:24:12.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.009+0000 7f14c27fc700 1 -- 172.21.15.2:0/1864798782 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f14cc069130 con 0x7f14b80a11e0 2023-11-16T18:24:12.010 INFO:teuthology.orchestra.run.smithi002.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-11-16T18:24:12.010 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300EV480DGN 447G No 2m ago LVM detected, locked 2023-11-16T18:24:12.010 INFO:teuthology.orchestra.run.smithi002.stdout:smithi040 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009J400BGN 372G No 2m ago LVM detected, locked 2023-11-16T18:24:12.010 INFO:teuthology.orchestra.run.smithi002.stdout:smithi188 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500MP400BGN 372G No 2m ago LVM detected, locked 2023-11-16T18:24:12.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.011+0000 7f14d3510700 1 -- 172.21.15.2:0/1864798782 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f14b80a11e0 msgr2=0x7f14b80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:12.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.011+0000 7f14d3510700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f14b80a11e0 0x7f14b80a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f14c80099d0 tx=0x7f14c8008040 comp rx=0 tx=0).stop 2023-11-16T18:24:12.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.011+0000 7f14d3510700 1 -- 172.21.15.2:0/1864798782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14cc0ff1e0 msgr2=0x7f14cc195230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:12.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.011+0000 7f14d3510700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14cc0ff1e0 0x7f14cc195230 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00b530 tx=0x7f14bc00b8f0 comp rx=0 tx=0).stop 2023-11-16T18:24:12.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.011+0000 7f14d3510700 1 -- 172.21.15.2:0/1864798782 shutdown_connections 2023-11-16T18:24:12.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.011+0000 7f14d3510700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f14b80a11e0 0x7f14b80a3690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:12.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.011+0000 7f14d3510700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14cc100d70 0x7f14cc19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:12.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.011+0000 7f14d3510700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14cc0ff1e0 0x7f14cc195230 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:12.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.011+0000 7f14d3510700 1 --2- 172.21.15.2:0/1864798782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14cc1003e0 0x7f14cc195770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:12.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.011+0000 7f14d3510700 1 -- 172.21.15.2:0/1864798782 >> 172.21.15.2:0/1864798782 conn(0x7f14cc0fa7f0 msgr2=0x7f14cc103950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:12.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.011+0000 7f14d3510700 1 -- 172.21.15.2:0/1864798782 shutdown_connections 2023-11-16T18:24:12.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:12.011+0000 7f14d3510700 1 -- 172.21.15.2:0/1864798782 wait complete. 2023-11-16T18:24:12.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:11 smithi188 ceph-mon[100639]: pgmap v304: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:12.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:11 smithi040 ceph-mon[100251]: pgmap v304: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:12.357 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-16T18:24:12.372 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T18:24:12.373 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph fs dump' 2023-11-16T18:24:13.028 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:24:13.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:13 smithi002 ceph-mon[98807]: pgmap v305: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T18:24:13.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:13 smithi002 ceph-mon[98807]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:14.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:13 smithi188 ceph-mon[100639]: pgmap v305: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T18:24:14.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:13 smithi188 ceph-mon[100639]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:14.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:13 smithi040 ceph-mon[100251]: pgmap v305: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T18:24:14.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:13 smithi040 ceph-mon[100251]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:14.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.336+0000 7f2fe314d700 1 -- 172.21.15.2:0/174579992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fdc100d70 msgr2=0x7f2fdc107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:14.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.336+0000 7f2fe314d700 1 --2- 172.21.15.2:0/174579992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fdc100d70 0x7f2fdc107630 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400b0a0 tx=0x7f2fd400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:14.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.336+0000 7f2fe314d700 1 -- 172.21.15.2:0/174579992 shutdown_connections 2023-11-16T18:24:14.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.336+0000 7f2fe314d700 1 --2- 172.21.15.2:0/174579992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fdc100d70 0x7f2fdc107630 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:14.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.336+0000 7f2fe314d700 1 --2- 172.21.15.2:0/174579992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc0ff1e0 0x7f2fdc0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:14.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.336+0000 7f2fe314d700 1 --2- 172.21.15.2:0/174579992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fdc1003e0 0x7f2fdc100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:14.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.336+0000 7f2fe314d700 1 -- 172.21.15.2:0/174579992 >> 172.21.15.2:0/174579992 conn(0x7f2fdc0fa7f0 msgr2=0x7f2fdc0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:14.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.336+0000 7f2fe314d700 1 -- 172.21.15.2:0/174579992 shutdown_connections 2023-11-16T18:24:14.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.336+0000 7f2fe314d700 1 -- 172.21.15.2:0/174579992 wait complete. 2023-11-16T18:24:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe314d700 1 Processor -- start 2023-11-16T18:24:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe314d700 1 -- start start 2023-11-16T18:24:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe314d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fdc0ff1e0 0x7f2fdc198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe314d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc1003e0 0x7f2fdc198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe314d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fdc100d70 0x7f2fdc19f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe314d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc19fed0 con 0x7f2fdc100d70 2023-11-16T18:24:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe314d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc1a0010 con 0x7f2fdc1003e0 2023-11-16T18:24:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe314d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc1a01a0 con 0x7f2fdc0ff1e0 2023-11-16T18:24:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe16ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fdc100d70 0x7f2fdc19f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe16ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fdc100d70 0x7f2fdc19f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49220/0 (socket says 172.21.15.2:49220) 2023-11-16T18:24:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe16ea700 1 -- 172.21.15.2:0/2789509350 learned_addr learned my addr 172.21.15.2:0/2789509350 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe0ee9700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fdc0ff1e0 0x7f2fdc198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fdbfff700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc1003e0 0x7f2fdc198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe16ea700 1 -- 172.21.15.2:0/2789509350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fdc0ff1e0 msgr2=0x7f2fdc198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe16ea700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fdc0ff1e0 0x7f2fdc198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe16ea700 1 -- 172.21.15.2:0/2789509350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc1003e0 msgr2=0x7f2fdc198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe16ea700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc1003e0 0x7f2fdc198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe16ea700 1 -- 172.21.15.2:0/2789509350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd4009d00 con 0x7f2fdc100d70 2023-11-16T18:24:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.337+0000 7f2fe0ee9700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fdc0ff1e0 0x7f2fdc198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.338+0000 7f2fe16ea700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fdc100d70 0x7f2fdc19f900 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400b070 tx=0x7f2fd4009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.338+0000 7f2fd9ffb700 1 -- 172.21.15.2:0/2789509350 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd400e040 con 0x7f2fdc100d70 2023-11-16T18:24:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.338+0000 7f2fe314d700 1 -- 172.21.15.2:0/2789509350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fdc10ab30 con 0x7f2fdc100d70 2023-11-16T18:24:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.338+0000 7f2fd9ffb700 1 -- 172.21.15.2:0/2789509350 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fd4004020 con 0x7f2fdc100d70 2023-11-16T18:24:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.338+0000 7f2fd9ffb700 1 -- 172.21.15.2:0/2789509350 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd401d6e0 con 0x7f2fdc100d70 2023-11-16T18:24:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.338+0000 7f2fe314d700 1 -- 172.21.15.2:0/2789509350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fdc10a9a0 con 0x7f2fdc100d70 2023-11-16T18:24:14.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.339+0000 7f2fe314d700 1 -- 172.21.15.2:0/2789509350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fdc1a08a0 con 0x7f2fdc100d70 2023-11-16T18:24:14.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.340+0000 7f2fd9ffb700 1 -- 172.21.15.2:0/2789509350 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f2fd4004470 con 0x7f2fdc100d70 2023-11-16T18:24:14.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.341+0000 7f2fd9ffb700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2fcc0a11e0 0x7f2fcc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:14.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.341+0000 7f2fe0ee9700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2fcc0a11e0 0x7f2fcc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:14.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.341+0000 7f2fd9ffb700 1 -- 172.21.15.2:0/2789509350 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2fd40cec00 con 0x7f2fdc100d70 2023-11-16T18:24:14.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.341+0000 7f2fe0ee9700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2fcc0a11e0 0x7f2fcc0a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0005950 tx=0x7f2fd00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:14.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.345+0000 7f2fd9ffb700 1 -- 172.21.15.2:0/2789509350 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2fd4097f70 con 0x7f2fdc100d70 2023-11-16T18:24:14.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.586+0000 7f2fe314d700 1 -- 172.21.15.2:0/2789509350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2fdc1a0360 con 0x7f2fdc100d70 2023-11-16T18:24:14.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.586+0000 7f2fd9ffb700 1 -- 172.21.15.2:0/2789509350 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+2878 (secure 0 0 0) 0x7f2fd409b590 con 0x7f2fdc100d70 2023-11-16T18:24:14.588 INFO:teuthology.orchestra.run.smithi002.stdout:e24 2023-11-16T18:24:14.588 INFO:teuthology.orchestra.run.smithi002.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T18:24:14.588 INFO:teuthology.orchestra.run.smithi002.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T18:24:14.588 INFO:teuthology.orchestra.run.smithi002.stdout:legacy client fscid: 1 2023-11-16T18:24:14.588 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:14.588 INFO:teuthology.orchestra.run.smithi002.stdout:Filesystem 'cephfs' (1) 2023-11-16T18:24:14.588 INFO:teuthology.orchestra.run.smithi002.stdout:fs_name cephfs 2023-11-16T18:24:14.588 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 24 2023-11-16T18:24:14.588 INFO:teuthology.orchestra.run.smithi002.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-11-16T18:24:14.588 INFO:teuthology.orchestra.run.smithi002.stdout:created 2023-11-16T18:22:46.806019+0000 2023-11-16T18:24:14.588 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2023-11-16T18:23:07.909835+0000 2023-11-16T18:24:14.588 INFO:teuthology.orchestra.run.smithi002.stdout:tableserver 0 2023-11-16T18:24:14.588 INFO:teuthology.orchestra.run.smithi002.stdout:root 0 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:session_timeout 300 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:session_autoclose 300 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:max_file_size 1099511627776 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:max_xattr_size 65536 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:required_client_features {} 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure 0 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure_osd_epoch 0 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:max_mds 3 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:in 0,1,2 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:up {0=24473,1=24460,2=24446} 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:failed 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:damaged 2023-11-16T18:24:14.589 INFO:teuthology.orchestra.run.smithi002.stdout:stopped 2023-11-16T18:24:14.590 INFO:teuthology.orchestra.run.smithi002.stdout:data_pools [3,4] 2023-11-16T18:24:14.590 INFO:teuthology.orchestra.run.smithi002.stdout:metadata_pool 2 2023-11-16T18:24:14.590 INFO:teuthology.orchestra.run.smithi002.stdout:inline_data disabled 2023-11-16T18:24:14.590 INFO:teuthology.orchestra.run.smithi002.stdout:balancer 2023-11-16T18:24:14.590 INFO:teuthology.orchestra.run.smithi002.stdout:bal_rank_mask -1 2023-11-16T18:24:14.590 INFO:teuthology.orchestra.run.smithi002.stdout:standby_count_wanted 0 2023-11-16T18:24:14.590 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.b{0:24473} state up:active seq 3 addr [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T18:24:14.590 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.c{0:24439} state up:standby-replay seq 1 addr [v2:172.21.15.188:6836/264232322,v1:172.21.15.188:6837/264232322] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T18:24:14.590 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.i{1:24460} state up:active seq 9 addr [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T18:24:14.590 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.f{1:24427} state up:standby-replay seq 1 addr [v2:172.21.15.188:6834/3630447129,v1:172.21.15.188:6835/3630447129] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T18:24:14.590 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.h{2:24446} state up:active seq 10 addr [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T18:24:14.590 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.e{2:24422} state up:standby-replay seq 1 addr [v2:172.21.15.40:6834/17956118,v1:172.21.15.40:6835/17956118] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T18:24:14.591 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:14.591 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:14.591 INFO:teuthology.orchestra.run.smithi002.stdout:Standby daemons: 2023-11-16T18:24:14.591 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:14.591 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.j{-1:14541} state up:standby seq 1 addr [v2:172.21.15.2:6834/223055355,v1:172.21.15.2:6835/223055355] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T18:24:14.591 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.d{-1:14553} state up:standby seq 1 addr [v2:172.21.15.2:6836/508165139,v1:172.21.15.2:6837/508165139] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T18:24:14.591 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.g{-1:14601} state up:standby seq 1 addr [v2:172.21.15.2:6838/1181439264,v1:172.21.15.2:6839/1181439264] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T18:24:14.591 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.a{-1:14613} state up:standby seq 1 addr [v2:172.21.15.2:6840/3213213799,v1:172.21.15.2:6841/3213213799] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T18:24:14.591 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.k{-1:24395} state up:standby seq 1 addr [v2:172.21.15.40:6832/1110523894,v1:172.21.15.40:6833/1110523894] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T18:24:14.591 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.l{-1:24400} state up:standby seq 1 addr [v2:172.21.15.188:6832/381028209,v1:172.21.15.188:6833/381028209] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T18:24:14.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.589+0000 7f2fe314d700 1 -- 172.21.15.2:0/2789509350 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2fcc0a11e0 msgr2=0x7f2fcc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:14.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.589+0000 7f2fe314d700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2fcc0a11e0 0x7f2fcc0a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0005950 tx=0x7f2fd00058e0 comp rx=0 tx=0).stop 2023-11-16T18:24:14.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.589+0000 7f2fe314d700 1 -- 172.21.15.2:0/2789509350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fdc100d70 msgr2=0x7f2fdc19f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:14.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.589+0000 7f2fe314d700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fdc100d70 0x7f2fdc19f900 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400b070 tx=0x7f2fd4009590 comp rx=0 tx=0).stop 2023-11-16T18:24:14.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.589+0000 7f2fe314d700 1 -- 172.21.15.2:0/2789509350 shutdown_connections 2023-11-16T18:24:14.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.589+0000 7f2fe314d700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f2fcc0a11e0 0x7f2fcc0a3690 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:14.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.589+0000 7f2fe314d700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fdc0ff1e0 0x7f2fdc198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:14.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.589+0000 7f2fe314d700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fdc100d70 0x7f2fdc19f900 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:14.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.589+0000 7f2fe314d700 1 --2- 172.21.15.2:0/2789509350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc1003e0 0x7f2fdc198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:14.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.589+0000 7f2fe314d700 1 -- 172.21.15.2:0/2789509350 >> 172.21.15.2:0/2789509350 conn(0x7f2fdc0fa7f0 msgr2=0x7f2fdc068610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:14.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.589+0000 7f2fe314d700 1 -- 172.21.15.2:0/2789509350 shutdown_connections 2023-11-16T18:24:14.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:14.589+0000 7f2fe314d700 1 -- 172.21.15.2:0/2789509350 wait complete. 2023-11-16T18:24:14.594 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T18:24:14.925 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:14 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2789509350' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-11-16T18:24:14.993 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph osd dump' 2023-11-16T18:24:15.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:14 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2789509350' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-11-16T18:24:15.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:14 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2789509350' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-11-16T18:24:15.653 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:24:16.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:15 smithi188 ceph-mon[100639]: pgmap v306: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:16.058 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:15 smithi002 ceph-mon[98807]: pgmap v306: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:16.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:15 smithi040 ceph-mon[100251]: pgmap v306: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:16.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.854+0000 7f8a563c6700 1 -- 172.21.15.2:0/1814774227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a50100d70 msgr2=0x7f8a50107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:16.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.854+0000 7f8a563c6700 1 --2- 172.21.15.2:0/1814774227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a50100d70 0x7f8a50107630 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800b0a0 tx=0x7f8a4800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:16.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.854+0000 7f8a563c6700 1 -- 172.21.15.2:0/1814774227 shutdown_connections 2023-11-16T18:24:16.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.854+0000 7f8a563c6700 1 --2- 172.21.15.2:0/1814774227 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a500ff1e0 0x7f8a500ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:16.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.854+0000 7f8a563c6700 1 --2- 172.21.15.2:0/1814774227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a50100d70 0x7f8a50107630 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:16.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.854+0000 7f8a563c6700 1 --2- 172.21.15.2:0/1814774227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a501003e0 0x7f8a50100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:16.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.854+0000 7f8a563c6700 1 -- 172.21.15.2:0/1814774227 >> 172.21.15.2:0/1814774227 conn(0x7f8a500fa7f0 msgr2=0x7f8a500fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:16.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.854+0000 7f8a563c6700 1 -- 172.21.15.2:0/1814774227 shutdown_connections 2023-11-16T18:24:16.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.854+0000 7f8a563c6700 1 -- 172.21.15.2:0/1814774227 wait complete. 2023-11-16T18:24:16.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a563c6700 1 Processor -- start 2023-11-16T18:24:16.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a563c6700 1 -- start start 2023-11-16T18:24:16.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a563c6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a500ff1e0 0x7f8a50196160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:16.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a563c6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a501003e0 0x7f8a501966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:16.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a563c6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a50100d70 0x7f8a5019d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:16.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a563c6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5019dcf0 con 0x7f8a50100d70 2023-11-16T18:24:16.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a563c6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5019de30 con 0x7f8a500ff1e0 2023-11-16T18:24:16.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a563c6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5019dfc0 con 0x7f8a501003e0 2023-11-16T18:24:16.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a54963700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a50100d70 0x7f8a5019d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:16.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a4ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a500ff1e0 0x7f8a50196160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:16.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a4f7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a501003e0 0x7f8a501966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:16.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a4ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a500ff1e0 0x7f8a50196160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:39664/0 (socket says 172.21.15.2:39664) 2023-11-16T18:24:16.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a4f7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a501003e0 0x7f8a501966a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:40758/0 (socket says 172.21.15.2:40758) 2023-11-16T18:24:16.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.856+0000 7f8a4ffff700 1 -- 172.21.15.2:0/2655924891 learned_addr learned my addr 172.21.15.2:0/2655924891 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:16.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.855+0000 7f8a54963700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a50100d70 0x7f8a5019d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49230/0 (socket says 172.21.15.2:49230) 2023-11-16T18:24:16.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.856+0000 7f8a54963700 1 -- 172.21.15.2:0/2655924891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a501003e0 msgr2=0x7f8a501966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:16.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.856+0000 7f8a54963700 1 --2- 172.21.15.2:0/2655924891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a501003e0 0x7f8a501966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:16.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.856+0000 7f8a54963700 1 -- 172.21.15.2:0/2655924891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a500ff1e0 msgr2=0x7f8a50196160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:16.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.856+0000 7f8a54963700 1 --2- 172.21.15.2:0/2655924891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a500ff1e0 0x7f8a50196160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:16.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.856+0000 7f8a54963700 1 -- 172.21.15.2:0/2655924891 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a48009d00 con 0x7f8a50100d70 2023-11-16T18:24:16.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.856+0000 7f8a54963700 1 --2- 172.21.15.2:0/2655924891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a50100d70 0x7f8a5019d720 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800b0a0 tx=0x7f8a480094e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:16.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.856+0000 7f8a4d7fa700 1 -- 172.21.15.2:0/2655924891 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4800e040 con 0x7f8a50100d70 2023-11-16T18:24:16.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.856+0000 7f8a4d7fa700 1 -- 172.21.15.2:0/2655924891 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a48006080 con 0x7f8a50100d70 2023-11-16T18:24:16.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.856+0000 7f8a4d7fa700 1 -- 172.21.15.2:0/2655924891 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4801b850 con 0x7f8a50100d70 2023-11-16T18:24:16.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.856+0000 7f8a563c6700 1 -- 172.21.15.2:0/2655924891 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5010bce0 con 0x7f8a50100d70 2023-11-16T18:24:16.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.856+0000 7f8a563c6700 1 -- 172.21.15.2:0/2655924891 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5010bb50 con 0x7f8a50100d70 2023-11-16T18:24:16.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.857+0000 7f8a563c6700 1 -- 172.21.15.2:0/2655924891 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a50106170 con 0x7f8a50100d70 2023-11-16T18:24:16.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.858+0000 7f8a4d7fa700 1 -- 172.21.15.2:0/2655924891 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f8a4801d480 con 0x7f8a50100d70 2023-11-16T18:24:16.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.859+0000 7f8a4d7fa700 1 --2- 172.21.15.2:0/2655924891 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8a380a11e0 0x7f8a380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:16.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.859+0000 7f8a4ffff700 1 --2- 172.21.15.2:0/2655924891 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8a380a11e0 0x7f8a380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:16.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.859+0000 7f8a4d7fa700 1 -- 172.21.15.2:0/2655924891 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8a480cf0f0 con 0x7f8a50100d70 2023-11-16T18:24:16.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.859+0000 7f8a4ffff700 1 --2- 172.21.15.2:0/2655924891 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8a380a11e0 0x7f8a380a3690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000bd70 tx=0x7f8a4000b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:16.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:16.862+0000 7f8a4d7fa700 1 -- 172.21.15.2:0/2655924891 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8a48098460 con 0x7f8a50100d70 2023-11-16T18:24:17.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.059+0000 7f8a563c6700 1 -- 172.21.15.2:0/2655924891 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f8a50066ea0 con 0x7f8a50100d70 2023-11-16T18:24:17.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.060+0000 7f8a4d7fa700 1 -- 172.21.15.2:0/2655924891 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) v1 ==== 56+0+5561 (secure 0 0 0) 0x7f8a4809ba80 con 0x7f8a50100d70 2023-11-16T18:24:17.061 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 88 2023-11-16T18:24:17.061 INFO:teuthology.orchestra.run.smithi002.stdout:fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 2023-11-16T18:24:17.061 INFO:teuthology.orchestra.run.smithi002.stdout:created 2023-11-16T18:15:41.444339+0000 2023-11-16T18:24:17.062 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2023-11-16T18:23:39.497670+0000 2023-11-16T18:24:17.062 INFO:teuthology.orchestra.run.smithi002.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-11-16T18:24:17.062 INFO:teuthology.orchestra.run.smithi002.stdout:crush_version 27 2023-11-16T18:24:17.062 INFO:teuthology.orchestra.run.smithi002.stdout:full_ratio 0.95 2023-11-16T18:24:17.062 INFO:teuthology.orchestra.run.smithi002.stdout:backfillfull_ratio 0.9 2023-11-16T18:24:17.062 INFO:teuthology.orchestra.run.smithi002.stdout:nearfull_ratio 0.85 2023-11-16T18:24:17.062 INFO:teuthology.orchestra.run.smithi002.stdout:require_min_compat_client luminous 2023-11-16T18:24:17.062 INFO:teuthology.orchestra.run.smithi002.stdout:min_compat_client luminous 2023-11-16T18:24:17.062 INFO:teuthology.orchestra.run.smithi002.stdout:require_osd_release reef 2023-11-16T18:24:17.062 INFO:teuthology.orchestra.run.smithi002.stdout:stretch_mode_enabled false 2023-11-16T18:24:17.062 INFO:teuthology.orchestra.run.smithi002.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-11-16T18:24:17.063 INFO:teuthology.orchestra.run.smithi002.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 81 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2023-11-16T18:24:17.063 INFO:teuthology.orchestra.run.smithi002.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2023-11-16T18:24:17.063 INFO:teuthology.orchestra.run.smithi002.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2023-11-16T18:24:17.063 INFO:teuthology.orchestra.run.smithi002.stdout:max_osd 12 2023-11-16T18:24:17.063 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6802/4123843884,v1:172.21.15.2:6803/4123843884] [v2:172.21.15.2:6804/4123843884,v1:172.21.15.2:6805/4123843884] exists,up 7d70b5c0-08db-4d20-879d-c61212c48535 2023-11-16T18:24:17.063 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6800/3016934247,v1:172.21.15.40:6801/3016934247] [v2:172.21.15.40:6802/3016934247,v1:172.21.15.40:6803/3016934247] exists,up 69bb34ad-0d6c-4e75-911b-e366bb104d4c 2023-11-16T18:24:17.063 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6800/3338843401,v1:172.21.15.188:6801/3338843401] [v2:172.21.15.188:6802/3338843401,v1:172.21.15.188:6803/3338843401] exists,up 80f9c468-7dd5-494a-a6e2-98bea69fbe4b 2023-11-16T18:24:17.063 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 up in weight 1 up_from 25 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6810/1081663732,v1:172.21.15.2:6811/1081663732] [v2:172.21.15.2:6812/1081663732,v1:172.21.15.2:6813/1081663732] exists,up 9a13117e-ad7f-4576-8dcd-dee08b526577 2023-11-16T18:24:17.063 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 up in weight 1 up_from 32 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6808/865046130,v1:172.21.15.40:6809/865046130] [v2:172.21.15.40:6810/865046130,v1:172.21.15.40:6811/865046130] exists,up 0453e4fa-a79a-4aea-a701-aedd106bdb08 2023-11-16T18:24:17.063 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 up in weight 1 up_from 37 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6808/325018840,v1:172.21.15.188:6809/325018840] [v2:172.21.15.188:6810/325018840,v1:172.21.15.188:6811/325018840] exists,up 30994739-f13b-4840-b14d-af1553319d21 2023-11-16T18:24:17.064 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 up in weight 1 up_from 44 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6818/1942175812,v1:172.21.15.2:6819/1942175812] [v2:172.21.15.2:6820/1942175812,v1:172.21.15.2:6821/1942175812] exists,up 6d97c575-96ae-46ad-a165-36cc16633818 2023-11-16T18:24:17.064 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 up in weight 1 up_from 49 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6816/139194472,v1:172.21.15.40:6817/139194472] [v2:172.21.15.40:6818/139194472,v1:172.21.15.40:6819/139194472] exists,up bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36 2023-11-16T18:24:17.064 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8 up in weight 1 up_from 57 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6816/3337313264,v1:172.21.15.188:6817/3337313264] [v2:172.21.15.188:6818/3337313264,v1:172.21.15.188:6819/3337313264] exists,up 2589fd33-ad12-47e5-a70d-172ea07d61bc 2023-11-16T18:24:17.064 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9 up in weight 1 up_from 64 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6826/1605296180,v1:172.21.15.2:6827/1605296180] [v2:172.21.15.2:6828/1605296180,v1:172.21.15.2:6829/1605296180] exists,up 2b081d7a-d6be-433c-9989-d5159ee895da 2023-11-16T18:24:17.064 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10 up in weight 1 up_from 69 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6824/4286013944,v1:172.21.15.40:6825/4286013944] [v2:172.21.15.40:6826/4286013944,v1:172.21.15.40:6827/4286013944] exists,up a79f904a-1214-42bf-b36a-8a7709eb0c4a 2023-11-16T18:24:17.064 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11 up in weight 1 up_from 77 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6824/2926471675,v1:172.21.15.188:6825/2926471675] [v2:172.21.15.188:6826/2926471675,v1:172.21.15.188:6827/2926471675] exists,up f54ca334-9757-4c8b-90e3-cce7daa3b1b1 2023-11-16T18:24:17.064 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.6 [8,11] 2023-11-16T18:24:17.064 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.10 [8,11] 2023-11-16T18:24:17.064 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.14 [2,4] 2023-11-16T18:24:17.064 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.1b [2,1] 2023-11-16T18:24:17.064 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.14 [4,6] 2023-11-16T18:24:17.064 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.23 [4,8] 2023-11-16T18:24:17.065 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 4.c [9,10] 2023-11-16T18:24:17.065 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 4.10 [9,8] 2023-11-16T18:24:17.065 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 4.13 [9,8] 2023-11-16T18:24:17.065 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 4.15 [5,10] 2023-11-16T18:24:17.065 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/857143886 expires 2023-11-17T18:16:37.723675+0000 2023-11-16T18:24:17.065 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/251514255 expires 2023-11-17T18:16:37.723675+0000 2023-11-16T18:24:17.065 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6801/1605535303 expires 2023-11-17T18:16:37.723675+0000 2023-11-16T18:24:17.065 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6800/1605535303 expires 2023-11-17T18:16:37.723675+0000 2023-11-16T18:24:17.065 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6801/1609662822 expires 2023-11-17T18:16:13.217812+0000 2023-11-16T18:24:17.065 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6800/1609662822 expires 2023-11-17T18:16:13.217812+0000 2023-11-16T18:24:17.065 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/708195783 expires 2023-11-17T18:16:13.217812+0000 2023-11-16T18:24:17.066 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/1158002645 expires 2023-11-17T18:16:13.217812+0000 2023-11-16T18:24:17.066 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/1986390775 expires 2023-11-17T18:16:13.217812+0000 2023-11-16T18:24:17.066 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/2955607810 expires 2023-11-17T18:16:37.723675+0000 2023-11-16T18:24:17.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.062+0000 7f8a563c6700 1 -- 172.21.15.2:0/2655924891 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8a380a11e0 msgr2=0x7f8a380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:17.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.062+0000 7f8a563c6700 1 --2- 172.21.15.2:0/2655924891 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8a380a11e0 0x7f8a380a3690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000bd70 tx=0x7f8a4000b4a0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.062+0000 7f8a563c6700 1 -- 172.21.15.2:0/2655924891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a50100d70 msgr2=0x7f8a5019d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:17.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.062+0000 7f8a563c6700 1 --2- 172.21.15.2:0/2655924891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a50100d70 0x7f8a5019d720 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800b0a0 tx=0x7f8a480094e0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.062+0000 7f8a563c6700 1 -- 172.21.15.2:0/2655924891 shutdown_connections 2023-11-16T18:24:17.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.062+0000 7f8a563c6700 1 --2- 172.21.15.2:0/2655924891 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8a380a11e0 0x7f8a380a3690 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.062+0000 7f8a563c6700 1 --2- 172.21.15.2:0/2655924891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a50100d70 0x7f8a5019d720 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.062+0000 7f8a563c6700 1 --2- 172.21.15.2:0/2655924891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a500ff1e0 0x7f8a50196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.062+0000 7f8a563c6700 1 --2- 172.21.15.2:0/2655924891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a501003e0 0x7f8a501966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.062+0000 7f8a563c6700 1 -- 172.21.15.2:0/2655924891 >> 172.21.15.2:0/2655924891 conn(0x7f8a500fa7f0 msgr2=0x7f8a500fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:17.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.063+0000 7f8a563c6700 1 -- 172.21.15.2:0/2655924891 shutdown_connections 2023-11-16T18:24:17.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.063+0000 7f8a563c6700 1 -- 172.21.15.2:0/2655924891 wait complete. 2023-11-16T18:24:17.506 INFO:teuthology.run_tasks:Running task fs.ready... 2023-11-16T18:24:17.519 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:24:17.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.647+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/4147752252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ed4101ee0 msgr2=0x7f1ed41087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:17.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.647+0000 7f1ed9bf0700 1 --2- 172.21.15.2:0/4147752252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ed4101ee0 0x7f1ed41087a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00b0a0 tx=0x7f1ecc00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.647+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/4147752252 shutdown_connections 2023-11-16T18:24:17.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.647+0000 7f1ed9bf0700 1 --2- 172.21.15.2:0/4147752252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ed4101ee0 0x7f1ed41087a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.647+0000 7f1ed9bf0700 1 --2- 172.21.15.2:0/4147752252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ed4100360 0x7f1ed4100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.647+0000 7f1ed9bf0700 1 --2- 172.21.15.2:0/4147752252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ed4101550 0x7f1ed41019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.647+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/4147752252 >> 172.21.15.2:0/4147752252 conn(0x7f1ed40fb960 msgr2=0x7f1ed40fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:17.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.647+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/4147752252 shutdown_connections 2023-11-16T18:24:17.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.647+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/4147752252 wait complete. 2023-11-16T18:24:17.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.648+0000 7f1ed9bf0700 1 Processor -- start 2023-11-16T18:24:17.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.648+0000 7f1ed9bf0700 1 -- start start 2023-11-16T18:24:17.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.648+0000 7f1ed9bf0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ed4100360 0x7f1ed4195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:17.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.648+0000 7f1ed9bf0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ed4101550 0x7f1ed4196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:17.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.648+0000 7f1ed9bf0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ed4101ee0 0x7f1ed419d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:17.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.648+0000 7f1ed9bf0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed419db50 con 0x7f1ed4101ee0 2023-11-16T18:24:17.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.648+0000 7f1ed9bf0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed419dc90 con 0x7f1ed4101550 2023-11-16T18:24:17.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.648+0000 7f1ed9bf0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed419de20 con 0x7f1ed4100360 2023-11-16T18:24:17.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.648+0000 7f1ed3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ed4101ee0 0x7f1ed419d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:17.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.648+0000 7f1ed3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ed4101ee0 0x7f1ed419d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49242/0 (socket says 172.21.15.2:49242) 2023-11-16T18:24:17.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.648+0000 7f1ed3fff700 1 -- 172.21.15.2:0/1005875759 learned_addr learned my addr 172.21.15.2:0/1005875759 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:17.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.648+0000 7f1ed2ffd700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ed4101550 0x7f1ed4196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:17.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.648+0000 7f1ed37fe700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ed4100360 0x7f1ed4195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:17.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.649+0000 7f1ed3fff700 1 -- 172.21.15.2:0/1005875759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ed4100360 msgr2=0x7f1ed4195fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:17.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.649+0000 7f1ed3fff700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ed4100360 0x7f1ed4195fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.649+0000 7f1ed3fff700 1 -- 172.21.15.2:0/1005875759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ed4101550 msgr2=0x7f1ed4196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:17.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.649+0000 7f1ed3fff700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ed4101550 0x7f1ed4196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.649+0000 7f1ed3fff700 1 -- 172.21.15.2:0/1005875759 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ecc009d00 con 0x7f1ed4101ee0 2023-11-16T18:24:17.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.649+0000 7f1ed3fff700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ed4101ee0 0x7f1ed419d580 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00b830 tx=0x7f1ecc004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:17.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.649+0000 7f1ed0ff9700 1 -- 172.21.15.2:0/1005875759 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc00e040 con 0x7f1ed4101ee0 2023-11-16T18:24:17.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.649+0000 7f1ed0ff9700 1 -- 172.21.15.2:0/1005875759 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ecc0094d0 con 0x7f1ed4101ee0 2023-11-16T18:24:17.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.649+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/1005875759 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed410bc70 con 0x7f1ed4101ee0 2023-11-16T18:24:17.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.649+0000 7f1ed0ff9700 1 -- 172.21.15.2:0/1005875759 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc01bd20 con 0x7f1ed4101ee0 2023-11-16T18:24:17.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.649+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/1005875759 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed410bae0 con 0x7f1ed4101ee0 2023-11-16T18:24:17.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.650+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/1005875759 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed41076c0 con 0x7f1ed4101ee0 2023-11-16T18:24:17.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.652+0000 7f1ed0ff9700 1 -- 172.21.15.2:0/1005875759 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f1ecc01b450 con 0x7f1ed4101ee0 2023-11-16T18:24:17.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.652+0000 7f1ed0ff9700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1ebc0a11e0 0x7f1ebc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:17.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.652+0000 7f1ed37fe700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1ebc0a11e0 0x7f1ebc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:17.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.652+0000 7f1ed0ff9700 1 -- 172.21.15.2:0/1005875759 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1ecc0cf0d0 con 0x7f1ed4101ee0 2023-11-16T18:24:17.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.653+0000 7f1ed37fe700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1ebc0a11e0 0x7f1ebc0a3690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8005fd0 tx=0x7f1ec8005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:17.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.656+0000 7f1ed0ff9700 1 -- 172.21.15.2:0/1005875759 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1ecc098440 con 0x7f1ed4101ee0 2023-11-16T18:24:17.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.856+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/1005875759 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ed4067f40 con 0x7f1ed4101ee0 2023-11-16T18:24:17.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.858+0000 7f1ed0ff9700 1 -- 172.21.15.2:0/1005875759 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24383 (secure 0 0 0) 0x7f1ecc021090 con 0x7f1ed4101ee0 2023-11-16T18:24:17.859 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:17.860 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":88,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:23:39.497670+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:24:17.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.860+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/1005875759 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1ebc0a11e0 msgr2=0x7f1ebc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:17.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.860+0000 7f1ed9bf0700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1ebc0a11e0 0x7f1ebc0a3690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8005fd0 tx=0x7f1ec8005e50 comp rx=0 tx=0).stop 2023-11-16T18:24:17.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.860+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/1005875759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ed4101ee0 msgr2=0x7f1ed419d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:17.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.860+0000 7f1ed9bf0700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ed4101ee0 0x7f1ed419d580 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00b830 tx=0x7f1ecc004d00 comp rx=0 tx=0).stop 2023-11-16T18:24:17.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.860+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/1005875759 shutdown_connections 2023-11-16T18:24:17.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.860+0000 7f1ed9bf0700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f1ebc0a11e0 0x7f1ebc0a3690 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.860+0000 7f1ed9bf0700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ed4100360 0x7f1ed4195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.860+0000 7f1ed9bf0700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ed4101ee0 0x7f1ed419d580 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.860+0000 7f1ed9bf0700 1 --2- 172.21.15.2:0/1005875759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ed4101550 0x7f1ed4196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:17.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.860+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/1005875759 >> 172.21.15.2:0/1005875759 conn(0x7f1ed40fb960 msgr2=0x7f1ed4104800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:17.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.861+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/1005875759 shutdown_connections 2023-11-16T18:24:17.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:17.861+0000 7f1ed9bf0700 1 -- 172.21.15.2:0/1005875759 wait complete. 2023-11-16T18:24:17.874 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T18:22:44.902889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T18:22:46.187275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T18:22:49.539148+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T18:24:17.875 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T18:24:18.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.004+0000 7f6813c8b700 1 -- 172.21.15.2:0/846320914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c101f70 msgr2=0x7f680c108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.004+0000 7f6813c8b700 1 --2- 172.21.15.2:0/846320914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c101f70 0x7f680c108830 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f680400b0a0 tx=0x7f680400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.004+0000 7f6813c8b700 1 -- 172.21.15.2:0/846320914 shutdown_connections 2023-11-16T18:24:18.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.004+0000 7f6813c8b700 1 --2- 172.21.15.2:0/846320914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c101f70 0x7f680c108830 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.004+0000 7f6813c8b700 1 --2- 172.21.15.2:0/846320914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680c0fe570 0x7f680c0fe980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.004+0000 7f6813c8b700 1 --2- 172.21.15.2:0/846320914 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f680c1015e0 0x7f680c101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.004+0000 7f6813c8b700 1 -- 172.21.15.2:0/846320914 >> 172.21.15.2:0/846320914 conn(0x7f680c0f9f10 msgr2=0x7f680c0fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.004+0000 7f6813c8b700 1 -- 172.21.15.2:0/846320914 shutdown_connections 2023-11-16T18:24:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.004+0000 7f6813c8b700 1 -- 172.21.15.2:0/846320914 wait complete. 2023-11-16T18:24:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.004+0000 7f6813c8b700 1 Processor -- start 2023-11-16T18:24:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.004+0000 7f6813c8b700 1 -- start start 2023-11-16T18:24:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.004+0000 7f6813c8b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c0fe570 0x7f680c196000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.004+0000 7f6813c8b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680c1015e0 0x7f680c196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6813c8b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f680c101f70 0x7f680c19d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6813c8b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c19db80 con 0x7f680c0fe570 2023-11-16T18:24:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6813c8b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c19dcc0 con 0x7f680c1015e0 2023-11-16T18:24:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6813c8b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c19de50 con 0x7f680c101f70 2023-11-16T18:24:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6811a27700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c0fe570 0x7f680c196000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6811a27700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c0fe570 0x7f680c196000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49252/0 (socket says 172.21.15.2:49252) 2023-11-16T18:24:18.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6811a27700 1 -- 172.21.15.2:0/697697042 learned_addr learned my addr 172.21.15.2:0/697697042 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:18.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6811a27700 1 -- 172.21.15.2:0/697697042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f680c101f70 msgr2=0x7f680c19d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6811a27700 1 --2- 172.21.15.2:0/697697042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f680c101f70 0x7f680c19d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6811226700 1 --2- 172.21.15.2:0/697697042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680c1015e0 0x7f680c196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:18.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6811a27700 1 -- 172.21.15.2:0/697697042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680c1015e0 msgr2=0x7f680c196540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6811a27700 1 --2- 172.21.15.2:0/697697042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680c1015e0 0x7f680c196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6811a27700 1 -- 172.21.15.2:0/697697042 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6804009d00 con 0x7f680c0fe570 2023-11-16T18:24:18.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6811a27700 1 --2- 172.21.15.2:0/697697042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c0fe570 0x7f680c196000 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f680800c970 tx=0x7f680800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:18.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6802ffd700 1 -- 172.21.15.2:0/697697042 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808004220 con 0x7f680c0fe570 2023-11-16T18:24:18.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6813c8b700 1 -- 172.21.15.2:0/697697042 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680c10bc40 con 0x7f680c0fe570 2023-11-16T18:24:18.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6802ffd700 1 -- 172.21.15.2:0/697697042 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6808007d60 con 0x7f680c0fe570 2023-11-16T18:24:18.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6802ffd700 1 -- 172.21.15.2:0/697697042 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808003d00 con 0x7f680c0fe570 2023-11-16T18:24:18.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.005+0000 7f6813c8b700 1 -- 172.21.15.2:0/697697042 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680c10bab0 con 0x7f680c0fe570 2023-11-16T18:24:18.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.006+0000 7f6813c8b700 1 -- 172.21.15.2:0/697697042 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680c19e5a0 con 0x7f680c0fe570 2023-11-16T18:24:18.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.007+0000 7f6802ffd700 1 -- 172.21.15.2:0/697697042 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f6808004380 con 0x7f680c0fe570 2023-11-16T18:24:18.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.008+0000 7f6802ffd700 1 --2- 172.21.15.2:0/697697042 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f67f80a11e0 0x7f67f80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:18.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.008+0000 7f6802ffd700 1 -- 172.21.15.2:0/697697042 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f68080ccd90 con 0x7f680c0fe570 2023-11-16T18:24:18.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.008+0000 7f6811226700 1 --2- 172.21.15.2:0/697697042 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f67f80a11e0 0x7f67f80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:18.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.008+0000 7f6811226700 1 --2- 172.21.15.2:0/697697042 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f67f80a11e0 0x7f67f80a3690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00ba60 tx=0x7f67fc005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:18.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.011+0000 7f6802ffd700 1 -- 172.21.15.2:0/697697042 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6808096070 con 0x7f680c0fe570 2023-11-16T18:24:18.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:17 smithi188 ceph-mon[100639]: pgmap v307: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T18:24:18.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:17 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2655924891' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-11-16T18:24:18.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:17 smithi002 ceph-mon[98807]: pgmap v307: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T18:24:18.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:17 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2655924891' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-11-16T18:24:18.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.211+0000 7f6813c8b700 1 -- 172.21.15.2:0/697697042 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f680c107200 con 0x7f680c0fe570 2023-11-16T18:24:18.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.212+0000 7f6802ffd700 1 -- 172.21.15.2:0/697697042 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6808099690 con 0x7f680c0fe570 2023-11-16T18:24:18.213 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:24:18.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.214+0000 7f6813c8b700 1 -- 172.21.15.2:0/697697042 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f67f80a11e0 msgr2=0x7f67f80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.214+0000 7f6813c8b700 1 --2- 172.21.15.2:0/697697042 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f67f80a11e0 0x7f67f80a3690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00ba60 tx=0x7f67fc005d50 comp rx=0 tx=0).stop 2023-11-16T18:24:18.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.214+0000 7f6813c8b700 1 -- 172.21.15.2:0/697697042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c0fe570 msgr2=0x7f680c196000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.214+0000 7f6813c8b700 1 --2- 172.21.15.2:0/697697042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c0fe570 0x7f680c196000 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f680800c970 tx=0x7f680800cd30 comp rx=0 tx=0).stop 2023-11-16T18:24:18.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.214+0000 7f6813c8b700 1 -- 172.21.15.2:0/697697042 shutdown_connections 2023-11-16T18:24:18.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.214+0000 7f6813c8b700 1 --2- 172.21.15.2:0/697697042 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f67f80a11e0 0x7f67f80a3690 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.214+0000 7f6813c8b700 1 --2- 172.21.15.2:0/697697042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f680c101f70 0x7f680c19d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.214+0000 7f6813c8b700 1 --2- 172.21.15.2:0/697697042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f680c0fe570 0x7f680c196000 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.214+0000 7f6813c8b700 1 --2- 172.21.15.2:0/697697042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680c1015e0 0x7f680c196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.214+0000 7f6813c8b700 1 -- 172.21.15.2:0/697697042 >> 172.21.15.2:0/697697042 conn(0x7f680c0f9f10 msgr2=0x7f680c0fc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:18.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.214+0000 7f6813c8b700 1 -- 172.21.15.2:0/697697042 shutdown_connections 2023-11-16T18:24:18.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.214+0000 7f6813c8b700 1 -- 172.21.15.2:0/697697042 wait complete. 2023-11-16T18:24:18.228 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T18:24:18.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:17 smithi040 ceph-mon[100251]: pgmap v307: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T18:24:18.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:17 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2655924891' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-11-16T18:24:18.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.355+0000 7fd8e86de700 1 -- 172.21.15.2:0/4121316824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8e00ff1a0 msgr2=0x7fd8e00ff5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.355+0000 7fd8e86de700 1 --2- 172.21.15.2:0/4121316824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8e00ff1a0 0x7fd8e00ff5b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009990 tx=0x7fd8d0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.355+0000 7fd8e86de700 1 -- 172.21.15.2:0/4121316824 shutdown_connections 2023-11-16T18:24:18.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.355+0000 7fd8e86de700 1 --2- 172.21.15.2:0/4121316824 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8e0100d20 0x7fd8e01075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.355+0000 7fd8e86de700 1 --2- 172.21.15.2:0/4121316824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8e00ff1a0 0x7fd8e00ff5b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.355+0000 7fd8e86de700 1 --2- 172.21.15.2:0/4121316824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8e0100390 0x7fd8e01007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.355+0000 7fd8e86de700 1 -- 172.21.15.2:0/4121316824 >> 172.21.15.2:0/4121316824 conn(0x7fd8e00fa7a0 msgr2=0x7fd8e00fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:18.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.355+0000 7fd8e86de700 1 -- 172.21.15.2:0/4121316824 shutdown_connections 2023-11-16T18:24:18.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.355+0000 7fd8e86de700 1 -- 172.21.15.2:0/4121316824 wait complete. 2023-11-16T18:24:18.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.356+0000 7fd8e86de700 1 Processor -- start 2023-11-16T18:24:18.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.356+0000 7fd8e86de700 1 -- start start 2023-11-16T18:24:18.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.356+0000 7fd8e86de700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8e00ff1a0 0x7fd8e0069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:18.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.356+0000 7fd8e86de700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8e0100390 0x7fd8e019ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:18.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.356+0000 7fd8e86de700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8e0100d20 0x7fd8e019d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:18.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.356+0000 7fd8e86de700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e019db60 con 0x7fd8e0100390 2023-11-16T18:24:18.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.356+0000 7fd8e86de700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e019dcd0 con 0x7fd8e00ff1a0 2023-11-16T18:24:18.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.356+0000 7fd8e86de700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e019de90 con 0x7fd8e0100d20 2023-11-16T18:24:18.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.356+0000 7fd8e5c79700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8e0100390 0x7fd8e019ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:18.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.356+0000 7fd8e5c79700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8e0100390 0x7fd8e019ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49266/0 (socket says 172.21.15.2:49266) 2023-11-16T18:24:18.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.356+0000 7fd8e5c79700 1 -- 172.21.15.2:0/3932615242 learned_addr learned my addr 172.21.15.2:0/3932615242 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:18.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.356+0000 7fd8e647a700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8e00ff1a0 0x7fd8e0069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:18.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.357+0000 7fd8e6c7b700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8e0100d20 0x7fd8e019d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:18.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.357+0000 7fd8e5c79700 1 -- 172.21.15.2:0/3932615242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8e0100d20 msgr2=0x7fd8e019d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.357+0000 7fd8e5c79700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8e0100d20 0x7fd8e019d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.357+0000 7fd8e5c79700 1 -- 172.21.15.2:0/3932615242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8e00ff1a0 msgr2=0x7fd8e0069bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.357+0000 7fd8e5c79700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8e00ff1a0 0x7fd8e0069bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.357+0000 7fd8e5c79700 1 -- 172.21.15.2:0/3932615242 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d00095f0 con 0x7fd8e0100390 2023-11-16T18:24:18.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.357+0000 7fd8e6c7b700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8e0100d20 0x7fd8e019d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:18.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.357+0000 7fd8e5c79700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8e0100390 0x7fd8e019ad40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00d950 tx=0x7fd8dc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:18.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.357+0000 7fd8d77fe700 1 -- 172.21.15.2:0/3932615242 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc009940 con 0x7fd8e0100390 2023-11-16T18:24:18.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.357+0000 7fd8d77fe700 1 -- 172.21.15.2:0/3932615242 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8dc010460 con 0x7fd8e0100390 2023-11-16T18:24:18.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.357+0000 7fd8e86de700 1 -- 172.21.15.2:0/3932615242 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e010ab80 con 0x7fd8e0100390 2023-11-16T18:24:18.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.357+0000 7fd8d77fe700 1 -- 172.21.15.2:0/3932615242 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc00f620 con 0x7fd8e0100390 2023-11-16T18:24:18.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.357+0000 7fd8e86de700 1 -- 172.21.15.2:0/3932615242 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e010a9f0 con 0x7fd8e0100390 2023-11-16T18:24:18.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.358+0000 7fd8e86de700 1 -- 172.21.15.2:0/3932615242 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e0106150 con 0x7fd8e0100390 2023-11-16T18:24:18.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.359+0000 7fd8d77fe700 1 -- 172.21.15.2:0/3932615242 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fd8dc0108d0 con 0x7fd8e0100390 2023-11-16T18:24:18.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.360+0000 7fd8d77fe700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd8cc0a1260 0x7fd8cc0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:18.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.360+0000 7fd8e647a700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd8cc0a1260 0x7fd8cc0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:18.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.360+0000 7fd8d77fe700 1 -- 172.21.15.2:0/3932615242 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd8dc0ccc10 con 0x7fd8e0100390 2023-11-16T18:24:18.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.360+0000 7fd8e647a700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd8cc0a1260 0x7fd8cc0a3710 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd8d000b5c0 tx=0x7fd8d0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:18.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.363+0000 7fd8d77fe700 1 -- 172.21.15.2:0/3932615242 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd8dc095e90 con 0x7fd8e0100390 2023-11-16T18:24:18.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.563+0000 7fd8e86de700 1 -- 172.21.15.2:0/3932615242 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd8e0061700 con 0x7fd8e0100390 2023-11-16T18:24:18.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.564+0000 7fd8d77fe700 1 -- 172.21.15.2:0/3932615242 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd8dc020020 con 0x7fd8e0100390 2023-11-16T18:24:18.565 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:24:18.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.566+0000 7fd8e86de700 1 -- 172.21.15.2:0/3932615242 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd8cc0a1260 msgr2=0x7fd8cc0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.566+0000 7fd8e86de700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd8cc0a1260 0x7fd8cc0a3710 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd8d000b5c0 tx=0x7fd8d0005fb0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.566+0000 7fd8e86de700 1 -- 172.21.15.2:0/3932615242 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8e0100390 msgr2=0x7fd8e019ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.566+0000 7fd8e86de700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8e0100390 0x7fd8e019ad40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00d950 tx=0x7fd8dc00dc60 comp rx=0 tx=0).stop 2023-11-16T18:24:18.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.566+0000 7fd8e86de700 1 -- 172.21.15.2:0/3932615242 shutdown_connections 2023-11-16T18:24:18.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.566+0000 7fd8e86de700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd8cc0a1260 0x7fd8cc0a3710 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.566+0000 7fd8e86de700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8e0100d20 0x7fd8e019d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.566+0000 7fd8e86de700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8e00ff1a0 0x7fd8e0069bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.566+0000 7fd8e86de700 1 --2- 172.21.15.2:0/3932615242 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8e0100390 0x7fd8e019ad40 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.566+0000 7fd8e86de700 1 -- 172.21.15.2:0/3932615242 >> 172.21.15.2:0/3932615242 conn(0x7fd8e00fa7a0 msgr2=0x7fd8e00fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:18.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.566+0000 7fd8e86de700 1 -- 172.21.15.2:0/3932615242 shutdown_connections 2023-11-16T18:24:18.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.566+0000 7fd8e86de700 1 -- 172.21.15.2:0/3932615242 wait complete. 2023-11-16T18:24:18.580 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T18:24:18.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.707+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/441900323 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6e4100400 msgr2=0x7ff6e4100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.707+0000 7ff6ec2bd700 1 --2- 172.21.15.2:0/441900323 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6e4100400 0x7ff6e4100870 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0009990 tx=0x7ff6e0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.707+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/441900323 shutdown_connections 2023-11-16T18:24:18.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.707+0000 7ff6ec2bd700 1 --2- 172.21.15.2:0/441900323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6e4100db0 0x7ff6e4107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.707+0000 7ff6ec2bd700 1 --2- 172.21.15.2:0/441900323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6e40ff170 0x7ff6e40ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.707+0000 7ff6ec2bd700 1 --2- 172.21.15.2:0/441900323 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6e4100400 0x7ff6e4100870 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.707+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/441900323 >> 172.21.15.2:0/441900323 conn(0x7ff6e40fa750 msgr2=0x7ff6e40fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:18.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.707+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/441900323 shutdown_connections 2023-11-16T18:24:18.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.707+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/441900323 wait complete. 2023-11-16T18:24:18.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.708+0000 7ff6ec2bd700 1 Processor -- start 2023-11-16T18:24:18.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.708+0000 7ff6ec2bd700 1 -- start start 2023-11-16T18:24:18.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.708+0000 7ff6ec2bd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6e40ff170 0x7ff6e41031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:18.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.708+0000 7ff6ec2bd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6e4100400 0x7ff6e41017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:18.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.708+0000 7ff6ec2bd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6e4100db0 0x7ff6e4101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:18.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.708+0000 7ff6ec2bd700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e41024c0 con 0x7ff6e4100400 2023-11-16T18:24:18.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.708+0000 7ff6ec2bd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e4102630 con 0x7ff6e40ff170 2023-11-16T18:24:18.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.708+0000 7ff6ec2bd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e419ffe0 con 0x7ff6e4100db0 2023-11-16T18:24:18.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.708+0000 7ff6e9858700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6e4100400 0x7ff6e41017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:18.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.708+0000 7ff6e9858700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6e4100400 0x7ff6e41017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49280/0 (socket says 172.21.15.2:49280) 2023-11-16T18:24:18.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.708+0000 7ff6e9858700 1 -- 172.21.15.2:0/296473772 learned_addr learned my addr 172.21.15.2:0/296473772 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:18.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.708+0000 7ff6ea059700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6e40ff170 0x7ff6e41031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:18.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.709+0000 7ff6ea85a700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6e4100db0 0x7ff6e4101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:18.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.709+0000 7ff6e9858700 1 -- 172.21.15.2:0/296473772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6e4100db0 msgr2=0x7ff6e4101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.709+0000 7ff6e9858700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6e4100db0 0x7ff6e4101d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.709+0000 7ff6e9858700 1 -- 172.21.15.2:0/296473772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6e40ff170 msgr2=0x7ff6e41031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.709+0000 7ff6e9858700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6e40ff170 0x7ff6e41031a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.709+0000 7ff6e9858700 1 -- 172.21.15.2:0/296473772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e00095f0 con 0x7ff6e4100400 2023-11-16T18:24:18.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.709+0000 7ff6e9858700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6e4100400 0x7ff6e41017e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00059d0 tx=0x7ff6e000f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:18.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.709+0000 7ff6db7fe700 1 -- 172.21.15.2:0/296473772 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e001d070 con 0x7ff6e4100400 2023-11-16T18:24:18.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.709+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/296473772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e410ab70 con 0x7ff6e4100400 2023-11-16T18:24:18.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.709+0000 7ff6db7fe700 1 -- 172.21.15.2:0/296473772 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6e000fd60 con 0x7ff6e4100400 2023-11-16T18:24:18.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.709+0000 7ff6db7fe700 1 -- 172.21.15.2:0/296473772 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0017650 con 0x7ff6e4100400 2023-11-16T18:24:18.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.709+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/296473772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e410a9e0 con 0x7ff6e4100400 2023-11-16T18:24:18.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.710+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/296473772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e41a05e0 con 0x7ff6e4100400 2023-11-16T18:24:18.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.711+0000 7ff6db7fe700 1 -- 172.21.15.2:0/296473772 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ff6e0004830 con 0x7ff6e4100400 2023-11-16T18:24:18.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.712+0000 7ff6db7fe700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff6d00a11e0 0x7ff6d00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:18.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.712+0000 7ff6ea059700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff6d00a11e0 0x7ff6d00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:18.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.712+0000 7ff6db7fe700 1 -- 172.21.15.2:0/296473772 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff6e00ce5e0 con 0x7ff6e4100400 2023-11-16T18:24:18.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.712+0000 7ff6ea059700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff6d00a11e0 0x7ff6d00a3690 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009bf0 tx=0x7ff6d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:18.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.715+0000 7ff6db7fe700 1 -- 172.21.15.2:0/296473772 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff6e00978f0 con 0x7ff6e4100400 2023-11-16T18:24:18.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:18 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1005875759' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:18.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:18 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/697697042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:18.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:18 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3932615242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:18.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.918+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/296473772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff6e41a0cc0 con 0x7ff6e4100400 2023-11-16T18:24:18.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.918+0000 7ff6db7fe700 1 -- 172.21.15.2:0/296473772 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff6e009af10 con 0x7ff6e4100400 2023-11-16T18:24:18.920 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T18:24:18.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.920+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/296473772 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff6d00a11e0 msgr2=0x7ff6d00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.920+0000 7ff6ec2bd700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff6d00a11e0 0x7ff6d00a3690 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009bf0 tx=0x7ff6d4009260 comp rx=0 tx=0).stop 2023-11-16T18:24:18.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.921+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/296473772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6e4100400 msgr2=0x7ff6e41017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:18.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.921+0000 7ff6ec2bd700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6e4100400 0x7ff6e41017e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00059d0 tx=0x7ff6e000f510 comp rx=0 tx=0).stop 2023-11-16T18:24:18.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.921+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/296473772 shutdown_connections 2023-11-16T18:24:18.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.921+0000 7ff6ec2bd700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff6d00a11e0 0x7ff6d00a3690 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.921+0000 7ff6ec2bd700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6e4100db0 0x7ff6e4101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.921+0000 7ff6ec2bd700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6e40ff170 0x7ff6e41031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.921+0000 7ff6ec2bd700 1 --2- 172.21.15.2:0/296473772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6e4100400 0x7ff6e41017e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:18.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.921+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/296473772 >> 172.21.15.2:0/296473772 conn(0x7ff6e40fa750 msgr2=0x7ff6e4067dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:18.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.921+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/296473772 shutdown_connections 2023-11-16T18:24:18.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:18.921+0000 7ff6ec2bd700 1 -- 172.21.15.2:0/296473772 wait complete. 2023-11-16T18:24:18.935 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T18:24:19.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:18 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1005875759' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:19.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:18 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/697697042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:19.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:18 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3932615242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:19.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.063+0000 7f60762f9700 1 -- 172.21.15.2:0/2831535005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6070101e10 msgr2=0x7f60701086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:19.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.063+0000 7f60762f9700 1 --2- 172.21.15.2:0/2831535005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6070101e10 0x7f60701086d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f606800b0a0 tx=0x7f606800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.063+0000 7f60762f9700 1 -- 172.21.15.2:0/2831535005 shutdown_connections 2023-11-16T18:24:19.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.063+0000 7f60762f9700 1 --2- 172.21.15.2:0/2831535005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6070101e10 0x7f60701086d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.063+0000 7f60762f9700 1 --2- 172.21.15.2:0/2831535005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6070100290 0x7f60701006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.063+0000 7f60762f9700 1 --2- 172.21.15.2:0/2831535005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6070101480 0x7f60701018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.063+0000 7f60762f9700 1 -- 172.21.15.2:0/2831535005 >> 172.21.15.2:0/2831535005 conn(0x7f60700fb8b0 msgr2=0x7f60700fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:19.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.063+0000 7f60762f9700 1 -- 172.21.15.2:0/2831535005 shutdown_connections 2023-11-16T18:24:19.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.063+0000 7f60762f9700 1 -- 172.21.15.2:0/2831535005 wait complete. 2023-11-16T18:24:19.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.064+0000 7f60762f9700 1 Processor -- start 2023-11-16T18:24:19.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.064+0000 7f60762f9700 1 -- start start 2023-11-16T18:24:19.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.064+0000 7f60762f9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6070100290 0x7f6070196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:19.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.064+0000 7f60762f9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6070101480 0x7f60701965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:19.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.064+0000 7f60762f9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6070101e10 0x7f607019d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:19.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.064+0000 7f60762f9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607019dbf0 con 0x7f6070100290 2023-11-16T18:24:19.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.064+0000 7f60762f9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607019dd30 con 0x7f6070101e10 2023-11-16T18:24:19.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.064+0000 7f60762f9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607019dec0 con 0x7f6070101480 2023-11-16T18:24:19.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6070100290 0x7f6070196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:19.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6070100290 0x7f6070196070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49292/0 (socket says 172.21.15.2:49292) 2023-11-16T18:24:19.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606ffff700 1 -- 172.21.15.2:0/3730434070 learned_addr learned my addr 172.21.15.2:0/3730434070 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:19.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f6074896700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6070101e10 0x7f607019d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:19.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606f7fe700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6070101480 0x7f60701965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:19.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606ffff700 1 -- 172.21.15.2:0/3730434070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6070101480 msgr2=0x7f60701965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:19.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606ffff700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6070101480 0x7f60701965b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.076 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606ffff700 1 -- 172.21.15.2:0/3730434070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6070101e10 msgr2=0x7f607019d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:19.076 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606ffff700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6070101e10 0x7f607019d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.076 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606ffff700 1 -- 172.21.15.2:0/3730434070 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6068009d00 con 0x7f6070100290 2023-11-16T18:24:19.076 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606f7fe700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6070101480 0x7f60701965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:19.077 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606ffff700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6070100290 0x7f6070196070 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f606000c970 tx=0x7f606000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:19.077 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606d7fa700 1 -- 172.21.15.2:0/3730434070 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6060004220 con 0x7f6070100290 2023-11-16T18:24:19.077 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606d7fa700 1 -- 172.21.15.2:0/3730434070 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6060007d60 con 0x7f6070100290 2023-11-16T18:24:19.077 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f60762f9700 1 -- 172.21.15.2:0/3730434070 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607010bc20 con 0x7f6070100290 2023-11-16T18:24:19.077 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f606d7fa700 1 -- 172.21.15.2:0/3730434070 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6060003c90 con 0x7f6070100290 2023-11-16T18:24:19.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.065+0000 7f60762f9700 1 -- 172.21.15.2:0/3730434070 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607010ba90 con 0x7f6070100290 2023-11-16T18:24:19.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.066+0000 7f60762f9700 1 -- 172.21.15.2:0/3730434070 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6070107820 con 0x7f6070100290 2023-11-16T18:24:19.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.067+0000 7f606d7fa700 1 -- 172.21.15.2:0/3730434070 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f6060004380 con 0x7f6070100290 2023-11-16T18:24:19.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.068+0000 7f606d7fa700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f60580a1260 0x7f60580a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:19.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.068+0000 7f606f7fe700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f60580a1260 0x7f60580a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:19.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.068+0000 7f606d7fa700 1 -- 172.21.15.2:0/3730434070 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f60600cccf0 con 0x7f6070100290 2023-11-16T18:24:19.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.068+0000 7f606f7fe700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f60580a1260 0x7f60580a3710 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6064005fd0 tx=0x7f6064005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:19.080 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.071+0000 7f606d7fa700 1 -- 172.21.15.2:0/3730434070 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6060095ff0 con 0x7f6070100290 2023-11-16T18:24:19.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:18 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1005875759' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:19.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:18 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/697697042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:19.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:18 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3932615242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:19.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.274+0000 7f60762f9700 1 -- 172.21.15.2:0/3730434070 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f60700618b0 con 0x7f6070100290 2023-11-16T18:24:19.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.274+0000 7f606d7fa700 1 -- 172.21.15.2:0/3730434070 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7f606001d020 con 0x7f6070100290 2023-11-16T18:24:19.276 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:24:19.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.276+0000 7f60762f9700 1 -- 172.21.15.2:0/3730434070 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f60580a1260 msgr2=0x7f60580a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:19.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.276+0000 7f60762f9700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f60580a1260 0x7f60580a3710 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6064005fd0 tx=0x7f6064005ee0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.276+0000 7f60762f9700 1 -- 172.21.15.2:0/3730434070 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6070100290 msgr2=0x7f6070196070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:19.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.276+0000 7f60762f9700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6070100290 0x7f6070196070 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f606000c970 tx=0x7f606000cc80 comp rx=0 tx=0).stop 2023-11-16T18:24:19.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.276+0000 7f60762f9700 1 -- 172.21.15.2:0/3730434070 shutdown_connections 2023-11-16T18:24:19.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.276+0000 7f60762f9700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f60580a1260 0x7f60580a3710 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.276+0000 7f60762f9700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6070101e10 0x7f607019d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.276+0000 7f60762f9700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6070100290 0x7f6070196070 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.276+0000 7f60762f9700 1 --2- 172.21.15.2:0/3730434070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6070101480 0x7f60701965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.276+0000 7f60762f9700 1 -- 172.21.15.2:0/3730434070 >> 172.21.15.2:0/3730434070 conn(0x7f60700fb8b0 msgr2=0x7f6070104960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:19.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.277+0000 7f60762f9700 1 -- 172.21.15.2:0/3730434070 shutdown_connections 2023-11-16T18:24:19.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.277+0000 7f60762f9700 1 -- 172.21.15.2:0/3730434070 wait complete. 2023-11-16T18:24:19.291 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:24:19.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.415+0000 7fadfdd3d700 1 -- 172.21.15.2:0/819262400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fadf8100d40 msgr2=0x7fadf8107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.415+0000 7fadfdd3d700 1 --2- 172.21.15.2:0/819262400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fadf8100d40 0x7fadf8107600 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fadf000b0a0 tx=0x7fadf000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.415+0000 7fadfdd3d700 1 -- 172.21.15.2:0/819262400 shutdown_connections 2023-11-16T18:24:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.415+0000 7fadfdd3d700 1 --2- 172.21.15.2:0/819262400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadf80ff1c0 0x7fadf80ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.415+0000 7fadfdd3d700 1 --2- 172.21.15.2:0/819262400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fadf8100d40 0x7fadf8107600 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.415+0000 7fadfdd3d700 1 --2- 172.21.15.2:0/819262400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadf81003b0 0x7fadf8100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.415+0000 7fadfdd3d700 1 -- 172.21.15.2:0/819262400 >> 172.21.15.2:0/819262400 conn(0x7fadf80fa7c0 msgr2=0x7fadf80fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.415+0000 7fadfdd3d700 1 -- 172.21.15.2:0/819262400 shutdown_connections 2023-11-16T18:24:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.415+0000 7fadfdd3d700 1 -- 172.21.15.2:0/819262400 wait complete. 2023-11-16T18:24:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.416+0000 7fadfdd3d700 1 Processor -- start 2023-11-16T18:24:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.416+0000 7fadfdd3d700 1 -- start start 2023-11-16T18:24:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.416+0000 7fadfdd3d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fadf80ff1c0 0x7fadf8195060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.416+0000 7fadfdd3d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadf81003b0 0x7fadf81955a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.416+0000 7fadfdd3d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadf8100d40 0x7fadf819c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.416+0000 7fadfdd3d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf819cbe0 con 0x7fadf80ff1c0 2023-11-16T18:24:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.416+0000 7fadfdd3d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf819cd20 con 0x7fadf81003b0 2023-11-16T18:24:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.416+0000 7fadfdd3d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf819ceb0 con 0x7fadf8100d40 2023-11-16T18:24:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fadf80ff1c0 0x7fadf8195060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fadf80ff1c0 0x7fadf8195060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49306/0 (socket says 172.21.15.2:49306) 2023-11-16T18:24:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf77fe700 1 -- 172.21.15.2:0/4158064315 learned_addr learned my addr 172.21.15.2:0/4158064315 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf6ffd700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadf81003b0 0x7fadf81955a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf77fe700 1 -- 172.21.15.2:0/4158064315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadf8100d40 msgr2=0x7fadf819c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf77fe700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadf8100d40 0x7fadf819c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf77fe700 1 -- 172.21.15.2:0/4158064315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadf81003b0 msgr2=0x7fadf81955a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf77fe700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadf81003b0 0x7fadf81955a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf77fe700 1 -- 172.21.15.2:0/4158064315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf0009d00 con 0x7fadf80ff1c0 2023-11-16T18:24:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf6ffd700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadf81003b0 0x7fadf81955a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf77fe700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fadf80ff1c0 0x7fadf8195060 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fade800d980 tx=0x7fade800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf4ff9700 1 -- 172.21.15.2:0/4158064315 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade800de60 con 0x7fadf80ff1c0 2023-11-16T18:24:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadfdd3d700 1 -- 172.21.15.2:0/4158064315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf810ab40 con 0x7fadf80ff1c0 2023-11-16T18:24:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf4ff9700 1 -- 172.21.15.2:0/4158064315 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fade80044e0 con 0x7fadf80ff1c0 2023-11-16T18:24:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadf4ff9700 1 -- 172.21.15.2:0/4158064315 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8003be0 con 0x7fadf80ff1c0 2023-11-16T18:24:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.417+0000 7fadfdd3d700 1 -- 172.21.15.2:0/4158064315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf810a9b0 con 0x7fadf80ff1c0 2023-11-16T18:24:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.418+0000 7fadfdd3d700 1 -- 172.21.15.2:0/4158064315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf8106760 con 0x7fadf80ff1c0 2023-11-16T18:24:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.419+0000 7fadf4ff9700 1 -- 172.21.15.2:0/4158064315 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fade8009bb0 con 0x7fadf80ff1c0 2023-11-16T18:24:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.420+0000 7fadf4ff9700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fade00a11e0 0x7fade00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.420+0000 7fadf6ffd700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fade00a11e0 0x7fade00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:19.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.420+0000 7fadf4ff9700 1 -- 172.21.15.2:0/4158064315 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fade80ccb40 con 0x7fadf80ff1c0 2023-11-16T18:24:19.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.420+0000 7fadf6ffd700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fade00a11e0 0x7fade00a3690 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fadec005950 tx=0x7fadec005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:19.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.424+0000 7fadf4ff9700 1 -- 172.21.15.2:0/4158064315 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fade8095e20 con 0x7fadf80ff1c0 2023-11-16T18:24:19.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.669+0000 7fadfdd3d700 1 -- 172.21.15.2:0/4158064315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadf8061900 con 0x7fadf80ff1c0 2023-11-16T18:24:19.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.670+0000 7fadf4ff9700 1 -- 172.21.15.2:0/4158064315 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10774 (secure 0 0 0) 0x7fade8009940 con 0x7fadf80ff1c0 2023-11-16T18:24:19.671 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:19.671 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:23:07.909835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:24:19.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.672+0000 7fadfdd3d700 1 -- 172.21.15.2:0/4158064315 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fade00a11e0 msgr2=0x7fade00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:19.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.672+0000 7fadfdd3d700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fade00a11e0 0x7fade00a3690 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fadec005950 tx=0x7fadec005f50 comp rx=0 tx=0).stop 2023-11-16T18:24:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.672+0000 7fadfdd3d700 1 -- 172.21.15.2:0/4158064315 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fadf80ff1c0 msgr2=0x7fadf8195060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.672+0000 7fadfdd3d700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fadf80ff1c0 0x7fadf8195060 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fade800d980 tx=0x7fade800dd40 comp rx=0 tx=0).stop 2023-11-16T18:24:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.672+0000 7fadfdd3d700 1 -- 172.21.15.2:0/4158064315 shutdown_connections 2023-11-16T18:24:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.672+0000 7fadfdd3d700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fade00a11e0 0x7fade00a3690 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.672+0000 7fadfdd3d700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadf8100d40 0x7fadf819c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.672+0000 7fadfdd3d700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fadf80ff1c0 0x7fadf8195060 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.672+0000 7fadfdd3d700 1 --2- 172.21.15.2:0/4158064315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadf81003b0 0x7fadf81955a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.672+0000 7fadfdd3d700 1 -- 172.21.15.2:0/4158064315 >> 172.21.15.2:0/4158064315 conn(0x7fadf80fa7c0 msgr2=0x7fadf81038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.672+0000 7fadfdd3d700 1 -- 172.21.15.2:0/4158064315 shutdown_connections 2023-11-16T18:24:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.672+0000 7fadfdd3d700 1 -- 172.21.15.2:0/4158064315 wait complete. 2023-11-16T18:24:19.675 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T18:24:19.686 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:24:19.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.813+0000 7f8b0d872700 1 -- 172.21.15.2:0/4183468334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b081016d0 msgr2=0x7f8b08101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:19.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.813+0000 7f8b0d872700 1 --2- 172.21.15.2:0/4183468334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b081016d0 0x7f8b08101b20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8af0009990 tx=0x7f8af0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.813+0000 7f8b0d872700 1 -- 172.21.15.2:0/4183468334 shutdown_connections 2023-11-16T18:24:19.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.813+0000 7f8b0d872700 1 --2- 172.21.15.2:0/4183468334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b081004e0 0x7f8b081008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.813+0000 7f8b0d872700 1 --2- 172.21.15.2:0/4183468334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b08102060 0x7f8b08108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.813+0000 7f8b0d872700 1 --2- 172.21.15.2:0/4183468334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b081016d0 0x7f8b08101b20 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.813+0000 7f8b0d872700 1 -- 172.21.15.2:0/4183468334 >> 172.21.15.2:0/4183468334 conn(0x7f8b080fbac0 msgr2=0x7f8b080fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:19.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.813+0000 7f8b0d872700 1 -- 172.21.15.2:0/4183468334 shutdown_connections 2023-11-16T18:24:19.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.813+0000 7f8b0d872700 1 -- 172.21.15.2:0/4183468334 wait complete. 2023-11-16T18:24:19.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.813+0000 7f8b0d872700 1 Processor -- start 2023-11-16T18:24:19.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b0d872700 1 -- start start 2023-11-16T18:24:19.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b0d872700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b081004e0 0x7f8b08196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:19.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b0d872700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b081016d0 0x7f8b081967d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:19.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b0d872700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b08102060 0x7f8b0819d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:19.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b0d872700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0819de10 con 0x7f8b08102060 2023-11-16T18:24:19.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b0d872700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0819df50 con 0x7f8b081016d0 2023-11-16T18:24:19.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b0d872700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0819e0e0 con 0x7f8b081004e0 2023-11-16T18:24:19.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b077fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b08102060 0x7f8b0819d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:19.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b077fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b08102060 0x7f8b0819d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49324/0 (socket says 172.21.15.2:49324) 2023-11-16T18:24:19.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b077fe700 1 -- 172.21.15.2:0/2347479530 learned_addr learned my addr 172.21.15.2:0/2347479530 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:19.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b067fc700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b081016d0 0x7f8b081967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:19.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b06ffd700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b081004e0 0x7f8b08196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:19.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b077fe700 1 -- 172.21.15.2:0/2347479530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b081004e0 msgr2=0x7f8b08196290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:19.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b077fe700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b081004e0 0x7f8b08196290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b077fe700 1 -- 172.21.15.2:0/2347479530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b081016d0 msgr2=0x7f8b081967d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:19.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b077fe700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b081016d0 0x7f8b081967d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:19.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b077fe700 1 -- 172.21.15.2:0/2347479530 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af00095f0 con 0x7f8b08102060 2023-11-16T18:24:19.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b077fe700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b08102060 0x7f8b0819d840 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8b00007ee0 tx=0x7f8b0000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:19.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.814+0000 7f8b0c870700 1 -- 172.21.15.2:0/2347479530 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b00017070 con 0x7f8b08102060 2023-11-16T18:24:19.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.815+0000 7f8b0c870700 1 -- 172.21.15.2:0/2347479530 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b0000f040 con 0x7f8b08102060 2023-11-16T18:24:19.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.815+0000 7f8b0c870700 1 -- 172.21.15.2:0/2347479530 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b00005740 con 0x7f8b08102060 2023-11-16T18:24:19.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.815+0000 7f8b0d872700 1 -- 172.21.15.2:0/2347479530 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0810be40 con 0x7f8b08102060 2023-11-16T18:24:19.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.815+0000 7f8b0d872700 1 -- 172.21.15.2:0/2347479530 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0810bcb0 con 0x7f8b08102060 2023-11-16T18:24:19.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.815+0000 7f8b0d872700 1 -- 172.21.15.2:0/2347479530 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b08107a80 con 0x7f8b08102060 2023-11-16T18:24:19.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.817+0000 7f8b0c870700 1 -- 172.21.15.2:0/2347479530 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f8b000058a0 con 0x7f8b08102060 2023-11-16T18:24:19.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.818+0000 7f8b0c870700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8af40a1260 0x7f8af40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:19.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.818+0000 7f8b06ffd700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8af40a1260 0x7f8af40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:19.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.818+0000 7f8b0c870700 1 -- 172.21.15.2:0/2347479530 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8b000ccc20 con 0x7f8b08102060 2023-11-16T18:24:19.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.818+0000 7f8b06ffd700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8af40a1260 0x7f8af40a3710 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8af8005950 tx=0x7f8af80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:19.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:19.821+0000 7f8b0c870700 1 -- 172.21.15.2:0/2347479530 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8b000cce60 con 0x7f8b08102060 2023-11-16T18:24:20.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.019+0000 7f8b0d872700 1 -- 172.21.15.2:0/2347479530 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b0806a2f0 con 0x7f8b08102060 2023-11-16T18:24:20.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.021+0000 7f8b0c870700 1 -- 172.21.15.2:0/2347479530 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24383 (secure 0 0 0) 0x7f8b00099930 con 0x7f8b08102060 2023-11-16T18:24:20.023 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:20.023 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":88,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:23:39.497670+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:24:20.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.023+0000 7f8b0d872700 1 -- 172.21.15.2:0/2347479530 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8af40a1260 msgr2=0x7f8af40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.023+0000 7f8b0d872700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8af40a1260 0x7f8af40a3710 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8af8005950 tx=0x7f8af80058e0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.023+0000 7f8b0d872700 1 -- 172.21.15.2:0/2347479530 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b08102060 msgr2=0x7f8b0819d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.023+0000 7f8b0d872700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b08102060 0x7f8b0819d840 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8b00007ee0 tx=0x7f8b0000d3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.023+0000 7f8b0d872700 1 -- 172.21.15.2:0/2347479530 shutdown_connections 2023-11-16T18:24:20.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.023+0000 7f8b0d872700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f8af40a1260 0x7f8af40a3710 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.023+0000 7f8b0d872700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b081004e0 0x7f8b08196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.023+0000 7f8b0d872700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b08102060 0x7f8b0819d840 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.023+0000 7f8b0d872700 1 --2- 172.21.15.2:0/2347479530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b081016d0 0x7f8b081967d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.023+0000 7f8b0d872700 1 -- 172.21.15.2:0/2347479530 >> 172.21.15.2:0/2347479530 conn(0x7f8b080fbac0 msgr2=0x7f8b08104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:20.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.023+0000 7f8b0d872700 1 -- 172.21.15.2:0/2347479530 shutdown_connections 2023-11-16T18:24:20.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.023+0000 7f8b0d872700 1 -- 172.21.15.2:0/2347479530 wait complete. 2023-11-16T18:24:20.036 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:19 smithi002 ceph-mon[98807]: pgmap v308: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:24:20.036 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:19 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/296473772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:20.036 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:19 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3730434070' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:20.036 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:19 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/4158064315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:20.037 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T18:22:44.902889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T18:22:46.187275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T18:22:49.539148+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T18:24:20.038 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T18:24:20.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:19 smithi188 ceph-mon[100639]: pgmap v308: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:24:20.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:19 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/296473772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:20.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:19 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3730434070' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:20.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:19 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/4158064315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:20.167 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.166+0000 7f62cde9b700 1 -- 172.21.15.2:0/2041041119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62c80ff460 msgr2=0x7f62c80ff8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.167 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.166+0000 7f62cde9b700 1 --2- 172.21.15.2:0/2041041119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62c80ff460 0x7f62c80ff8d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f62b8009990 tx=0x7f62b8009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.167 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.166+0000 7f62cde9b700 1 -- 172.21.15.2:0/2041041119 shutdown_connections 2023-11-16T18:24:20.168 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.166+0000 7f62cde9b700 1 --2- 172.21.15.2:0/2041041119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62c80ffe10 0x7f62c81076c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.168 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.166+0000 7f62cde9b700 1 --2- 172.21.15.2:0/2041041119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c8100810 0x7f62c8100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.168 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.166+0000 7f62cde9b700 1 --2- 172.21.15.2:0/2041041119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62c80ff460 0x7f62c80ff8d0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.168 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.166+0000 7f62cde9b700 1 -- 172.21.15.2:0/2041041119 >> 172.21.15.2:0/2041041119 conn(0x7f62c80fa020 msgr2=0x7f62c80fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:20.168 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.166+0000 7f62cde9b700 1 -- 172.21.15.2:0/2041041119 shutdown_connections 2023-11-16T18:24:20.168 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.166+0000 7f62cde9b700 1 -- 172.21.15.2:0/2041041119 wait complete. 2023-11-16T18:24:20.168 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.166+0000 7f62cde9b700 1 Processor -- start 2023-11-16T18:24:20.168 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62cde9b700 1 -- start start 2023-11-16T18:24:20.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62cde9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62c80ff460 0x7f62c80feee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:20.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62cde9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80ffe10 0x7f62c819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:20.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62cde9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62c8100810 0x7f62c819d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:20.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62cde9b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c819daa0 con 0x7f62c8100810 2023-11-16T18:24:20.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62cde9b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c819dc10 con 0x7f62c80ffe10 2023-11-16T18:24:20.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62cde9b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c819ddd0 con 0x7f62c80ff460 2023-11-16T18:24:20.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62c7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62c8100810 0x7f62c819d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:20.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62c7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62c8100810 0x7f62c819d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49336/0 (socket says 172.21.15.2:49336) 2023-11-16T18:24:20.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62c7fff700 1 -- 172.21.15.2:0/2198266514 learned_addr learned my addr 172.21.15.2:0/2198266514 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:20.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62c77fe700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62c80ff460 0x7f62c80feee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:20.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62c6ffd700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80ffe10 0x7f62c819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:20.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62c7fff700 1 -- 172.21.15.2:0/2198266514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62c80ff460 msgr2=0x7f62c80feee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62c7fff700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62c80ff460 0x7f62c80feee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62c7fff700 1 -- 172.21.15.2:0/2198266514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80ffe10 msgr2=0x7f62c819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62c7fff700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80ffe10 0x7f62c819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62c7fff700 1 -- 172.21.15.2:0/2198266514 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b80095f0 con 0x7f62c8100810 2023-11-16T18:24:20.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62c6ffd700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80ffe10 0x7f62c819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:20.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62c7fff700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62c8100810 0x7f62c819d2d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f62c0007ee0 tx=0x7f62c000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:20.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.167+0000 7f62c4ff9700 1 -- 172.21.15.2:0/2198266514 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0017070 con 0x7f62c8100810 2023-11-16T18:24:20.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.168+0000 7f62c4ff9700 1 -- 172.21.15.2:0/2198266514 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62c000f040 con 0x7f62c8100810 2023-11-16T18:24:20.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.168+0000 7f62cde9b700 1 -- 172.21.15.2:0/2198266514 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c810abb0 con 0x7f62c8100810 2023-11-16T18:24:20.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.168+0000 7f62c4ff9700 1 -- 172.21.15.2:0/2198266514 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0005740 con 0x7f62c8100810 2023-11-16T18:24:20.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.168+0000 7f62cde9b700 1 -- 172.21.15.2:0/2198266514 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c810aa20 con 0x7f62c8100810 2023-11-16T18:24:20.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.169+0000 7f62cde9b700 1 -- 172.21.15.2:0/2198266514 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c81067c0 con 0x7f62c8100810 2023-11-16T18:24:20.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.170+0000 7f62c4ff9700 1 -- 172.21.15.2:0/2198266514 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f62c000dc60 con 0x7f62c8100810 2023-11-16T18:24:20.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.171+0000 7f62c4ff9700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f62b40a11e0 0x7f62b40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:20.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.171+0000 7f62c77fe700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f62b40a11e0 0x7f62b40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:20.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.171+0000 7f62c4ff9700 1 -- 172.21.15.2:0/2198266514 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f62c00ccc60 con 0x7f62c8100810 2023-11-16T18:24:20.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.171+0000 7f62c77fe700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f62b40a11e0 0x7f62b40a3690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f62b000ac50 tx=0x7f62b000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:20.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.174+0000 7f62c4ff9700 1 -- 172.21.15.2:0/2198266514 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f62c0095f40 con 0x7f62c8100810 2023-11-16T18:24:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:19 smithi040 ceph-mon[100251]: pgmap v308: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:24:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:19 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/296473772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:19 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3730434070' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:20.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:19 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/4158064315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:20.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.375+0000 7f62cde9b700 1 -- 172.21.15.2:0/2198266514 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f62c8061700 con 0x7f62c8100810 2023-11-16T18:24:20.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.376+0000 7f62c4ff9700 1 -- 172.21.15.2:0/2198266514 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f62c0099560 con 0x7f62c8100810 2023-11-16T18:24:20.377 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:24:20.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.378+0000 7f62cde9b700 1 -- 172.21.15.2:0/2198266514 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f62b40a11e0 msgr2=0x7f62b40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.378+0000 7f62cde9b700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f62b40a11e0 0x7f62b40a3690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f62b000ac50 tx=0x7f62b000a380 comp rx=0 tx=0).stop 2023-11-16T18:24:20.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.378+0000 7f62cde9b700 1 -- 172.21.15.2:0/2198266514 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62c8100810 msgr2=0x7f62c819d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.378+0000 7f62cde9b700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62c8100810 0x7f62c819d2d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f62c0007ee0 tx=0x7f62c000d3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.378+0000 7f62cde9b700 1 -- 172.21.15.2:0/2198266514 shutdown_connections 2023-11-16T18:24:20.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.378+0000 7f62cde9b700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f62b40a11e0 0x7f62b40a3690 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.378+0000 7f62cde9b700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62c80ff460 0x7f62c80feee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.378+0000 7f62cde9b700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62c8100810 0x7f62c819d2d0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.378+0000 7f62cde9b700 1 --2- 172.21.15.2:0/2198266514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80ffe10 0x7f62c819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.378+0000 7f62cde9b700 1 -- 172.21.15.2:0/2198266514 >> 172.21.15.2:0/2198266514 conn(0x7f62c80fa020 msgr2=0x7f62c8103900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:20.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.378+0000 7f62cde9b700 1 -- 172.21.15.2:0/2198266514 shutdown_connections 2023-11-16T18:24:20.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.378+0000 7f62cde9b700 1 -- 172.21.15.2:0/2198266514 wait complete. 2023-11-16T18:24:20.391 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T18:24:20.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.520+0000 7f4025421700 1 -- 172.21.15.2:0/2228158528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4020101ef0 msgr2=0x7f40201087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.520+0000 7f4025421700 1 --2- 172.21.15.2:0/2228158528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4020101ef0 0x7f40201087b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f401800b0a0 tx=0x7f401800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.520+0000 7f4025421700 1 -- 172.21.15.2:0/2228158528 shutdown_connections 2023-11-16T18:24:20.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.520+0000 7f4025421700 1 --2- 172.21.15.2:0/2228158528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4020101ef0 0x7f40201087b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.520+0000 7f4025421700 1 --2- 172.21.15.2:0/2228158528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40200fe590 0x7f40200fe9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.520+0000 7f4025421700 1 --2- 172.21.15.2:0/2228158528 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4020101560 0x7f40201019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.520+0000 7f4025421700 1 -- 172.21.15.2:0/2228158528 >> 172.21.15.2:0/2228158528 conn(0x7f40200f9f70 msgr2=0x7f40200fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:20.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.520+0000 7f4025421700 1 -- 172.21.15.2:0/2228158528 shutdown_connections 2023-11-16T18:24:20.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.520+0000 7f4025421700 1 -- 172.21.15.2:0/2228158528 wait complete. 2023-11-16T18:24:20.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.521+0000 7f4025421700 1 Processor -- start 2023-11-16T18:24:20.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.521+0000 7f4025421700 1 -- start start 2023-11-16T18:24:20.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.521+0000 7f4025421700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40200fe590 0x7f4020100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:20.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.521+0000 7f4025421700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4020101560 0x7f40200ff510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:20.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.521+0000 7f4025421700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4020101ef0 0x7f40200ffa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:20.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.521+0000 7f4025421700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4020199f60 con 0x7f4020101ef0 2023-11-16T18:24:20.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.521+0000 7f4025421700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40200fff90 con 0x7f4020101560 2023-11-16T18:24:20.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.521+0000 7f4025421700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4020100150 con 0x7f40200fe590 2023-11-16T18:24:20.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.521+0000 7f401f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4020101ef0 0x7f40200ffa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:20.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.521+0000 7f401f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4020101ef0 0x7f40200ffa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49350/0 (socket says 172.21.15.2:49350) 2023-11-16T18:24:20.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.521+0000 7f401f7fe700 1 -- 172.21.15.2:0/2446826592 learned_addr learned my addr 172.21.15.2:0/2446826592 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:20.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.521+0000 7f401effd700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40200fe590 0x7f4020100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:20.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.521+0000 7f401e7fc700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4020101560 0x7f40200ff510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:20.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.522+0000 7f401f7fe700 1 -- 172.21.15.2:0/2446826592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40200fe590 msgr2=0x7f4020100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.522+0000 7f401f7fe700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40200fe590 0x7f4020100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.522+0000 7f401f7fe700 1 -- 172.21.15.2:0/2446826592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4020101560 msgr2=0x7f40200ff510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.522+0000 7f401f7fe700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4020101560 0x7f40200ff510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.522+0000 7f401f7fe700 1 -- 172.21.15.2:0/2446826592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4018009d00 con 0x7f4020101ef0 2023-11-16T18:24:20.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.522+0000 7f401e7fc700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4020101560 0x7f40200ff510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:20.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.522+0000 7f401f7fe700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4020101ef0 0x7f40200ffa50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f40180062a0 tx=0x7f4018004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:20.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.522+0000 7f4017fff700 1 -- 172.21.15.2:0/2446826592 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401800e040 con 0x7f4020101ef0 2023-11-16T18:24:20.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.522+0000 7f4025421700 1 -- 172.21.15.2:0/2446826592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f402010bcc0 con 0x7f4020101ef0 2023-11-16T18:24:20.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.522+0000 7f4017fff700 1 -- 172.21.15.2:0/2446826592 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f401800cec0 con 0x7f4020101ef0 2023-11-16T18:24:20.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.522+0000 7f4017fff700 1 -- 172.21.15.2:0/2446826592 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4018004020 con 0x7f4020101ef0 2023-11-16T18:24:20.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.522+0000 7f4025421700 1 -- 172.21.15.2:0/2446826592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f402010bb30 con 0x7f4020101ef0 2023-11-16T18:24:20.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.523+0000 7f4025421700 1 -- 172.21.15.2:0/2446826592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4020107250 con 0x7f4020101ef0 2023-11-16T18:24:20.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.524+0000 7f4017fff700 1 -- 172.21.15.2:0/2446826592 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f401801d690 con 0x7f4020101ef0 2023-11-16T18:24:20.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.525+0000 7f4017fff700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f400c0a11e0 0x7f400c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:20.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.525+0000 7f4017fff700 1 -- 172.21.15.2:0/2446826592 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f40180cf0d0 con 0x7f4020101ef0 2023-11-16T18:24:20.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.525+0000 7f401effd700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f400c0a11e0 0x7f400c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:20.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.525+0000 7f401effd700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f400c0a11e0 0x7f400c0a3690 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4020068eb0 tx=0x7f4008007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:20.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.528+0000 7f4017fff700 1 -- 172.21.15.2:0/2446826592 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4018098440 con 0x7f4020101ef0 2023-11-16T18:24:20.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.728+0000 7f4025421700 1 -- 172.21.15.2:0/2446826592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4020067ea0 con 0x7f4020101ef0 2023-11-16T18:24:20.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.729+0000 7f4017fff700 1 -- 172.21.15.2:0/2446826592 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f40180253b0 con 0x7f4020101ef0 2023-11-16T18:24:20.730 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:24:20.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.731+0000 7f4025421700 1 -- 172.21.15.2:0/2446826592 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f400c0a11e0 msgr2=0x7f400c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.731+0000 7f4025421700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f400c0a11e0 0x7f400c0a3690 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4020068eb0 tx=0x7f4008007400 comp rx=0 tx=0).stop 2023-11-16T18:24:20.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.731+0000 7f4025421700 1 -- 172.21.15.2:0/2446826592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4020101ef0 msgr2=0x7f40200ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.731+0000 7f4025421700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4020101ef0 0x7f40200ffa50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f40180062a0 tx=0x7f4018004d00 comp rx=0 tx=0).stop 2023-11-16T18:24:20.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.731+0000 7f4025421700 1 -- 172.21.15.2:0/2446826592 shutdown_connections 2023-11-16T18:24:20.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.731+0000 7f4025421700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f400c0a11e0 0x7f400c0a3690 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.731+0000 7f4025421700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40200fe590 0x7f4020100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.731+0000 7f4025421700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4020101ef0 0x7f40200ffa50 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.731+0000 7f4025421700 1 --2- 172.21.15.2:0/2446826592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4020101560 0x7f40200ff510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.731+0000 7f4025421700 1 -- 172.21.15.2:0/2446826592 >> 172.21.15.2:0/2446826592 conn(0x7f40200f9f70 msgr2=0x7f40200fc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:20.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.731+0000 7f4025421700 1 -- 172.21.15.2:0/2446826592 shutdown_connections 2023-11-16T18:24:20.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.731+0000 7f4025421700 1 -- 172.21.15.2:0/2446826592 wait complete. 2023-11-16T18:24:20.745 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T18:24:20.878 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.874+0000 7fef53c4a700 1 -- 172.21.15.2:0/650556147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef4c0ff170 msgr2=0x7fef4c0ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.879 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.874+0000 7fef53c4a700 1 --2- 172.21.15.2:0/650556147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef4c0ff170 0x7fef4c0ff580 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fef48009990 tx=0x7fef48009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.879 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.874+0000 7fef53c4a700 1 -- 172.21.15.2:0/650556147 shutdown_connections 2023-11-16T18:24:20.879 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.874+0000 7fef53c4a700 1 --2- 172.21.15.2:0/650556147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef4c100db0 0x7fef4c107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.879 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.874+0000 7fef53c4a700 1 --2- 172.21.15.2:0/650556147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef4c0ff170 0x7fef4c0ff580 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.879 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.874+0000 7fef53c4a700 1 --2- 172.21.15.2:0/650556147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef4c100400 0x7fef4c100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.879 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.874+0000 7fef53c4a700 1 -- 172.21.15.2:0/650556147 >> 172.21.15.2:0/650556147 conn(0x7fef4c0fa750 msgr2=0x7fef4c0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:20.879 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.874+0000 7fef53c4a700 1 -- 172.21.15.2:0/650556147 shutdown_connections 2023-11-16T18:24:20.879 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.874+0000 7fef53c4a700 1 -- 172.21.15.2:0/650556147 wait complete. 2023-11-16T18:24:20.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.874+0000 7fef53c4a700 1 Processor -- start 2023-11-16T18:24:20.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.874+0000 7fef53c4a700 1 -- start start 2023-11-16T18:24:20.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef53c4a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef4c0ff170 0x7fef4c1960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:20.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef53c4a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef4c100400 0x7fef4c1965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:20.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef53c4a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef4c100db0 0x7fef4c19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:20.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef53c4a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c19dc20 con 0x7fef4c100db0 2023-11-16T18:24:20.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef53c4a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c19dd60 con 0x7fef4c100400 2023-11-16T18:24:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef53c4a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c19def0 con 0x7fef4c0ff170 2023-11-16T18:24:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef521e7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef4c100db0 0x7fef4c19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef521e7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef4c100db0 0x7fef4c19d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49364/0 (socket says 172.21.15.2:49364) 2023-11-16T18:24:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef521e7700 1 -- 172.21.15.2:0/2800939474 learned_addr learned my addr 172.21.15.2:0/2800939474 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef511e5700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef4c100400 0x7fef4c1965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef519e6700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef4c0ff170 0x7fef4c1960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef521e7700 1 -- 172.21.15.2:0/2800939474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef4c0ff170 msgr2=0x7fef4c1960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef521e7700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef4c0ff170 0x7fef4c1960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef521e7700 1 -- 172.21.15.2:0/2800939474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef4c100400 msgr2=0x7fef4c1965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:20.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef521e7700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef4c100400 0x7fef4c1965e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:20.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef521e7700 1 -- 172.21.15.2:0/2800939474 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef480095f0 con 0x7fef4c100db0 2023-11-16T18:24:20.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef519e6700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef4c0ff170 0x7fef4c1960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:20.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef521e7700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef4c100db0 0x7fef4c19d650 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fef4400b970 tx=0x7fef4400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:20.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef42ffd700 1 -- 172.21.15.2:0/2800939474 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4400c930 con 0x7fef4c100db0 2023-11-16T18:24:20.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef42ffd700 1 -- 172.21.15.2:0/2800939474 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef44014440 con 0x7fef4c100db0 2023-11-16T18:24:20.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef53c4a700 1 -- 172.21.15.2:0/2800939474 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4c10bd20 con 0x7fef4c100db0 2023-11-16T18:24:20.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef42ffd700 1 -- 172.21.15.2:0/2800939474 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef440075d0 con 0x7fef4c100db0 2023-11-16T18:24:20.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.875+0000 7fef53c4a700 1 -- 172.21.15.2:0/2800939474 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef4c10bb90 con 0x7fef4c100db0 2023-11-16T18:24:20.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.876+0000 7fef53c4a700 1 -- 172.21.15.2:0/2800939474 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4c106190 con 0x7fef4c100db0 2023-11-16T18:24:20.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.878+0000 7fef42ffd700 1 -- 172.21.15.2:0/2800939474 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fef440145b0 con 0x7fef4c100db0 2023-11-16T18:24:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.878+0000 7fef42ffd700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fef380a11e0 0x7fef380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.878+0000 7fef519e6700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fef380a11e0 0x7fef380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.878+0000 7fef42ffd700 1 -- 172.21.15.2:0/2800939474 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fef44006e40 con 0x7fef4c100db0 2023-11-16T18:24:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.879+0000 7fef519e6700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fef380a11e0 0x7fef380a3690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fef4800b5c0 tx=0x7fef48005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:20.882+0000 7fef42ffd700 1 -- 172.21.15.2:0/2800939474 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fef4404e730 con 0x7fef4c100db0 2023-11-16T18:24:21.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:20 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2347479530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:21.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:20 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2198266514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:21.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:20 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2446826592' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:21.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.082+0000 7fef53c4a700 1 -- 172.21.15.2:0/2800939474 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fef4c061700 con 0x7fef4c100db0 2023-11-16T18:24:21.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.082+0000 7fef42ffd700 1 -- 172.21.15.2:0/2800939474 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fef44095730 con 0x7fef4c100db0 2023-11-16T18:24:21.083 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T18:24:21.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.084+0000 7fef53c4a700 1 -- 172.21.15.2:0/2800939474 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fef380a11e0 msgr2=0x7fef380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.084+0000 7fef53c4a700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fef380a11e0 0x7fef380a3690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fef4800b5c0 tx=0x7fef48005fb0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.084+0000 7fef53c4a700 1 -- 172.21.15.2:0/2800939474 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef4c100db0 msgr2=0x7fef4c19d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.084+0000 7fef53c4a700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef4c100db0 0x7fef4c19d650 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fef4400b970 tx=0x7fef4400bc80 comp rx=0 tx=0).stop 2023-11-16T18:24:21.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.084+0000 7fef53c4a700 1 -- 172.21.15.2:0/2800939474 shutdown_connections 2023-11-16T18:24:21.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.084+0000 7fef53c4a700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fef380a11e0 0x7fef380a3690 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.084+0000 7fef53c4a700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef4c0ff170 0x7fef4c1960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.084+0000 7fef53c4a700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef4c100db0 0x7fef4c19d650 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.084+0000 7fef53c4a700 1 --2- 172.21.15.2:0/2800939474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef4c100400 0x7fef4c1965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.084+0000 7fef53c4a700 1 -- 172.21.15.2:0/2800939474 >> 172.21.15.2:0/2800939474 conn(0x7fef4c0fa750 msgr2=0x7fef4c0fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:21.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.084+0000 7fef53c4a700 1 -- 172.21.15.2:0/2800939474 shutdown_connections 2023-11-16T18:24:21.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.084+0000 7fef53c4a700 1 -- 172.21.15.2:0/2800939474 wait complete. 2023-11-16T18:24:21.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:20 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2347479530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:21.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:20 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2198266514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:21.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:20 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2446826592' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:21.098 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T18:24:21.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.226+0000 7ff0d5895700 1 -- 172.21.15.2:0/563794231 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0d0105fd0 msgr2=0x7ff0d01083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.226+0000 7ff0d5895700 1 --2- 172.21.15.2:0/563794231 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0d0105fd0 0x7ff0d01083b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800b0a0 tx=0x7ff0c800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.226+0000 7ff0d5895700 1 -- 172.21.15.2:0/563794231 shutdown_connections 2023-11-16T18:24:21.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.226+0000 7ff0d5895700 1 --2- 172.21.15.2:0/563794231 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0d0105fd0 0x7ff0d01083b0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.226+0000 7ff0d5895700 1 --2- 172.21.15.2:0/563794231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0d0100d90 0x7ff0d0103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.226+0000 7ff0d5895700 1 --2- 172.21.15.2:0/563794231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0d01036b0 0x7ff0d0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.226+0000 7ff0d5895700 1 -- 172.21.15.2:0/563794231 >> 172.21.15.2:0/563794231 conn(0x7ff0d00fa770 msgr2=0x7ff0d00fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:21.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.226+0000 7ff0d5895700 1 -- 172.21.15.2:0/563794231 shutdown_connections 2023-11-16T18:24:21.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.226+0000 7ff0d5895700 1 -- 172.21.15.2:0/563794231 wait complete. 2023-11-16T18:24:21.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.226+0000 7ff0d5895700 1 Processor -- start 2023-11-16T18:24:21.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.226+0000 7ff0d5895700 1 -- start start 2023-11-16T18:24:21.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0d5895700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0d0100d90 0x7ff0d0196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:21.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0d5895700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0d01036b0 0x7ff0d01965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:21.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0d5895700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0d0105fd0 0x7ff0d019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0d5895700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d019dc10 con 0x7ff0d0100d90 2023-11-16T18:24:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0d5895700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d019dd50 con 0x7ff0d0105fd0 2023-11-16T18:24:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0d5895700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d019dee0 con 0x7ff0d01036b0 2023-11-16T18:24:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0ceffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0d0100d90 0x7ff0d0196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0ceffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0d0100d90 0x7ff0d0196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49374/0 (socket says 172.21.15.2:49374) 2023-11-16T18:24:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0ceffd700 1 -- 172.21.15.2:0/3792473498 learned_addr learned my addr 172.21.15.2:0/3792473498 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0ceffd700 1 -- 172.21.15.2:0/3792473498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0d01036b0 msgr2=0x7ff0d01965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0ceffd700 1 --2- 172.21.15.2:0/3792473498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0d01036b0 0x7ff0d01965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0ceffd700 1 -- 172.21.15.2:0/3792473498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0d0105fd0 msgr2=0x7ff0d019d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0ceffd700 1 --2- 172.21.15.2:0/3792473498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0d0105fd0 0x7ff0d019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0ceffd700 1 -- 172.21.15.2:0/3792473498 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c8009d00 con 0x7ff0d0100d90 2023-11-16T18:24:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0ceffd700 1 --2- 172.21.15.2:0/3792473498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0d0100d90 0x7ff0d0196090 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400e970 tx=0x7ff0c400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0d4893700 1 -- 172.21.15.2:0/3792473498 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4009790 con 0x7ff0d0100d90 2023-11-16T18:24:21.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0d5895700 1 -- 172.21.15.2:0/3792473498 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d010bc00 con 0x7ff0d0100d90 2023-11-16T18:24:21.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0d4893700 1 -- 172.21.15.2:0/3792473498 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff0c4004cf0 con 0x7ff0d0100d90 2023-11-16T18:24:21.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0d4893700 1 -- 172.21.15.2:0/3792473498 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4005680 con 0x7ff0d0100d90 2023-11-16T18:24:21.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.227+0000 7ff0d5895700 1 -- 172.21.15.2:0/3792473498 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d010ba70 con 0x7ff0d0100d90 2023-11-16T18:24:21.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.229+0000 7ff0d5895700 1 -- 172.21.15.2:0/3792473498 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d00fc2e0 con 0x7ff0d0100d90 2023-11-16T18:24:21.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:20 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2347479530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:21.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:20 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2198266514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:21.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:20 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2446826592' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:21.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.230+0000 7ff0d4893700 1 -- 172.21.15.2:0/3792473498 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ff0c4004750 con 0x7ff0d0100d90 2023-11-16T18:24:21.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.230+0000 7ff0d4893700 1 --2- 172.21.15.2:0/3792473498 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff0b80a11e0 0x7ff0b80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:21.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.230+0000 7ff0d4893700 1 -- 172.21.15.2:0/3792473498 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff0c4014070 con 0x7ff0d0100d90 2023-11-16T18:24:21.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.230+0000 7ff0ce7fc700 1 --2- 172.21.15.2:0/3792473498 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff0b80a11e0 0x7ff0b80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:21.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.231+0000 7ff0ce7fc700 1 --2- 172.21.15.2:0/3792473498 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff0b80a11e0 0x7ff0b80a3690 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0005950 tx=0x7ff0c00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:21.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.234+0000 7ff0d4893700 1 -- 172.21.15.2:0/3792473498 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff0c4096580 con 0x7ff0d0100d90 2023-11-16T18:24:21.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.432+0000 7ff0d5895700 1 -- 172.21.15.2:0/3792473498 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff0d0061700 con 0x7ff0d0100d90 2023-11-16T18:24:21.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.432+0000 7ff0d4893700 1 -- 172.21.15.2:0/3792473498 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff0c4099ba0 con 0x7ff0d0100d90 2023-11-16T18:24:21.434 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:24:21.436 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.435+0000 7ff0d5895700 1 -- 172.21.15.2:0/3792473498 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff0b80a11e0 msgr2=0x7ff0b80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.436 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.435+0000 7ff0d5895700 1 --2- 172.21.15.2:0/3792473498 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff0b80a11e0 0x7ff0b80a3690 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0005950 tx=0x7ff0c00058e0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.436 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.435+0000 7ff0d5895700 1 -- 172.21.15.2:0/3792473498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0d0100d90 msgr2=0x7ff0d0196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.436 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.435+0000 7ff0d5895700 1 --2- 172.21.15.2:0/3792473498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0d0100d90 0x7ff0d0196090 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400e970 tx=0x7ff0c400ed30 comp rx=0 tx=0).stop 2023-11-16T18:24:21.436 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.435+0000 7ff0d5895700 1 -- 172.21.15.2:0/3792473498 shutdown_connections 2023-11-16T18:24:21.436 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.435+0000 7ff0d5895700 1 --2- 172.21.15.2:0/3792473498 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff0b80a11e0 0x7ff0b80a3690 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.435+0000 7ff0d5895700 1 --2- 172.21.15.2:0/3792473498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0d0105fd0 0x7ff0d019d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.435+0000 7ff0d5895700 1 --2- 172.21.15.2:0/3792473498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0d0100d90 0x7ff0d0196090 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.435+0000 7ff0d5895700 1 --2- 172.21.15.2:0/3792473498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0d01036b0 0x7ff0d01965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.435+0000 7ff0d5895700 1 -- 172.21.15.2:0/3792473498 >> 172.21.15.2:0/3792473498 conn(0x7ff0d00fa770 msgr2=0x7ff0d01019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:21.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.435+0000 7ff0d5895700 1 -- 172.21.15.2:0/3792473498 shutdown_connections 2023-11-16T18:24:21.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.435+0000 7ff0d5895700 1 -- 172.21.15.2:0/3792473498 wait complete. 2023-11-16T18:24:21.448 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:24:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.575+0000 7fd7cb625700 1 -- 172.21.15.2:0/1485252665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c40ff1f0 msgr2=0x7fd7c40ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.575+0000 7fd7cb625700 1 --2- 172.21.15.2:0/1485252665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c40ff1f0 0x7fd7c40ff600 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0009990 tx=0x7fd7c0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.575+0000 7fd7cb625700 1 -- 172.21.15.2:0/1485252665 shutdown_connections 2023-11-16T18:24:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.575+0000 7fd7cb625700 1 --2- 172.21.15.2:0/1485252665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c4100d70 0x7fd7c4107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.575+0000 7fd7cb625700 1 --2- 172.21.15.2:0/1485252665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c40ff1f0 0x7fd7c40ff600 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.575+0000 7fd7cb625700 1 --2- 172.21.15.2:0/1485252665 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c41003e0 0x7fd7c4100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.575+0000 7fd7cb625700 1 -- 172.21.15.2:0/1485252665 >> 172.21.15.2:0/1485252665 conn(0x7fd7c40fa7d0 msgr2=0x7fd7c40fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:21.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.575+0000 7fd7cb625700 1 -- 172.21.15.2:0/1485252665 shutdown_connections 2023-11-16T18:24:21.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.575+0000 7fd7cb625700 1 -- 172.21.15.2:0/1485252665 wait complete. 2023-11-16T18:24:21.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.575+0000 7fd7cb625700 1 Processor -- start 2023-11-16T18:24:21.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7cb625700 1 -- start start 2023-11-16T18:24:21.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7cb625700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c40ff1f0 0x7fd7c4195010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:21.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7cb625700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c41003e0 0x7fd7c4195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:21.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7cb625700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c4100d70 0x7fd7c419c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:21.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7cb625700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c419cb90 con 0x7fd7c40ff1f0 2023-11-16T18:24:21.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7cb625700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c419ccd0 con 0x7fd7c41003e0 2023-11-16T18:24:21.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7cb625700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c419ce60 con 0x7fd7c4100d70 2023-11-16T18:24:21.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7c93c1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c40ff1f0 0x7fd7c4195010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7c93c1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c40ff1f0 0x7fd7c4195010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49388/0 (socket says 172.21.15.2:49388) 2023-11-16T18:24:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7c93c1700 1 -- 172.21.15.2:0/2457315983 learned_addr learned my addr 172.21.15.2:0/2457315983 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7c8bc0700 1 --2- 172.21.15.2:0/2457315983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c41003e0 0x7fd7c4195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7c93c1700 1 -- 172.21.15.2:0/2457315983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c4100d70 msgr2=0x7fd7c419c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7c93c1700 1 --2- 172.21.15.2:0/2457315983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c4100d70 0x7fd7c419c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7c93c1700 1 -- 172.21.15.2:0/2457315983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c41003e0 msgr2=0x7fd7c4195550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7c93c1700 1 --2- 172.21.15.2:0/2457315983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c41003e0 0x7fd7c4195550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7c93c1700 1 -- 172.21.15.2:0/2457315983 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c00095f0 con 0x7fd7c40ff1f0 2023-11-16T18:24:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.576+0000 7fd7c93c1700 1 --2- 172.21.15.2:0/2457315983 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c40ff1f0 0x7fd7c4195010 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0006010 tx=0x7fd7c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.577+0000 7fd7ba7fc700 1 -- 172.21.15.2:0/2457315983 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c001f070 con 0x7fd7c40ff1f0 2023-11-16T18:24:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.577+0000 7fd7ba7fc700 1 -- 172.21.15.2:0/2457315983 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7c001ab50 con 0x7fd7c40ff1f0 2023-11-16T18:24:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.577+0000 7fd7cb625700 1 -- 172.21.15.2:0/2457315983 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c410ab60 con 0x7fd7c40ff1f0 2023-11-16T18:24:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.577+0000 7fd7ba7fc700 1 -- 172.21.15.2:0/2457315983 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c0003f30 con 0x7fd7c40ff1f0 2023-11-16T18:24:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.577+0000 7fd7cb625700 1 -- 172.21.15.2:0/2457315983 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c410a9d0 con 0x7fd7c40ff1f0 2023-11-16T18:24:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.577+0000 7fd7cb625700 1 -- 172.21.15.2:0/2457315983 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c40699c0 con 0x7fd7c40ff1f0 2023-11-16T18:24:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.579+0000 7fd7ba7fc700 1 -- 172.21.15.2:0/2457315983 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fd7c0023070 con 0x7fd7c40ff1f0 2023-11-16T18:24:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.579+0000 7fd7ba7fc700 1 --2- 172.21.15.2:0/2457315983 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd7b00a1260 0x7fd7b00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:21.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.579+0000 7fd7c8bc0700 1 --2- 172.21.15.2:0/2457315983 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd7b00a1260 0x7fd7b00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:21.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.579+0000 7fd7ba7fc700 1 -- 172.21.15.2:0/2457315983 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd7c00d05f0 con 0x7fd7c40ff1f0 2023-11-16T18:24:21.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.580+0000 7fd7c8bc0700 1 --2- 172.21.15.2:0/2457315983 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd7b00a1260 0x7fd7b00a3710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4005950 tx=0x7fd7b400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:21.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.583+0000 7fd7ba7fc700 1 -- 172.21.15.2:0/2457315983 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd7c0099960 con 0x7fd7c40ff1f0 2023-11-16T18:24:21.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.825+0000 7fd7cb625700 1 -- 172.21.15.2:0/2457315983 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7c4106620 con 0x7fd7c40ff1f0 2023-11-16T18:24:21.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.826+0000 7fd7ba7fc700 1 -- 172.21.15.2:0/2457315983 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10774 (secure 0 0 0) 0x7fd7c009cf80 con 0x7fd7c40ff1f0 2023-11-16T18:24:21.827 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:21.827 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:23:07.909835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:24:21.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.828+0000 7fd7cb625700 1 -- 172.21.15.2:0/2457315983 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd7b00a1260 msgr2=0x7fd7b00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.828+0000 7fd7cb625700 1 --2- 172.21.15.2:0/2457315983 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd7b00a1260 0x7fd7b00a3710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4005950 tx=0x7fd7b400b410 comp rx=0 tx=0).stop 2023-11-16T18:24:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.828+0000 7fd7cb625700 1 -- 172.21.15.2:0/2457315983 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c40ff1f0 msgr2=0x7fd7c4195010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.828+0000 7fd7cb625700 1 --2- 172.21.15.2:0/2457315983 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c40ff1f0 0x7fd7c4195010 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0006010 tx=0x7fd7c0004730 comp rx=0 tx=0).stop 2023-11-16T18:24:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.828+0000 7fd7cb625700 1 -- 172.21.15.2:0/2457315983 shutdown_connections 2023-11-16T18:24:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.828+0000 7fd7cb625700 1 --2- 172.21.15.2:0/2457315983 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd7b00a1260 0x7fd7b00a3710 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.828+0000 7fd7cb625700 1 --2- 172.21.15.2:0/2457315983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c4100d70 0x7fd7c419c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.828+0000 7fd7cb625700 1 --2- 172.21.15.2:0/2457315983 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c40ff1f0 0x7fd7c4195010 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.828+0000 7fd7cb625700 1 --2- 172.21.15.2:0/2457315983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c41003e0 0x7fd7c4195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.828+0000 7fd7cb625700 1 -- 172.21.15.2:0/2457315983 >> 172.21.15.2:0/2457315983 conn(0x7fd7c40fa7d0 msgr2=0x7fd7c41038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.829+0000 7fd7cb625700 1 -- 172.21.15.2:0/2457315983 shutdown_connections 2023-11-16T18:24:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.829+0000 7fd7cb625700 1 -- 172.21.15.2:0/2457315983 wait complete. 2023-11-16T18:24:21.831 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T18:24:21.842 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:24:21.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.967+0000 7f46cb124700 1 -- 172.21.15.2:0/70929629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c41003e0 msgr2=0x7f46c4100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.967+0000 7f46cb124700 1 --2- 172.21.15.2:0/70929629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c41003e0 0x7f46c4100830 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009990 tx=0x7f46b0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.967+0000 7f46cb124700 1 -- 172.21.15.2:0/70929629 shutdown_connections 2023-11-16T18:24:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.967+0000 7f46cb124700 1 --2- 172.21.15.2:0/70929629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46c40ff1f0 0x7f46c40ff600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.967+0000 7f46cb124700 1 --2- 172.21.15.2:0/70929629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c4100d70 0x7f46c4107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.967+0000 7f46cb124700 1 --2- 172.21.15.2:0/70929629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c41003e0 0x7f46c4100830 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.967+0000 7f46cb124700 1 -- 172.21.15.2:0/70929629 >> 172.21.15.2:0/70929629 conn(0x7f46c40fa7d0 msgr2=0x7f46c40fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.967+0000 7f46cb124700 1 -- 172.21.15.2:0/70929629 shutdown_connections 2023-11-16T18:24:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.967+0000 7f46cb124700 1 -- 172.21.15.2:0/70929629 wait complete. 2023-11-16T18:24:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46cb124700 1 Processor -- start 2023-11-16T18:24:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46cb124700 1 -- start start 2023-11-16T18:24:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46cb124700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c40ff1f0 0x7f46c4196000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46cb124700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46c41003e0 0x7f46c4196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46cb124700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c4100d70 0x7f46c419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46cb124700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c419db80 con 0x7f46c4100d70 2023-11-16T18:24:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46cb124700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c419dcc0 con 0x7f46c40ff1f0 2023-11-16T18:24:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46cb124700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c419de50 con 0x7f46c41003e0 2023-11-16T18:24:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46c96c1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c4100d70 0x7f46c419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46c96c1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c4100d70 0x7f46c419d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49406/0 (socket says 172.21.15.2:49406) 2023-11-16T18:24:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46c96c1700 1 -- 172.21.15.2:0/636506978 learned_addr learned my addr 172.21.15.2:0/636506978 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46c8ec0700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c40ff1f0 0x7f46c4196000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46c3fff700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46c41003e0 0x7f46c4196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46c96c1700 1 -- 172.21.15.2:0/636506978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46c41003e0 msgr2=0x7f46c4196540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46c96c1700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46c41003e0 0x7f46c4196540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46c96c1700 1 -- 172.21.15.2:0/636506978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c40ff1f0 msgr2=0x7f46c4196000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46c96c1700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c40ff1f0 0x7f46c4196000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.968+0000 7f46c96c1700 1 -- 172.21.15.2:0/636506978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b00095f0 con 0x7f46c4100d70 2023-11-16T18:24:21.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.969+0000 7f46c96c1700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c4100d70 0x7f46c419d5b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00ef30 tx=0x7f46bc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:21.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.969+0000 7f46c1ffb700 1 -- 172.21.15.2:0/636506978 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc018070 con 0x7f46c4100d70 2023-11-16T18:24:21.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.969+0000 7f46c1ffb700 1 -- 172.21.15.2:0/636506978 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46bc00f040 con 0x7f46c4100d70 2023-11-16T18:24:21.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.969+0000 7f46cb124700 1 -- 172.21.15.2:0/636506978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c410ab50 con 0x7f46c4100d70 2023-11-16T18:24:21.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.969+0000 7f46c1ffb700 1 -- 172.21.15.2:0/636506978 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc00cd00 con 0x7f46c4100d70 2023-11-16T18:24:21.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.969+0000 7f46cb124700 1 -- 172.21.15.2:0/636506978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c410a9c0 con 0x7f46c4100d70 2023-11-16T18:24:21.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.970+0000 7f46cb124700 1 -- 172.21.15.2:0/636506978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c4107220 con 0x7f46c4100d70 2023-11-16T18:24:21.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.971+0000 7f46c1ffb700 1 -- 172.21.15.2:0/636506978 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f46bc003a00 con 0x7f46c4100d70 2023-11-16T18:24:21.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.972+0000 7f46c1ffb700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f46b40a11e0 0x7f46b40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:21.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.972+0000 7f46c1ffb700 1 -- 172.21.15.2:0/636506978 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f46bc0cc900 con 0x7f46c4100d70 2023-11-16T18:24:21.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.972+0000 7f46c8ec0700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f46b40a11e0 0x7f46b40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:21.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.972+0000 7f46c8ec0700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f46b40a11e0 0x7f46b40a3690 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f46c4100240 tx=0x7f46b800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:21.976+0000 7f46c1ffb700 1 -- 172.21.15.2:0/636506978 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f46bc095c70 con 0x7f46c4100d70 2023-11-16T18:24:22.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:21 smithi188 ceph-mon[100639]: pgmap v309: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:22.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:21 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2800939474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:22.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:21 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3792473498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:22.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:21 smithi002 ceph-mon[98807]: pgmap v309: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:22.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:21 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2800939474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:22.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:21 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3792473498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:22.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.176+0000 7f46cb124700 1 -- 172.21.15.2:0/636506978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46c402cfe0 con 0x7f46c4100d70 2023-11-16T18:24:22.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.178+0000 7f46c1ffb700 1 -- 172.21.15.2:0/636506978 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24383 (secure 0 0 0) 0x7f46bc099290 con 0x7f46c4100d70 2023-11-16T18:24:22.180 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:22.180 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":88,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:23:39.497670+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:24:22.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.180+0000 7f46cb124700 1 -- 172.21.15.2:0/636506978 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f46b40a11e0 msgr2=0x7f46b40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.180+0000 7f46cb124700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f46b40a11e0 0x7f46b40a3690 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f46c4100240 tx=0x7f46b800b410 comp rx=0 tx=0).stop 2023-11-16T18:24:22.183 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.180+0000 7f46cb124700 1 -- 172.21.15.2:0/636506978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c4100d70 msgr2=0x7f46c419d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.183 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.180+0000 7f46cb124700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c4100d70 0x7f46c419d5b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00ef30 tx=0x7f46bc00c3a0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.183 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.180+0000 7f46cb124700 1 -- 172.21.15.2:0/636506978 shutdown_connections 2023-11-16T18:24:22.183 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.180+0000 7f46cb124700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f46b40a11e0 0x7f46b40a3690 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.183 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.180+0000 7f46cb124700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c4100d70 0x7f46c419d5b0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.183 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.180+0000 7f46cb124700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c40ff1f0 0x7f46c4196000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.183 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.180+0000 7f46cb124700 1 --2- 172.21.15.2:0/636506978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46c41003e0 0x7f46c4196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.183 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.180+0000 7f46cb124700 1 -- 172.21.15.2:0/636506978 >> 172.21.15.2:0/636506978 conn(0x7f46c40fa7d0 msgr2=0x7f46c40fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.183 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.180+0000 7f46cb124700 1 -- 172.21.15.2:0/636506978 shutdown_connections 2023-11-16T18:24:22.183 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.180+0000 7f46cb124700 1 -- 172.21.15.2:0/636506978 wait complete. 2023-11-16T18:24:22.194 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-11-16T18:22:46.806019+0000', 'modified': '2023-11-16T18:23:07.909835+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24473, 'mds_1': 24460, 'mds_2': 24446}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24422': {'gid': 24422, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6835/17956118', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 17956118}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 17956118}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24427': {'gid': 24427, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6835/3630447129', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6834', 'nonce': 3630447129}, {'type': 'v1', 'addr': '172.21.15.188:6835', 'nonce': 3630447129}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24439': {'gid': 24439, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6837/264232322', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6836', 'nonce': 264232322}, {'type': 'v1', 'addr': '172.21.15.188:6837', 'nonce': 264232322}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24446': {'gid': 24446, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.40:6837/248461642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6836', 'nonce': 248461642}, {'type': 'v1', 'addr': '172.21.15.40:6837', 'nonce': 248461642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.188:6839/1646912892', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6838', 'nonce': 1646912892}, {'type': 'v1', 'addr': '172.21.15.188:6839', 'nonce': 1646912892}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24473': {'gid': 24473, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6839/3162585010', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 3162585010}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 3162585010}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-16T18:24:22.194 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-11-16T18:24:22.194 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-11-16T18:24:22.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:21 smithi040 ceph-mon[100251]: pgmap v309: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:22.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:21 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2800939474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:22.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:21 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3792473498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:22.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.332+0000 7fc5f4970700 1 -- 172.21.15.2:0/2656865776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5f0100320 msgr2=0x7fc5f0100770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.332+0000 7fc5f4970700 1 --2- 172.21.15.2:0/2656865776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5f0100320 0x7fc5f0100770 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009990 tx=0x7fc5e4009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.333+0000 7fc5f4970700 1 -- 172.21.15.2:0/2656865776 shutdown_connections 2023-11-16T18:24:22.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.333+0000 7fc5f4970700 1 --2- 172.21.15.2:0/2656865776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5f0100cb0 0x7fc5f0107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.333+0000 7fc5f4970700 1 --2- 172.21.15.2:0/2656865776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5f00ff130 0x7fc5f00ff540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.333+0000 7fc5f4970700 1 --2- 172.21.15.2:0/2656865776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5f0100320 0x7fc5f0100770 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.333+0000 7fc5f4970700 1 -- 172.21.15.2:0/2656865776 >> 172.21.15.2:0/2656865776 conn(0x7fc5f00fa710 msgr2=0x7fc5f00fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.333+0000 7fc5f4970700 1 -- 172.21.15.2:0/2656865776 shutdown_connections 2023-11-16T18:24:22.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.333+0000 7fc5f4970700 1 -- 172.21.15.2:0/2656865776 wait complete. 2023-11-16T18:24:22.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5f4970700 1 Processor -- start 2023-11-16T18:24:22.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5f4970700 1 -- start start 2023-11-16T18:24:22.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5f4970700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5f00ff130 0x7fc5f00699f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5f4970700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5f0100320 0x7fc5f019ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5f4970700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5f0100cb0 0x7fc5f019d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5f4970700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f019da60 con 0x7fc5f0100320 2023-11-16T18:24:22.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5f4970700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f019dbd0 con 0x7fc5f0100cb0 2023-11-16T18:24:22.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5f4970700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f019dd90 con 0x7fc5f00ff130 2023-11-16T18:24:22.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5edd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5f0100320 0x7fc5f019ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5edd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5f0100320 0x7fc5f019ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49418/0 (socket says 172.21.15.2:49418) 2023-11-16T18:24:22.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5edd9b700 1 -- 172.21.15.2:0/2940672088 learned_addr learned my addr 172.21.15.2:0/2940672088 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5f00ff130 0x7fc5f00699f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5eed9d700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5f0100cb0 0x7fc5f019d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5edd9b700 1 -- 172.21.15.2:0/2940672088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5f00ff130 msgr2=0x7fc5f00699f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5edd9b700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5f00ff130 0x7fc5f00699f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5edd9b700 1 -- 172.21.15.2:0/2940672088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5f0100cb0 msgr2=0x7fc5f019d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5edd9b700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5f0100cb0 0x7fc5f019d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5edd9b700 1 -- 172.21.15.2:0/2940672088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e40095f0 con 0x7fc5f0100320 2023-11-16T18:24:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5f00ff130 0x7fc5f00699f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.334+0000 7fc5eed9d700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5f0100cb0 0x7fc5f019d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.335+0000 7fc5edd9b700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5f0100320 0x7fc5f019ac30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4004760 tx=0x7fc5e4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.335+0000 7fc5df7fe700 1 -- 172.21.15.2:0/2940672088 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e401d070 con 0x7fc5f0100320 2023-11-16T18:24:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.335+0000 7fc5df7fe700 1 -- 172.21.15.2:0/2940672088 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5e4004200 con 0x7fc5f0100320 2023-11-16T18:24:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.335+0000 7fc5f4970700 1 -- 172.21.15.2:0/2940672088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f010aa90 con 0x7fc5f0100320 2023-11-16T18:24:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.335+0000 7fc5df7fe700 1 -- 172.21.15.2:0/2940672088 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e40049f0 con 0x7fc5f0100320 2023-11-16T18:24:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.335+0000 7fc5f4970700 1 -- 172.21.15.2:0/2940672088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f010a900 con 0x7fc5f0100320 2023-11-16T18:24:22.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.337+0000 7fc5df7fe700 1 -- 172.21.15.2:0/2940672088 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fc5e4003d40 con 0x7fc5f0100320 2023-11-16T18:24:22.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.338+0000 7fc5df7fe700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc5d80a11e0 0x7fc5d80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.338+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc5d80a11e0 0x7fc5d80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.338+0000 7fc5df7fe700 1 -- 172.21.15.2:0/2940672088 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc5e40cdd00 con 0x7fc5f0100320 2023-11-16T18:24:22.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.338+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc5d80a11e0 0x7fc5d80a3690 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0005fd0 tx=0x7fc5e0005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc600afcb80 1 Processor -- start 2023-11-16T18:24:22.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc600afcb80 1 -- start start 2023-11-16T18:24:22.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc600afcb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368050 0x55a11c368460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc600afcb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368ee0 0x55a11c369350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc600afcb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 0x55a11c370130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc600afcb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a11c269010 con 0x55a11c369890 2023-11-16T18:24:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc600afcb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a11c2bb850 con 0x55a11c368ee0 2023-11-16T18:24:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc600afcb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a11c2b4ce0 con 0x55a11c368050 2023-11-16T18:24:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc5eed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 0x55a11c370130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc5eed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 0x55a11c370130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49428/0 (socket says 172.21.15.2:49428) 2023-11-16T18:24:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc5eed9d700 1 -- 172.21.15.2:0/3622333304 learned_addr learned my addr 172.21.15.2:0/3622333304 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc5edd9b700 1 --2- 172.21.15.2:0/3622333304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368ee0 0x55a11c369350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/3622333304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368050 0x55a11c368460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.339+0000 7fc5eed9d700 1 -- 172.21.15.2:0/3622333304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368050 msgr2=0x55a11c368460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.340+0000 7fc5eed9d700 1 --2- 172.21.15.2:0/3622333304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368050 0x55a11c368460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.340+0000 7fc5eed9d700 1 -- 172.21.15.2:0/3622333304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368ee0 msgr2=0x55a11c369350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.340+0000 7fc5eed9d700 1 --2- 172.21.15.2:0/3622333304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368ee0 0x55a11c369350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.340+0000 7fc5eed9d700 1 -- 172.21.15.2:0/3622333304 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a11c1365e0 con 0x55a11c369890 2023-11-16T18:24:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.340+0000 7fc5eed9d700 1 --2- 172.21.15.2:0/3622333304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 0x55a11c370130 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009d50 tx=0x7fc5e80077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8520db904c110db server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.340+0000 7fc5dcff9700 1 -- 172.21.15.2:0/3622333304 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8009ea0 con 0x55a11c369890 2023-11-16T18:24:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.340+0000 7fc5dcff9700 1 -- 172.21.15.2:0/3622333304 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5e8014ac0 con 0x55a11c369890 2023-11-16T18:24:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.340+0000 7fc5dcff9700 1 -- 172.21.15.2:0/3622333304 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8013aa0 con 0x55a11c369890 2023-11-16T18:24:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.341+0000 7fc600afcb80 1 -- 172.21.15.2:0/3622333304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 msgr2=0x55a11c370130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.341+0000 7fc600afcb80 1 --2- 172.21.15.2:0/3622333304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 0x55a11c370130 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009d50 tx=0x7fc5e80077e0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.341+0000 7fc600afcb80 1 -- 172.21.15.2:0/3622333304 shutdown_connections 2023-11-16T18:24:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.341+0000 7fc600afcb80 1 --2- 172.21.15.2:0/3622333304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368050 0x55a11c368460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.341+0000 7fc600afcb80 1 --2- 172.21.15.2:0/3622333304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 0x55a11c370130 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.341+0000 7fc600afcb80 1 --2- 172.21.15.2:0/3622333304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368ee0 0x55a11c369350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.341+0000 7fc600afcb80 1 -- 172.21.15.2:0/3622333304 >> 172.21.15.2:0/3622333304 conn(0x55a11c359310 msgr2=0x55a11c35e620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.341+0000 7fc600afcb80 1 -- 172.21.15.2:0/3622333304 shutdown_connections 2023-11-16T18:24:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.341+0000 7fc600afcb80 1 -- 172.21.15.2:0/3622333304 wait complete. 2023-11-16T18:24:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.341+0000 7fc600afcb80 1 -- start start 2023-11-16T18:24:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.341+0000 7fc600afcb80 1 Processor -- start 2023-11-16T18:24:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc600afcb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368050 0x55a11c3719e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc600afcb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368ee0 0x55a11c371f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc600afcb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c369890 0x55a11c378f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc600afcb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a11c256890 con 0x55a11c368ee0 2023-11-16T18:24:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc600afcb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a11c256530 con 0x55a11c369890 2023-11-16T18:24:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc600afcb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a11c1388d0 con 0x55a11c368050 2023-11-16T18:24:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc600afcb80 10 client.0 _ll_register_callbacks cb 0x55a11c274730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:24:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5edd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368ee0 0x55a11c371f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5edd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368ee0 0x55a11c371f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49432/0 (socket says 172.21.15.2:49432) 2023-11-16T18:24:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5edd9b700 1 -- 172.21.15.2:0/2147275940 learned_addr learned my addr 172.21.15.2:0/2147275940 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368050 0x55a11c3719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5eed9d700 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c369890 0x55a11c378f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5edd9b700 1 -- 172.21.15.2:0/2147275940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368050 msgr2=0x55a11c3719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5edd9b700 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368050 0x55a11c3719e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5edd9b700 1 -- 172.21.15.2:0/2147275940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c369890 msgr2=0x55a11c378f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5edd9b700 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c369890 0x55a11c378f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5edd9b700 1 -- 172.21.15.2:0/2147275940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e8009980 con 0x55a11c368ee0 2023-11-16T18:24:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5eed9d700 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c369890 0x55a11c378f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368050 0x55a11c3719e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5edd9b700 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368ee0 0x55a11c371f20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40cb420 tx=0x7fc5e4005550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102f9bbb8a7448f4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5d5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:24:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5d5ffb700 1 -- 172.21.15.2:0/2147275940 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e40a30f0 con 0x55a11c368ee0 2023-11-16T18:24:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc5d5ffb700 1 -- 172.21.15.2:0/2147275940 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc5e409d030 con 0x55a11c368ee0 2023-11-16T18:24:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.342+0000 7fc600afcb80 1 -- 172.21.15.2:0/2147275940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a11c356ad0 con 0x55a11c368ee0 2023-11-16T18:24:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.343+0000 7fc5d5ffb700 1 -- 172.21.15.2:0/2147275940 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5e40b34b0 con 0x55a11c368ee0 2023-11-16T18:24:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.343+0000 7fc600afcb80 10 client.15363 fetch_fsmap learned FSMap version 24 2023-11-16T18:24:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.343+0000 7fc600afcb80 1 -- 172.21.15.2:0/2147275940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a11c27b140 con 0x55a11c368ee0 2023-11-16T18:24:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.343+0000 7fc5d5ffb700 1 -- 172.21.15.2:0/2147275940 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a11c356ad0 con 0x55a11c368ee0 2023-11-16T18:24:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.343+0000 7fc5d5ffb700 1 -- 172.21.15.2:0/2147275940 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc5e4099c90 con 0x55a11c368ee0 2023-11-16T18:24:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.343+0000 7fc600afcb80 10 client.15363 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:24:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.343+0000 7fc600afcb80 10 client.15363 resolve_mds: resolved name 'h' to daemon mds.h 2023-11-16T18:24:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.343+0000 7fc600afcb80 20 client.15363 populate_metadata read hostname 'smithi002' 2023-11-16T18:24:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.343+0000 7fc600afcb80 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a11c37c810 0x55a11c37be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.343+0000 7fc600afcb80 4 client.15363 mds_command: new command op to 24446 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:24:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.343+0000 7fc600afcb80 1 -- 172.21.15.2:0/2147275940 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a11c277050 con 0x55a11c37c810 2023-11-16T18:24:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.343+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a11c37c810 0x55a11c37be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.344+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a11c37c810 0x55a11c37be90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=62489ed712479c35 server_cookie=3ca77dbebdc3d7ba in_seq=0 out_seq=0 2023-11-16T18:24:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.344+0000 7fc5d5ffb700 10 client.15363 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:24:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc5d5ffb700 1 -- 172.21.15.2:0/2147275940 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a11c277050 con 0x55a11c37c810 2023-11-16T18:24:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc5d5ffb700 10 client.15363 handle_command_reply: tid=0 2023-11-16T18:24:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc600afcb80 1 client.15363 shutdown 2023-11-16T18:24:22.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc600afcb80 1 -- 172.21.15.2:0/2147275940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368ee0 msgr2=0x55a11c371f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc600afcb80 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368ee0 0x55a11c371f20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40cb420 tx=0x7fc5e4005550 comp rx=0 tx=0).stop 2023-11-16T18:24:22.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc600afcb80 1 -- 172.21.15.2:0/2147275940 shutdown_connections 2023-11-16T18:24:22.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc600afcb80 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a11c37c810 0x55a11c37be90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc600afcb80 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368050 0x55a11c3719e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc5d5ffb700 0 client.15363 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:24:22.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc600afcb80 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c369890 0x55a11c378f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc600afcb80 1 --2- 172.21.15.2:0/2147275940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368ee0 0x55a11c371f20 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc600afcb80 1 -- 172.21.15.2:0/2147275940 >> 172.21.15.2:0/2147275940 conn(0x55a11c359310 msgr2=0x55a11c35dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc600afcb80 1 -- 172.21.15.2:0/2147275940 shutdown_connections 2023-11-16T18:24:22.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc600afcb80 1 -- 172.21.15.2:0/2147275940 wait complete. 2023-11-16T18:24:22.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.345+0000 7fc600afcb80 20 client.15363 trim_cache size 0 max 16384 2023-11-16T18:24:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.363+0000 7fc600afcb80 1 Processor -- start 2023-11-16T18:24:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.363+0000 7fc600afcb80 1 -- start start 2023-11-16T18:24:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc600afcb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368050 0x55a11c370700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc600afcb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368ee0 0x55a11c378f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc600afcb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c369890 0x55a11c37be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc600afcb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a11c37db30 con 0x55a11c368050 2023-11-16T18:24:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc600afcb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a11c277050 con 0x55a11c368ee0 2023-11-16T18:24:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc600afcb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a11c36c360 con 0x55a11c369890 2023-11-16T18:24:22.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5ee59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368050 0x55a11c370700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5ee59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368050 0x55a11c370700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49438/0 (socket says 172.21.15.2:49438) 2023-11-16T18:24:22.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5ee59c700 1 -- 172.21.15.2:0/2568617178 learned_addr learned my addr 172.21.15.2:0/2568617178 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5edd9b700 1 --2- 172.21.15.2:0/2568617178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368ee0 0x55a11c378f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5ee59c700 1 -- 172.21.15.2:0/2568617178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c369890 msgr2=0x55a11c37be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/2568617178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c369890 0x55a11c37be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5ee59c700 1 -- 172.21.15.2:0/2568617178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368ee0 msgr2=0x55a11c378f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/2568617178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368ee0 0x55a11c378f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5ee59c700 1 -- 172.21.15.2:0/2568617178 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a11c27ae90 con 0x55a11c368050 2023-11-16T18:24:22.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5edd9b700 1 --2- 172.21.15.2:0/2568617178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368ee0 0x55a11c378f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:22.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/2568617178 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368050 0x55a11c370700 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x55a11c1389e0 tx=0x7fc5e0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdeb69985d19662d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5dd7fa700 1 -- 172.21.15.2:0/2568617178 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0011d60 con 0x55a11c368050 2023-11-16T18:24:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5dd7fa700 1 -- 172.21.15.2:0/2568617178 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5e0026040 con 0x55a11c368050 2023-11-16T18:24:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.364+0000 7fc5dd7fa700 1 -- 172.21.15.2:0/2568617178 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0004a70 con 0x55a11c368050 2023-11-16T18:24:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.365+0000 7fc600afcb80 1 -- 172.21.15.2:0/2568617178 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368050 msgr2=0x55a11c370700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.365+0000 7fc600afcb80 1 --2- 172.21.15.2:0/2568617178 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368050 0x55a11c370700 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x55a11c1389e0 tx=0x7fc5e0014000 comp rx=0 tx=0).stop 2023-11-16T18:24:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.365+0000 7fc600afcb80 1 -- 172.21.15.2:0/2568617178 shutdown_connections 2023-11-16T18:24:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.365+0000 7fc600afcb80 1 --2- 172.21.15.2:0/2568617178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c369890 0x55a11c37be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.365+0000 7fc600afcb80 1 --2- 172.21.15.2:0/2568617178 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c368050 0x55a11c370700 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.365+0000 7fc600afcb80 1 --2- 172.21.15.2:0/2568617178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368ee0 0x55a11c378f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.365+0000 7fc600afcb80 1 -- 172.21.15.2:0/2568617178 >> 172.21.15.2:0/2568617178 conn(0x55a11c359310 msgr2=0x55a11c3855a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.365+0000 7fc600afcb80 1 -- 172.21.15.2:0/2568617178 shutdown_connections 2023-11-16T18:24:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.365+0000 7fc600afcb80 1 -- 172.21.15.2:0/2568617178 wait complete. 2023-11-16T18:24:22.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.365+0000 7fc600afcb80 1 -- start start 2023-11-16T18:24:22.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.365+0000 7fc600afcb80 1 Processor -- start 2023-11-16T18:24:22.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc600afcb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368050 0x55a11c375ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc600afcb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368ee0 0x55a11c3761e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc600afcb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 0x55a11c3d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc600afcb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a11c3767b0 con 0x55a11c369890 2023-11-16T18:24:22.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc600afcb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a11c376920 con 0x55a11c368050 2023-11-16T18:24:22.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc600afcb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a11c3d0c90 con 0x55a11c368ee0 2023-11-16T18:24:22.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc600afcb80 10 client.0 _ll_register_callbacks cb 0x55a11c274730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:24:22.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5eed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 0x55a11c3d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5eed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 0x55a11c3d06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49448/0 (socket says 172.21.15.2:49448) 2023-11-16T18:24:22.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5eed9d700 1 -- 172.21.15.2:0/631790587 learned_addr learned my addr 172.21.15.2:0/631790587 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5edd9b700 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368ee0 0x55a11c3761e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5eed9d700 1 -- 172.21.15.2:0/631790587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368ee0 msgr2=0x55a11c3761e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5eed9d700 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368ee0 0x55a11c3761e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368050 0x55a11c375ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5eed9d700 1 -- 172.21.15.2:0/631790587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368050 msgr2=0x55a11c375ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5eed9d700 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368050 0x55a11c375ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5eed9d700 1 -- 172.21.15.2:0/631790587 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e000edb0 con 0x55a11c369890 2023-11-16T18:24:22.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368050 0x55a11c375ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:22.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5eed9d700 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 0x55a11c3d06d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009d50 tx=0x7fc5e8004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ce9350051ad6d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5d5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:24:22.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5d5ffb700 1 -- 172.21.15.2:0/631790587 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8014610 con 0x55a11c369890 2023-11-16T18:24:22.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc5d5ffb700 1 -- 172.21.15.2:0/631790587 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc5e8014770 con 0x55a11c369890 2023-11-16T18:24:22.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.366+0000 7fc600afcb80 1 -- 172.21.15.2:0/631790587 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc5e8014ac0 con 0x55a11c369890 2023-11-16T18:24:22.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.367+0000 7fc5d5ffb700 1 -- 172.21.15.2:0/631790587 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5e8014930 con 0x55a11c369890 2023-11-16T18:24:22.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.367+0000 7fc600afcb80 10 client.15369 fetch_fsmap learned FSMap version 24 2023-11-16T18:24:22.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.367+0000 7fc600afcb80 1 -- 172.21.15.2:0/631790587 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a11c0cbc10 con 0x55a11c369890 2023-11-16T18:24:22.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.367+0000 7fc5d5ffb700 1 -- 172.21.15.2:0/631790587 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5e8014ac0 con 0x55a11c369890 2023-11-16T18:24:22.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.367+0000 7fc5d5ffb700 1 -- 172.21.15.2:0/631790587 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc5e8028540 con 0x55a11c369890 2023-11-16T18:24:22.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.367+0000 7fc600afcb80 10 client.15369 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:24:22.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.367+0000 7fc600afcb80 10 client.15369 resolve_mds: resolved name 'h' to daemon mds.h 2023-11-16T18:24:22.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.367+0000 7fc600afcb80 20 client.15369 populate_metadata read hostname 'smithi002' 2023-11-16T18:24:22.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.367+0000 7fc600afcb80 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a11c3d59b0 0x55a11c3d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.367+0000 7fc600afcb80 4 client.15369 mds_command: new command op to 24446 tid=0[{"prefix": "status"}] 2023-11-16T18:24:22.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.367+0000 7fc600afcb80 1 -- 172.21.15.2:0/631790587 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a11c283610 con 0x55a11c3d59b0 2023-11-16T18:24:22.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.367+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a11c3d59b0 0x55a11c3d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc5d5ffb700 10 client.15369 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:24:22.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc5ee59c700 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a11c3d59b0 0x55a11c3d2430 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e7519b92a036af42 server_cookie=c48b3f6791b5a483 in_seq=0 out_seq=0 2023-11-16T18:24:22.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc5d5ffb700 1 -- 172.21.15.2:0/631790587 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55a11c283610 con 0x55a11c3d59b0 2023-11-16T18:24:22.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc5d5ffb700 10 client.15369 handle_command_reply: tid=0 2023-11-16T18:24:22.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc600afcb80 1 client.15369 shutdown 2023-11-16T18:24:22.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc600afcb80 1 -- 172.21.15.2:0/631790587 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 msgr2=0x55a11c3d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc600afcb80 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 0x55a11c3d06d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009d50 tx=0x7fc5e8004c60 comp rx=0 tx=0).stop 2023-11-16T18:24:22.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc600afcb80 1 -- 172.21.15.2:0/631790587 shutdown_connections 2023-11-16T18:24:22.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc600afcb80 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a11c3d59b0 0x55a11c3d2430 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc600afcb80 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a11c369890 0x55a11c3d06d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc5d5ffb700 0 client.15369 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:24:22.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc600afcb80 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a11c368050 0x55a11c375ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc600afcb80 1 --2- 172.21.15.2:0/631790587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a11c368ee0 0x55a11c3761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc600afcb80 1 -- 172.21.15.2:0/631790587 >> 172.21.15.2:0/631790587 conn(0x55a11c359310 msgr2=0x55a11c37be90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc600afcb80 1 -- 172.21.15.2:0/631790587 shutdown_connections 2023-11-16T18:24:22.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc600afcb80 1 -- 172.21.15.2:0/631790587 wait complete. 2023-11-16T18:24:22.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.368+0000 7fc600afcb80 20 client.15369 trim_cache size 0 max 16384 2023-11-16T18:24:22.379 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:24:22.379 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "08160fac-84ac-11ee-8dbe-212e2dc638e7", 2023-11-16T18:24:22.379 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 2, 2023-11-16T18:24:22.379 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24446, 2023-11-16T18:24:22.379 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T18:24:22.379 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T18:24:22.379 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T18:24:22.379 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 75.465181731000001, 2023-11-16T18:24:22.379 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 24, 2023-11-16T18:24:22.380 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 88, 2023-11-16T18:24:22.380 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 86, 2023-11-16T18:24:22.380 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 110.011493883 2023-11-16T18:24:22.380 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:24:22.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.369+0000 7fc5f4970700 1 -- 172.21.15.2:0/2940672088 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc5d80a11e0 msgr2=0x7fc5d80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.369+0000 7fc5f4970700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc5d80a11e0 0x7fc5d80a3690 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0005fd0 tx=0x7fc5e0005f00 comp rx=0 tx=0).stop 2023-11-16T18:24:22.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.369+0000 7fc5f4970700 1 -- 172.21.15.2:0/2940672088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5f0100320 msgr2=0x7fc5f019ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.369+0000 7fc5f4970700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5f0100320 0x7fc5f019ac30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4004760 tx=0x7fc5e4009f90 comp rx=0 tx=0).stop 2023-11-16T18:24:22.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.369+0000 7fc5f4970700 1 -- 172.21.15.2:0/2940672088 shutdown_connections 2023-11-16T18:24:22.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.369+0000 7fc5f4970700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fc5d80a11e0 0x7fc5d80a3690 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.369+0000 7fc5f4970700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5f00ff130 0x7fc5f00699f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.369+0000 7fc5f4970700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5f0100cb0 0x7fc5f019d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.369+0000 7fc5f4970700 1 --2- 172.21.15.2:0/2940672088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5f0100320 0x7fc5f019ac30 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.369+0000 7fc5f4970700 1 -- 172.21.15.2:0/2940672088 >> 172.21.15.2:0/2940672088 conn(0x7fc5f00fa710 msgr2=0x7fc5f00fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.369+0000 7fc5f4970700 1 -- 172.21.15.2:0/2940672088 shutdown_connections 2023-11-16T18:24:22.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.369+0000 7fc5f4970700 1 -- 172.21.15.2:0/2940672088 wait complete. 2023-11-16T18:24:22.383 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-11-16T18:24:22.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.509+0000 7f33b4c29700 1 -- 172.21.15.2:0/1712888317 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33b0100d50 msgr2=0x7f33b0103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.509+0000 7f33b4c29700 1 --2- 172.21.15.2:0/1712888317 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33b0100d50 0x7f33b0103130 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f33a0009990 tx=0x7f33a0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.509+0000 7f33b4c29700 1 -- 172.21.15.2:0/1712888317 shutdown_connections 2023-11-16T18:24:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.509+0000 7f33b4c29700 1 --2- 172.21.15.2:0/1712888317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33b0105f90 0x7f33b0108370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.509+0000 7f33b4c29700 1 --2- 172.21.15.2:0/1712888317 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33b0100d50 0x7f33b0103130 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.509+0000 7f33b4c29700 1 --2- 172.21.15.2:0/1712888317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33b0103670 0x7f33b0105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.509+0000 7f33b4c29700 1 -- 172.21.15.2:0/1712888317 >> 172.21.15.2:0/1712888317 conn(0x7f33b00fa770 msgr2=0x7f33b00fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.509+0000 7f33b4c29700 1 -- 172.21.15.2:0/1712888317 shutdown_connections 2023-11-16T18:24:22.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.509+0000 7f33b4c29700 1 -- 172.21.15.2:0/1712888317 wait complete. 2023-11-16T18:24:22.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.510+0000 7f33b4c29700 1 Processor -- start 2023-11-16T18:24:22.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.510+0000 7f33b4c29700 1 -- start start 2023-11-16T18:24:22.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.510+0000 7f33b4c29700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33b0100d50 0x7f33b0195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.510+0000 7f33b4c29700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33b0103670 0x7f33b01964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.510+0000 7f33b4c29700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33b0105f90 0x7f33b019d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.510+0000 7f33b4c29700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b019db30 con 0x7f33b0105f90 2023-11-16T18:24:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.510+0000 7f33b4c29700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b019dc70 con 0x7f33b0100d50 2023-11-16T18:24:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.510+0000 7f33b4c29700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b019de00 con 0x7f33b0103670 2023-11-16T18:24:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.510+0000 7f33aed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33b0105f90 0x7f33b019d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.510+0000 7f33aed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33b0105f90 0x7f33b019d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49462/0 (socket says 172.21.15.2:49462) 2023-11-16T18:24:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.510+0000 7f33aed9d700 1 -- 172.21.15.2:0/203667326 learned_addr learned my addr 172.21.15.2:0/203667326 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33ae59c700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33b0100d50 0x7f33b0195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33aed9d700 1 -- 172.21.15.2:0/203667326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33b0103670 msgr2=0x7f33b01964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33add9b700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33b0103670 0x7f33b01964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33aed9d700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33b0103670 0x7f33b01964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33aed9d700 1 -- 172.21.15.2:0/203667326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33b0100d50 msgr2=0x7f33b0195fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33aed9d700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33b0100d50 0x7f33b0195fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33aed9d700 1 -- 172.21.15.2:0/203667326 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a00095f0 con 0x7f33b0105f90 2023-11-16T18:24:22.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33add9b700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33b0103670 0x7f33b01964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:22.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33ae59c700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33b0100d50 0x7f33b0195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:22.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33aed9d700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33b0105f90 0x7f33b019d560 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f33a800ef30 tx=0x7f33a800c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33a77fe700 1 -- 172.21.15.2:0/203667326 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8018070 con 0x7f33b0105f90 2023-11-16T18:24:22.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33a77fe700 1 -- 172.21.15.2:0/203667326 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33a800f040 con 0x7f33b0105f90 2023-11-16T18:24:22.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33a77fe700 1 -- 172.21.15.2:0/203667326 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8003dc0 con 0x7f33b0105f90 2023-11-16T18:24:22.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33b4c29700 1 -- 172.21.15.2:0/203667326 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b010bc00 con 0x7f33b0105f90 2023-11-16T18:24:22.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.511+0000 7f33b4c29700 1 -- 172.21.15.2:0/203667326 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b010ba70 con 0x7f33b0105f90 2023-11-16T18:24:22.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.514+0000 7f33a77fe700 1 -- 172.21.15.2:0/203667326 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f33a800caa0 con 0x7f33b0105f90 2023-11-16T18:24:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.514+0000 7f33a77fe700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f339c0a11e0 0x7f339c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.514+0000 7f33a77fe700 1 -- 172.21.15.2:0/203667326 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f33a80cc9e0 con 0x7f33b0105f90 2023-11-16T18:24:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.514+0000 7f33ae59c700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f339c0a11e0 0x7f339c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.515+0000 7f33ae59c700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f339c0a11e0 0x7f339c0a3690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f33a0000c00 tx=0x7f33a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.515+0000 7f33c0db5b80 1 Processor -- start 2023-11-16T18:24:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.515+0000 7f33c0db5b80 1 -- start start 2023-11-16T18:24:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.515+0000 7f33c0db5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 0x561d4fd73fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.515+0000 7f33c0db5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd74ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.515+0000 7f33c0db5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd750c0 0x561d4fd75530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.515+0000 7f33c0db5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d4fc79010 con 0x561d4fd74500 2023-11-16T18:24:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.515+0000 7f33c0db5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d4fccb850 con 0x561d4fd73bb0 2023-11-16T18:24:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.515+0000 7f33c0db5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d4fcc4ce0 con 0x561d4fd750c0 2023-11-16T18:24:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.515+0000 7f33add9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd74ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.515+0000 7f33add9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd74ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49468/0 (socket says 172.21.15.2:49468) 2023-11-16T18:24:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.515+0000 7f33add9b700 1 -- 172.21.15.2:0/2074872024 learned_addr learned my addr 172.21.15.2:0/2074872024 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.515+0000 7f33ae59c700 1 --2- 172.21.15.2:0/2074872024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 0x561d4fd73fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.516+0000 7f33aed9d700 1 --2- 172.21.15.2:0/2074872024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd750c0 0x561d4fd75530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.516+0000 7f33add9b700 1 -- 172.21.15.2:0/2074872024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd750c0 msgr2=0x561d4fd75530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.516+0000 7f33add9b700 1 --2- 172.21.15.2:0/2074872024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd750c0 0x561d4fd75530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.516+0000 7f33add9b700 1 -- 172.21.15.2:0/2074872024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 msgr2=0x561d4fd73fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.516+0000 7f33add9b700 1 --2- 172.21.15.2:0/2074872024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 0x561d4fd73fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.516+0000 7f33add9b700 1 -- 172.21.15.2:0/2074872024 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d4fb465e0 con 0x561d4fd74500 2023-11-16T18:24:22.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.516+0000 7f33add9b700 1 --2- 172.21.15.2:0/2074872024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd74ae0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3398009d50 tx=0x7f339800e5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98ce448bb8b5dcc1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.516+0000 7f33a4ff9700 1 -- 172.21.15.2:0/2074872024 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3398009ea0 con 0x561d4fd74500 2023-11-16T18:24:22.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.516+0000 7f33a4ff9700 1 -- 172.21.15.2:0/2074872024 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f339800ed30 con 0x561d4fd74500 2023-11-16T18:24:22.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.516+0000 7f33a4ff9700 1 -- 172.21.15.2:0/2074872024 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33980056a0 con 0x561d4fd74500 2023-11-16T18:24:22.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.517+0000 7f33c0db5b80 1 -- 172.21.15.2:0/2074872024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 msgr2=0x561d4fd74ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.517+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/2074872024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd74ae0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3398009d50 tx=0x7f339800e5d0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.517+0000 7f33c0db5b80 1 -- 172.21.15.2:0/2074872024 shutdown_connections 2023-11-16T18:24:22.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.517+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/2074872024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd750c0 0x561d4fd75530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.517+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/2074872024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 0x561d4fd73fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.517+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/2074872024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd74ae0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.517+0000 7f33c0db5b80 1 -- 172.21.15.2:0/2074872024 >> 172.21.15.2:0/2074872024 conn(0x561d4fd69310 msgr2=0x561d4fd6e620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.517+0000 7f33c0db5b80 1 -- 172.21.15.2:0/2074872024 shutdown_connections 2023-11-16T18:24:22.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.517+0000 7f33c0db5b80 1 -- 172.21.15.2:0/2074872024 wait complete. 2023-11-16T18:24:22.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.517+0000 7f33c0db5b80 1 -- start start 2023-11-16T18:24:22.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.517+0000 7f33c0db5b80 1 Processor -- start 2023-11-16T18:24:22.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33c0db5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd73bb0 0x561d4fd7d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33c0db5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd7dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33c0db5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd750c0 0x561d4fd84af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33c0db5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d4fc66890 con 0x561d4fd74500 2023-11-16T18:24:22.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33c0db5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d4fc66530 con 0x561d4fd750c0 2023-11-16T18:24:22.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33c0db5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d4fb488d0 con 0x561d4fd73bb0 2023-11-16T18:24:22.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33c0db5b80 10 client.0 _ll_register_callbacks cb 0x561d4fc84730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:24:22.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33add9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd7dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33add9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd7dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49474/0 (socket says 172.21.15.2:49474) 2023-11-16T18:24:22.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33add9b700 1 -- 172.21.15.2:0/3643785990 learned_addr learned my addr 172.21.15.2:0/3643785990 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33ae59c700 1 --2- 172.21.15.2:0/3643785990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd73bb0 0x561d4fd7d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33aed9d700 1 --2- 172.21.15.2:0/3643785990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd750c0 0x561d4fd84af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33add9b700 1 -- 172.21.15.2:0/3643785990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd73bb0 msgr2=0x561d4fd7d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33add9b700 1 --2- 172.21.15.2:0/3643785990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd73bb0 0x561d4fd7d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33add9b700 1 -- 172.21.15.2:0/3643785990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd750c0 msgr2=0x561d4fd84af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33add9b700 1 --2- 172.21.15.2:0/3643785990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd750c0 0x561d4fd84af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.518+0000 7f33add9b700 1 -- 172.21.15.2:0/3643785990 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3398009980 con 0x561d4fd74500 2023-11-16T18:24:22.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f33add9b700 1 --2- 172.21.15.2:0/3643785990 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd7dab0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f33980052b0 tx=0x7f3398005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65a10ef4cb63bf8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f3395ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:24:22.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f3395ffb700 1 -- 172.21.15.2:0/3643785990 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3398021070 con 0x561d4fd74500 2023-11-16T18:24:22.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f3395ffb700 1 -- 172.21.15.2:0/3643785990 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f339800eb30 con 0x561d4fd74500 2023-11-16T18:24:22.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f33c0db5b80 1 -- 172.21.15.2:0/3643785990 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561d4fd66ad0 con 0x561d4fd74500 2023-11-16T18:24:22.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f3395ffb700 1 -- 172.21.15.2:0/3643785990 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f339800eea0 con 0x561d4fd74500 2023-11-16T18:24:22.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f33c0db5b80 10 client.15381 fetch_fsmap learned FSMap version 24 2023-11-16T18:24:22.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f33c0db5b80 1 -- 172.21.15.2:0/3643785990 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x561d4fc8b140 con 0x561d4fd74500 2023-11-16T18:24:22.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f3395ffb700 1 -- 172.21.15.2:0/3643785990 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561d4fd66ad0 con 0x561d4fd74500 2023-11-16T18:24:22.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f3395ffb700 1 -- 172.21.15.2:0/3643785990 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f339800c070 con 0x561d4fd74500 2023-11-16T18:24:22.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f33c0db5b80 10 client.15381 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:24:22.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f33c0db5b80 10 client.15381 resolve_mds: resolved name 'i' to daemon mds.i 2023-11-16T18:24:22.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f33c0db5b80 20 client.15381 populate_metadata read hostname 'smithi002' 2023-11-16T18:24:22.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/3643785990 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x561d4fd88790 0x561d4fd87b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f33c0db5b80 4 client.15381 mds_command: new command op to 24460 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:24:22.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.519+0000 7f33c0db5b80 1 -- 172.21.15.2:0/3643785990 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561d4fc87050 con 0x561d4fd88790 2023-11-16T18:24:22.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.520+0000 7f33ae59c700 1 --2- 172.21.15.2:0/3643785990 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x561d4fd88790 0x561d4fd87b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.520+0000 7f3395ffb700 10 client.15381 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:24:22.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.520+0000 7f33ae59c700 1 --2- 172.21.15.2:0/3643785990 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x561d4fd88790 0x561d4fd87b90 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1aa57b07e1cb054c server_cookie=98bafcf6d581ec9b in_seq=0 out_seq=0 2023-11-16T18:24:22.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.521+0000 7f3395ffb700 1 -- 172.21.15.2:0/3643785990 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561d4fc87050 con 0x561d4fd88790 2023-11-16T18:24:22.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.521+0000 7f3395ffb700 10 client.15381 handle_command_reply: tid=0 2023-11-16T18:24:22.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.521+0000 7f33c0db5b80 1 client.15381 shutdown 2023-11-16T18:24:22.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.521+0000 7f33c0db5b80 1 -- 172.21.15.2:0/3643785990 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 msgr2=0x561d4fd7dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.521+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/3643785990 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd7dab0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f33980052b0 tx=0x7f3398005810 comp rx=0 tx=0).stop 2023-11-16T18:24:22.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.521+0000 7f33c0db5b80 1 -- 172.21.15.2:0/3643785990 shutdown_connections 2023-11-16T18:24:22.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.521+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/3643785990 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x561d4fd88790 0x561d4fd87b90 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.521+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/3643785990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd73bb0 0x561d4fd7d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.521+0000 7f3395ffb700 0 client.15381 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:24:22.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.521+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/3643785990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd750c0 0x561d4fd84af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.521+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/3643785990 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd7dab0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.521+0000 7f33c0db5b80 1 -- 172.21.15.2:0/3643785990 >> 172.21.15.2:0/3643785990 conn(0x561d4fd69310 msgr2=0x561d4fd6dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.522+0000 7f33c0db5b80 1 -- 172.21.15.2:0/3643785990 shutdown_connections 2023-11-16T18:24:22.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.522+0000 7f33c0db5b80 1 -- 172.21.15.2:0/3643785990 wait complete. 2023-11-16T18:24:22.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.522+0000 7f33c0db5b80 20 client.15381 trim_cache size 0 max 16384 2023-11-16T18:24:22.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.539+0000 7f33c0db5b80 1 Processor -- start 2023-11-16T18:24:22.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.539+0000 7f33c0db5b80 1 -- start start 2023-11-16T18:24:22.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.539+0000 7f33c0db5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 0x561d4fd91500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.539+0000 7f33c0db5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd74500 0x561d4fd84ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.539+0000 7f33c0db5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd750c0 0x561d4fd87b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.539+0000 7f33c0db5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d4fd759b0 con 0x561d4fd750c0 2023-11-16T18:24:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.539+0000 7f33c0db5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d4fc87050 con 0x561d4fd73bb0 2023-11-16T18:24:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.539+0000 7f33c0db5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d4fbd0ed0 con 0x561d4fd74500 2023-11-16T18:24:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33aed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd750c0 0x561d4fd87b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33aed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd750c0 0x561d4fd87b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49484/0 (socket says 172.21.15.2:49484) 2023-11-16T18:24:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33aed9d700 1 -- 172.21.15.2:0/1443194008 learned_addr learned my addr 172.21.15.2:0/1443194008 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33ae59c700 1 --2- 172.21.15.2:0/1443194008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 0x561d4fd91500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33add9b700 1 --2- 172.21.15.2:0/1443194008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd74500 0x561d4fd84ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33aed9d700 1 -- 172.21.15.2:0/1443194008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd74500 msgr2=0x561d4fd84ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33aed9d700 1 --2- 172.21.15.2:0/1443194008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd74500 0x561d4fd84ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33aed9d700 1 -- 172.21.15.2:0/1443194008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 msgr2=0x561d4fd91500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33aed9d700 1 --2- 172.21.15.2:0/1443194008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 0x561d4fd91500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33aed9d700 1 -- 172.21.15.2:0/1443194008 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d4fc8ae90 con 0x561d4fd750c0 2023-11-16T18:24:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33aed9d700 1 --2- 172.21.15.2:0/1443194008 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd750c0 0x561d4fd87b90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f33a8096480 tx=0x7f33a809f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70cb2fca94322bfa server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33a57fa700 1 -- 172.21.15.2:0/1443194008 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a80b1070 con 0x561d4fd750c0 2023-11-16T18:24:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33a57fa700 1 -- 172.21.15.2:0/1443194008 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33a8099a10 con 0x561d4fd750c0 2023-11-16T18:24:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.540+0000 7f33a57fa700 1 -- 172.21.15.2:0/1443194008 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a80ca2d0 con 0x561d4fd750c0 2023-11-16T18:24:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 -- 172.21.15.2:0/1443194008 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd750c0 msgr2=0x561d4fd87b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/1443194008 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd750c0 0x561d4fd87b90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f33a8096480 tx=0x7f33a809f040 comp rx=0 tx=0).stop 2023-11-16T18:24:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 -- 172.21.15.2:0/1443194008 shutdown_connections 2023-11-16T18:24:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/1443194008 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd750c0 0x561d4fd87b90 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/1443194008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 0x561d4fd91500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/1443194008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd74500 0x561d4fd84ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 -- 172.21.15.2:0/1443194008 >> 172.21.15.2:0/1443194008 conn(0x561d4fd69310 msgr2=0x561d4fd7c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 -- 172.21.15.2:0/1443194008 shutdown_connections 2023-11-16T18:24:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 -- 172.21.15.2:0/1443194008 wait complete. 2023-11-16T18:24:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 -- start start 2023-11-16T18:24:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 Processor -- start 2023-11-16T18:24:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 0x561d4fd7f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd7fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd750c0 0x561d4fddc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d4fd84c90 con 0x561d4fd74500 2023-11-16T18:24:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d4fd84e00 con 0x561d4fd73bb0 2023-11-16T18:24:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d4fddcb70 con 0x561d4fd750c0 2023-11-16T18:24:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33c0db5b80 10 client.0 _ll_register_callbacks cb 0x7f33a0005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:24:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.541+0000 7f33add9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd7fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33add9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd7fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49488/0 (socket says 172.21.15.2:49488) 2023-11-16T18:24:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33add9b700 1 -- 172.21.15.2:0/248129090 learned_addr learned my addr 172.21.15.2:0/248129090 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33ae59c700 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 0x561d4fd7f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33aed9d700 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd750c0 0x561d4fddc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33add9b700 1 -- 172.21.15.2:0/248129090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd750c0 msgr2=0x561d4fddc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33add9b700 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd750c0 0x561d4fddc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33add9b700 1 -- 172.21.15.2:0/248129090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 msgr2=0x561d4fd7f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33add9b700 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 0x561d4fd7f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33add9b700 1 -- 172.21.15.2:0/248129090 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a809ae70 con 0x561d4fd74500 2023-11-16T18:24:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33aed9d700 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd750c0 0x561d4fddc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:22.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33add9b700 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd7fcb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3398000c00 tx=0x7f33980056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=186f10d7e5b9c127 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f3395ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:24:22.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f3395ffb700 1 -- 172.21.15.2:0/248129090 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3398021040 con 0x561d4fd74500 2023-11-16T18:24:22.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f3395ffb700 1 -- 172.21.15.2:0/248129090 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3398000e40 con 0x561d4fd74500 2023-11-16T18:24:22.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33c0db5b80 1 -- 172.21.15.2:0/248129090 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f339800ed30 con 0x561d4fd74500 2023-11-16T18:24:22.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f3395ffb700 1 -- 172.21.15.2:0/248129090 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3398023310 con 0x561d4fd74500 2023-11-16T18:24:22.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33c0db5b80 10 client.15387 fetch_fsmap learned FSMap version 24 2023-11-16T18:24:22.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f33c0db5b80 1 -- 172.21.15.2:0/248129090 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x561d4fadbc10 con 0x561d4fd74500 2023-11-16T18:24:22.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.542+0000 7f3395ffb700 1 -- 172.21.15.2:0/248129090 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f339800ed30 con 0x561d4fd74500 2023-11-16T18:24:22.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.543+0000 7f3395ffb700 1 -- 172.21.15.2:0/248129090 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f339800e2f0 con 0x561d4fd74500 2023-11-16T18:24:22.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.543+0000 7f33c0db5b80 10 client.15387 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:24:22.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.543+0000 7f33c0db5b80 10 client.15387 resolve_mds: resolved name 'i' to daemon mds.i 2023-11-16T18:24:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.543+0000 7f33c0db5b80 20 client.15387 populate_metadata read hostname 'smithi002' 2023-11-16T18:24:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.543+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x561d4fde1360 0x561d4fdddd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.543+0000 7f33c0db5b80 4 client.15387 mds_command: new command op to 24460 tid=0[{"prefix": "status"}] 2023-11-16T18:24:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.543+0000 7f33c0db5b80 1 -- 172.21.15.2:0/248129090 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561d4fc93610 con 0x561d4fde1360 2023-11-16T18:24:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.543+0000 7f33ae59c700 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x561d4fde1360 0x561d4fdddd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.543+0000 7f3395ffb700 10 client.15387 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:24:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.543+0000 7f33ae59c700 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x561d4fde1360 0x561d4fdddd40 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b42840ecacde8fd9 server_cookie=92f1a5a2a19e6abe in_seq=0 out_seq=0 2023-11-16T18:24:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f3395ffb700 1 -- 172.21.15.2:0/248129090 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x561d4fc93610 con 0x561d4fde1360 2023-11-16T18:24:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f3395ffb700 10 client.15387 handle_command_reply: tid=0 2023-11-16T18:24:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f33c0db5b80 1 client.15387 shutdown 2023-11-16T18:24:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f33c0db5b80 1 -- 172.21.15.2:0/248129090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 msgr2=0x561d4fd7fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd7fcb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3398000c00 tx=0x7f33980056f0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f33c0db5b80 1 -- 172.21.15.2:0/248129090 shutdown_connections 2023-11-16T18:24:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x561d4fde1360 0x561d4fdddd40 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d4fd750c0 0x561d4fddc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d4fd73bb0 0x561d4fd7f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f3395ffb700 0 client.15387 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:24:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f33c0db5b80 1 --2- 172.21.15.2:0/248129090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561d4fd74500 0x561d4fd7fcb0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f33c0db5b80 1 -- 172.21.15.2:0/248129090 >> 172.21.15.2:0/248129090 conn(0x561d4fd69310 msgr2=0x561d4fd87b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f33c0db5b80 1 -- 172.21.15.2:0/248129090 shutdown_connections 2023-11-16T18:24:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f33c0db5b80 1 -- 172.21.15.2:0/248129090 wait complete. 2023-11-16T18:24:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.544+0000 7f33c0db5b80 20 client.15387 trim_cache size 0 max 16384 2023-11-16T18:24:22.566 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:24:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "08160fac-84ac-11ee-8dbe-212e2dc638e7", 2023-11-16T18:24:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 1, 2023-11-16T18:24:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24460, 2023-11-16T18:24:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T18:24:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T18:24:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T18:24:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 76.648928705000003, 2023-11-16T18:24:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 24, 2023-11-16T18:24:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 88, 2023-11-16T18:24:22.567 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 86, 2023-11-16T18:24:22.567 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 107.918917537 2023-11-16T18:24:22.567 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:24:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.545+0000 7f33b4c29700 1 -- 172.21.15.2:0/203667326 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f339c0a11e0 msgr2=0x7f339c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.545+0000 7f33b4c29700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f339c0a11e0 0x7f339c0a3690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f33a0000c00 tx=0x7f33a0005fb0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.545+0000 7f33b4c29700 1 -- 172.21.15.2:0/203667326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33b0105f90 msgr2=0x7f33b019d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.545+0000 7f33b4c29700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33b0105f90 0x7f33b019d560 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f33a800ef30 tx=0x7f33a800c450 comp rx=0 tx=0).stop 2023-11-16T18:24:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.545+0000 7f33b4c29700 1 -- 172.21.15.2:0/203667326 shutdown_connections 2023-11-16T18:24:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.545+0000 7f33b4c29700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f339c0a11e0 0x7f339c0a3690 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.545+0000 7f33b4c29700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33b0105f90 0x7f33b019d560 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.545+0000 7f33b4c29700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33b0100d50 0x7f33b0195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.545+0000 7f33b4c29700 1 --2- 172.21.15.2:0/203667326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33b0103670 0x7f33b01964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.545+0000 7f33b4c29700 1 -- 172.21.15.2:0/203667326 >> 172.21.15.2:0/203667326 conn(0x7f33b00fa770 msgr2=0x7f33b0101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.545+0000 7f33b4c29700 1 -- 172.21.15.2:0/203667326 shutdown_connections 2023-11-16T18:24:22.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.545+0000 7f33b4c29700 1 -- 172.21.15.2:0/203667326 wait complete. 2023-11-16T18:24:22.569 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-16T18:24:22.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.695+0000 7fcb4325e700 1 -- 172.21.15.2:0/2942046683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb3c100250 msgr2=0x7fcb3c100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.695+0000 7fcb4325e700 1 --2- 172.21.15.2:0/2942046683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb3c100250 0x7fcb3c100660 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009990 tx=0x7fcb30009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.695+0000 7fcb4325e700 1 -- 172.21.15.2:0/2942046683 shutdown_connections 2023-11-16T18:24:22.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.695+0000 7fcb4325e700 1 --2- 172.21.15.2:0/2942046683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb3c101dd0 0x7fcb3c108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.695+0000 7fcb4325e700 1 --2- 172.21.15.2:0/2942046683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb3c100250 0x7fcb3c100660 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.695+0000 7fcb4325e700 1 --2- 172.21.15.2:0/2942046683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb3c101440 0x7fcb3c101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.695+0000 7fcb4325e700 1 -- 172.21.15.2:0/2942046683 >> 172.21.15.2:0/2942046683 conn(0x7fcb3c0fb850 msgr2=0x7fcb3c0fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb4325e700 1 -- 172.21.15.2:0/2942046683 shutdown_connections 2023-11-16T18:24:22.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb4325e700 1 -- 172.21.15.2:0/2942046683 wait complete. 2023-11-16T18:24:22.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb4325e700 1 Processor -- start 2023-11-16T18:24:22.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb4325e700 1 -- start start 2023-11-16T18:24:22.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb4325e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb3c100250 0x7fcb3c195f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb4325e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb3c101440 0x7fcb3c196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb4325e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb3c101dd0 0x7fcb3c19d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb4325e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c19dac0 con 0x7fcb3c101dd0 2023-11-16T18:24:22.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb4325e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c19dc00 con 0x7fcb3c101440 2023-11-16T18:24:22.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb4325e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c19dd90 con 0x7fcb3c100250 2023-11-16T18:24:22.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb417fb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb3c101dd0 0x7fcb3c19d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb417fb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb3c101dd0 0x7fcb3c19d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49504/0 (socket says 172.21.15.2:49504) 2023-11-16T18:24:22.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb417fb700 1 -- 172.21.15.2:0/2668397211 learned_addr learned my addr 172.21.15.2:0/2668397211 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.696+0000 7fcb3bfff700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb3c101440 0x7fcb3c196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.697+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb3c100250 0x7fcb3c195f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.697+0000 7fcb417fb700 1 -- 172.21.15.2:0/2668397211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb3c100250 msgr2=0x7fcb3c195f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.697+0000 7fcb417fb700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb3c100250 0x7fcb3c195f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.697+0000 7fcb417fb700 1 -- 172.21.15.2:0/2668397211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb3c101440 msgr2=0x7fcb3c196480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.697+0000 7fcb417fb700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb3c101440 0x7fcb3c196480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.697+0000 7fcb417fb700 1 -- 172.21.15.2:0/2668397211 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb300095f0 con 0x7fcb3c101dd0 2023-11-16T18:24:22.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.697+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb3c100250 0x7fcb3c195f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:22.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.697+0000 7fcb417fb700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb3c101dd0 0x7fcb3c19d4f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcb34007ee0 tx=0x7fcb3400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.697+0000 7fcb39ffb700 1 -- 172.21.15.2:0/2668397211 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34017070 con 0x7fcb3c101dd0 2023-11-16T18:24:22.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.697+0000 7fcb39ffb700 1 -- 172.21.15.2:0/2668397211 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb3400f040 con 0x7fcb3c101dd0 2023-11-16T18:24:22.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.697+0000 7fcb4325e700 1 -- 172.21.15.2:0/2668397211 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3c10bbc0 con 0x7fcb3c101dd0 2023-11-16T18:24:22.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.697+0000 7fcb39ffb700 1 -- 172.21.15.2:0/2668397211 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34005740 con 0x7fcb3c101dd0 2023-11-16T18:24:22.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.697+0000 7fcb4325e700 1 -- 172.21.15.2:0/2668397211 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3c10ba30 con 0x7fcb3c101dd0 2023-11-16T18:24:22.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.699+0000 7fcb39ffb700 1 -- 172.21.15.2:0/2668397211 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fcb3400dc60 con 0x7fcb3c101dd0 2023-11-16T18:24:22.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.700+0000 7fcb39ffb700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcb2c0a11e0 0x7fcb2c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.700+0000 7fcb39ffb700 1 -- 172.21.15.2:0/2668397211 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcb340cc5e0 con 0x7fcb3c101dd0 2023-11-16T18:24:22.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.700+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcb2c0a11e0 0x7fcb2c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.700+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcb2c0a11e0 0x7fcb2c0a3690 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcb30006010 tx=0x7fcb3000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb4f3eab80 1 Processor -- start 2023-11-16T18:24:22.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb4f3eab80 1 -- start start 2023-11-16T18:24:22.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb4f3eab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cf9bb0 0x5619d5cf9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb4f3eab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfa500 0x5619d5cfaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb4f3eab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 0x5619d5cfb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb4f3eab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619d5bff010 con 0x5619d5cfb0c0 2023-11-16T18:24:22.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb4f3eab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619d5c51850 con 0x5619d5cfa500 2023-11-16T18:24:22.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb4f3eab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619d5c4ace0 con 0x5619d5cf9bb0 2023-11-16T18:24:22.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb417fb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 0x5619d5cfb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb417fb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 0x5619d5cfb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49512/0 (socket says 172.21.15.2:49512) 2023-11-16T18:24:22.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb417fb700 1 -- 172.21.15.2:0/323817126 learned_addr learned my addr 172.21.15.2:0/323817126 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb3bfff700 1 --2- 172.21.15.2:0/323817126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfa500 0x5619d5cfaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/323817126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cf9bb0 0x5619d5cf9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb417fb700 1 -- 172.21.15.2:0/323817126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cf9bb0 msgr2=0x5619d5cf9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb417fb700 1 --2- 172.21.15.2:0/323817126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cf9bb0 0x5619d5cf9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb417fb700 1 -- 172.21.15.2:0/323817126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfa500 msgr2=0x5619d5cfaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb417fb700 1 --2- 172.21.15.2:0/323817126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfa500 0x5619d5cfaae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb417fb700 1 -- 172.21.15.2:0/323817126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619d5acc5e0 con 0x5619d5cfb0c0 2023-11-16T18:24:22.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.701+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/323817126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cf9bb0 0x5619d5cf9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:22.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.702+0000 7fcb417fb700 1 --2- 172.21.15.2:0/323817126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 0x5619d5cfb530 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcb34095a20 tx=0x7fcb3400df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170fbac6b7202bd5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.702+0000 7fcb26ffd700 1 -- 172.21.15.2:0/323817126 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34095040 con 0x5619d5cfb0c0 2023-11-16T18:24:22.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.702+0000 7fcb26ffd700 1 -- 172.21.15.2:0/323817126 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb3409d550 con 0x5619d5cfb0c0 2023-11-16T18:24:22.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.702+0000 7fcb26ffd700 1 -- 172.21.15.2:0/323817126 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3409d170 con 0x5619d5cfb0c0 2023-11-16T18:24:22.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.702+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/323817126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 msgr2=0x5619d5cfb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.702+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/323817126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 0x5619d5cfb530 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcb34095a20 tx=0x7fcb3400df60 comp rx=0 tx=0).stop 2023-11-16T18:24:22.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.702+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/323817126 shutdown_connections 2023-11-16T18:24:22.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.702+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/323817126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cf9bb0 0x5619d5cf9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.702+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/323817126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 0x5619d5cfb530 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.703+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/323817126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfa500 0x5619d5cfaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.703+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/323817126 >> 172.21.15.2:0/323817126 conn(0x5619d5cef310 msgr2=0x5619d5cf4620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.703+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/323817126 shutdown_connections 2023-11-16T18:24:22.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.703+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/323817126 wait complete. 2023-11-16T18:24:22.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.703+0000 7fcb4f3eab80 1 -- start start 2023-11-16T18:24:22.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.703+0000 7fcb4f3eab80 1 Processor -- start 2023-11-16T18:24:22.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.703+0000 7fcb4f3eab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cf9bb0 0x5619d5d03610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.711 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.703+0000 7fcb4f3eab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfa500 0x5619d5d03b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.703+0000 7fcb4f3eab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 0x5619d5d0ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.703+0000 7fcb4f3eab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619d5bec890 con 0x5619d5cfb0c0 2023-11-16T18:24:22.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.703+0000 7fcb4f3eab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619d5bec530 con 0x5619d5cf9bb0 2023-11-16T18:24:22.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.703+0000 7fcb4f3eab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619d5ace8d0 con 0x5619d5cfa500 2023-11-16T18:24:22.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.703+0000 7fcb4f3eab80 10 client.0 _ll_register_callbacks cb 0x5619d5c0a730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:24:22.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb417fb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 0x5619d5d0ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb417fb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 0x5619d5d0ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49518/0 (socket says 172.21.15.2:49518) 2023-11-16T18:24:22.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb40ffa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cf9bb0 0x5619d5d03610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb417fb700 1 -- 172.21.15.2:0/2390151622 learned_addr learned my addr 172.21.15.2:0/2390151622 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb3bfff700 1 --2- 172.21.15.2:0/2390151622 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfa500 0x5619d5d03b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb417fb700 1 -- 172.21.15.2:0/2390151622 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfa500 msgr2=0x5619d5d03b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb417fb700 1 --2- 172.21.15.2:0/2390151622 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfa500 0x5619d5d03b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb417fb700 1 -- 172.21.15.2:0/2390151622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cf9bb0 msgr2=0x5619d5d03610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb417fb700 1 --2- 172.21.15.2:0/2390151622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cf9bb0 0x5619d5d03610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb417fb700 1 -- 172.21.15.2:0/2390151622 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3c10ba30 con 0x5619d5cfb0c0 2023-11-16T18:24:22.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb417fb700 1 --2- 172.21.15.2:0/2390151622 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 0x5619d5d0ab90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcb340a1440 tx=0x7fcb340996a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1849aadf37bc1dac server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb1bfff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:24:22.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb1bfff700 1 -- 172.21.15.2:0/2390151622 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb340b5070 con 0x5619d5cfb0c0 2023-11-16T18:24:22.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb1bfff700 1 -- 172.21.15.2:0/2390151622 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcb34084050 con 0x5619d5cfb0c0 2023-11-16T18:24:22.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2390151622 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5619d5cecad0 con 0x5619d5cfb0c0 2023-11-16T18:24:22.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb1bfff700 1 -- 172.21.15.2:0/2390151622 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb34095040 con 0x5619d5cfb0c0 2023-11-16T18:24:22.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb4f3eab80 10 client.15399 fetch_fsmap learned FSMap version 24 2023-11-16T18:24:22.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.704+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2390151622 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5619d5c11140 con 0x5619d5cfb0c0 2023-11-16T18:24:22.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.705+0000 7fcb1bfff700 1 -- 172.21.15.2:0/2390151622 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5619d5cecad0 con 0x5619d5cfb0c0 2023-11-16T18:24:22.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.705+0000 7fcb1bfff700 1 -- 172.21.15.2:0/2390151622 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcb34098a10 con 0x5619d5cfb0c0 2023-11-16T18:24:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.705+0000 7fcb4f3eab80 10 client.15399 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:24:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.705+0000 7fcb4f3eab80 10 client.15399 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T18:24:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.705+0000 7fcb4f3eab80 20 client.15399 populate_metadata read hostname 'smithi002' 2023-11-16T18:24:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.705+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/2390151622 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5619d5d0e2f0 0x5619d5d0da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.705+0000 7fcb4f3eab80 4 client.15399 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:24:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.705+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2390151622 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5619d5c0d050 con 0x5619d5d0e2f0 2023-11-16T18:24:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.705+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/2390151622 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5619d5d0e2f0 0x5619d5d0da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.706+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/2390151622 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5619d5d0e2f0 0x5619d5d0da30 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=551ba2cabe4c4d94 server_cookie=2d5a22cfbbdef04c in_seq=0 out_seq=0 2023-11-16T18:24:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.706+0000 7fcb1bfff700 10 client.15399 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:24:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.706+0000 7fcb1bfff700 1 -- 172.21.15.2:0/2390151622 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5619d5c0d050 con 0x5619d5d0e2f0 2023-11-16T18:24:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.706+0000 7fcb1bfff700 10 client.15399 handle_command_reply: tid=0 2023-11-16T18:24:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.706+0000 7fcb4f3eab80 1 client.15399 shutdown 2023-11-16T18:24:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.707+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2390151622 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 msgr2=0x5619d5d0ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.707+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/2390151622 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 0x5619d5d0ab90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcb340a1440 tx=0x7fcb340996a0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.707+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2390151622 shutdown_connections 2023-11-16T18:24:22.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.707+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/2390151622 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5619d5d0e2f0 0x5619d5d0da30 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.707+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/2390151622 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cfb0c0 0x5619d5d0ab90 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.707+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/2390151622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cf9bb0 0x5619d5d03610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.707+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/2390151622 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfa500 0x5619d5d03b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.707+0000 7fcb1bfff700 0 client.15399 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:24:22.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.707+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2390151622 >> 172.21.15.2:0/2390151622 conn(0x5619d5cef310 msgr2=0x5619d5cf3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.707+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2390151622 shutdown_connections 2023-11-16T18:24:22.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.707+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2390151622 wait complete. 2023-11-16T18:24:22.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.707+0000 7fcb4f3eab80 20 client.15399 trim_cache size 0 max 16384 2023-11-16T18:24:22.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.725+0000 7fcb4f3eab80 1 Processor -- start 2023-11-16T18:24:22.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.725+0000 7fcb4f3eab80 1 -- start start 2023-11-16T18:24:22.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.725+0000 7fcb4f3eab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 0x5619d5d022a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.725+0000 7fcb4f3eab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfa500 0x5619d5d0aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.725+0000 7fcb4f3eab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfb0c0 0x5619d5cfe050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.725+0000 7fcb4f3eab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619d5bec890 con 0x5619d5cf9bb0 2023-11-16T18:24:22.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.725+0000 7fcb4f3eab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619d5d043a0 con 0x5619d5cfb0c0 2023-11-16T18:24:22.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.725+0000 7fcb4f3eab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619d5c0d050 con 0x5619d5cfa500 2023-11-16T18:24:22.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.725+0000 7fcb40ffa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 0x5619d5d022a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.725+0000 7fcb40ffa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 0x5619d5d022a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49522/0 (socket says 172.21.15.2:49522) 2023-11-16T18:24:22.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.725+0000 7fcb40ffa700 1 -- 172.21.15.2:0/4037780468 learned_addr learned my addr 172.21.15.2:0/4037780468 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.725+0000 7fcb3bfff700 1 --2- 172.21.15.2:0/4037780468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfa500 0x5619d5d0aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.725+0000 7fcb417fb700 1 --2- 172.21.15.2:0/4037780468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfb0c0 0x5619d5cfe050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.726+0000 7fcb40ffa700 1 -- 172.21.15.2:0/4037780468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfa500 msgr2=0x5619d5d0aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.726+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/4037780468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfa500 0x5619d5d0aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.726+0000 7fcb40ffa700 1 -- 172.21.15.2:0/4037780468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfb0c0 msgr2=0x5619d5cfe050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.726+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/4037780468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfb0c0 0x5619d5cfe050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.726+0000 7fcb40ffa700 1 -- 172.21.15.2:0/4037780468 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619d5cf48c0 con 0x5619d5cf9bb0 2023-11-16T18:24:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.726+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/4037780468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 0x5619d5d022a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c1012a0 tx=0x7fcb3001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15aa5c87f098af30 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.726+0000 7fcb277fe700 1 -- 172.21.15.2:0/4037780468 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3000bdd0 con 0x5619d5cf9bb0 2023-11-16T18:24:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.726+0000 7fcb277fe700 1 -- 172.21.15.2:0/4037780468 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb3001e040 con 0x5619d5cf9bb0 2023-11-16T18:24:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.726+0000 7fcb277fe700 1 -- 172.21.15.2:0/4037780468 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb30005310 con 0x5619d5cf9bb0 2023-11-16T18:24:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.726+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/4037780468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 msgr2=0x5619d5d022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.726+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/4037780468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 0x5619d5d022a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c1012a0 tx=0x7fcb3001f000 comp rx=0 tx=0).stop 2023-11-16T18:24:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.726+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/4037780468 shutdown_connections 2023-11-16T18:24:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/4037780468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfb0c0 0x5619d5cfe050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/4037780468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 0x5619d5d022a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/4037780468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfa500 0x5619d5d0aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/4037780468 >> 172.21.15.2:0/4037780468 conn(0x5619d5cef310 msgr2=0x5619d5d17020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/4037780468 shutdown_connections 2023-11-16T18:24:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/4037780468 wait complete. 2023-11-16T18:24:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 -- start start 2023-11-16T18:24:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 Processor -- start 2023-11-16T18:24:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 0x5619d5d0ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfa500 0x5619d5d075c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfb0c0 0x5619d5d07b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619d5d081f0 con 0x5619d5cf9bb0 2023-11-16T18:24:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619d5d62150 con 0x5619d5cfa500 2023-11-16T18:24:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619d5d62310 con 0x5619d5cfb0c0 2023-11-16T18:24:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb4f3eab80 10 client.0 _ll_register_callbacks cb 0x7fcb2800a620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:24:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb40ffa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 0x5619d5d0ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb40ffa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 0x5619d5d0ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49528/0 (socket says 172.21.15.2:49528) 2023-11-16T18:24:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb40ffa700 1 -- 172.21.15.2:0/2904978727 learned_addr learned my addr 172.21.15.2:0/2904978727 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.727+0000 7fcb3bfff700 1 --2- 172.21.15.2:0/2904978727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfa500 0x5619d5d075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb417fb700 1 --2- 172.21.15.2:0/2904978727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfb0c0 0x5619d5d07b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb40ffa700 1 -- 172.21.15.2:0/2904978727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfb0c0 msgr2=0x5619d5d07b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/2904978727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfb0c0 0x5619d5d07b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb40ffa700 1 -- 172.21.15.2:0/2904978727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfa500 msgr2=0x5619d5d075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/2904978727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfa500 0x5619d5d075c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb40ffa700 1 -- 172.21.15.2:0/2904978727 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3c108fc0 con 0x5619d5cf9bb0 2023-11-16T18:24:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb40ffa700 1 --2- 172.21.15.2:0/2904978727 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 0x5619d5d0ab20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fcb30005f50 tx=0x7fcb30005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd5f4e51c2a9b19 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb1bfff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:24:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb1bfff700 1 -- 172.21.15.2:0/2904978727 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb300050b0 con 0x5619d5cf9bb0 2023-11-16T18:24:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb1bfff700 1 -- 172.21.15.2:0/2904978727 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcb30005210 con 0x5619d5cf9bb0 2023-11-16T18:24:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2904978727 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcb3409d550 con 0x5619d5cf9bb0 2023-11-16T18:24:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb1bfff700 1 -- 172.21.15.2:0/2904978727 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb300053d0 con 0x5619d5cf9bb0 2023-11-16T18:24:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb4f3eab80 10 client.15405 fetch_fsmap learned FSMap version 24 2023-11-16T18:24:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2904978727 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5619d5a78d90 con 0x5619d5cf9bb0 2023-11-16T18:24:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.728+0000 7fcb1bfff700 1 -- 172.21.15.2:0/2904978727 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb3409d550 con 0x5619d5cf9bb0 2023-11-16T18:24:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.729+0000 7fcb1bfff700 1 -- 172.21.15.2:0/2904978727 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcb300039b0 con 0x5619d5cf9bb0 2023-11-16T18:24:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.729+0000 7fcb4f3eab80 10 client.15405 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:24:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.729+0000 7fcb4f3eab80 10 client.15405 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T18:24:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.729+0000 7fcb4f3eab80 20 client.15405 populate_metadata read hostname 'smithi002' 2023-11-16T18:24:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.729+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/2904978727 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5619d5d66c80 0x5619d5d63610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.729+0000 7fcb4f3eab80 4 client.15405 mds_command: new command op to 24473 tid=0[{"prefix": "status"}] 2023-11-16T18:24:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.729+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2904978727 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5619d5c19610 con 0x5619d5d66c80 2023-11-16T18:24:22.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.729+0000 7fcb3bfff700 1 --2- 172.21.15.2:0/2904978727 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5619d5d66c80 0x5619d5d63610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:22.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb3bfff700 1 --2- 172.21.15.2:0/2904978727 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5619d5d66c80 0x5619d5d63610 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=648715faaf4d3945 server_cookie=361bc6f35ad6d7b0 in_seq=0 out_seq=0 2023-11-16T18:24:22.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb1bfff700 10 client.15405 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:24:22.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb1bfff700 1 -- 172.21.15.2:0/2904978727 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5619d5c19610 con 0x5619d5d66c80 2023-11-16T18:24:22.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb1bfff700 10 client.15405 handle_command_reply: tid=0 2023-11-16T18:24:22.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb4f3eab80 1 client.15405 shutdown 2023-11-16T18:24:22.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2904978727 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 msgr2=0x5619d5d0ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/2904978727 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 0x5619d5d0ab20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fcb30005f50 tx=0x7fcb30005a00 comp rx=0 tx=0).stop 2023-11-16T18:24:22.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2904978727 shutdown_connections 2023-11-16T18:24:22.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/2904978727 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5619d5d66c80 0x5619d5d63610 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/2904978727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619d5cfb0c0 0x5619d5d07b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb1bfff700 0 client.15405 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:24:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/2904978727 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5619d5cf9bb0 0x5619d5d0ab20 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb4f3eab80 1 --2- 172.21.15.2:0/2904978727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619d5cfa500 0x5619d5d075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2904978727 >> 172.21.15.2:0/2904978727 conn(0x5619d5cef310 msgr2=0x5619d5d022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2904978727 shutdown_connections 2023-11-16T18:24:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb4f3eab80 1 -- 172.21.15.2:0/2904978727 wait complete. 2023-11-16T18:24:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.730+0000 7fcb4f3eab80 20 client.15405 trim_cache size 0 max 16384 2023-11-16T18:24:22.739 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:24:22.740 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "08160fac-84ac-11ee-8dbe-212e2dc638e7", 2023-11-16T18:24:22.740 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2023-11-16T18:24:22.740 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24473, 2023-11-16T18:24:22.740 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T18:24:22.740 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T18:24:22.740 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T18:24:22.740 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 95.921785427000003, 2023-11-16T18:24:22.740 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 24, 2023-11-16T18:24:22.740 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 88, 2023-11-16T18:24:22.740 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 86, 2023-11-16T18:24:22.740 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 101.521959208 2023-11-16T18:24:22.741 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:24:22.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.731+0000 7fcb4325e700 1 -- 172.21.15.2:0/2668397211 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcb2c0a11e0 msgr2=0x7fcb2c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.731+0000 7fcb4325e700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcb2c0a11e0 0x7fcb2c0a3690 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcb30006010 tx=0x7fcb3000b540 comp rx=0 tx=0).stop 2023-11-16T18:24:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.731+0000 7fcb4325e700 1 -- 172.21.15.2:0/2668397211 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb3c101dd0 msgr2=0x7fcb3c19d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.731+0000 7fcb4325e700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb3c101dd0 0x7fcb3c19d4f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcb34007ee0 tx=0x7fcb3400d3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.731+0000 7fcb4325e700 1 -- 172.21.15.2:0/2668397211 shutdown_connections 2023-11-16T18:24:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.731+0000 7fcb4325e700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcb2c0a11e0 0x7fcb2c0a3690 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.731+0000 7fcb4325e700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb3c100250 0x7fcb3c195f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.731+0000 7fcb4325e700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb3c101dd0 0x7fcb3c19d4f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.731+0000 7fcb4325e700 1 --2- 172.21.15.2:0/2668397211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb3c101440 0x7fcb3c196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.731+0000 7fcb4325e700 1 -- 172.21.15.2:0/2668397211 >> 172.21.15.2:0/2668397211 conn(0x7fcb3c0fb850 msgr2=0x7fcb3c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.731+0000 7fcb4325e700 1 -- 172.21.15.2:0/2668397211 shutdown_connections 2023-11-16T18:24:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:22.731+0000 7fcb4325e700 1 -- 172.21.15.2:0/2668397211 wait complete. 2023-11-16T18:24:22.744 INFO:teuthology.run_tasks:Running task kclient... 2023-11-16T18:24:22.757 INFO:tasks.kclient:Mounting kernel clients... 2023-11-16T18:24:22.757 INFO:tasks.kclient:config is {'client.0': {}} 2023-11-16T18:24:22.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:22.758 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2023-11-16T18:24:22.776 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-11-16T18:24:22.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:22.776 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-16T18:24:22.841 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:22 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2457315983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:22.841 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:22 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/636506978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:22.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:22.850 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link delete ceph-brx 2023-11-16T18:24:22.947 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2023-11-16T18:24:22.948 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-16T18:24:22.948 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-16T18:24:22.948 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph fs ls 2023-11-16T18:24:23.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:22 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2457315983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:23.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:22 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/636506978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:23.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:22 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2457315983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:23.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:22 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/636506978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:23.604 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:24:24.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:23 smithi188 ceph-mon[100639]: pgmap v310: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T18:24:24.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:23 smithi002 ceph-mon[98807]: pgmap v310: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T18:24:24.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:23 smithi040 ceph-mon[100251]: pgmap v310: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T18:24:24.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.800+0000 7f59746d2700 1 -- 172.21.15.2:0/2641545562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f596c104500 msgr2=0x7f596c10c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:32.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.800+0000 7f59746d2700 1 --2- 172.21.15.2:0/2641545562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f596c104500 0x7f596c10c080 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f596400b0d0 tx=0x7f596400b3e0 comp rx=0 tx=0).stop 2023-11-16T18:24:32.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.800+0000 7f59746d2700 1 -- 172.21.15.2:0/2641545562 shutdown_connections 2023-11-16T18:24:32.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.800+0000 7f59746d2700 1 --2- 172.21.15.2:0/2641545562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f596c0ff460 0x7f596c0ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:32.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.800+0000 7f59746d2700 1 --2- 172.21.15.2:0/2641545562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f596c104500 0x7f596c10c080 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.800+0000 7f59746d2700 1 --2- 172.21.15.2:0/2641545562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f596c0ffe00 0x7f596c103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.800+0000 7f59746d2700 1 -- 172.21.15.2:0/2641545562 >> 172.21.15.2:0/2641545562 conn(0x7f596c0770a0 msgr2=0x7f596c0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.800+0000 7f59746d2700 1 -- 172.21.15.2:0/2641545562 shutdown_connections 2023-11-16T18:24:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.800+0000 7f59746d2700 1 -- 172.21.15.2:0/2641545562 wait complete. 2023-11-16T18:24:32.570 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:25 smithi040 ceph-mon[100251]: pgmap v311: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:32.570 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:25 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2054410594' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T18:24:32.570 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:27 smithi040 ceph-mon[100251]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T18:24:32.570 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:29 smithi040 ceph-mon[100251]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:24:32.570 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:31 smithi040 ceph-mon[100251]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:32.570 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:25 smithi188 ceph-mon[100639]: pgmap v311: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:32.570 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:25 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2054410594' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T18:24:32.570 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:27 smithi188 ceph-mon[100639]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T18:24:32.570 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:29 smithi188 ceph-mon[100639]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:24:32.570 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:31 smithi188 ceph-mon[100639]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:32.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.801+0000 7f59746d2700 1 Processor -- start 2023-11-16T18:24:32.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.801+0000 7f59746d2700 1 -- start start 2023-11-16T18:24:32.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.801+0000 7f59746d2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f596c0ff460 0x7f596c196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:32.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.801+0000 7f59746d2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f596c0ffe00 0x7f596c196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:32.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.801+0000 7f59746d2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f596c104500 0x7f596c19aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:32.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.801+0000 7f59746d2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c19b4f0 con 0x7f596c104500 2023-11-16T18:24:32.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.801+0000 7f59746d2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c19b660 con 0x7f596c0ff460 2023-11-16T18:24:32.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.801+0000 7f59746d2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c19b820 con 0x7f596c0ffe00 2023-11-16T18:24:32.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.801+0000 7f5972c6f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f596c104500 0x7f596c19aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:32.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.801+0000 7f5972c6f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f596c104500 0x7f596c19aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49546/0 (socket says 172.21.15.2:49546) 2023-11-16T18:24:32.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.801+0000 7f5972c6f700 1 -- 172.21.15.2:0/2054410594 learned_addr learned my addr 172.21.15.2:0/2054410594 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:32.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.801+0000 7f597246e700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f596c0ff460 0x7f596c196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.801+0000 7f5971c6d700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f596c0ffe00 0x7f596c196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.802+0000 7f5972c6f700 1 -- 172.21.15.2:0/2054410594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f596c0ffe00 msgr2=0x7f596c196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.802+0000 7f5972c6f700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f596c0ffe00 0x7f596c196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.802+0000 7f5972c6f700 1 -- 172.21.15.2:0/2054410594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f596c0ff460 msgr2=0x7f596c196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.802+0000 7f5972c6f700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f596c0ff460 0x7f596c196530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.802+0000 7f5972c6f700 1 -- 172.21.15.2:0/2054410594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5964009d00 con 0x7f596c104500 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.802+0000 7f5972c6f700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f596c104500 0x7f596c19aee0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5964007870 tx=0x7f59640078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.802+0000 7f595f7fe700 1 -- 172.21.15.2:0/2054410594 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596400e040 con 0x7f596c104500 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.802+0000 7f595f7fe700 1 -- 172.21.15.2:0/2054410594 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5964003e10 con 0x7f596c104500 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.802+0000 7f59746d2700 1 -- 172.21.15.2:0/2054410594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c10f240 con 0x7f596c104500 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.802+0000 7f595f7fe700 1 -- 172.21.15.2:0/2054410594 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596401ba10 con 0x7f596c104500 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.802+0000 7f59746d2700 1 -- 172.21.15.2:0/2054410594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c10f0b0 con 0x7f596c104500 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.803+0000 7f59746d2700 1 -- 172.21.15.2:0/2054410594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c10f3d0 con 0x7f596c104500 2023-11-16T18:24:32.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.805+0000 7f595f7fe700 1 -- 172.21.15.2:0/2054410594 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f59640050d0 con 0x7f596c104500 2023-11-16T18:24:32.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.805+0000 7f595f7fe700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f59580a11e0 0x7f59580a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:32.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.805+0000 7f597246e700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f59580a11e0 0x7f59580a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:32.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.805+0000 7f595f7fe700 1 -- 172.21.15.2:0/2054410594 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f59640cebc0 con 0x7f596c104500 2023-11-16T18:24:32.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.806+0000 7f597246e700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f59580a11e0 0x7f59580a3690 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f596800ac50 tx=0x7f596800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:32.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:24.809+0000 7f595f7fe700 1 -- 172.21.15.2:0/2054410594 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5964097ed0 con 0x7f596c104500 2023-11-16T18:24:32.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.051+0000 7f59746d2700 1 -- 172.21.15.2:0/2054410594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f596c197990 con 0x7f596c104500 2023-11-16T18:24:32.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.051+0000 7f595f7fe700 1 -- 172.21.15.2:0/2054410594 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+88 (secure 0 0 0) 0x7f596409b4f0 con 0x7f596c104500 2023-11-16T18:24:32.617 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-16T18:24:32.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.053+0000 7f59746d2700 1 -- 172.21.15.2:0/2054410594 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f59580a11e0 msgr2=0x7f59580a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:32.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.053+0000 7f59746d2700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f59580a11e0 0x7f59580a3690 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f596800ac50 tx=0x7f596800a380 comp rx=0 tx=0).stop 2023-11-16T18:24:32.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.053+0000 7f59746d2700 1 -- 172.21.15.2:0/2054410594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f596c104500 msgr2=0x7f596c19aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:32.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.053+0000 7f59746d2700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f596c104500 0x7f596c19aee0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5964007870 tx=0x7f59640078a0 comp rx=0 tx=0).stop 2023-11-16T18:24:32.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.053+0000 7f59746d2700 1 -- 172.21.15.2:0/2054410594 shutdown_connections 2023-11-16T18:24:32.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.053+0000 7f59746d2700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f59580a11e0 0x7f59580a3690 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:32.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.053+0000 7f59746d2700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f596c104500 0x7f596c19aee0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:32.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.053+0000 7f59746d2700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f596c0ff460 0x7f596c196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:32.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.053+0000 7f59746d2700 1 --2- 172.21.15.2:0/2054410594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f596c0ffe00 0x7f596c196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:32.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.053+0000 7f59746d2700 1 -- 172.21.15.2:0/2054410594 >> 172.21.15.2:0/2054410594 conn(0x7f596c0770a0 msgr2=0x7f596c0685d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:32.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.053+0000 7f59746d2700 1 -- 172.21.15.2:0/2054410594 shutdown_connections 2023-11-16T18:24:32.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:25.053+0000 7f59746d2700 1 -- 172.21.15.2:0/2054410594 wait complete. 2023-11-16T18:24:32.662 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-16T18:24:32.663 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-16T18:24:32.663 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2023-11-16T18:24:32.663 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-16T18:24:32.663 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-16T18:24:32.663 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-16T18:24:32.663 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-16T18:24:32.663 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-16T18:24:32.663 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-16T18:24:32.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:32.663 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2023-11-16T18:24:32.664 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:25 smithi002 ceph-mon[98807]: pgmap v311: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:32.664 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:25 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2054410594' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T18:24:32.664 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:27 smithi002 ceph-mon[98807]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T18:24:32.665 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:29 smithi002 ceph-mon[98807]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:24:32.665 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:31 smithi002 ceph-mon[98807]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:32.688 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-16T18:24:32.689 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-16T18:24:32.689 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2023-11-16T18:24:32.689 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2023-11-16T18:24:32.689 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2023-11-16T18:24:32.689 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2023-11-16T18:24:32.689 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T18:24:32.690 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2023-11-16T18:24:32.690 INFO:teuthology.orchestra.run.smithi002.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-16T18:24:32.690 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2023-11-16T18:24:32.690 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-16T18:24:32.690 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 41688sec preferred_lft 41688sec 2023-11-16T18:24:32.691 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link 2023-11-16T18:24:32.691 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2023-11-16T18:24:32.691 INFO:teuthology.orchestra.run.smithi002.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T18:24:32.691 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2023-11-16T18:24:32.691 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T18:24:32.691 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2023-11-16T18:24:32.692 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-16T18:24:32.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:32.693 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T18:24:32.693 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T18:24:32.693 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add name ceph-brx type bridge 2023-11-16T18:24:32.693 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr flush dev ceph-brx 2023-11-16T18:24:32.693 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set ceph-brx up 2023-11-16T18:24:32.695 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-16T18:24:32.695 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T18:24:32.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:32.825 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-16T18:24:32.855 INFO:teuthology.orchestra.run.smithi002.stdout:1 2023-11-16T18:24:32.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:32.856 DEBUG:teuthology.orchestra.run.smithi002:> route 2023-11-16T18:24:32.918 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2023-11-16T18:24:32.918 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-16T18:24:32.918 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-16T18:24:32.919 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-16T18:24:32.919 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-16T18:24:32.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:32.919 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T18:24:32.920 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T18:24:32.920 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-16T18:24:32.920 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-16T18:24:32.920 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-16T18:24:32.920 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T18:24:33.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:33.028 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2023-11-16T18:24:33.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:33.051 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list-id 2023-11-16T18:24:33.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:33.108 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T18:24:33.108 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T18:24:33.109 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-16T18:24:33.109 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-11-16T18:24:33.109 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T18:24:33.205 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-11-16T18:24:33.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:33.206 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T18:24:33.206 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T18:24:33.206 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-11-16T18:24:33.206 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-16T18:24:33.206 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-11-16T18:24:33.207 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-11-16T18:24:33.207 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-11-16T18:24:33.207 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T18:24:33.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:33.397 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T18:24:33.397 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T18:24:33.397 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2023-11-16T18:24:33.397 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set dev brx.0 master ceph-brx 2023-11-16T18:24:33.397 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T18:24:33.525 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T18:24:33.568 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-16T18:24:33.585 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-16T18:24:33.586 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T18:24:33.586 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-16T18:24:33.643 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2023-11-16T18:24:33.643 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2023-11-16T18:24:33.643 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e 2023-11-16T18:24:33.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:33.643 DEBUG:teuthology.orchestra.run.smithi002:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2023-11-16T18:24:33.729 INFO:teuthology.orchestra.run.smithi002.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2023-11-16T18:24:33.730 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2023-11-16T18:24:33.903 INFO:teuthology.orchestra.run.smithi002.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2023-11-16T18:24:33.903 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2023-11-16T18:24:33.903 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: resolved to: "172.21.15.2:3300,172.21.15.40:3300,172.21.15.188:3300" 2023-11-16T18:24:33.903 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: trying mount with new device syntax: 0@08160fac-84ac-11ee-8dbe-212e2dc638e7.cephfs=/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e 2023-11-16T18:24:33.903 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.2:3300/172.21.15.40:3300/172.21.15.188:3300" will pass to kernel 2023-11-16T18:24:33.903 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: v2 mount support check returned 2 2023-11-16T18:24:33.904 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: kernel does not support v2 syntax 2023-11-16T18:24:33.904 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.2:3300,172.21.15.40:3300,172.21.15.188:3300:/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e 2023-11-16T18:24:33.904 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mds_namespace=cephfs,fsid=08160fac-84ac-11ee-8dbe-212e2dc638e7" will pass to kernel 2023-11-16T18:24:33.922 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-16T18:24:33.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:33.923 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-16T18:24:33.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:33.955 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-16T18:24:34.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:33 smithi188 ceph-mon[100639]: pgmap v315: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T18:24:34.058 INFO:teuthology.orchestra.run.smithi002.stdout:client25006 2023-11-16T18:24:34.058 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:24:34.059 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T18:24:34.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:34.078 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-16T18:24:34.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:33 smithi002 ceph-mon[98807]: pgmap v315: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T18:24:34.161 INFO:teuthology.orchestra.run.smithi002.stdout:client25006 2023-11-16T18:24:34.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:34.162 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-16T18:24:34.221 INFO:teuthology.orchestra.run.smithi002.stdout:client25006 2023-11-16T18:24:34.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:24:34.222 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7.client25006/status') 2023-11-16T18:24:34.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:33 smithi040 ceph-mon[100251]: pgmap v315: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T18:24:34.290 INFO:teuthology.orchestra.run.smithi002.stdout:instance: client.25006 (3)172.21.15.2:0/4148164237 2023-11-16T18:24:34.290 INFO:teuthology.orchestra.run.smithi002.stdout:blocklisted: false 2023-11-16T18:24:34.291 INFO:teuthology.orchestra.run.smithi002.stderr:0+1 records in 2023-11-16T18:24:34.291 INFO:teuthology.orchestra.run.smithi002.stderr:0+1 records out 2023-11-16T18:24:34.291 INFO:teuthology.orchestra.run.smithi002.stderr:70 bytes copied, 2.2991e-05 s, 3.0 MB/s 2023-11-16T18:24:34.292 INFO:teuthology.run_tasks:Running task check-counter... 2023-11-16T18:24:34.302 INFO:teuthology.run_tasks:Running task fwd_scrub... 2023-11-16T18:24:34.350 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:24:34.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.474+0000 7f45fc438700 1 -- 172.21.15.2:0/364851207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45f40ff1c0 msgr2=0x7f45f40ff5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:34.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.474+0000 7f45fc438700 1 --2- 172.21.15.2:0/364851207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45f40ff1c0 0x7f45f40ff5d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009990 tx=0x7f45f0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 -- 172.21.15.2:0/364851207 shutdown_connections 2023-11-16T18:24:34.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 --2- 172.21.15.2:0/364851207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45f4100d40 0x7f45f4107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 --2- 172.21.15.2:0/364851207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45f40ff1c0 0x7f45f40ff5d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 --2- 172.21.15.2:0/364851207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41003b0 0x7f45f4100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 -- 172.21.15.2:0/364851207 >> 172.21.15.2:0/364851207 conn(0x7f45f40fa7c0 msgr2=0x7f45f40fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:34.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 -- 172.21.15.2:0/364851207 shutdown_connections 2023-11-16T18:24:34.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 -- 172.21.15.2:0/364851207 wait complete. 2023-11-16T18:24:34.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 Processor -- start 2023-11-16T18:24:34.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 -- start start 2023-11-16T18:24:34.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45f40ff1c0 0x7f45f41960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:34.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41003b0 0x7f45f41965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:34.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45f4100d40 0x7f45f419d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:34.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f419dc30 con 0x7f45f40ff1c0 2023-11-16T18:24:34.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f419dd70 con 0x7f45f41003b0 2023-11-16T18:24:34.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.475+0000 7f45fc438700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f419df00 con 0x7f45f4100d40 2023-11-16T18:24:34.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45fa1d4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45f40ff1c0 0x7f45f41960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:34.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45fa1d4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45f40ff1c0 0x7f45f41960b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49560/0 (socket says 172.21.15.2:49560) 2023-11-16T18:24:34.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45fa1d4700 1 -- 172.21.15.2:0/2711044657 learned_addr learned my addr 172.21.15.2:0/2711044657 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:34.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45fa9d5700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45f4100d40 0x7f45f419d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:34.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45fa1d4700 1 -- 172.21.15.2:0/2711044657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45f4100d40 msgr2=0x7f45f419d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:34.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45fa1d4700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45f4100d40 0x7f45f419d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45fa1d4700 1 -- 172.21.15.2:0/2711044657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41003b0 msgr2=0x7f45f41965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:34.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45f99d3700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41003b0 0x7f45f41965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:34.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45fa1d4700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41003b0 0x7f45f41965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45fa1d4700 1 -- 172.21.15.2:0/2711044657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f00095f0 con 0x7f45f40ff1c0 2023-11-16T18:24:34.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45fa9d5700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45f4100d40 0x7f45f419d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:34.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45fa1d4700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45f40ff1c0 0x7f45f41960b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f45f0006010 tx=0x7f45f0005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:34.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45e77fe700 1 -- 172.21.15.2:0/2711044657 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f001d070 con 0x7f45f40ff1c0 2023-11-16T18:24:34.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45e77fe700 1 -- 172.21.15.2:0/2711044657 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45f00040e0 con 0x7f45f40ff1c0 2023-11-16T18:24:34.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45fc438700 1 -- 172.21.15.2:0/2711044657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f410ab20 con 0x7f45f40ff1c0 2023-11-16T18:24:34.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45e77fe700 1 -- 172.21.15.2:0/2711044657 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f0022e70 con 0x7f45f40ff1c0 2023-11-16T18:24:34.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.476+0000 7f45fc438700 1 -- 172.21.15.2:0/2711044657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f410a990 con 0x7f45f40ff1c0 2023-11-16T18:24:34.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.477+0000 7f45fc438700 1 -- 172.21.15.2:0/2711044657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f4067af0 con 0x7f45f40ff1c0 2023-11-16T18:24:34.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.478+0000 7f45e77fe700 1 -- 172.21.15.2:0/2711044657 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f45f0022570 con 0x7f45f40ff1c0 2023-11-16T18:24:34.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.479+0000 7f45e77fe700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45e00a11e0 0x7f45e00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:34.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.479+0000 7f45e77fe700 1 -- 172.21.15.2:0/2711044657 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f45f00ce500 con 0x7f45f40ff1c0 2023-11-16T18:24:34.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.479+0000 7f45f99d3700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45e00a11e0 0x7f45e00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:34.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.479+0000 7f45f99d3700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45e00a11e0 0x7f45e00a3690 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f45e8005950 tx=0x7f45e80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:34.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.482+0000 7f45e77fe700 1 -- 172.21.15.2:0/2711044657 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f45f0097870 con 0x7f45f40ff1c0 2023-11-16T18:24:34.686 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.684+0000 7f45fc438700 1 -- 172.21.15.2:0/2711044657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45f419ea70 con 0x7f45f40ff1c0 2023-11-16T18:24:34.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.686+0000 7f45e77fe700 1 -- 172.21.15.2:0/2711044657 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24383 (secure 0 0 0) 0x7f45f009ae90 con 0x7f45f40ff1c0 2023-11-16T18:24:34.688 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:34.689 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":88,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:23:39.497670+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:24:34.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.688+0000 7f45fc438700 1 -- 172.21.15.2:0/2711044657 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45e00a11e0 msgr2=0x7f45e00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:34.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.688+0000 7f45fc438700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45e00a11e0 0x7f45e00a3690 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f45e8005950 tx=0x7f45e80058e0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.688+0000 7f45fc438700 1 -- 172.21.15.2:0/2711044657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45f40ff1c0 msgr2=0x7f45f41960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:34.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.688+0000 7f45fc438700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45f40ff1c0 0x7f45f41960b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f45f0006010 tx=0x7f45f0005ef0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.688+0000 7f45fc438700 1 -- 172.21.15.2:0/2711044657 shutdown_connections 2023-11-16T18:24:34.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.688+0000 7f45fc438700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f45e00a11e0 0x7f45e00a3690 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.688+0000 7f45fc438700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45f4100d40 0x7f45f419d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.688+0000 7f45fc438700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45f40ff1c0 0x7f45f41960b0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.688+0000 7f45fc438700 1 --2- 172.21.15.2:0/2711044657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41003b0 0x7f45f41965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.688+0000 7f45fc438700 1 -- 172.21.15.2:0/2711044657 >> 172.21.15.2:0/2711044657 conn(0x7f45f40fa7c0 msgr2=0x7f45f4067de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:34.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.688+0000 7f45fc438700 1 -- 172.21.15.2:0/2711044657 shutdown_connections 2023-11-16T18:24:34.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.688+0000 7f45fc438700 1 -- 172.21.15.2:0/2711044657 wait complete. 2023-11-16T18:24:34.705 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T18:22:44.902889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T18:22:46.187275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T18:22:49.539148+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T18:24:34.706 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T18:24:34.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.834+0000 7fee549d9700 1 -- 172.21.15.2:0/1174841910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee50100d30 msgr2=0x7fee501075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:34.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.834+0000 7fee549d9700 1 --2- 172.21.15.2:0/1174841910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee50100d30 0x7fee501075f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fee4800b0a0 tx=0x7fee4800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.835+0000 7fee549d9700 1 -- 172.21.15.2:0/1174841910 shutdown_connections 2023-11-16T18:24:34.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.835+0000 7fee549d9700 1 --2- 172.21.15.2:0/1174841910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee50100d30 0x7fee501075f0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.835+0000 7fee549d9700 1 --2- 172.21.15.2:0/1174841910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee500ff1b0 0x7fee500ff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.835+0000 7fee549d9700 1 --2- 172.21.15.2:0/1174841910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee501003a0 0x7fee501007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.835+0000 7fee549d9700 1 -- 172.21.15.2:0/1174841910 >> 172.21.15.2:0/1174841910 conn(0x7fee500fa7b0 msgr2=0x7fee500fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:34.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.835+0000 7fee549d9700 1 -- 172.21.15.2:0/1174841910 shutdown_connections 2023-11-16T18:24:34.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.835+0000 7fee549d9700 1 -- 172.21.15.2:0/1174841910 wait complete. 2023-11-16T18:24:34.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.835+0000 7fee549d9700 1 Processor -- start 2023-11-16T18:24:34.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.835+0000 7fee549d9700 1 -- start start 2023-11-16T18:24:34.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee549d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee500ff1b0 0x7fee50196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:34.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee549d9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee501003a0 0x7fee501965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:34.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee549d9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee50100d30 0x7fee5019d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:34.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee549d9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5019dbe0 con 0x7fee500ff1b0 2023-11-16T18:24:34.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee549d9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5019dd20 con 0x7fee501003a0 2023-11-16T18:24:34.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee549d9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5019deb0 con 0x7fee50100d30 2023-11-16T18:24:34.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee4e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee500ff1b0 0x7fee50196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:34.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee4e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee500ff1b0 0x7fee50196060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49574/0 (socket says 172.21.15.2:49574) 2023-11-16T18:24:34.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee4e59c700 1 -- 172.21.15.2:0/859307637 learned_addr learned my addr 172.21.15.2:0/859307637 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:34.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee4ed9d700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee50100d30 0x7fee5019d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:34.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee4dd9b700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee501003a0 0x7fee501965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:34.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee4e59c700 1 -- 172.21.15.2:0/859307637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee50100d30 msgr2=0x7fee5019d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:34.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee4e59c700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee50100d30 0x7fee5019d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee4e59c700 1 -- 172.21.15.2:0/859307637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee501003a0 msgr2=0x7fee501965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:34.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee4e59c700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee501003a0 0x7fee501965a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:34.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee4e59c700 1 -- 172.21.15.2:0/859307637 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee48009d00 con 0x7fee500ff1b0 2023-11-16T18:24:34.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee4ed9d700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee50100d30 0x7fee5019d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:34.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee4e59c700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee500ff1b0 0x7fee50196060 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fee4000d980 tx=0x7fee4000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:34.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee3f7fe700 1 -- 172.21.15.2:0/859307637 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4000de60 con 0x7fee500ff1b0 2023-11-16T18:24:34.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee3f7fe700 1 -- 172.21.15.2:0/859307637 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee400044e0 con 0x7fee500ff1b0 2023-11-16T18:24:34.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee3f7fe700 1 -- 172.21.15.2:0/859307637 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40003be0 con 0x7fee500ff1b0 2023-11-16T18:24:34.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.836+0000 7fee549d9700 1 -- 172.21.15.2:0/859307637 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5010ab30 con 0x7fee500ff1b0 2023-11-16T18:24:34.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.837+0000 7fee549d9700 1 -- 172.21.15.2:0/859307637 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5010a9a0 con 0x7fee500ff1b0 2023-11-16T18:24:34.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.837+0000 7fee549d9700 1 -- 172.21.15.2:0/859307637 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5019e600 con 0x7fee500ff1b0 2023-11-16T18:24:34.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.839+0000 7fee3f7fe700 1 -- 172.21.15.2:0/859307637 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fee40009bb0 con 0x7fee500ff1b0 2023-11-16T18:24:34.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.839+0000 7fee3f7fe700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fee380a1260 0x7fee380a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:34.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.839+0000 7fee3f7fe700 1 -- 172.21.15.2:0/859307637 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fee400cce50 con 0x7fee500ff1b0 2023-11-16T18:24:34.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.839+0000 7fee4dd9b700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fee380a1260 0x7fee380a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:34.844 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.840+0000 7fee4dd9b700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fee380a1260 0x7fee380a3710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fee44005950 tx=0x7fee440058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:34.844 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:34.843+0000 7fee3f7fe700 1 -- 172.21.15.2:0/859307637 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fee400961b0 con 0x7fee500ff1b0 2023-11-16T18:24:35.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.045+0000 7fee549d9700 1 -- 172.21.15.2:0/859307637 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fee5019ebb0 con 0x7fee500ff1b0 2023-11-16T18:24:35.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.045+0000 7fee3f7fe700 1 -- 172.21.15.2:0/859307637 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fee40009940 con 0x7fee500ff1b0 2023-11-16T18:24:35.047 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:24:35.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.047+0000 7fee549d9700 1 -- 172.21.15.2:0/859307637 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fee380a1260 msgr2=0x7fee380a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.047+0000 7fee549d9700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fee380a1260 0x7fee380a3710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fee44005950 tx=0x7fee440058e0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.047+0000 7fee549d9700 1 -- 172.21.15.2:0/859307637 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee500ff1b0 msgr2=0x7fee50196060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.047+0000 7fee549d9700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee500ff1b0 0x7fee50196060 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fee4000d980 tx=0x7fee4000dd40 comp rx=0 tx=0).stop 2023-11-16T18:24:35.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.047+0000 7fee549d9700 1 -- 172.21.15.2:0/859307637 shutdown_connections 2023-11-16T18:24:35.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.047+0000 7fee549d9700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fee380a1260 0x7fee380a3710 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.047+0000 7fee549d9700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee50100d30 0x7fee5019d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.047+0000 7fee549d9700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee500ff1b0 0x7fee50196060 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.047+0000 7fee549d9700 1 --2- 172.21.15.2:0/859307637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee501003a0 0x7fee501965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.047+0000 7fee549d9700 1 -- 172.21.15.2:0/859307637 >> 172.21.15.2:0/859307637 conn(0x7fee500fa7b0 msgr2=0x7fee50069050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:35.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.047+0000 7fee549d9700 1 -- 172.21.15.2:0/859307637 shutdown_connections 2023-11-16T18:24:35.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.048+0000 7fee549d9700 1 -- 172.21.15.2:0/859307637 wait complete. 2023-11-16T18:24:35.061 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:34 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2711044657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:35.062 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T18:24:35.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.191+0000 7f20532f9700 1 -- 172.21.15.2:0/3732077580 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c105e80 msgr2=0x7f204c1082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.191+0000 7f20532f9700 1 --2- 172.21.15.2:0/3732077580 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c105e80 0x7f204c1082e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f204400b0a0 tx=0x7f204400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.191+0000 7f20532f9700 1 -- 172.21.15.2:0/3732077580 shutdown_connections 2023-11-16T18:24:35.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.191+0000 7f20532f9700 1 --2- 172.21.15.2:0/3732077580 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f204c100c40 0x7f204c103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.191+0000 7f20532f9700 1 --2- 172.21.15.2:0/3732077580 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c105e80 0x7f204c1082e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.191+0000 7f20532f9700 1 --2- 172.21.15.2:0/3732077580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c103560 0x7f204c105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.191+0000 7f20532f9700 1 -- 172.21.15.2:0/3732077580 >> 172.21.15.2:0/3732077580 conn(0x7f204c0fa6e0 msgr2=0x7f204c0fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.191+0000 7f20532f9700 1 -- 172.21.15.2:0/3732077580 shutdown_connections 2023-11-16T18:24:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.191+0000 7f20532f9700 1 -- 172.21.15.2:0/3732077580 wait complete. 2023-11-16T18:24:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f20532f9700 1 Processor -- start 2023-11-16T18:24:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f20532f9700 1 -- start start 2023-11-16T18:24:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f20532f9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c100c40 0x7f204c1005b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f20532f9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f204c103560 0x7f204c0febf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f20532f9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c105e80 0x7f204c0ff130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f20532f9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c0ff8d0 con 0x7f204c100c40 2023-11-16T18:24:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f20532f9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c0ffa40 con 0x7f204c105e80 2023-11-16T18:24:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f20532f9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c1a0040 con 0x7f204c103560 2023-11-16T18:24:35.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f2051095700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c100c40 0x7f204c1005b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:35.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f2051095700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c100c40 0x7f204c1005b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49588/0 (socket says 172.21.15.2:49588) 2023-11-16T18:24:35.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f2051095700 1 -- 172.21.15.2:0/2953054814 learned_addr learned my addr 172.21.15.2:0/2953054814 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:35.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f2051896700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c105e80 0x7f204c0ff130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:35.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f2050894700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f204c103560 0x7f204c0febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:35.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f2051095700 1 -- 172.21.15.2:0/2953054814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f204c103560 msgr2=0x7f204c0febf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f2051095700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f204c103560 0x7f204c0febf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f2051095700 1 -- 172.21.15.2:0/2953054814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c105e80 msgr2=0x7f204c0ff130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f2051095700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c105e80 0x7f204c0ff130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f2051095700 1 -- 172.21.15.2:0/2953054814 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2044009d00 con 0x7f204c100c40 2023-11-16T18:24:35.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.192+0000 7f2051095700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c100c40 0x7f204c1005b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f203c00e9a0 tx=0x7f203c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:35.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.193+0000 7f20427fc700 1 -- 172.21.15.2:0/2953054814 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c009790 con 0x7f204c100c40 2023-11-16T18:24:35.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.193+0000 7f20427fc700 1 -- 172.21.15.2:0/2953054814 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f203c00cbd0 con 0x7f204c100c40 2023-11-16T18:24:35.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.193+0000 7f20532f9700 1 -- 172.21.15.2:0/2953054814 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204c10bb80 con 0x7f204c100c40 2023-11-16T18:24:35.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.193+0000 7f20427fc700 1 -- 172.21.15.2:0/2953054814 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c018680 con 0x7f204c100c40 2023-11-16T18:24:35.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.193+0000 7f20532f9700 1 -- 172.21.15.2:0/2953054814 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204c10b9f0 con 0x7f204c100c40 2023-11-16T18:24:35.199 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.194+0000 7f20532f9700 1 -- 172.21.15.2:0/2953054814 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f204c0fc250 con 0x7f204c100c40 2023-11-16T18:24:35.199 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.195+0000 7f20427fc700 1 -- 172.21.15.2:0/2953054814 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f203c00cd40 con 0x7f204c100c40 2023-11-16T18:24:35.199 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.195+0000 7f20427fc700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f20380a1260 0x7f20380a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:35.199 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.196+0000 7f2050894700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f20380a1260 0x7f20380a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:35.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.196+0000 7f20427fc700 1 -- 172.21.15.2:0/2953054814 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f203c014070 con 0x7f204c100c40 2023-11-16T18:24:35.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.196+0000 7f2050894700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f20380a1260 0x7f20380a3710 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2048005950 tx=0x7f20480058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:35.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.199+0000 7f20427fc700 1 -- 172.21.15.2:0/2953054814 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f203c096520 con 0x7f204c100c40 2023-11-16T18:24:35.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:34 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2711044657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:35.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:34 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2711044657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.401+0000 7f20532f9700 1 -- 172.21.15.2:0/2953054814 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f204c061700 con 0x7f204c100c40 2023-11-16T18:24:35.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.401+0000 7f20427fc700 1 -- 172.21.15.2:0/2953054814 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f203c099b40 con 0x7f204c100c40 2023-11-16T18:24:35.406 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:24:35.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.403+0000 7f20532f9700 1 -- 172.21.15.2:0/2953054814 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f20380a1260 msgr2=0x7f20380a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.403+0000 7f20532f9700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f20380a1260 0x7f20380a3710 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2048005950 tx=0x7f20480058e0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.403+0000 7f20532f9700 1 -- 172.21.15.2:0/2953054814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c100c40 msgr2=0x7f204c1005b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.403+0000 7f20532f9700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c100c40 0x7f204c1005b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f203c00e9a0 tx=0x7f203c00ecb0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.403+0000 7f20532f9700 1 -- 172.21.15.2:0/2953054814 shutdown_connections 2023-11-16T18:24:35.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.403+0000 7f20532f9700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f20380a1260 0x7f20380a3710 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.403+0000 7f20532f9700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c105e80 0x7f204c0ff130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.403+0000 7f20532f9700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c100c40 0x7f204c1005b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.403+0000 7f20532f9700 1 --2- 172.21.15.2:0/2953054814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f204c103560 0x7f204c0febf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.403+0000 7f20532f9700 1 -- 172.21.15.2:0/2953054814 >> 172.21.15.2:0/2953054814 conn(0x7f204c0fa6e0 msgr2=0x7f204c1018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.403+0000 7f20532f9700 1 -- 172.21.15.2:0/2953054814 shutdown_connections 2023-11-16T18:24:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.403+0000 7f20532f9700 1 -- 172.21.15.2:0/2953054814 wait complete. 2023-11-16T18:24:35.418 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T18:24:35.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.545+0000 7fcc85a0d700 1 -- 172.21.15.2:0/4002129631 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc80100dc0 msgr2=0x7fcc801031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.545+0000 7fcc85a0d700 1 --2- 172.21.15.2:0/4002129631 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc80100dc0 0x7fcc801031a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009990 tx=0x7fcc74009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.545+0000 7fcc85a0d700 1 -- 172.21.15.2:0/4002129631 shutdown_connections 2023-11-16T18:24:35.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.545+0000 7fcc85a0d700 1 --2- 172.21.15.2:0/4002129631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80106000 0x7fcc801083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.545+0000 7fcc85a0d700 1 --2- 172.21.15.2:0/4002129631 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc80100dc0 0x7fcc801031a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.545+0000 7fcc85a0d700 1 --2- 172.21.15.2:0/4002129631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc801036e0 0x7fcc80105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.545+0000 7fcc85a0d700 1 -- 172.21.15.2:0/4002129631 >> 172.21.15.2:0/4002129631 conn(0x7fcc800fa800 msgr2=0x7fcc800fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:35.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.545+0000 7fcc85a0d700 1 -- 172.21.15.2:0/4002129631 shutdown_connections 2023-11-16T18:24:35.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.545+0000 7fcc85a0d700 1 -- 172.21.15.2:0/4002129631 wait complete. 2023-11-16T18:24:35.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.545+0000 7fcc85a0d700 1 Processor -- start 2023-11-16T18:24:35.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.545+0000 7fcc85a0d700 1 -- start start 2023-11-16T18:24:35.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc85a0d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80100dc0 0x7fcc80195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:35.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc85a0d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc801036e0 0x7fcc80196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:35.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc85a0d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc80106000 0x7fcc8019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:35.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc85a0d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8019db70 con 0x7fcc801036e0 2023-11-16T18:24:35.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc85a0d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8019dcb0 con 0x7fcc80100dc0 2023-11-16T18:24:35.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc85a0d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8019de40 con 0x7fcc80106000 2023-11-16T18:24:35.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc7e7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc801036e0 0x7fcc80196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:35.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc7e7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc801036e0 0x7fcc80196530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49602/0 (socket says 172.21.15.2:49602) 2023-11-16T18:24:35.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc7e7fc700 1 -- 172.21.15.2:0/495597508 learned_addr learned my addr 172.21.15.2:0/495597508 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:35.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc7f7fe700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc80106000 0x7fcc8019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:35.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc7effd700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80100dc0 0x7fcc80195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:35.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc7e7fc700 1 -- 172.21.15.2:0/495597508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc80106000 msgr2=0x7fcc8019d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc7e7fc700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc80106000 0x7fcc8019d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc7e7fc700 1 -- 172.21.15.2:0/495597508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80100dc0 msgr2=0x7fcc80195ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc7e7fc700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80100dc0 0x7fcc80195ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc7e7fc700 1 -- 172.21.15.2:0/495597508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc740095f0 con 0x7fcc801036e0 2023-11-16T18:24:35.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc7effd700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80100dc0 0x7fcc80195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:35.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc7e7fc700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc801036e0 0x7fcc80196530 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000e9a0 tx=0x7fcc7000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:35.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc84a0b700 1 -- 172.21.15.2:0/495597508 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70009790 con 0x7fcc801036e0 2023-11-16T18:24:35.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc84a0b700 1 -- 172.21.15.2:0/495597508 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc7000cbd0 con 0x7fcc801036e0 2023-11-16T18:24:35.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc85a0d700 1 -- 172.21.15.2:0/495597508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8010bc70 con 0x7fcc801036e0 2023-11-16T18:24:35.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc84a0b700 1 -- 172.21.15.2:0/495597508 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70018680 con 0x7fcc801036e0 2023-11-16T18:24:35.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.546+0000 7fcc85a0d700 1 -- 172.21.15.2:0/495597508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8010bae0 con 0x7fcc801036e0 2023-11-16T18:24:35.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.547+0000 7fcc85a0d700 1 -- 172.21.15.2:0/495597508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc800fc370 con 0x7fcc801036e0 2023-11-16T18:24:35.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.549+0000 7fcc84a0b700 1 -- 172.21.15.2:0/495597508 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fcc7000cd40 con 0x7fcc801036e0 2023-11-16T18:24:35.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.550+0000 7fcc84a0b700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcc680a11e0 0x7fcc680a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:35.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.550+0000 7fcc7effd700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcc680a11e0 0x7fcc680a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:35.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.550+0000 7fcc84a0b700 1 -- 172.21.15.2:0/495597508 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcc70014070 con 0x7fcc801036e0 2023-11-16T18:24:35.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.550+0000 7fcc7effd700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcc680a11e0 0x7fcc680a3690 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcc7400b2e0 tx=0x7fcc74000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:35.555 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.553+0000 7fcc84a0b700 1 -- 172.21.15.2:0/495597508 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcc70096880 con 0x7fcc801036e0 2023-11-16T18:24:35.754 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.752+0000 7fcc85a0d700 1 -- 172.21.15.2:0/495597508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcc80067ea0 con 0x7fcc801036e0 2023-11-16T18:24:35.755 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.753+0000 7fcc84a0b700 1 -- 172.21.15.2:0/495597508 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcc70099ea0 con 0x7fcc801036e0 2023-11-16T18:24:35.755 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T18:24:35.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.755+0000 7fcc85a0d700 1 -- 172.21.15.2:0/495597508 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcc680a11e0 msgr2=0x7fcc680a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.755+0000 7fcc85a0d700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcc680a11e0 0x7fcc680a3690 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcc7400b2e0 tx=0x7fcc74000bc0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.755+0000 7fcc85a0d700 1 -- 172.21.15.2:0/495597508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc801036e0 msgr2=0x7fcc80196530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.755+0000 7fcc85a0d700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc801036e0 0x7fcc80196530 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000e9a0 tx=0x7fcc7000ed60 comp rx=0 tx=0).stop 2023-11-16T18:24:35.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.755+0000 7fcc85a0d700 1 -- 172.21.15.2:0/495597508 shutdown_connections 2023-11-16T18:24:35.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.755+0000 7fcc85a0d700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fcc680a11e0 0x7fcc680a3690 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.755+0000 7fcc85a0d700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc80106000 0x7fcc8019d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.755+0000 7fcc85a0d700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80100dc0 0x7fcc80195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.755+0000 7fcc85a0d700 1 --2- 172.21.15.2:0/495597508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc801036e0 0x7fcc80196530 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.755+0000 7fcc85a0d700 1 -- 172.21.15.2:0/495597508 >> 172.21.15.2:0/495597508 conn(0x7fcc800fa800 msgr2=0x7fcc80101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:35.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.755+0000 7fcc85a0d700 1 -- 172.21.15.2:0/495597508 shutdown_connections 2023-11-16T18:24:35.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.755+0000 7fcc85a0d700 1 -- 172.21.15.2:0/495597508 wait complete. 2023-11-16T18:24:35.770 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T18:24:35.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.897+0000 7f1139dc9700 1 -- 172.21.15.2:0/3548091057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1134100e00 msgr2=0x7f11341031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.897+0000 7f1139dc9700 1 --2- 172.21.15.2:0/3548091057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1134100e00 0x7f11341031e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1124009990 tx=0x7f1124009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.897+0000 7f1139dc9700 1 -- 172.21.15.2:0/3548091057 shutdown_connections 2023-11-16T18:24:35.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.897+0000 7f1139dc9700 1 --2- 172.21.15.2:0/3548091057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134106040 0x7f1134108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.897+0000 7f1139dc9700 1 --2- 172.21.15.2:0/3548091057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1134100e00 0x7f11341031e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.897+0000 7f1139dc9700 1 --2- 172.21.15.2:0/3548091057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1134103720 0x7f1134105b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.897+0000 7f1139dc9700 1 -- 172.21.15.2:0/3548091057 >> 172.21.15.2:0/3548091057 conn(0x7f11340fa800 msgr2=0x7f11340fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:35.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.897+0000 7f1139dc9700 1 -- 172.21.15.2:0/3548091057 shutdown_connections 2023-11-16T18:24:35.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.897+0000 7f1139dc9700 1 -- 172.21.15.2:0/3548091057 wait complete. 2023-11-16T18:24:35.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1139dc9700 1 Processor -- start 2023-11-16T18:24:35.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1139dc9700 1 -- start start 2023-11-16T18:24:35.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1139dc9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1134100e00 0x7f1134195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:35.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1139dc9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134103720 0x7f1134196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:35.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1139dc9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1134106040 0x7f113419d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:35.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1139dc9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113419db70 con 0x7f1134106040 2023-11-16T18:24:35.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1139dc9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113419dcb0 con 0x7f1134103720 2023-11-16T18:24:35.908 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1139dc9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113419de40 con 0x7f1134100e00 2023-11-16T18:24:35.908 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1133fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1134106040 0x7f113419d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:35.908 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1133fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1134106040 0x7f113419d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49618/0 (socket says 172.21.15.2:49618) 2023-11-16T18:24:35.908 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f11337fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1134100e00 0x7f1134195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:35.908 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1133fff700 1 -- 172.21.15.2:0/282408252 learned_addr learned my addr 172.21.15.2:0/282408252 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:35.908 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1133fff700 1 -- 172.21.15.2:0/282408252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1134100e00 msgr2=0x7f1134195ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1133fff700 1 --2- 172.21.15.2:0/282408252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1134100e00 0x7f1134195ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1133fff700 1 -- 172.21.15.2:0/282408252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134103720 msgr2=0x7f1134196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:35.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1133fff700 1 --2- 172.21.15.2:0/282408252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134103720 0x7f1134196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:35.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1133fff700 1 -- 172.21.15.2:0/282408252 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11240095f0 con 0x7f1134106040 2023-11-16T18:24:35.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.898+0000 7f1133fff700 1 --2- 172.21.15.2:0/282408252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1134106040 0x7f113419d5a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f112c007ee0 tx=0x7f112c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:35.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.899+0000 7f11317fa700 1 -- 172.21.15.2:0/282408252 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c00ddd0 con 0x7f1134106040 2023-11-16T18:24:35.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.899+0000 7f11317fa700 1 -- 172.21.15.2:0/282408252 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f112c00f040 con 0x7f1134106040 2023-11-16T18:24:35.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.899+0000 7f1139dc9700 1 -- 172.21.15.2:0/282408252 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113410bc90 con 0x7f1134106040 2023-11-16T18:24:35.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.899+0000 7f11317fa700 1 -- 172.21.15.2:0/282408252 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c0048e0 con 0x7f1134106040 2023-11-16T18:24:35.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.899+0000 7f1139dc9700 1 -- 172.21.15.2:0/282408252 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113410bb00 con 0x7f1134106040 2023-11-16T18:24:35.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.900+0000 7f1139dc9700 1 -- 172.21.15.2:0/282408252 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11340fc370 con 0x7f1134106040 2023-11-16T18:24:35.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.901+0000 7f11317fa700 1 -- 172.21.15.2:0/282408252 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f112c003a00 con 0x7f1134106040 2023-11-16T18:24:35.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.901+0000 7f11317fa700 1 --2- 172.21.15.2:0/282408252 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f11200a1260 0x7f11200a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:35.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.902+0000 7f11317fa700 1 -- 172.21.15.2:0/282408252 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f112c0cc690 con 0x7f1134106040 2023-11-16T18:24:35.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.902+0000 7f11337fe700 1 --2- 172.21.15.2:0/282408252 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f11200a1260 0x7f11200a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:35.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.902+0000 7f11337fe700 1 --2- 172.21.15.2:0/282408252 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f11200a1260 0x7f11200a3710 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f112400b5c0 tx=0x7f1124005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:35.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:35.906+0000 7f11317fa700 1 -- 172.21.15.2:0/282408252 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f112c04e730 con 0x7f1134106040 2023-11-16T18:24:36.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:35 smithi002 ceph-mon[98807]: pgmap v316: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:36.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:35 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/859307637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:36.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:35 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2953054814' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:36.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:35 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/495597508' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.107+0000 7f1139dc9700 1 -- 172.21.15.2:0/282408252 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1134061700 con 0x7f1134106040 2023-11-16T18:24:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.107+0000 7f11317fa700 1 -- 172.21.15.2:0/282408252 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7f112c095970 con 0x7f1134106040 2023-11-16T18:24:36.110 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:24:36.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.110+0000 7f1139dc9700 1 -- 172.21.15.2:0/282408252 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f11200a1260 msgr2=0x7f11200a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.110+0000 7f1139dc9700 1 --2- 172.21.15.2:0/282408252 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f11200a1260 0x7f11200a3710 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f112400b5c0 tx=0x7f1124005fb0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.110+0000 7f1139dc9700 1 -- 172.21.15.2:0/282408252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1134106040 msgr2=0x7f113419d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.110+0000 7f1139dc9700 1 --2- 172.21.15.2:0/282408252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1134106040 0x7f113419d5a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f112c007ee0 tx=0x7f112c00d3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.110+0000 7f1139dc9700 1 -- 172.21.15.2:0/282408252 shutdown_connections 2023-11-16T18:24:36.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.110+0000 7f1139dc9700 1 --2- 172.21.15.2:0/282408252 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f11200a1260 0x7f11200a3710 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.110+0000 7f1139dc9700 1 --2- 172.21.15.2:0/282408252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1134100e00 0x7f1134195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.110+0000 7f1139dc9700 1 --2- 172.21.15.2:0/282408252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1134106040 0x7f113419d5a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.110+0000 7f1139dc9700 1 --2- 172.21.15.2:0/282408252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134103720 0x7f1134196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.110+0000 7f1139dc9700 1 -- 172.21.15.2:0/282408252 >> 172.21.15.2:0/282408252 conn(0x7f11340fa800 msgr2=0x7f1134101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:36.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.110+0000 7f1139dc9700 1 -- 172.21.15.2:0/282408252 shutdown_connections 2023-11-16T18:24:36.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.110+0000 7f1139dc9700 1 -- 172.21.15.2:0/282408252 wait complete. 2023-11-16T18:24:36.124 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:24:36.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:35 smithi040 ceph-mon[100251]: pgmap v316: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:36.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:35 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/859307637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:36.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:35 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2953054814' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:36.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:35 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/495597508' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:36.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.251+0000 7f7b9193f700 1 -- 172.21.15.2:0/2247832628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b8c1081a0 msgr2=0x7f7b8c10a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.251+0000 7f7b9193f700 1 --2- 172.21.15.2:0/2247832628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b8c1081a0 0x7f7b8c10a620 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7b8400b0a0 tx=0x7f7b8400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.251+0000 7f7b9193f700 1 -- 172.21.15.2:0/2247832628 shutdown_connections 2023-11-16T18:24:36.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.251+0000 7f7b9193f700 1 --2- 172.21.15.2:0/2247832628 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b8c074ad0 0x7f7b8c072f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.251+0000 7f7b9193f700 1 --2- 172.21.15.2:0/2247832628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b8c1081a0 0x7f7b8c10a620 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.251+0000 7f7b9193f700 1 --2- 172.21.15.2:0/2247832628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b8c0734f0 0x7f7b8c073960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.251+0000 7f7b9193f700 1 -- 172.21.15.2:0/2247832628 >> 172.21.15.2:0/2247832628 conn(0x7f7b8c0fb930 msgr2=0x7f7b8c0fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:36.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.251+0000 7f7b9193f700 1 -- 172.21.15.2:0/2247832628 shutdown_connections 2023-11-16T18:24:36.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.251+0000 7f7b9193f700 1 -- 172.21.15.2:0/2247832628 wait complete. 2023-11-16T18:24:36.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.251+0000 7f7b9193f700 1 Processor -- start 2023-11-16T18:24:36.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b9193f700 1 -- start start 2023-11-16T18:24:36.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b9193f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b8c0734f0 0x7f7b8c198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:36.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b9193f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b8c074ad0 0x7f7b8c198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:36.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b9193f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b8c1081a0 0x7f7b8c19f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:36.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b9193f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c19fec0 con 0x7f7b8c074ad0 2023-11-16T18:24:36.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b9193f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c1a0000 con 0x7f7b8c1081a0 2023-11-16T18:24:36.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b9193f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c1a0190 con 0x7f7b8c0734f0 2023-11-16T18:24:36.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b8a7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b8c074ad0 0x7f7b8c198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:36.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b8a7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b8c074ad0 0x7f7b8c198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49630/0 (socket says 172.21.15.2:49630) 2023-11-16T18:24:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b8a7fc700 1 -- 172.21.15.2:0/3932495711 learned_addr learned my addr 172.21.15.2:0/3932495711 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b8affd700 1 --2- 172.21.15.2:0/3932495711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b8c0734f0 0x7f7b8c198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b8a7fc700 1 -- 172.21.15.2:0/3932495711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b8c0734f0 msgr2=0x7f7b8c198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b8a7fc700 1 --2- 172.21.15.2:0/3932495711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b8c0734f0 0x7f7b8c198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b8a7fc700 1 -- 172.21.15.2:0/3932495711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b8c1081a0 msgr2=0x7f7b8c19f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b8a7fc700 1 --2- 172.21.15.2:0/3932495711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b8c1081a0 0x7f7b8c19f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b8a7fc700 1 -- 172.21.15.2:0/3932495711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b84009d00 con 0x7f7b8c074ad0 2023-11-16T18:24:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.252+0000 7f7b8a7fc700 1 --2- 172.21.15.2:0/3932495711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b8c074ad0 0x7f7b8c198880 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400e9a0 tx=0x7f7b7400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.253+0000 7f7b9093d700 1 -- 172.21.15.2:0/3932495711 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74009790 con 0x7f7b8c074ad0 2023-11-16T18:24:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.253+0000 7f7b9193f700 1 -- 172.21.15.2:0/3932495711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8c10deb0 con 0x7f7b8c074ad0 2023-11-16T18:24:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.253+0000 7f7b9093d700 1 -- 172.21.15.2:0/3932495711 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b7400cbd0 con 0x7f7b8c074ad0 2023-11-16T18:24:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.253+0000 7f7b9093d700 1 -- 172.21.15.2:0/3932495711 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74018680 con 0x7f7b8c074ad0 2023-11-16T18:24:36.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.253+0000 7f7b9193f700 1 -- 172.21.15.2:0/3932495711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8c10dd20 con 0x7f7b8c074ad0 2023-11-16T18:24:36.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.254+0000 7f7b9193f700 1 -- 172.21.15.2:0/3932495711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8c106ff0 con 0x7f7b8c074ad0 2023-11-16T18:24:36.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.255+0000 7f7b9093d700 1 -- 172.21.15.2:0/3932495711 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f7b7400cd40 con 0x7f7b8c074ad0 2023-11-16T18:24:36.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.256+0000 7f7b9093d700 1 --2- 172.21.15.2:0/3932495711 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7b780a11e0 0x7f7b780a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:36.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.256+0000 7f7b9093d700 1 -- 172.21.15.2:0/3932495711 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7b74014070 con 0x7f7b8c074ad0 2023-11-16T18:24:36.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.256+0000 7f7b8affd700 1 --2- 172.21.15.2:0/3932495711 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7b780a11e0 0x7f7b780a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:36.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.256+0000 7f7b8affd700 1 --2- 172.21.15.2:0/3932495711 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7b780a11e0 0x7f7b780a3690 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009c60 tx=0x7f7b7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:36.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.259+0000 7f7b9093d700 1 -- 172.21.15.2:0/3932495711 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7b740965c0 con 0x7f7b8c074ad0 2023-11-16T18:24:36.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:35 smithi188 ceph-mon[100639]: pgmap v316: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T18:24:36.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:35 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/859307637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:36.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:35 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2953054814' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:36.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:35 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/495597508' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:36.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.474+0000 7f7b9193f700 1 -- 172.21.15.2:0/3932495711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b8c02cfe0 con 0x7f7b8c074ad0 2023-11-16T18:24:36.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.475+0000 7f7b9093d700 1 -- 172.21.15.2:0/3932495711 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24383 (secure 0 0 0) 0x7f7b74099be0 con 0x7f7b8c074ad0 2023-11-16T18:24:36.478 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:36.478 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":88,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:23:39.497670+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:24:36.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.477+0000 7f7b9193f700 1 -- 172.21.15.2:0/3932495711 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7b780a11e0 msgr2=0x7f7b780a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.477+0000 7f7b9193f700 1 --2- 172.21.15.2:0/3932495711 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7b780a11e0 0x7f7b780a3690 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009c60 tx=0x7f7b7c009260 comp rx=0 tx=0).stop 2023-11-16T18:24:36.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.477+0000 7f7b9193f700 1 -- 172.21.15.2:0/3932495711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b8c074ad0 msgr2=0x7f7b8c198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.477+0000 7f7b9193f700 1 --2- 172.21.15.2:0/3932495711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b8c074ad0 0x7f7b8c198880 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400e9a0 tx=0x7f7b7400ecb0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.478+0000 7f7b9193f700 1 -- 172.21.15.2:0/3932495711 shutdown_connections 2023-11-16T18:24:36.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.478+0000 7f7b9193f700 1 --2- 172.21.15.2:0/3932495711 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f7b780a11e0 0x7f7b780a3690 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.478+0000 7f7b9193f700 1 --2- 172.21.15.2:0/3932495711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b8c0734f0 0x7f7b8c198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.478+0000 7f7b9193f700 1 --2- 172.21.15.2:0/3932495711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b8c1081a0 0x7f7b8c19f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.478+0000 7f7b9193f700 1 --2- 172.21.15.2:0/3932495711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b8c074ad0 0x7f7b8c198880 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.478+0000 7f7b9193f700 1 -- 172.21.15.2:0/3932495711 >> 172.21.15.2:0/3932495711 conn(0x7f7b8c0fb930 msgr2=0x7f7b8c104130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:36.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.478+0000 7f7b9193f700 1 -- 172.21.15.2:0/3932495711 shutdown_connections 2023-11-16T18:24:36.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.478+0000 7f7b9193f700 1 -- 172.21.15.2:0/3932495711 wait complete. 2023-11-16T18:24:36.492 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T18:22:44.902889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T18:22:46.187275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T18:22:49.539148+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T18:24:36.493 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T18:24:36.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.619+0000 7fa0c2953700 1 -- 172.21.15.2:0/3957820526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc101e10 msgr2=0x7fa0bc1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.619+0000 7fa0c2953700 1 --2- 172.21.15.2:0/3957820526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc101e10 0x7fa0bc1086d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400b0a0 tx=0x7fa0b400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.619+0000 7fa0c2953700 1 -- 172.21.15.2:0/3957820526 shutdown_connections 2023-11-16T18:24:36.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.619+0000 7fa0c2953700 1 --2- 172.21.15.2:0/3957820526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0bc100290 0x7fa0bc1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.619+0000 7fa0c2953700 1 --2- 172.21.15.2:0/3957820526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc101e10 0x7fa0bc1086d0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.619+0000 7fa0c2953700 1 --2- 172.21.15.2:0/3957820526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0bc101480 0x7fa0bc1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.619+0000 7fa0c2953700 1 -- 172.21.15.2:0/3957820526 >> 172.21.15.2:0/3957820526 conn(0x7fa0bc0fb8b0 msgr2=0x7fa0bc0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:36.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.619+0000 7fa0c2953700 1 -- 172.21.15.2:0/3957820526 shutdown_connections 2023-11-16T18:24:36.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.619+0000 7fa0c2953700 1 -- 172.21.15.2:0/3957820526 wait complete. 2023-11-16T18:24:36.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0c2953700 1 Processor -- start 2023-11-16T18:24:36.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0c2953700 1 -- start start 2023-11-16T18:24:36.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0c2953700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0bc100290 0x7fa0bc196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:36.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0c2953700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc101480 0x7fa0bc1965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:36.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0c2953700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0bc101e10 0x7fa0bc19d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:36.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0c2953700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc19dbf0 con 0x7fa0bc101480 2023-11-16T18:24:36.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0c2953700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc19dd30 con 0x7fa0bc101e10 2023-11-16T18:24:36.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0c2953700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc19dec0 con 0x7fa0bc100290 2023-11-16T18:24:36.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0bb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc101480 0x7fa0bc1965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:36.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0bb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc101480 0x7fa0bc1965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49644/0 (socket says 172.21.15.2:49644) 2023-11-16T18:24:36.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0bbfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0bc100290 0x7fa0bc196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:36.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0bb7fe700 1 -- 172.21.15.2:0/1012602797 learned_addr learned my addr 172.21.15.2:0/1012602797 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:36.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0c0ef0700 1 --2- 172.21.15.2:0/1012602797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0bc101e10 0x7fa0bc19d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:36.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0bb7fe700 1 -- 172.21.15.2:0/1012602797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0bc100290 msgr2=0x7fa0bc196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0bb7fe700 1 --2- 172.21.15.2:0/1012602797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0bc100290 0x7fa0bc196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0bb7fe700 1 -- 172.21.15.2:0/1012602797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0bc101e10 msgr2=0x7fa0bc19d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0bb7fe700 1 --2- 172.21.15.2:0/1012602797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0bc101e10 0x7fa0bc19d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.620+0000 7fa0bb7fe700 1 -- 172.21.15.2:0/1012602797 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b4009d00 con 0x7fa0bc101480 2023-11-16T18:24:36.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.621+0000 7fa0bb7fe700 1 --2- 172.21.15.2:0/1012602797 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc101480 0x7fa0bc1965b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000d980 tx=0x7fa0b000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.621+0000 7fa0b97fa700 1 -- 172.21.15.2:0/1012602797 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0004220 con 0x7fa0bc101480 2023-11-16T18:24:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.621+0000 7fa0b97fa700 1 -- 172.21.15.2:0/1012602797 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0b0007640 con 0x7fa0bc101480 2023-11-16T18:24:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.621+0000 7fa0c2953700 1 -- 172.21.15.2:0/1012602797 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0bc10bc20 con 0x7fa0bc101480 2023-11-16T18:24:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.621+0000 7fa0b97fa700 1 -- 172.21.15.2:0/1012602797 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0005610 con 0x7fa0bc101480 2023-11-16T18:24:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.621+0000 7fa0c2953700 1 -- 172.21.15.2:0/1012602797 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0bc10ba90 con 0x7fa0bc101480 2023-11-16T18:24:36.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.622+0000 7fa0c2953700 1 -- 172.21.15.2:0/1012602797 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0bc107820 con 0x7fa0bc101480 2023-11-16T18:24:36.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.623+0000 7fa0b97fa700 1 -- 172.21.15.2:0/1012602797 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fa0b0004750 con 0x7fa0bc101480 2023-11-16T18:24:36.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.623+0000 7fa0b97fa700 1 --2- 172.21.15.2:0/1012602797 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa0a40a11e0 0x7fa0a40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:36.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.623+0000 7fa0b97fa700 1 -- 172.21.15.2:0/1012602797 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa0b00ccbd0 con 0x7fa0bc101480 2023-11-16T18:24:36.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.623+0000 7fa0bbfff700 1 --2- 172.21.15.2:0/1012602797 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa0a40a11e0 0x7fa0a40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:36.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.624+0000 7fa0bbfff700 1 --2- 172.21.15.2:0/1012602797 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa0a40a11e0 0x7fa0a40a3690 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc1012e0 tx=0x7fa0ac007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:36.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.628+0000 7fa0b97fa700 1 -- 172.21.15.2:0/1012602797 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa0b0095eb0 con 0x7fa0bc101480 2023-11-16T18:24:36.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.827+0000 7fa0c2953700 1 -- 172.21.15.2:0/1012602797 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa0bc061700 con 0x7fa0bc101480 2023-11-16T18:24:36.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.828+0000 7fa0b97fa700 1 -- 172.21.15.2:0/1012602797 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa0b0016020 con 0x7fa0bc101480 2023-11-16T18:24:36.830 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:24:36.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.830+0000 7fa0c2953700 1 -- 172.21.15.2:0/1012602797 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa0a40a11e0 msgr2=0x7fa0a40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.830+0000 7fa0c2953700 1 --2- 172.21.15.2:0/1012602797 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa0a40a11e0 0x7fa0a40a3690 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc1012e0 tx=0x7fa0ac007400 comp rx=0 tx=0).stop 2023-11-16T18:24:36.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.830+0000 7fa0c2953700 1 -- 172.21.15.2:0/1012602797 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc101480 msgr2=0x7fa0bc1965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.830+0000 7fa0c2953700 1 --2- 172.21.15.2:0/1012602797 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc101480 0x7fa0bc1965b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000d980 tx=0x7fa0b000dc90 comp rx=0 tx=0).stop 2023-11-16T18:24:36.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.830+0000 7fa0c2953700 1 -- 172.21.15.2:0/1012602797 shutdown_connections 2023-11-16T18:24:36.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.830+0000 7fa0c2953700 1 --2- 172.21.15.2:0/1012602797 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa0a40a11e0 0x7fa0a40a3690 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.830+0000 7fa0c2953700 1 --2- 172.21.15.2:0/1012602797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0bc100290 0x7fa0bc196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.830+0000 7fa0c2953700 1 --2- 172.21.15.2:0/1012602797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0bc101e10 0x7fa0bc19d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.830+0000 7fa0c2953700 1 --2- 172.21.15.2:0/1012602797 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc101480 0x7fa0bc1965b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.830+0000 7fa0c2953700 1 -- 172.21.15.2:0/1012602797 >> 172.21.15.2:0/1012602797 conn(0x7fa0bc0fb8b0 msgr2=0x7fa0bc104960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:36.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.830+0000 7fa0c2953700 1 -- 172.21.15.2:0/1012602797 shutdown_connections 2023-11-16T18:24:36.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.830+0000 7fa0c2953700 1 -- 172.21.15.2:0/1012602797 wait complete. 2023-11-16T18:24:36.845 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T18:24:36.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.971+0000 7fb481a11700 1 -- 172.21.15.2:0/3574198764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb47c100db0 msgr2=0x7fb47c107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.971+0000 7fb481a11700 1 --2- 172.21.15.2:0/3574198764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb47c100db0 0x7fb47c107650 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb47400b0a0 tx=0x7fb47400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.972+0000 7fb481a11700 1 -- 172.21.15.2:0/3574198764 shutdown_connections 2023-11-16T18:24:36.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.972+0000 7fb481a11700 1 --2- 172.21.15.2:0/3574198764 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb47c0ff170 0x7fb47c0ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.972+0000 7fb481a11700 1 --2- 172.21.15.2:0/3574198764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb47c100db0 0x7fb47c107650 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.972+0000 7fb481a11700 1 --2- 172.21.15.2:0/3574198764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c100400 0x7fb47c100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.972+0000 7fb481a11700 1 -- 172.21.15.2:0/3574198764 >> 172.21.15.2:0/3574198764 conn(0x7fb47c0fa770 msgr2=0x7fb47c0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:36.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.972+0000 7fb481a11700 1 -- 172.21.15.2:0/3574198764 shutdown_connections 2023-11-16T18:24:36.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.972+0000 7fb481a11700 1 -- 172.21.15.2:0/3574198764 wait complete. 2023-11-16T18:24:36.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.972+0000 7fb481a11700 1 Processor -- start 2023-11-16T18:24:36.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.972+0000 7fb481a11700 1 -- start start 2023-11-16T18:24:36.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.972+0000 7fb481a11700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb47c0ff170 0x7fb47c194f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:36.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.972+0000 7fb481a11700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb47c100400 0x7fb47c1954d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:36.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb481a11700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c100db0 0x7fb47c19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:36.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb481a11700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c19cb10 con 0x7fb47c0ff170 2023-11-16T18:24:36.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb481a11700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c19cc50 con 0x7fb47c100db0 2023-11-16T18:24:36.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb481a11700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c19cde0 con 0x7fb47c100400 2023-11-16T18:24:36.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb47affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb47c0ff170 0x7fb47c194f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:36.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb47affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb47c0ff170 0x7fb47c194f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49656/0 (socket says 172.21.15.2:49656) 2023-11-16T18:24:36.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb47affd700 1 -- 172.21.15.2:0/1323852695 learned_addr learned my addr 172.21.15.2:0/1323852695 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:36.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb47a7fc700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb47c100400 0x7fb47c1954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:36.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb47affd700 1 -- 172.21.15.2:0/1323852695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb47c100400 msgr2=0x7fb47c1954d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb47affd700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb47c100400 0x7fb47c1954d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb47affd700 1 -- 172.21.15.2:0/1323852695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c100db0 msgr2=0x7fb47c19c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:36.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb47b7fe700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c100db0 0x7fb47c19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:36.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb47affd700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c100db0 0x7fb47c19c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:36.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb47affd700 1 -- 172.21.15.2:0/1323852695 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb474009d00 con 0x7fb47c0ff170 2023-11-16T18:24:36.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb47a7fc700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb47c100400 0x7fb47c1954d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:36.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb47affd700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb47c0ff170 0x7fb47c194f90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb47000c970 tx=0x7fb47000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:36.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb480a0f700 1 -- 172.21.15.2:0/1323852695 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470007aa0 con 0x7fb47c0ff170 2023-11-16T18:24:36.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb480a0f700 1 -- 172.21.15.2:0/1323852695 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb470004cf0 con 0x7fb47c0ff170 2023-11-16T18:24:36.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb481a11700 1 -- 172.21.15.2:0/1323852695 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c10ab70 con 0x7fb47c0ff170 2023-11-16T18:24:36.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb480a0f700 1 -- 172.21.15.2:0/1323852695 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470005600 con 0x7fb47c0ff170 2023-11-16T18:24:36.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.973+0000 7fb481a11700 1 -- 172.21.15.2:0/1323852695 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c10a9e0 con 0x7fb47c0ff170 2023-11-16T18:24:36.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.974+0000 7fb481a11700 1 -- 172.21.15.2:0/1323852695 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c106190 con 0x7fb47c0ff170 2023-11-16T18:24:36.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.975+0000 7fb480a0f700 1 -- 172.21.15.2:0/1323852695 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fb470004750 con 0x7fb47c0ff170 2023-11-16T18:24:36.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.976+0000 7fb480a0f700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb4640a1260 0x7fb4640a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:36.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.976+0000 7fb47a7fc700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb4640a1260 0x7fb4640a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:36.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.976+0000 7fb480a0f700 1 -- 172.21.15.2:0/1323852695 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb470013070 con 0x7fb47c0ff170 2023-11-16T18:24:36.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.977+0000 7fb47a7fc700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb4640a1260 0x7fb4640a3710 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb46c005d90 tx=0x7fb46c005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:36.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:36.981+0000 7fb480a0f700 1 -- 172.21.15.2:0/1323852695 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb470095ea0 con 0x7fb47c0ff170 2023-11-16T18:24:37.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:36 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/282408252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:37.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:36 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3932495711' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:37.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.179+0000 7fb481a11700 1 -- 172.21.15.2:0/1323852695 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb47c061700 con 0x7fb47c0ff170 2023-11-16T18:24:37.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.179+0000 7fb480a0f700 1 -- 172.21.15.2:0/1323852695 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb4700994c0 con 0x7fb47c0ff170 2023-11-16T18:24:37.182 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:24:37.184 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.182+0000 7fb481a11700 1 -- 172.21.15.2:0/1323852695 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb4640a1260 msgr2=0x7fb4640a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:37.184 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.182+0000 7fb481a11700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb4640a1260 0x7fb4640a3710 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb46c005d90 tx=0x7fb46c005d00 comp rx=0 tx=0).stop 2023-11-16T18:24:37.184 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.182+0000 7fb481a11700 1 -- 172.21.15.2:0/1323852695 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb47c0ff170 msgr2=0x7fb47c194f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:37.184 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.182+0000 7fb481a11700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb47c0ff170 0x7fb47c194f90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb47000c970 tx=0x7fb47000cd30 comp rx=0 tx=0).stop 2023-11-16T18:24:37.184 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.182+0000 7fb481a11700 1 -- 172.21.15.2:0/1323852695 shutdown_connections 2023-11-16T18:24:37.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.182+0000 7fb481a11700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fb4640a1260 0x7fb4640a3710 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.182+0000 7fb481a11700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c100db0 0x7fb47c19c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.182+0000 7fb481a11700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb47c0ff170 0x7fb47c194f90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.182+0000 7fb481a11700 1 --2- 172.21.15.2:0/1323852695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb47c100400 0x7fb47c1954d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.182+0000 7fb481a11700 1 -- 172.21.15.2:0/1323852695 >> 172.21.15.2:0/1323852695 conn(0x7fb47c0fa770 msgr2=0x7fb47c0fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:37.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.182+0000 7fb481a11700 1 -- 172.21.15.2:0/1323852695 shutdown_connections 2023-11-16T18:24:37.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.182+0000 7fb481a11700 1 -- 172.21.15.2:0/1323852695 wait complete. 2023-11-16T18:24:37.196 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T18:24:37.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:36 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/282408252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:37.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:36 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3932495711' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:37.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:36 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/282408252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:37.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:36 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3932495711' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:37.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.323+0000 7fbffa99e700 1 -- 172.21.15.2:0/577406173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbff4101d10 msgr2=0x7fbff41085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:37.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.323+0000 7fbffa99e700 1 --2- 172.21.15.2:0/577406173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbff4101d10 0x7fbff41085d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00b0a0 tx=0x7fbfec00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.323+0000 7fbffa99e700 1 -- 172.21.15.2:0/577406173 shutdown_connections 2023-11-16T18:24:37.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.323+0000 7fbffa99e700 1 --2- 172.21.15.2:0/577406173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbff40fe310 0x7fbff40fe720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.323+0000 7fbffa99e700 1 --2- 172.21.15.2:0/577406173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbff4101d10 0x7fbff41085d0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.323+0000 7fbffa99e700 1 --2- 172.21.15.2:0/577406173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff4101380 0x7fbff41017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.323+0000 7fbffa99e700 1 -- 172.21.15.2:0/577406173 >> 172.21.15.2:0/577406173 conn(0x7fbff40f9cf0 msgr2=0x7fbff40fc120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:37.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.323+0000 7fbffa99e700 1 -- 172.21.15.2:0/577406173 shutdown_connections 2023-11-16T18:24:37.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.323+0000 7fbffa99e700 1 -- 172.21.15.2:0/577406173 wait complete. 2023-11-16T18:24:37.327 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbffa99e700 1 Processor -- start 2023-11-16T18:24:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbffa99e700 1 -- start start 2023-11-16T18:24:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbffa99e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbff40fe310 0x7fbff4100cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbffa99e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbff4101380 0x7fbff40ff330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbffa99e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff4101d10 0x7fbff40ff870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbffa99e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff4100010 con 0x7fbff40fe310 2023-11-16T18:24:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbffa99e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff4100180 con 0x7fbff4101d10 2023-11-16T18:24:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbffa99e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff419feb0 con 0x7fbff4101380 2023-11-16T18:24:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbff999c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbff40fe310 0x7fbff4100cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbff999c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbff40fe310 0x7fbff4100cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49670/0 (socket says 172.21.15.2:49670) 2023-11-16T18:24:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbff999c700 1 -- 172.21.15.2:0/3503535338 learned_addr learned my addr 172.21.15.2:0/3503535338 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbffa19d700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff4101d10 0x7fbff40ff870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbff999c700 1 -- 172.21.15.2:0/3503535338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbff4101380 msgr2=0x7fbff40ff330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbff919b700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbff4101380 0x7fbff40ff330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:37.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbff999c700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbff4101380 0x7fbff40ff330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbff999c700 1 -- 172.21.15.2:0/3503535338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff4101d10 msgr2=0x7fbff40ff870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:37.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbff999c700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff4101d10 0x7fbff40ff870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbff999c700 1 -- 172.21.15.2:0/3503535338 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec009d00 con 0x7fbff40fe310 2023-11-16T18:24:37.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.324+0000 7fbff999c700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbff40fe310 0x7fbff4100cf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400d980 tx=0x7fbfe400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:37.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.325+0000 7fbfeaffd700 1 -- 172.21.15.2:0/3503535338 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe400de60 con 0x7fbff40fe310 2023-11-16T18:24:37.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.325+0000 7fbfeaffd700 1 -- 172.21.15.2:0/3503535338 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbfe40044e0 con 0x7fbff40fe310 2023-11-16T18:24:37.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.325+0000 7fbffa99e700 1 -- 172.21.15.2:0/3503535338 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff410b9c0 con 0x7fbff40fe310 2023-11-16T18:24:37.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.325+0000 7fbfeaffd700 1 -- 172.21.15.2:0/3503535338 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4003be0 con 0x7fbff40fe310 2023-11-16T18:24:37.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.325+0000 7fbffa99e700 1 -- 172.21.15.2:0/3503535338 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff410b830 con 0x7fbff40fe310 2023-11-16T18:24:37.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.326+0000 7fbffa99e700 1 -- 172.21.15.2:0/3503535338 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff4107620 con 0x7fbff40fe310 2023-11-16T18:24:37.330 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.327+0000 7fbfeaffd700 1 -- 172.21.15.2:0/3503535338 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fbfe4009bb0 con 0x7fbff40fe310 2023-11-16T18:24:37.330 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.327+0000 7fbfeaffd700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbfe00a11e0 0x7fbfe00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:37.330 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.328+0000 7fbff919b700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbfe00a11e0 0x7fbfe00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:37.330 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.328+0000 7fbfeaffd700 1 -- 172.21.15.2:0/3503535338 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbfe40ccf50 con 0x7fbff40fe310 2023-11-16T18:24:37.330 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.328+0000 7fbff919b700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbfe00a11e0 0x7fbfe00a3690 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009b30 tx=0x7fbff0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:37.333 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.331+0000 7fbfeaffd700 1 -- 172.21.15.2:0/3503535338 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbfe40961d0 con 0x7fbff40fe310 2023-11-16T18:24:37.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.526+0000 7fbffa99e700 1 -- 172.21.15.2:0/3503535338 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbff4061700 con 0x7fbff40fe310 2023-11-16T18:24:37.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.527+0000 7fbfeaffd700 1 -- 172.21.15.2:0/3503535338 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbfe4009940 con 0x7fbff40fe310 2023-11-16T18:24:37.529 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T18:24:37.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.529+0000 7fbffa99e700 1 -- 172.21.15.2:0/3503535338 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbfe00a11e0 msgr2=0x7fbfe00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:37.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.529+0000 7fbffa99e700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbfe00a11e0 0x7fbfe00a3690 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009b30 tx=0x7fbff0009260 comp rx=0 tx=0).stop 2023-11-16T18:24:37.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.529+0000 7fbffa99e700 1 -- 172.21.15.2:0/3503535338 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbff40fe310 msgr2=0x7fbff4100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:37.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.529+0000 7fbffa99e700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbff40fe310 0x7fbff4100cf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400d980 tx=0x7fbfe400dd40 comp rx=0 tx=0).stop 2023-11-16T18:24:37.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.529+0000 7fbffa99e700 1 -- 172.21.15.2:0/3503535338 shutdown_connections 2023-11-16T18:24:37.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.529+0000 7fbffa99e700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fbfe00a11e0 0x7fbfe00a3690 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.529+0000 7fbffa99e700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff4101d10 0x7fbff40ff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.529+0000 7fbffa99e700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbff40fe310 0x7fbff4100cf0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.529+0000 7fbffa99e700 1 --2- 172.21.15.2:0/3503535338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbff4101380 0x7fbff40ff330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.529+0000 7fbffa99e700 1 -- 172.21.15.2:0/3503535338 >> 172.21.15.2:0/3503535338 conn(0x7fbff40f9cf0 msgr2=0x7fbff4104760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.529+0000 7fbffa99e700 1 -- 172.21.15.2:0/3503535338 shutdown_connections 2023-11-16T18:24:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.529+0000 7fbffa99e700 1 -- 172.21.15.2:0/3503535338 wait complete. 2023-11-16T18:24:37.544 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T18:24:37.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.670+0000 7fd1545f4700 1 -- 172.21.15.2:0/1137395621 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd14c0ff1c0 msgr2=0x7fd14c0ff5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:37.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.670+0000 7fd1545f4700 1 --2- 172.21.15.2:0/1137395621 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd14c0ff1c0 0x7fd14c0ff5d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd140009990 tx=0x7fd140009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.671+0000 7fd1545f4700 1 -- 172.21.15.2:0/1137395621 shutdown_connections 2023-11-16T18:24:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.671+0000 7fd1545f4700 1 --2- 172.21.15.2:0/1137395621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd14c100d40 0x7fd14c107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.671+0000 7fd1545f4700 1 --2- 172.21.15.2:0/1137395621 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd14c0ff1c0 0x7fd14c0ff5d0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.671+0000 7fd1545f4700 1 --2- 172.21.15.2:0/1137395621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd14c1003b0 0x7fd14c100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.671+0000 7fd1545f4700 1 -- 172.21.15.2:0/1137395621 >> 172.21.15.2:0/1137395621 conn(0x7fd14c0fa800 msgr2=0x7fd14c0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.671+0000 7fd1545f4700 1 -- 172.21.15.2:0/1137395621 shutdown_connections 2023-11-16T18:24:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.671+0000 7fd1545f4700 1 -- 172.21.15.2:0/1137395621 wait complete. 2023-11-16T18:24:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.671+0000 7fd1545f4700 1 Processor -- start 2023-11-16T18:24:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.671+0000 7fd1545f4700 1 -- start start 2023-11-16T18:24:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.671+0000 7fd1545f4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd14c0ff1c0 0x7fd14c069950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.671+0000 7fd1545f4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd14c1003b0 0x7fd14c19acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.671+0000 7fd1545f4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd14c100d40 0x7fd14c19d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.671+0000 7fd1545f4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c19dab0 con 0x7fd14c1003b0 2023-11-16T18:24:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd1545f4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c19dc20 con 0x7fd14c0ff1c0 2023-11-16T18:24:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd1545f4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c19dde0 con 0x7fd14c100d40 2023-11-16T18:24:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd151b8f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd14c1003b0 0x7fd14c19acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd151b8f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd14c1003b0 0x7fd14c19acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49686/0 (socket says 172.21.15.2:49686) 2023-11-16T18:24:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd151b8f700 1 -- 172.21.15.2:0/2962856659 learned_addr learned my addr 172.21.15.2:0/2962856659 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd152390700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd14c0ff1c0 0x7fd14c069950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd152b91700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd14c100d40 0x7fd14c19d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd151b8f700 1 -- 172.21.15.2:0/2962856659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd14c100d40 msgr2=0x7fd14c19d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd151b8f700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd14c100d40 0x7fd14c19d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd151b8f700 1 -- 172.21.15.2:0/2962856659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd14c0ff1c0 msgr2=0x7fd14c069950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd151b8f700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd14c0ff1c0 0x7fd14c069950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd151b8f700 1 -- 172.21.15.2:0/2962856659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1400095f0 con 0x7fd14c1003b0 2023-11-16T18:24:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd152b91700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd14c100d40 0x7fd14c19d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd151b8f700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd14c1003b0 0x7fd14c19acd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd14800e9a0 tx=0x7fd14800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd13f7fe700 1 -- 172.21.15.2:0/2962856659 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148009790 con 0x7fd14c1003b0 2023-11-16T18:24:37.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd13f7fe700 1 -- 172.21.15.2:0/2962856659 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd14800cbd0 con 0x7fd14c1003b0 2023-11-16T18:24:37.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd1545f4700 1 -- 172.21.15.2:0/2962856659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14c10ab20 con 0x7fd14c1003b0 2023-11-16T18:24:37.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd13f7fe700 1 -- 172.21.15.2:0/2962856659 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148018680 con 0x7fd14c1003b0 2023-11-16T18:24:37.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.672+0000 7fd1545f4700 1 -- 172.21.15.2:0/2962856659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14c10a990 con 0x7fd14c1003b0 2023-11-16T18:24:37.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.673+0000 7fd1545f4700 1 -- 172.21.15.2:0/2962856659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14c106750 con 0x7fd14c1003b0 2023-11-16T18:24:37.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.674+0000 7fd13f7fe700 1 -- 172.21.15.2:0/2962856659 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fd14800cd40 con 0x7fd14c1003b0 2023-11-16T18:24:37.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.675+0000 7fd13f7fe700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd1380a1260 0x7fd1380a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:37.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.675+0000 7fd13f7fe700 1 -- 172.21.15.2:0/2962856659 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd148014070 con 0x7fd14c1003b0 2023-11-16T18:24:37.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.675+0000 7fd152390700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd1380a1260 0x7fd1380a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:37.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.675+0000 7fd152390700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd1380a1260 0x7fd1380a3710 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd140006010 tx=0x7fd140005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:37.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.679+0000 7fd13f7fe700 1 -- 172.21.15.2:0/2962856659 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd148051730 con 0x7fd14c1003b0 2023-11-16T18:24:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.905+0000 7fd1545f4700 1 -- 172.21.15.2:0/2962856659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd14c061700 con 0x7fd14c1003b0 2023-11-16T18:24:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.905+0000 7fd13f7fe700 1 -- 172.21.15.2:0/2962856659 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd148096260 con 0x7fd14c1003b0 2023-11-16T18:24:37.907 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:24:37.909 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.907+0000 7fd1545f4700 1 -- 172.21.15.2:0/2962856659 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd1380a1260 msgr2=0x7fd1380a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.907+0000 7fd1545f4700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd1380a1260 0x7fd1380a3710 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd140006010 tx=0x7fd140005960 comp rx=0 tx=0).stop 2023-11-16T18:24:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.907+0000 7fd1545f4700 1 -- 172.21.15.2:0/2962856659 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd14c1003b0 msgr2=0x7fd14c19acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.907+0000 7fd1545f4700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd14c1003b0 0x7fd14c19acd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd14800e9a0 tx=0x7fd14800ecb0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.907+0000 7fd1545f4700 1 -- 172.21.15.2:0/2962856659 shutdown_connections 2023-11-16T18:24:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.907+0000 7fd1545f4700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd1380a1260 0x7fd1380a3710 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.907+0000 7fd1545f4700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd14c100d40 0x7fd14c19d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.907+0000 7fd1545f4700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd14c0ff1c0 0x7fd14c069950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.907+0000 7fd1545f4700 1 --2- 172.21.15.2:0/2962856659 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd14c1003b0 0x7fd14c19acd0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.907+0000 7fd1545f4700 1 -- 172.21.15.2:0/2962856659 >> 172.21.15.2:0/2962856659 conn(0x7fd14c0fa800 msgr2=0x7fd14c103890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.907+0000 7fd1545f4700 1 -- 172.21.15.2:0/2962856659 shutdown_connections 2023-11-16T18:24:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:37.907+0000 7fd1545f4700 1 -- 172.21.15.2:0/2962856659 wait complete. 2023-11-16T18:24:37.921 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2023-11-16T18:24:37.922 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:24:38.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.045+0000 7ff9b8171700 1 -- 172.21.15.2:0/318548909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9b0100340 msgr2=0x7ff9b0100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:38.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.045+0000 7ff9b8171700 1 --2- 172.21.15.2:0/318548909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9b0100340 0x7ff9b0100750 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009990 tx=0x7ff9a4009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.045+0000 7ff9b8171700 1 -- 172.21.15.2:0/318548909 shutdown_connections 2023-11-16T18:24:38.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.045+0000 7ff9b8171700 1 --2- 172.21.15.2:0/318548909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b0101ec0 0x7ff9b0108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.045+0000 7ff9b8171700 1 --2- 172.21.15.2:0/318548909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9b0100340 0x7ff9b0100750 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.045+0000 7ff9b8171700 1 --2- 172.21.15.2:0/318548909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9b0101530 0x7ff9b0101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.045+0000 7ff9b8171700 1 -- 172.21.15.2:0/318548909 >> 172.21.15.2:0/318548909 conn(0x7ff9b00fb940 msgr2=0x7ff9b00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:38.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.045+0000 7ff9b8171700 1 -- 172.21.15.2:0/318548909 shutdown_connections 2023-11-16T18:24:38.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.045+0000 7ff9b8171700 1 -- 172.21.15.2:0/318548909 wait complete. 2023-11-16T18:24:38.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.046+0000 7ff9b8171700 1 Processor -- start 2023-11-16T18:24:38.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.046+0000 7ff9b8171700 1 -- start start 2023-11-16T18:24:38.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.046+0000 7ff9b8171700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9b0100340 0x7ff9b0196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:38.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.046+0000 7ff9b8171700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b0101530 0x7ff9b0196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:38.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.046+0000 7ff9b8171700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9b0101ec0 0x7ff9b019d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:38.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.046+0000 7ff9b8171700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b019dbc0 con 0x7ff9b0101ec0 2023-11-16T18:24:38.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.046+0000 7ff9b8171700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b019dd00 con 0x7ff9b0101530 2023-11-16T18:24:38.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.046+0000 7ff9b8171700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b019de90 con 0x7ff9b0100340 2023-11-16T18:24:38.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b670e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9b0101ec0 0x7ff9b019d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:38.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b670e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9b0101ec0 0x7ff9b019d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49702/0 (socket says 172.21.15.2:49702) 2023-11-16T18:24:38.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b670e700 1 -- 172.21.15.2:0/519250188 learned_addr learned my addr 172.21.15.2:0/519250188 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:38.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b570c700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b0101530 0x7ff9b0196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:38.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b5f0d700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9b0100340 0x7ff9b0196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:38.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b670e700 1 -- 172.21.15.2:0/519250188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9b0100340 msgr2=0x7ff9b0196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:38.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b670e700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9b0100340 0x7ff9b0196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b670e700 1 -- 172.21.15.2:0/519250188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b0101530 msgr2=0x7ff9b0196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:38.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b670e700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b0101530 0x7ff9b0196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b670e700 1 -- 172.21.15.2:0/519250188 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a40095f0 con 0x7ff9b0101ec0 2023-11-16T18:24:38.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b5f0d700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9b0100340 0x7ff9b0196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:38.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b670e700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9b0101ec0 0x7ff9b019d5f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8007d70 tx=0x7ff9a800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:38.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9aeffd700 1 -- 172.21.15.2:0/519250188 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a800ad40 con 0x7ff9b0101ec0 2023-11-16T18:24:38.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b8171700 1 -- 172.21.15.2:0/519250188 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b010bcc0 con 0x7ff9b0101ec0 2023-11-16T18:24:38.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9aeffd700 1 -- 172.21.15.2:0/519250188 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9a80128e0 con 0x7ff9b0101ec0 2023-11-16T18:24:38.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9aeffd700 1 -- 172.21.15.2:0/519250188 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a80116b0 con 0x7ff9b0101ec0 2023-11-16T18:24:38.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.047+0000 7ff9b8171700 1 -- 172.21.15.2:0/519250188 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b010bb30 con 0x7ff9b0101ec0 2023-11-16T18:24:38.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.048+0000 7ff9b8171700 1 -- 172.21.15.2:0/519250188 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b01071f0 con 0x7ff9b0101ec0 2023-11-16T18:24:38.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.050+0000 7ff9aeffd700 1 -- 172.21.15.2:0/519250188 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ff9a8012420 con 0x7ff9b0101ec0 2023-11-16T18:24:38.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.051+0000 7ff9aeffd700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff9a00a11e0 0x7ff9a00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:38.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.051+0000 7ff9b5f0d700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff9a00a11e0 0x7ff9a00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:38.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.051+0000 7ff9aeffd700 1 -- 172.21.15.2:0/519250188 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff9a80cd370 con 0x7ff9b0101ec0 2023-11-16T18:24:38.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.051+0000 7ff9b5f0d700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff9a00a11e0 0x7ff9a00a3690 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4006010 tx=0x7ff9a400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:38.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.054+0000 7ff9aeffd700 1 -- 172.21.15.2:0/519250188 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff9a8096700 con 0x7ff9b0101ec0 2023-11-16T18:24:38.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:37 smithi002 ceph-mon[98807]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2023-11-16T18:24:38.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:37 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1012602797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:38.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:37 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1323852695' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:38.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:37 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3503535338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:38.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:37 smithi040 ceph-mon[100251]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2023-11-16T18:24:38.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:37 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1012602797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:38.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:37 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1323852695' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:38.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:37 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3503535338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:38.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:37 smithi188 ceph-mon[100639]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2023-11-16T18:24:38.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:37 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1012602797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:38.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:37 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1323852695' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:38.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:37 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3503535338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:38.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.302+0000 7ff9b8171700 1 -- 172.21.15.2:0/519250188 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9b002cfe0 con 0x7ff9b0101ec0 2023-11-16T18:24:38.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.303+0000 7ff9aeffd700 1 -- 172.21.15.2:0/519250188 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10774 (secure 0 0 0) 0x7ff9a8023350 con 0x7ff9b0101ec0 2023-11-16T18:24:38.305 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:38.305 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:23:07.909835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:24:38.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.305+0000 7ff9b8171700 1 -- 172.21.15.2:0/519250188 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff9a00a11e0 msgr2=0x7ff9a00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:38.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.305+0000 7ff9b8171700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff9a00a11e0 0x7ff9a00a3690 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4006010 tx=0x7ff9a400b540 comp rx=0 tx=0).stop 2023-11-16T18:24:38.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.305+0000 7ff9b8171700 1 -- 172.21.15.2:0/519250188 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9b0101ec0 msgr2=0x7ff9b019d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:38.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.305+0000 7ff9b8171700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9b0101ec0 0x7ff9b019d5f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8007d70 tx=0x7ff9a800a340 comp rx=0 tx=0).stop 2023-11-16T18:24:38.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.305+0000 7ff9b8171700 1 -- 172.21.15.2:0/519250188 shutdown_connections 2023-11-16T18:24:38.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.305+0000 7ff9b8171700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7ff9a00a11e0 0x7ff9a00a3690 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.305+0000 7ff9b8171700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9b0100340 0x7ff9b0196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.305+0000 7ff9b8171700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9b0101ec0 0x7ff9b019d5f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.305+0000 7ff9b8171700 1 --2- 172.21.15.2:0/519250188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b0101530 0x7ff9b0196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.305+0000 7ff9b8171700 1 -- 172.21.15.2:0/519250188 >> 172.21.15.2:0/519250188 conn(0x7ff9b00fb940 msgr2=0x7ff9b00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:38.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.305+0000 7ff9b8171700 1 -- 172.21.15.2:0/519250188 shutdown_connections 2023-11-16T18:24:38.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.305+0000 7ff9b8171700 1 -- 172.21.15.2:0/519250188 wait complete. 2023-11-16T18:24:38.308 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T18:24:38.320 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2023-11-16T18:24:38.320 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2023-11-16T18:24:38.320 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T18:24:38.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.445+0000 7f03dd4df700 1 -- 172.21.15.2:0/2551133743 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d8101ec0 msgr2=0x7f03d8108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:38.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.445+0000 7f03dd4df700 1 --2- 172.21.15.2:0/2551133743 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d8101ec0 0x7f03d8108780 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f03d000b0a0 tx=0x7f03d000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.445+0000 7f03dd4df700 1 -- 172.21.15.2:0/2551133743 shutdown_connections 2023-11-16T18:24:38.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.445+0000 7f03dd4df700 1 --2- 172.21.15.2:0/2551133743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03d8100340 0x7f03d8100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.445+0000 7f03dd4df700 1 --2- 172.21.15.2:0/2551133743 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d8101ec0 0x7f03d8108780 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.445+0000 7f03dd4df700 1 --2- 172.21.15.2:0/2551133743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d8101530 0x7f03d8101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.445+0000 7f03dd4df700 1 -- 172.21.15.2:0/2551133743 >> 172.21.15.2:0/2551133743 conn(0x7f03d80fb940 msgr2=0x7f03d80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:38.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.445+0000 7f03dd4df700 1 -- 172.21.15.2:0/2551133743 shutdown_connections 2023-11-16T18:24:38.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.445+0000 7f03dd4df700 1 -- 172.21.15.2:0/2551133743 wait complete. 2023-11-16T18:24:38.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.446+0000 7f03dd4df700 1 Processor -- start 2023-11-16T18:24:38.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.446+0000 7f03dd4df700 1 -- start start 2023-11-16T18:24:38.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.446+0000 7f03dd4df700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d8100340 0x7f03d8196120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:38.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.446+0000 7f03dd4df700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03d8101530 0x7f03d8196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:38.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.446+0000 7f03dd4df700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d8101ec0 0x7f03d819d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:38.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.446+0000 7f03dd4df700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d819dca0 con 0x7f03d8101ec0 2023-11-16T18:24:38.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.446+0000 7f03dd4df700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d819dde0 con 0x7f03d8100340 2023-11-16T18:24:38.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.446+0000 7f03dd4df700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d819df70 con 0x7f03d8101530 2023-11-16T18:24:38.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.446+0000 7f03d77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d8101ec0 0x7f03d819d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:38.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.446+0000 7f03d77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d8101ec0 0x7f03d819d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49714/0 (socket says 172.21.15.2:49714) 2023-11-16T18:24:38.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.446+0000 7f03d77fe700 1 -- 172.21.15.2:0/3238526172 learned_addr learned my addr 172.21.15.2:0/3238526172 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:38.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.446+0000 7f03d67fc700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03d8101530 0x7f03d8196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:38.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.446+0000 7f03d6ffd700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d8100340 0x7f03d8196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:38.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.447+0000 7f03d77fe700 1 -- 172.21.15.2:0/3238526172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03d8101530 msgr2=0x7f03d8196660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:38.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.447+0000 7f03d77fe700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03d8101530 0x7f03d8196660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.447+0000 7f03d77fe700 1 -- 172.21.15.2:0/3238526172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d8100340 msgr2=0x7f03d8196120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:38.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.447+0000 7f03d77fe700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d8100340 0x7f03d8196120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.447+0000 7f03d77fe700 1 -- 172.21.15.2:0/3238526172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d0009d00 con 0x7f03d8101ec0 2023-11-16T18:24:38.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.447+0000 7f03d77fe700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d8101ec0 0x7f03d819d6d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f03d000b830 tx=0x7f03d0004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:38.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.447+0000 7f03bffff700 1 -- 172.21.15.2:0/3238526172 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d000e040 con 0x7f03d8101ec0 2023-11-16T18:24:38.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.447+0000 7f03dd4df700 1 -- 172.21.15.2:0/3238526172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d810bc60 con 0x7f03d8101ec0 2023-11-16T18:24:38.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.447+0000 7f03bffff700 1 -- 172.21.15.2:0/3238526172 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03d001db50 con 0x7f03d8101ec0 2023-11-16T18:24:38.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.447+0000 7f03bffff700 1 -- 172.21.15.2:0/3238526172 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d001bb20 con 0x7f03d8101ec0 2023-11-16T18:24:38.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.447+0000 7f03dd4df700 1 -- 172.21.15.2:0/3238526172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d810bad0 con 0x7f03d8101ec0 2023-11-16T18:24:38.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.448+0000 7f03dd4df700 1 -- 172.21.15.2:0/3238526172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d810bdf0 con 0x7f03d8101ec0 2023-11-16T18:24:38.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.449+0000 7f03bffff700 1 -- 172.21.15.2:0/3238526172 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f03d001dcc0 con 0x7f03d8101ec0 2023-11-16T18:24:38.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.450+0000 7f03bffff700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f03c00a11e0 0x7f03c00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:38.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.450+0000 7f03d6ffd700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f03c00a11e0 0x7f03c00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:38.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.450+0000 7f03bffff700 1 -- 172.21.15.2:0/3238526172 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f03d00ce9f0 con 0x7f03d8101ec0 2023-11-16T18:24:38.455 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.450+0000 7f03d6ffd700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f03c00a11e0 0x7f03c00a3690 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f03d8101390 tx=0x7f03cc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:38.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.454+0000 7f03bffff700 1 -- 172.21.15.2:0/3238526172 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f03d0097d00 con 0x7f03d8101ec0 2023-11-16T18:24:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.653+0000 7f03dd4df700 1 -- 172.21.15.2:0/3238526172 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f03d80619a0 con 0x7f03c00a11e0 2023-11-16T18:24:38.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.668+0000 7f03bffff700 1 -- 172.21.15.2:0/3238526172 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426750 (secure 0 0 0) 0x7f03d80619a0 con 0x7f03c00a11e0 2023-11-16T18:24:38.680 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:38.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.672+0000 7f03dd4df700 1 -- 172.21.15.2:0/3238526172 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f03c00a11e0 msgr2=0x7f03c00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:38.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.672+0000 7f03dd4df700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f03c00a11e0 0x7f03c00a3690 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f03d8101390 tx=0x7f03cc00b410 comp rx=0 tx=0).stop 2023-11-16T18:24:38.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.672+0000 7f03dd4df700 1 -- 172.21.15.2:0/3238526172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d8101ec0 msgr2=0x7f03d819d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:38.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.672+0000 7f03dd4df700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d8101ec0 0x7f03d819d6d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f03d000b830 tx=0x7f03d0004c80 comp rx=0 tx=0).stop 2023-11-16T18:24:38.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.672+0000 7f03dd4df700 1 -- 172.21.15.2:0/3238526172 shutdown_connections 2023-11-16T18:24:38.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.672+0000 7f03dd4df700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f03c00a11e0 0x7f03c00a3690 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.672+0000 7f03dd4df700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d8101ec0 0x7f03d819d6d0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.672+0000 7f03dd4df700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d8100340 0x7f03d8196120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.672+0000 7f03dd4df700 1 --2- 172.21.15.2:0/3238526172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03d8101530 0x7f03d8196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.672+0000 7f03dd4df700 1 -- 172.21.15.2:0/3238526172 >> 172.21.15.2:0/3238526172 conn(0x7f03d80fb940 msgr2=0x7f03d8104920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:38.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.672+0000 7f03dd4df700 1 -- 172.21.15.2:0/3238526172 shutdown_connections 2023-11-16T18:24:38.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.672+0000 7f03dd4df700 1 -- 172.21.15.2:0/3238526172 wait complete. 2023-11-16T18:24:38.691 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-16T18:24:38.692 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2023-11-16T18:24:37.846306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656381,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":680,"num_read_kb":459,"num_write":341,"num_write_kb":3054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1242772,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80796,"kb_used_data":11708,"kb_used_omap":0,"kb_used_meta":68992,"kb_avail":1124861028,"statfs":{"total":1151940427776,"available":1151857692672,"internally_reserved":0,"allocated":11988992,"data_stored":7772273,"data_compressed":329214,"data_compressed_allocated":442368,"data_compressed_original":2416640,"omap_allocated":0,"internal_metadata":70647808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":636,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":32,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.016959"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506980+0000","last_change":"2023-11-16T18:22:46.829814+0000","last_active":"2023-11-16T18:23:39.506980+0000","last_peered":"2023-11-16T18:23:39.506980+0000","last_clean":"2023-11-16T18:23:39.506980+0000","last_became_active":"2023-11-16T18:22:46.829407+0000","last_became_peered":"2023-11-16T18:22:46.829407+0000","last_unstale":"2023-11-16T18:23:39.506980+0000","last_undegraded":"2023-11-16T18:23:39.506980+0000","last_fullsized":"2023-11-16T18:23:39.506980+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:02:05.303420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505209+0000","last_change":"2023-11-16T18:22:46.818700+0000","last_active":"2023-11-16T18:23:39.505209+0000","last_peered":"2023-11-16T18:23:39.505209+0000","last_clean":"2023-11-16T18:23:39.505209+0000","last_became_active":"2023-11-16T18:22:46.818388+0000","last_became_peered":"2023-11-16T18:22:46.818388+0000","last_unstale":"2023-11-16T18:23:39.505209+0000","last_undegraded":"2023-11-16T18:23:39.505209+0000","last_fullsized":"2023-11-16T18:23:39.505209+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:05:25.862565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504017+0000","last_change":"2023-11-16T18:22:46.827732+0000","last_active":"2023-11-16T18:23:39.504017+0000","last_peered":"2023-11-16T18:23:39.504017+0000","last_clean":"2023-11-16T18:23:39.504017+0000","last_became_active":"2023-11-16T18:22:46.827328+0000","last_became_peered":"2023-11-16T18:22:46.827328+0000","last_unstale":"2023-11-16T18:23:39.504017+0000","last_undegraded":"2023-11-16T18:23:39.504017+0000","last_fullsized":"2023-11-16T18:23:39.504017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:44:05.572178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504758+0000","last_change":"2023-11-16T18:22:46.826837+0000","last_active":"2023-11-16T18:23:39.504758+0000","last_peered":"2023-11-16T18:23:39.504758+0000","last_clean":"2023-11-16T18:23:39.504758+0000","last_became_active":"2023-11-16T18:22:46.826532+0000","last_became_peered":"2023-11-16T18:22:46.826532+0000","last_unstale":"2023-11-16T18:23:39.504758+0000","last_undegraded":"2023-11-16T18:23:39.504758+0000","last_fullsized":"2023-11-16T18:23:39.504758+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:29:24.407020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504437+0000","last_change":"2023-11-16T18:22:46.841164+0000","last_active":"2023-11-16T18:23:39.504437+0000","last_peered":"2023-11-16T18:23:39.504437+0000","last_clean":"2023-11-16T18:23:39.504437+0000","last_became_active":"2023-11-16T18:22:46.840855+0000","last_became_peered":"2023-11-16T18:22:46.840855+0000","last_unstale":"2023-11-16T18:23:39.504437+0000","last_undegraded":"2023-11-16T18:23:39.504437+0000","last_fullsized":"2023-11-16T18:23:39.504437+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:46.380315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507548+0000","last_change":"2023-11-16T18:22:46.822171+0000","last_active":"2023-11-16T18:23:39.507548+0000","last_peered":"2023-11-16T18:23:39.507548+0000","last_clean":"2023-11-16T18:23:39.507548+0000","last_became_active":"2023-11-16T18:22:46.821860+0000","last_became_peered":"2023-11-16T18:22:46.821860+0000","last_unstale":"2023-11-16T18:23:39.507548+0000","last_undegraded":"2023-11-16T18:23:39.507548+0000","last_fullsized":"2023-11-16T18:23:39.507548+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:28:18.589285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502347+0000","last_change":"2023-11-16T18:22:46.832463+0000","last_active":"2023-11-16T18:23:39.502347+0000","last_peered":"2023-11-16T18:23:39.502347+0000","last_clean":"2023-11-16T18:23:39.502347+0000","last_became_active":"2023-11-16T18:22:46.832141+0000","last_became_peered":"2023-11-16T18:22:46.832141+0000","last_unstale":"2023-11-16T18:23:39.502347+0000","last_undegraded":"2023-11-16T18:23:39.502347+0000","last_fullsized":"2023-11-16T18:23:39.502347+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:12:03.361742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504425+0000","last_change":"2023-11-16T18:22:46.844239+0000","last_active":"2023-11-16T18:23:39.504425+0000","last_peered":"2023-11-16T18:23:39.504425+0000","last_clean":"2023-11-16T18:23:39.504425+0000","last_became_active":"2023-11-16T18:22:46.843906+0000","last_became_peered":"2023-11-16T18:22:46.843906+0000","last_unstale":"2023-11-16T18:23:39.504425+0000","last_undegraded":"2023-11-16T18:23:39.504425+0000","last_fullsized":"2023-11-16T18:23:39.504425+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:16:15.202242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508804+0000","last_change":"2023-11-16T18:22:46.834625+0000","last_active":"2023-11-16T18:23:39.508804+0000","last_peered":"2023-11-16T18:23:39.508804+0000","last_clean":"2023-11-16T18:23:39.508804+0000","last_became_active":"2023-11-16T18:22:46.834145+0000","last_became_peered":"2023-11-16T18:22:46.834145+0000","last_unstale":"2023-11-16T18:23:39.508804+0000","last_undegraded":"2023-11-16T18:23:39.508804+0000","last_fullsized":"2023-11-16T18:23:39.508804+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:25:16.196016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509061+0000","last_change":"2023-11-16T18:22:46.827569+0000","last_active":"2023-11-16T18:23:39.509061+0000","last_peered":"2023-11-16T18:23:39.509061+0000","last_clean":"2023-11-16T18:23:39.509061+0000","last_became_active":"2023-11-16T18:22:46.827234+0000","last_became_peered":"2023-11-16T18:22:46.827234+0000","last_unstale":"2023-11-16T18:23:39.509061+0000","last_undegraded":"2023-11-16T18:23:39.509061+0000","last_fullsized":"2023-11-16T18:23:39.509061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:38:30.584402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503061+0000","last_change":"2023-11-16T18:22:46.833701+0000","last_active":"2023-11-16T18:23:39.503061+0000","last_peered":"2023-11-16T18:23:39.503061+0000","last_clean":"2023-11-16T18:23:39.503061+0000","last_became_active":"2023-11-16T18:22:46.833333+0000","last_became_peered":"2023-11-16T18:22:46.833333+0000","last_unstale":"2023-11-16T18:23:39.503061+0000","last_undegraded":"2023-11-16T18:23:39.503061+0000","last_fullsized":"2023-11-16T18:23:39.503061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:31:52.429141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504924+0000","last_change":"2023-11-16T18:22:46.823222+0000","last_active":"2023-11-16T18:23:39.504924+0000","last_peered":"2023-11-16T18:23:39.504924+0000","last_clean":"2023-11-16T18:23:39.504924+0000","last_became_active":"2023-11-16T18:22:46.822224+0000","last_became_peered":"2023-11-16T18:22:46.822224+0000","last_unstale":"2023-11-16T18:23:39.504924+0000","last_undegraded":"2023-11-16T18:23:39.504924+0000","last_fullsized":"2023-11-16T18:23:39.504924+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:23:38.463744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584444+0000","last_change":"2023-11-16T18:22:46.818631+0000","last_active":"2023-11-16T18:23:39.584444+0000","last_peered":"2023-11-16T18:23:39.584444+0000","last_clean":"2023-11-16T18:23:39.584444+0000","last_became_active":"2023-11-16T18:22:46.818305+0000","last_became_peered":"2023-11-16T18:22:46.818305+0000","last_unstale":"2023-11-16T18:23:39.584444+0000","last_undegraded":"2023-11-16T18:23:39.584444+0000","last_fullsized":"2023-11-16T18:23:39.584444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:06:53.083409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503715+0000","last_change":"2023-11-16T18:22:46.825439+0000","last_active":"2023-11-16T18:23:39.503715+0000","last_peered":"2023-11-16T18:23:39.503715+0000","last_clean":"2023-11-16T18:23:39.503715+0000","last_became_active":"2023-11-16T18:22:46.825062+0000","last_became_peered":"2023-11-16T18:22:46.825062+0000","last_unstale":"2023-11-16T18:23:39.503715+0000","last_undegraded":"2023-11-16T18:23:39.503715+0000","last_fullsized":"2023-11-16T18:23:39.503715+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:32:03.819033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503546+0000","last_change":"2023-11-16T18:22:46.827730+0000","last_active":"2023-11-16T18:23:39.503546+0000","last_peered":"2023-11-16T18:23:39.503546+0000","last_clean":"2023-11-16T18:23:39.503546+0000","last_became_active":"2023-11-16T18:22:46.827416+0000","last_became_peered":"2023-11-16T18:22:46.827416+0000","last_unstale":"2023-11-16T18:23:39.503546+0000","last_undegraded":"2023-11-16T18:23:39.503546+0000","last_fullsized":"2023-11-16T18:23:39.503546+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:18:08.691944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507323+0000","last_change":"2023-11-16T18:22:46.834259+0000","last_active":"2023-11-16T18:23:39.507323+0000","last_peered":"2023-11-16T18:23:39.507323+0000","last_clean":"2023-11-16T18:23:39.507323+0000","last_became_active":"2023-11-16T18:22:46.833775+0000","last_became_peered":"2023-11-16T18:22:46.833775+0000","last_unstale":"2023-11-16T18:23:39.507323+0000","last_undegraded":"2023-11-16T18:23:39.507323+0000","last_fullsized":"2023-11-16T18:23:39.507323+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:02:17.762712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506738+0000","last_change":"2023-11-16T18:22:46.829475+0000","last_active":"2023-11-16T18:23:39.506738+0000","last_peered":"2023-11-16T18:23:39.506738+0000","last_clean":"2023-11-16T18:23:39.506738+0000","last_became_active":"2023-11-16T18:22:46.829157+0000","last_became_peered":"2023-11-16T18:22:46.829157+0000","last_unstale":"2023-11-16T18:23:39.506738+0000","last_undegraded":"2023-11-16T18:23:39.506738+0000","last_fullsized":"2023-11-16T18:23:39.506738+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:29.122041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501933+0000","last_change":"2023-11-16T18:22:46.821716+0000","last_active":"2023-11-16T18:23:39.501933+0000","last_peered":"2023-11-16T18:23:39.501933+0000","last_clean":"2023-11-16T18:23:39.501933+0000","last_became_active":"2023-11-16T18:22:46.821393+0000","last_became_peered":"2023-11-16T18:22:46.821393+0000","last_unstale":"2023-11-16T18:23:39.501933+0000","last_undegraded":"2023-11-16T18:23:39.501933+0000","last_fullsized":"2023-11-16T18:23:39.501933+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:10.453727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510952+0000","last_change":"2023-11-16T18:22:46.837433+0000","last_active":"2023-11-16T18:23:39.510952+0000","last_peered":"2023-11-16T18:23:39.510952+0000","last_clean":"2023-11-16T18:23:39.510952+0000","last_became_active":"2023-11-16T18:22:46.837143+0000","last_became_peered":"2023-11-16T18:22:46.837143+0000","last_unstale":"2023-11-16T18:23:39.510952+0000","last_undegraded":"2023-11-16T18:23:39.510952+0000","last_fullsized":"2023-11-16T18:23:39.510952+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:45:40.959535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510180+0000","last_change":"2023-11-16T18:22:51.123749+0000","last_active":"2023-11-16T18:23:39.510180+0000","last_peered":"2023-11-16T18:23:39.510180+0000","last_clean":"2023-11-16T18:23:39.510180+0000","last_became_active":"2023-11-16T18:22:51.123185+0000","last_became_peered":"2023-11-16T18:22:51.123185+0000","last_unstale":"2023-11-16T18:23:39.510180+0000","last_undegraded":"2023-11-16T18:23:39.510180+0000","last_fullsized":"2023-11-16T18:23:39.510180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:11:31.023936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501272+0000","last_change":"2023-11-16T18:22:46.385440+0000","last_active":"2023-11-16T18:23:39.501272+0000","last_peered":"2023-11-16T18:23:39.501272+0000","last_clean":"2023-11-16T18:23:39.501272+0000","last_became_active":"2023-11-16T18:22:46.385098+0000","last_became_peered":"2023-11-16T18:22:46.385098+0000","last_unstale":"2023-11-16T18:23:39.501272+0000","last_undegraded":"2023-11-16T18:23:39.501272+0000","last_fullsized":"2023-11-16T18:23:39.501272+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:08:49.547450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504280+0000","last_change":"2023-11-16T18:22:46.842681+0000","last_active":"2023-11-16T18:23:39.504280+0000","last_peered":"2023-11-16T18:23:39.504280+0000","last_clean":"2023-11-16T18:23:39.504280+0000","last_became_active":"2023-11-16T18:22:46.842408+0000","last_became_peered":"2023-11-16T18:22:46.842408+0000","last_unstale":"2023-11-16T18:23:39.504280+0000","last_undegraded":"2023-11-16T18:23:39.504280+0000","last_fullsized":"2023-11-16T18:23:39.504280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:37:57.050474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503905+0000","last_change":"2023-11-16T18:22:51.123450+0000","last_active":"2023-11-16T18:23:39.503905+0000","last_peered":"2023-11-16T18:23:39.503905+0000","last_clean":"2023-11-16T18:23:39.503905+0000","last_became_active":"2023-11-16T18:22:51.123002+0000","last_became_peered":"2023-11-16T18:22:51.123002+0000","last_unstale":"2023-11-16T18:23:39.503905+0000","last_undegraded":"2023-11-16T18:23:39.503905+0000","last_fullsized":"2023-11-16T18:23:39.503905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:48:43.003189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504416+0000","last_change":"2023-11-16T18:22:46.384768+0000","last_active":"2023-11-16T18:23:39.504416+0000","last_peered":"2023-11-16T18:23:39.504416+0000","last_clean":"2023-11-16T18:23:39.504416+0000","last_became_active":"2023-11-16T18:22:46.384246+0000","last_became_peered":"2023-11-16T18:22:46.384246+0000","last_unstale":"2023-11-16T18:23:39.504416+0000","last_undegraded":"2023-11-16T18:23:39.504416+0000","last_fullsized":"2023-11-16T18:23:39.504416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:20:51.259854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504317+0000","last_change":"2023-11-16T18:22:46.826069+0000","last_active":"2023-11-16T18:23:39.504317+0000","last_peered":"2023-11-16T18:23:39.504317+0000","last_clean":"2023-11-16T18:23:39.504317+0000","last_became_active":"2023-11-16T18:22:46.824676+0000","last_became_peered":"2023-11-16T18:22:46.824676+0000","last_unstale":"2023-11-16T18:23:39.504317+0000","last_undegraded":"2023-11-16T18:23:39.504317+0000","last_fullsized":"2023-11-16T18:23:39.504317+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:36:58.519456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504272+0000","last_change":"2023-11-16T18:22:51.127529+0000","last_active":"2023-11-16T18:23:39.504272+0000","last_peered":"2023-11-16T18:23:39.504272+0000","last_clean":"2023-11-16T18:23:39.504272+0000","last_became_active":"2023-11-16T18:22:51.126748+0000","last_became_peered":"2023-11-16T18:22:51.126748+0000","last_unstale":"2023-11-16T18:23:39.504272+0000","last_undegraded":"2023-11-16T18:23:39.504272+0000","last_fullsized":"2023-11-16T18:23:39.504272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:46:02.931821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505594+0000","last_change":"2023-11-16T18:22:46.385890+0000","last_active":"2023-11-16T18:23:39.505594+0000","last_peered":"2023-11-16T18:23:39.505594+0000","last_clean":"2023-11-16T18:23:39.505594+0000","last_became_active":"2023-11-16T18:22:46.385568+0000","last_became_peered":"2023-11-16T18:22:46.385568+0000","last_unstale":"2023-11-16T18:23:39.505594+0000","last_undegraded":"2023-11-16T18:23:39.505594+0000","last_fullsized":"2023-11-16T18:23:39.505594+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:45:23.088120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508492+0000","last_change":"2023-11-16T18:22:46.824796+0000","last_active":"2023-11-16T18:23:39.508492+0000","last_peered":"2023-11-16T18:23:39.508492+0000","last_clean":"2023-11-16T18:23:39.508492+0000","last_became_active":"2023-11-16T18:22:46.823895+0000","last_became_peered":"2023-11-16T18:22:46.823895+0000","last_unstale":"2023-11-16T18:23:39.508492+0000","last_undegraded":"2023-11-16T18:23:39.508492+0000","last_fullsized":"2023-11-16T18:23:39.508492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:57:28.297178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504555+0000","last_change":"2023-11-16T18:22:51.118714+0000","last_active":"2023-11-16T18:23:39.504555+0000","last_peered":"2023-11-16T18:23:39.504555+0000","last_clean":"2023-11-16T18:23:39.504555+0000","last_became_active":"2023-11-16T18:22:51.118216+0000","last_became_peered":"2023-11-16T18:22:51.118216+0000","last_unstale":"2023-11-16T18:23:39.504555+0000","last_undegraded":"2023-11-16T18:23:39.504555+0000","last_fullsized":"2023-11-16T18:23:39.504555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:27:57.339401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505119+0000","last_change":"2023-11-16T18:22:46.376431+0000","last_active":"2023-11-16T18:23:39.505119+0000","last_peered":"2023-11-16T18:23:39.505119+0000","last_clean":"2023-11-16T18:23:39.505119+0000","last_became_active":"2023-11-16T18:22:46.375914+0000","last_became_peered":"2023-11-16T18:22:46.375914+0000","last_unstale":"2023-11-16T18:23:39.505119+0000","last_undegraded":"2023-11-16T18:23:39.505119+0000","last_fullsized":"2023-11-16T18:23:39.505119+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:31:34.447838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508396+0000","last_change":"2023-11-16T18:22:46.825002+0000","last_active":"2023-11-16T18:23:39.508396+0000","last_peered":"2023-11-16T18:23:39.508396+0000","last_clean":"2023-11-16T18:23:39.508396+0000","last_became_active":"2023-11-16T18:22:46.824012+0000","last_became_peered":"2023-11-16T18:22:46.824012+0000","last_unstale":"2023-11-16T18:23:39.508396+0000","last_undegraded":"2023-11-16T18:23:39.508396+0000","last_fullsized":"2023-11-16T18:23:39.508396+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:04:41.816487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510910+0000","last_change":"2023-11-16T18:22:51.117623+0000","last_active":"2023-11-16T18:23:39.510910+0000","last_peered":"2023-11-16T18:23:39.510910+0000","last_clean":"2023-11-16T18:23:39.510910+0000","last_became_active":"2023-11-16T18:22:51.117183+0000","last_became_peered":"2023-11-16T18:22:51.117183+0000","last_unstale":"2023-11-16T18:23:39.510910+0000","last_undegraded":"2023-11-16T18:23:39.510910+0000","last_fullsized":"2023-11-16T18:23:39.510910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:16:00.063607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:35.602608+0000","last_change":"2023-11-16T18:23:39.664514+0000","last_active":"2023-11-16T18:24:35.602608+0000","last_peered":"2023-11-16T18:24:35.602608+0000","last_clean":"2023-11-16T18:24:35.602608+0000","last_became_active":"2023-11-16T18:23:39.511648+0000","last_became_peered":"2023-11-16T18:23:39.511648+0000","last_unstale":"2023-11-16T18:24:35.602608+0000","last_undegraded":"2023-11-16T18:24:35.602608+0000","last_fullsized":"2023-11-16T18:24:35.602608+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:08:22.914647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1282,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505046+0000","last_change":"2023-11-16T18:22:46.820073+0000","last_active":"2023-11-16T18:23:39.505046+0000","last_peered":"2023-11-16T18:23:39.505046+0000","last_clean":"2023-11-16T18:23:39.505046+0000","last_became_active":"2023-11-16T18:22:46.819764+0000","last_became_peered":"2023-11-16T18:22:46.819764+0000","last_unstale":"2023-11-16T18:23:39.505046+0000","last_undegraded":"2023-11-16T18:23:39.505046+0000","last_fullsized":"2023-11-16T18:23:39.505046+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:58.766286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510964+0000","last_change":"2023-11-16T18:22:51.116619+0000","last_active":"2023-11-16T18:23:39.510964+0000","last_peered":"2023-11-16T18:23:39.510964+0000","last_clean":"2023-11-16T18:23:39.510964+0000","last_became_active":"2023-11-16T18:22:51.116264+0000","last_became_peered":"2023-11-16T18:22:51.116264+0000","last_unstale":"2023-11-16T18:23:39.510964+0000","last_undegraded":"2023-11-16T18:23:39.510964+0000","last_fullsized":"2023-11-16T18:23:39.510964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:32:46.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507700+0000","last_change":"2023-11-16T18:22:46.389590+0000","last_active":"2023-11-16T18:23:39.507700+0000","last_peered":"2023-11-16T18:23:39.507700+0000","last_clean":"2023-11-16T18:23:39.507700+0000","last_became_active":"2023-11-16T18:22:46.389354+0000","last_became_peered":"2023-11-16T18:22:46.389354+0000","last_unstale":"2023-11-16T18:23:39.507700+0000","last_undegraded":"2023-11-16T18:23:39.507700+0000","last_fullsized":"2023-11-16T18:23:39.507700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:58:58.086957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509671+0000","last_change":"2023-11-16T18:22:46.841106+0000","last_active":"2023-11-16T18:23:39.509671+0000","last_peered":"2023-11-16T18:23:39.509671+0000","last_clean":"2023-11-16T18:23:39.509671+0000","last_became_active":"2023-11-16T18:22:46.840876+0000","last_became_peered":"2023-11-16T18:22:46.840876+0000","last_unstale":"2023-11-16T18:23:39.509671+0000","last_undegraded":"2023-11-16T18:23:39.509671+0000","last_fullsized":"2023-11-16T18:23:39.509671+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:46:23.740026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506599+0000","last_change":"2023-11-16T18:22:51.125813+0000","last_active":"2023-11-16T18:23:39.506599+0000","last_peered":"2023-11-16T18:23:39.506599+0000","last_clean":"2023-11-16T18:23:39.506599+0000","last_became_active":"2023-11-16T18:22:51.125277+0000","last_became_peered":"2023-11-16T18:22:51.125277+0000","last_unstale":"2023-11-16T18:23:39.506599+0000","last_undegraded":"2023-11-16T18:23:39.506599+0000","last_fullsized":"2023-11-16T18:23:39.506599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:02:51.083984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584572+0000","last_change":"2023-11-16T18:22:46.381208+0000","last_active":"2023-11-16T18:23:39.584572+0000","last_peered":"2023-11-16T18:23:39.584572+0000","last_clean":"2023-11-16T18:23:39.584572+0000","last_became_active":"2023-11-16T18:22:46.380351+0000","last_became_peered":"2023-11-16T18:22:46.380351+0000","last_unstale":"2023-11-16T18:23:39.584572+0000","last_undegraded":"2023-11-16T18:23:39.584572+0000","last_fullsized":"2023-11-16T18:23:39.584572+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:27:31.749114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504765+0000","last_change":"2023-11-16T18:22:46.843043+0000","last_active":"2023-11-16T18:23:39.504765+0000","last_peered":"2023-11-16T18:23:39.504765+0000","last_clean":"2023-11-16T18:23:39.504765+0000","last_became_active":"2023-11-16T18:22:46.841522+0000","last_became_peered":"2023-11-16T18:22:46.841522+0000","last_unstale":"2023-11-16T18:23:39.504765+0000","last_undegraded":"2023-11-16T18:23:39.504765+0000","last_fullsized":"2023-11-16T18:23:39.504765+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:53:29.309989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505013+0000","last_change":"2023-11-16T18:22:51.127729+0000","last_active":"2023-11-16T18:23:39.505013+0000","last_peered":"2023-11-16T18:23:39.505013+0000","last_clean":"2023-11-16T18:23:39.505013+0000","last_became_active":"2023-11-16T18:22:51.127303+0000","last_became_peered":"2023-11-16T18:22:51.127303+0000","last_unstale":"2023-11-16T18:23:39.505013+0000","last_undegraded":"2023-11-16T18:23:39.505013+0000","last_fullsized":"2023-11-16T18:23:39.505013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:16:37.982862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508172+0000","last_change":"2023-11-16T18:22:46.377437+0000","last_active":"2023-11-16T18:23:39.508172+0000","last_peered":"2023-11-16T18:23:39.508172+0000","last_clean":"2023-11-16T18:23:39.508172+0000","last_became_active":"2023-11-16T18:22:46.377120+0000","last_became_peered":"2023-11-16T18:22:46.377120+0000","last_unstale":"2023-11-16T18:23:39.508172+0000","last_undegraded":"2023-11-16T18:23:39.508172+0000","last_fullsized":"2023-11-16T18:23:39.508172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:02:31.673863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500756+0000","last_change":"2023-11-16T18:22:46.837804+0000","last_active":"2023-11-16T18:23:39.500756+0000","last_peered":"2023-11-16T18:23:39.500756+0000","last_clean":"2023-11-16T18:23:39.500756+0000","last_became_active":"2023-11-16T18:22:46.837432+0000","last_became_peered":"2023-11-16T18:22:46.837432+0000","last_unstale":"2023-11-16T18:23:39.500756+0000","last_undegraded":"2023-11-16T18:23:39.500756+0000","last_fullsized":"2023-11-16T18:23:39.500756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:10:14.516300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503848+0000","last_change":"2023-11-16T18:22:51.127546+0000","last_active":"2023-11-16T18:23:39.503848+0000","last_peered":"2023-11-16T18:23:39.503848+0000","last_clean":"2023-11-16T18:23:39.503848+0000","last_became_active":"2023-11-16T18:22:51.127133+0000","last_became_peered":"2023-11-16T18:22:51.127133+0000","last_unstale":"2023-11-16T18:23:39.503848+0000","last_undegraded":"2023-11-16T18:23:39.503848+0000","last_fullsized":"2023-11-16T18:23:39.503848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:02:33.244659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":117,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:33.883048+0000","last_change":"2023-11-16T18:22:46.386309+0000","last_active":"2023-11-16T18:24:33.883048+0000","last_peered":"2023-11-16T18:24:33.883048+0000","last_clean":"2023-11-16T18:24:33.883048+0000","last_became_active":"2023-11-16T18:22:46.385960+0000","last_became_peered":"2023-11-16T18:22:46.385960+0000","last_unstale":"2023-11-16T18:24:33.883048+0000","last_undegraded":"2023-11-16T18:24:33.883048+0000","last_fullsized":"2023-11-16T18:24:33.883048+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:46:45.743666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508584+0000","last_change":"2023-11-16T18:22:46.827211+0000","last_active":"2023-11-16T18:23:39.508584+0000","last_peered":"2023-11-16T18:23:39.508584+0000","last_clean":"2023-11-16T18:23:39.508584+0000","last_became_active":"2023-11-16T18:22:46.826396+0000","last_became_peered":"2023-11-16T18:22:46.826396+0000","last_unstale":"2023-11-16T18:23:39.508584+0000","last_undegraded":"2023-11-16T18:23:39.508584+0000","last_fullsized":"2023-11-16T18:23:39.508584+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:58:36.202010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.513065+0000","last_change":"2023-11-16T18:23:39.512554+0000","last_active":"2023-11-16T18:23:39.513065+0000","last_peered":"2023-11-16T18:23:39.513065+0000","last_clean":"2023-11-16T18:23:39.513065+0000","last_became_active":"2023-11-16T18:23:39.512118+0000","last_became_peered":"2023-11-16T18:23:39.512118+0000","last_unstale":"2023-11-16T18:23:39.513065+0000","last_undegraded":"2023-11-16T18:23:39.513065+0000","last_fullsized":"2023-11-16T18:23:39.513065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:55:33.269875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503214+0000","last_change":"2023-11-16T18:22:46.384680+0000","last_active":"2023-11-16T18:23:39.503214+0000","last_peered":"2023-11-16T18:23:39.503214+0000","last_clean":"2023-11-16T18:23:39.503214+0000","last_became_active":"2023-11-16T18:22:46.384290+0000","last_became_peered":"2023-11-16T18:22:46.384290+0000","last_unstale":"2023-11-16T18:23:39.503214+0000","last_undegraded":"2023-11-16T18:23:39.503214+0000","last_fullsized":"2023-11-16T18:23:39.503214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:05:40.782246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504447+0000","last_change":"2023-11-16T18:22:46.834726+0000","last_active":"2023-11-16T18:23:39.504447+0000","last_peered":"2023-11-16T18:23:39.504447+0000","last_clean":"2023-11-16T18:23:39.504447+0000","last_became_active":"2023-11-16T18:22:46.834424+0000","last_became_peered":"2023-11-16T18:22:46.834424+0000","last_unstale":"2023-11-16T18:23:39.504447+0000","last_undegraded":"2023-11-16T18:23:39.504447+0000","last_fullsized":"2023-11-16T18:23:39.504447+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:52:11.738947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.513064+0000","last_change":"2023-11-16T18:23:39.513064+0000","last_active":"2023-11-16T18:23:39.513064+0000","last_peered":"2023-11-16T18:23:39.513064+0000","last_clean":"2023-11-16T18:23:39.513064+0000","last_became_active":"2023-11-16T18:23:39.512606+0000","last_became_peered":"2023-11-16T18:23:39.512606+0000","last_unstale":"2023-11-16T18:23:39.513064+0000","last_undegraded":"2023-11-16T18:23:39.513064+0000","last_fullsized":"2023-11-16T18:23:39.513064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:08:18.327854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508258+0000","last_change":"2023-11-16T18:22:46.380804+0000","last_active":"2023-11-16T18:23:39.508258+0000","last_peered":"2023-11-16T18:23:39.508258+0000","last_clean":"2023-11-16T18:23:39.508258+0000","last_became_active":"2023-11-16T18:22:46.380477+0000","last_became_peered":"2023-11-16T18:22:46.380477+0000","last_unstale":"2023-11-16T18:23:39.508258+0000","last_undegraded":"2023-11-16T18:23:39.508258+0000","last_fullsized":"2023-11-16T18:23:39.508258+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:17:45.128995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.586650+0000","last_change":"2023-11-16T18:23:39.586650+0000","last_active":"2023-11-16T18:23:39.586650+0000","last_peered":"2023-11-16T18:23:39.586650+0000","last_clean":"2023-11-16T18:23:39.586650+0000","last_became_active":"2023-11-16T18:23:39.586241+0000","last_became_peered":"2023-11-16T18:23:39.586241+0000","last_unstale":"2023-11-16T18:23:39.586650+0000","last_undegraded":"2023-11-16T18:23:39.586650+0000","last_fullsized":"2023-11-16T18:23:39.586650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:15:14.907282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500754+0000","last_change":"2023-11-16T18:22:51.119711+0000","last_active":"2023-11-16T18:23:39.500754+0000","last_peered":"2023-11-16T18:23:39.500754+0000","last_clean":"2023-11-16T18:23:39.500754+0000","last_became_active":"2023-11-16T18:22:51.119304+0000","last_became_peered":"2023-11-16T18:22:51.119304+0000","last_unstale":"2023-11-16T18:23:39.500754+0000","last_undegraded":"2023-11-16T18:23:39.500754+0000","last_fullsized":"2023-11-16T18:23:39.500754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:37:34.450684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":260,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:34.600234+0000","last_change":"2023-11-16T18:23:39.671486+0000","last_active":"2023-11-16T18:24:34.600234+0000","last_peered":"2023-11-16T18:24:34.600234+0000","last_clean":"2023-11-16T18:24:34.600234+0000","last_became_active":"2023-11-16T18:23:39.513413+0000","last_became_peered":"2023-11-16T18:23:39.513413+0000","last_unstale":"2023-11-16T18:24:34.600234+0000","last_undegraded":"2023-11-16T18:24:34.600234+0000","last_fullsized":"2023-11-16T18:24:34.600234+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:07:37.199058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61155,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":178,"num_read_kb":147,"num_write":34,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60837,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504736+0000","last_change":"2023-11-16T18:22:46.833537+0000","last_active":"2023-11-16T18:23:39.504736+0000","last_peered":"2023-11-16T18:23:39.504736+0000","last_clean":"2023-11-16T18:23:39.504736+0000","last_became_active":"2023-11-16T18:22:46.833217+0000","last_became_peered":"2023-11-16T18:22:46.833217+0000","last_unstale":"2023-11-16T18:23:39.504736+0000","last_undegraded":"2023-11-16T18:23:39.504736+0000","last_fullsized":"2023-11-16T18:23:39.504736+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:54:32.987198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585712+0000","last_change":"2023-11-16T18:23:39.585247+0000","last_active":"2023-11-16T18:23:39.585712+0000","last_peered":"2023-11-16T18:23:39.585712+0000","last_clean":"2023-11-16T18:23:39.585712+0000","last_became_active":"2023-11-16T18:23:39.584802+0000","last_became_peered":"2023-11-16T18:23:39.584802+0000","last_unstale":"2023-11-16T18:23:39.585712+0000","last_undegraded":"2023-11-16T18:23:39.585712+0000","last_fullsized":"2023-11-16T18:23:39.585712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:45:43.860286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509671+0000","last_change":"2023-11-16T18:22:46.382812+0000","last_active":"2023-11-16T18:23:39.509671+0000","last_peered":"2023-11-16T18:23:39.509671+0000","last_clean":"2023-11-16T18:23:39.509671+0000","last_became_active":"2023-11-16T18:22:46.382499+0000","last_became_peered":"2023-11-16T18:22:46.382499+0000","last_unstale":"2023-11-16T18:23:39.509671+0000","last_undegraded":"2023-11-16T18:23:39.509671+0000","last_fullsized":"2023-11-16T18:23:39.509671+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:48:39.263999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584764+0000","last_change":"2023-11-16T18:22:46.840917+0000","last_active":"2023-11-16T18:23:39.584764+0000","last_peered":"2023-11-16T18:23:39.584764+0000","last_clean":"2023-11-16T18:23:39.584764+0000","last_became_active":"2023-11-16T18:22:46.840579+0000","last_became_peered":"2023-11-16T18:22:46.840579+0000","last_unstale":"2023-11-16T18:23:39.584764+0000","last_undegraded":"2023-11-16T18:23:39.584764+0000","last_fullsized":"2023-11-16T18:23:39.584764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:05:17.920819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506477+0000","last_change":"2023-11-16T18:22:51.117852+0000","last_active":"2023-11-16T18:23:39.506477+0000","last_peered":"2023-11-16T18:23:39.506477+0000","last_clean":"2023-11-16T18:23:39.506477+0000","last_became_active":"2023-11-16T18:22:51.117390+0000","last_became_peered":"2023-11-16T18:22:51.117390+0000","last_unstale":"2023-11-16T18:23:39.506477+0000","last_undegraded":"2023-11-16T18:23:39.506477+0000","last_fullsized":"2023-11-16T18:23:39.506477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:11:37.167591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508136+0000","last_change":"2023-11-16T18:22:46.385137+0000","last_active":"2023-11-16T18:23:39.508136+0000","last_peered":"2023-11-16T18:23:39.508136+0000","last_clean":"2023-11-16T18:23:39.508136+0000","last_became_active":"2023-11-16T18:22:46.384636+0000","last_became_peered":"2023-11-16T18:22:46.384636+0000","last_unstale":"2023-11-16T18:23:39.508136+0000","last_undegraded":"2023-11-16T18:23:39.508136+0000","last_fullsized":"2023-11-16T18:23:39.508136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:11:10.885960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502619+0000","last_change":"2023-11-16T18:22:46.834786+0000","last_active":"2023-11-16T18:23:39.502619+0000","last_peered":"2023-11-16T18:23:39.502619+0000","last_clean":"2023-11-16T18:23:39.502619+0000","last_became_active":"2023-11-16T18:22:46.834338+0000","last_became_peered":"2023-11-16T18:22:46.834338+0000","last_unstale":"2023-11-16T18:23:39.502619+0000","last_undegraded":"2023-11-16T18:23:39.502619+0000","last_fullsized":"2023-11-16T18:23:39.502619+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:52:01.046978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500921+0000","last_change":"2023-11-16T18:22:51.124667+0000","last_active":"2023-11-16T18:23:39.500921+0000","last_peered":"2023-11-16T18:23:39.500921+0000","last_clean":"2023-11-16T18:23:39.500921+0000","last_became_active":"2023-11-16T18:22:51.124211+0000","last_became_peered":"2023-11-16T18:22:51.124211+0000","last_unstale":"2023-11-16T18:23:39.500921+0000","last_undegraded":"2023-11-16T18:23:39.500921+0000","last_fullsized":"2023-11-16T18:23:39.500921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:22:02.360209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":121,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:34.054214+0000","last_change":"2023-11-16T18:22:46.380829+0000","last_active":"2023-11-16T18:24:34.054214+0000","last_peered":"2023-11-16T18:24:34.054214+0000","last_clean":"2023-11-16T18:24:34.054214+0000","last_became_active":"2023-11-16T18:22:46.380466+0000","last_became_peered":"2023-11-16T18:22:46.380466+0000","last_unstale":"2023-11-16T18:24:34.054214+0000","last_undegraded":"2023-11-16T18:24:34.054214+0000","last_fullsized":"2023-11-16T18:24:34.054214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:22:52.401291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502257+0000","last_change":"2023-11-16T18:22:46.835900+0000","last_active":"2023-11-16T18:23:39.502257+0000","last_peered":"2023-11-16T18:23:39.502257+0000","last_clean":"2023-11-16T18:23:39.502257+0000","last_became_active":"2023-11-16T18:22:46.835438+0000","last_became_peered":"2023-11-16T18:22:46.835438+0000","last_unstale":"2023-11-16T18:23:39.502257+0000","last_undegraded":"2023-11-16T18:23:39.502257+0000","last_fullsized":"2023-11-16T18:23:39.502257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:36:23.733440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585779+0000","last_change":"2023-11-16T18:22:51.120005+0000","last_active":"2023-11-16T18:23:39.585779+0000","last_peered":"2023-11-16T18:23:39.585779+0000","last_clean":"2023-11-16T18:23:39.585779+0000","last_became_active":"2023-11-16T18:22:51.119540+0000","last_became_peered":"2023-11-16T18:22:51.119540+0000","last_unstale":"2023-11-16T18:23:39.585779+0000","last_undegraded":"2023-11-16T18:23:39.585779+0000","last_fullsized":"2023-11-16T18:23:39.585779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:33:29.490015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:41.897854+0000","last_change":"2023-11-16T18:23:41.897338+0000","last_active":"2023-11-16T18:23:41.897854+0000","last_peered":"2023-11-16T18:23:41.897854+0000","last_clean":"2023-11-16T18:23:41.897854+0000","last_became_active":"2023-11-16T18:23:39.512018+0000","last_became_peered":"2023-11-16T18:23:39.512018+0000","last_unstale":"2023-11-16T18:23:41.897854+0000","last_undegraded":"2023-11-16T18:23:41.897854+0000","last_fullsized":"2023-11-16T18:23:41.897854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:53:22.707297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507902+0000","last_change":"2023-11-16T18:22:46.823838+0000","last_active":"2023-11-16T18:23:39.507902+0000","last_peered":"2023-11-16T18:23:39.507902+0000","last_clean":"2023-11-16T18:23:39.507902+0000","last_became_active":"2023-11-16T18:22:46.823545+0000","last_became_peered":"2023-11-16T18:22:46.823545+0000","last_unstale":"2023-11-16T18:23:39.507902+0000","last_undegraded":"2023-11-16T18:23:39.507902+0000","last_fullsized":"2023-11-16T18:23:39.507902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:11:32.214804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507865+0000","last_change":"2023-11-16T18:22:51.127371+0000","last_active":"2023-11-16T18:23:39.507865+0000","last_peered":"2023-11-16T18:23:39.507865+0000","last_clean":"2023-11-16T18:23:39.507865+0000","last_became_active":"2023-11-16T18:22:51.126951+0000","last_became_peered":"2023-11-16T18:22:51.126951+0000","last_unstale":"2023-11-16T18:23:39.507865+0000","last_undegraded":"2023-11-16T18:23:39.507865+0000","last_fullsized":"2023-11-16T18:23:39.507865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:55.418916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504323+0000","last_change":"2023-11-16T18:22:46.376561+0000","last_active":"2023-11-16T18:23:39.504323+0000","last_peered":"2023-11-16T18:23:39.504323+0000","last_clean":"2023-11-16T18:23:39.504323+0000","last_became_active":"2023-11-16T18:22:46.375426+0000","last_became_peered":"2023-11-16T18:22:46.375426+0000","last_unstale":"2023-11-16T18:23:39.504323+0000","last_undegraded":"2023-11-16T18:23:39.504323+0000","last_fullsized":"2023-11-16T18:23:39.504323+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:29:00.383105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505100+0000","last_change":"2023-11-16T18:22:46.826657+0000","last_active":"2023-11-16T18:23:39.505100+0000","last_peered":"2023-11-16T18:23:39.505100+0000","last_clean":"2023-11-16T18:23:39.505100+0000","last_became_active":"2023-11-16T18:22:46.826126+0000","last_became_peered":"2023-11-16T18:22:46.826126+0000","last_unstale":"2023-11-16T18:23:39.505100+0000","last_undegraded":"2023-11-16T18:23:39.505100+0000","last_fullsized":"2023-11-16T18:23:39.505100+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:13:34.794628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504904+0000","last_change":"2023-11-16T18:22:51.127949+0000","last_active":"2023-11-16T18:23:39.504904+0000","last_peered":"2023-11-16T18:23:39.504904+0000","last_clean":"2023-11-16T18:23:39.504904+0000","last_became_active":"2023-11-16T18:22:51.127388+0000","last_became_peered":"2023-11-16T18:22:51.127388+0000","last_unstale":"2023-11-16T18:23:39.504904+0000","last_undegraded":"2023-11-16T18:23:39.504904+0000","last_fullsized":"2023-11-16T18:23:39.504904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:37:30.745978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500985+0000","last_change":"2023-11-16T18:22:46.378568+0000","last_active":"2023-11-16T18:23:39.500985+0000","last_peered":"2023-11-16T18:23:39.500985+0000","last_clean":"2023-11-16T18:23:39.500985+0000","last_became_active":"2023-11-16T18:22:46.378018+0000","last_became_peered":"2023-11-16T18:22:46.378018+0000","last_unstale":"2023-11-16T18:23:39.500985+0000","last_undegraded":"2023-11-16T18:23:39.500985+0000","last_fullsized":"2023-11-16T18:23:39.500985+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:05:59.328232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509189+0000","last_change":"2023-11-16T18:22:46.841129+0000","last_active":"2023-11-16T18:23:39.509189+0000","last_peered":"2023-11-16T18:23:39.509189+0000","last_clean":"2023-11-16T18:23:39.509189+0000","last_became_active":"2023-11-16T18:22:46.840780+0000","last_became_peered":"2023-11-16T18:22:46.840780+0000","last_unstale":"2023-11-16T18:23:39.509189+0000","last_undegraded":"2023-11-16T18:23:39.509189+0000","last_fullsized":"2023-11-16T18:23:39.509189+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:09:39.912158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585594+0000","last_change":"2023-11-16T18:22:51.126221+0000","last_active":"2023-11-16T18:23:39.585594+0000","last_peered":"2023-11-16T18:23:39.585594+0000","last_clean":"2023-11-16T18:23:39.585594+0000","last_became_active":"2023-11-16T18:22:51.125770+0000","last_became_peered":"2023-11-16T18:22:51.125770+0000","last_unstale":"2023-11-16T18:23:39.585594+0000","last_undegraded":"2023-11-16T18:23:39.585594+0000","last_fullsized":"2023-11-16T18:23:39.585594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:35:06.867124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502402+0000","last_change":"2023-11-16T18:22:46.384872+0000","last_active":"2023-11-16T18:23:39.502402+0000","last_peered":"2023-11-16T18:23:39.502402+0000","last_clean":"2023-11-16T18:23:39.502402+0000","last_became_active":"2023-11-16T18:22:46.384499+0000","last_became_peered":"2023-11-16T18:22:46.384499+0000","last_unstale":"2023-11-16T18:23:39.502402+0000","last_undegraded":"2023-11-16T18:23:39.502402+0000","last_fullsized":"2023-11-16T18:23:39.502402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:26:53.535471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507989+0000","last_change":"2023-11-16T18:22:46.822574+0000","last_active":"2023-11-16T18:23:39.507989+0000","last_peered":"2023-11-16T18:23:39.507989+0000","last_clean":"2023-11-16T18:23:39.507989+0000","last_became_active":"2023-11-16T18:22:46.822230+0000","last_became_peered":"2023-11-16T18:22:46.822230+0000","last_unstale":"2023-11-16T18:23:39.507989+0000","last_undegraded":"2023-11-16T18:23:39.507989+0000","last_fullsized":"2023-11-16T18:23:39.507989+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:16:44.677701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585706+0000","last_change":"2023-11-16T18:22:51.117838+0000","last_active":"2023-11-16T18:23:39.585706+0000","last_peered":"2023-11-16T18:23:39.585706+0000","last_clean":"2023-11-16T18:23:39.585706+0000","last_became_active":"2023-11-16T18:22:51.117275+0000","last_became_peered":"2023-11-16T18:22:51.117275+0000","last_unstale":"2023-11-16T18:23:39.585706+0000","last_undegraded":"2023-11-16T18:23:39.585706+0000","last_fullsized":"2023-11-16T18:23:39.585706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:42:14.482945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508453+0000","last_change":"2023-11-16T18:22:46.827499+0000","last_active":"2023-11-16T18:23:39.508453+0000","last_peered":"2023-11-16T18:23:39.508453+0000","last_clean":"2023-11-16T18:23:39.508453+0000","last_became_active":"2023-11-16T18:22:46.827166+0000","last_became_peered":"2023-11-16T18:22:46.827166+0000","last_unstale":"2023-11-16T18:23:39.508453+0000","last_undegraded":"2023-11-16T18:23:39.508453+0000","last_fullsized":"2023-11-16T18:23:39.508453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:23:43.219136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":116,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:32.050352+0000","last_change":"2023-11-16T18:22:46.381397+0000","last_active":"2023-11-16T18:24:32.050352+0000","last_peered":"2023-11-16T18:24:32.050352+0000","last_clean":"2023-11-16T18:24:32.050352+0000","last_became_active":"2023-11-16T18:22:46.380939+0000","last_became_peered":"2023-11-16T18:22:46.380939+0000","last_unstale":"2023-11-16T18:24:32.050352+0000","last_undegraded":"2023-11-16T18:24:32.050352+0000","last_fullsized":"2023-11-16T18:24:32.050352+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:44:30.574771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502915+0000","last_change":"2023-11-16T18:22:51.125044+0000","last_active":"2023-11-16T18:23:39.502915+0000","last_peered":"2023-11-16T18:23:39.502915+0000","last_clean":"2023-11-16T18:23:39.502915+0000","last_became_active":"2023-11-16T18:22:51.124422+0000","last_became_peered":"2023-11-16T18:22:51.124422+0000","last_unstale":"2023-11-16T18:23:39.502915+0000","last_undegraded":"2023-11-16T18:23:39.502915+0000","last_fullsized":"2023-11-16T18:23:39.502915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:11.651599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503647+0000","last_change":"2023-11-16T18:22:46.841500+0000","last_active":"2023-11-16T18:23:39.503647+0000","last_peered":"2023-11-16T18:23:39.503647+0000","last_clean":"2023-11-16T18:23:39.503647+0000","last_became_active":"2023-11-16T18:22:46.841186+0000","last_became_peered":"2023-11-16T18:22:46.841186+0000","last_unstale":"2023-11-16T18:23:39.503647+0000","last_undegraded":"2023-11-16T18:23:39.503647+0000","last_fullsized":"2023-11-16T18:23:39.503647+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:38:25.407837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504887+0000","last_change":"2023-11-16T18:22:46.380988+0000","last_active":"2023-11-16T18:23:39.504887+0000","last_peered":"2023-11-16T18:23:39.504887+0000","last_clean":"2023-11-16T18:23:39.504887+0000","last_became_active":"2023-11-16T18:22:46.380616+0000","last_became_peered":"2023-11-16T18:22:46.380616+0000","last_unstale":"2023-11-16T18:23:39.504887+0000","last_undegraded":"2023-11-16T18:23:39.504887+0000","last_fullsized":"2023-11-16T18:23:39.504887+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:24:26.911536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502037+0000","last_change":"2023-11-16T18:21:44.282706+0000","last_active":"2023-11-16T18:23:39.502037+0000","last_peered":"2023-11-16T18:23:39.502037+0000","last_clean":"2023-11-16T18:23:39.502037+0000","last_became_active":"2023-11-16T18:21:44.282201+0000","last_became_peered":"2023-11-16T18:21:44.282201+0000","last_unstale":"2023-11-16T18:23:39.502037+0000","last_undegraded":"2023-11-16T18:23:39.502037+0000","last_fullsized":"2023-11-16T18:23:39.502037+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_clean_scrub_stamp":"2023-11-16T18:18:46.711191+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:41:17.702365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508868+0000","last_change":"2023-11-16T18:22:51.123051+0000","last_active":"2023-11-16T18:23:39.508868+0000","last_peered":"2023-11-16T18:23:39.508868+0000","last_clean":"2023-11-16T18:23:39.508868+0000","last_became_active":"2023-11-16T18:22:51.122635+0000","last_became_peered":"2023-11-16T18:22:51.122635+0000","last_unstale":"2023-11-16T18:23:39.508868+0000","last_undegraded":"2023-11-16T18:23:39.508868+0000","last_fullsized":"2023-11-16T18:23:39.508868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:59:09.977278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584296+0000","last_change":"2023-11-16T18:22:46.822625+0000","last_active":"2023-11-16T18:23:39.584296+0000","last_peered":"2023-11-16T18:23:39.584296+0000","last_clean":"2023-11-16T18:23:39.584296+0000","last_became_active":"2023-11-16T18:22:46.822296+0000","last_became_peered":"2023-11-16T18:22:46.822296+0000","last_unstale":"2023-11-16T18:23:39.584296+0000","last_undegraded":"2023-11-16T18:23:39.584296+0000","last_fullsized":"2023-11-16T18:23:39.584296+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:43:40.832756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502155+0000","last_change":"2023-11-16T18:22:46.380445+0000","last_active":"2023-11-16T18:23:39.502155+0000","last_peered":"2023-11-16T18:23:39.502155+0000","last_clean":"2023-11-16T18:23:39.502155+0000","last_became_active":"2023-11-16T18:22:46.380113+0000","last_became_peered":"2023-11-16T18:22:46.380113+0000","last_unstale":"2023-11-16T18:23:39.502155+0000","last_undegraded":"2023-11-16T18:23:39.502155+0000","last_fullsized":"2023-11-16T18:23:39.502155+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:17:48.769928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504943+0000","last_change":"2023-11-16T18:22:51.118457+0000","last_active":"2023-11-16T18:23:39.504943+0000","last_peered":"2023-11-16T18:23:39.504943+0000","last_clean":"2023-11-16T18:23:39.504943+0000","last_became_active":"2023-11-16T18:22:51.118013+0000","last_became_peered":"2023-11-16T18:22:51.118013+0000","last_unstale":"2023-11-16T18:23:39.504943+0000","last_undegraded":"2023-11-16T18:23:39.504943+0000","last_fullsized":"2023-11-16T18:23:39.504943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:11.432801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506267+0000","last_change":"2023-11-16T18:22:46.384799+0000","last_active":"2023-11-16T18:23:39.506267+0000","last_peered":"2023-11-16T18:23:39.506267+0000","last_clean":"2023-11-16T18:23:39.506267+0000","last_became_active":"2023-11-16T18:22:46.384482+0000","last_became_peered":"2023-11-16T18:22:46.384482+0000","last_unstale":"2023-11-16T18:23:39.506267+0000","last_undegraded":"2023-11-16T18:23:39.506267+0000","last_fullsized":"2023-11-16T18:23:39.506267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:42:48.220989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585377+0000","last_change":"2023-11-16T18:22:46.829541+0000","last_active":"2023-11-16T18:23:39.585377+0000","last_peered":"2023-11-16T18:23:39.585377+0000","last_clean":"2023-11-16T18:23:39.585377+0000","last_became_active":"2023-11-16T18:22:46.829006+0000","last_became_peered":"2023-11-16T18:22:46.829006+0000","last_unstale":"2023-11-16T18:23:39.585377+0000","last_undegraded":"2023-11-16T18:23:39.585377+0000","last_fullsized":"2023-11-16T18:23:39.585377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:13:02.956336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504640+0000","last_change":"2023-11-16T18:22:51.122751+0000","last_active":"2023-11-16T18:23:39.504640+0000","last_peered":"2023-11-16T18:23:39.504640+0000","last_clean":"2023-11-16T18:23:39.504640+0000","last_became_active":"2023-11-16T18:22:51.122423+0000","last_became_peered":"2023-11-16T18:22:51.122423+0000","last_unstale":"2023-11-16T18:23:39.504640+0000","last_undegraded":"2023-11-16T18:23:39.504640+0000","last_fullsized":"2023-11-16T18:23:39.504640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:27:49.338555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'5","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.516672+0000","last_change":"2023-11-16T18:23:39.516672+0000","last_active":"2023-11-16T18:23:39.516672+0000","last_peered":"2023-11-16T18:23:39.516672+0000","last_clean":"2023-11-16T18:23:39.516672+0000","last_became_active":"2023-11-16T18:23:39.509087+0000","last_became_peered":"2023-11-16T18:23:39.509087+0000","last_unstale":"2023-11-16T18:23:39.516672+0000","last_undegraded":"2023-11-16T18:23:39.516672+0000","last_fullsized":"2023-11-16T18:23:39.516672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:57:28.479767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506041+0000","last_change":"2023-11-16T18:22:46.843802+0000","last_active":"2023-11-16T18:23:39.506041+0000","last_peered":"2023-11-16T18:23:39.506041+0000","last_clean":"2023-11-16T18:23:39.506041+0000","last_became_active":"2023-11-16T18:22:46.842210+0000","last_became_peered":"2023-11-16T18:22:46.842210+0000","last_unstale":"2023-11-16T18:23:39.506041+0000","last_undegraded":"2023-11-16T18:23:39.506041+0000","last_fullsized":"2023-11-16T18:23:39.506041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:20:02.893140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509951+0000","last_change":"2023-11-16T18:22:51.124632+0000","last_active":"2023-11-16T18:23:39.509951+0000","last_peered":"2023-11-16T18:23:39.509951+0000","last_clean":"2023-11-16T18:23:39.509951+0000","last_became_active":"2023-11-16T18:22:51.124220+0000","last_became_peered":"2023-11-16T18:22:51.124220+0000","last_unstale":"2023-11-16T18:23:39.509951+0000","last_undegraded":"2023-11-16T18:23:39.509951+0000","last_fullsized":"2023-11-16T18:23:39.509951+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:51:57.981627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505140+0000","last_change":"2023-11-16T18:22:46.381456+0000","last_active":"2023-11-16T18:23:39.505140+0000","last_peered":"2023-11-16T18:23:39.505140+0000","last_clean":"2023-11-16T18:23:39.505140+0000","last_became_active":"2023-11-16T18:22:46.381137+0000","last_became_peered":"2023-11-16T18:22:46.381137+0000","last_unstale":"2023-11-16T18:23:39.505140+0000","last_undegraded":"2023-11-16T18:23:39.505140+0000","last_fullsized":"2023-11-16T18:23:39.505140+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:34:36.759190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505363+0000","last_change":"2023-11-16T18:22:46.838249+0000","last_active":"2023-11-16T18:23:39.505363+0000","last_peered":"2023-11-16T18:23:39.505363+0000","last_clean":"2023-11-16T18:23:39.505363+0000","last_became_active":"2023-11-16T18:22:46.837925+0000","last_became_peered":"2023-11-16T18:22:46.837925+0000","last_unstale":"2023-11-16T18:23:39.505363+0000","last_undegraded":"2023-11-16T18:23:39.505363+0000","last_fullsized":"2023-11-16T18:23:39.505363+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:02:17.287183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504877+0000","last_change":"2023-11-16T18:22:51.123687+0000","last_active":"2023-11-16T18:23:39.504877+0000","last_peered":"2023-11-16T18:23:39.504877+0000","last_clean":"2023-11-16T18:23:39.504877+0000","last_became_active":"2023-11-16T18:22:51.123209+0000","last_became_peered":"2023-11-16T18:22:51.123209+0000","last_unstale":"2023-11-16T18:23:39.504877+0000","last_undegraded":"2023-11-16T18:23:39.504877+0000","last_fullsized":"2023-11-16T18:23:39.504877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:09:15.998485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508037+0000","last_change":"2023-11-16T18:22:46.378038+0000","last_active":"2023-11-16T18:23:39.508037+0000","last_peered":"2023-11-16T18:23:39.508037+0000","last_clean":"2023-11-16T18:23:39.508037+0000","last_became_active":"2023-11-16T18:22:46.377680+0000","last_became_peered":"2023-11-16T18:22:46.377680+0000","last_unstale":"2023-11-16T18:23:39.508037+0000","last_undegraded":"2023-11-16T18:23:39.508037+0000","last_fullsized":"2023-11-16T18:23:39.508037+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:31.194156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504247+0000","last_change":"2023-11-16T18:22:46.821888+0000","last_active":"2023-11-16T18:23:39.504247+0000","last_peered":"2023-11-16T18:23:39.504247+0000","last_clean":"2023-11-16T18:23:39.504247+0000","last_became_active":"2023-11-16T18:22:46.821446+0000","last_became_peered":"2023-11-16T18:22:46.821446+0000","last_unstale":"2023-11-16T18:23:39.504247+0000","last_undegraded":"2023-11-16T18:23:39.504247+0000","last_fullsized":"2023-11-16T18:23:39.504247+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:11:30.632655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501162+0000","last_change":"2023-11-16T18:22:51.119717+0000","last_active":"2023-11-16T18:23:39.501162+0000","last_peered":"2023-11-16T18:23:39.501162+0000","last_clean":"2023-11-16T18:23:39.501162+0000","last_became_active":"2023-11-16T18:22:51.119256+0000","last_became_peered":"2023-11-16T18:22:51.119256+0000","last_unstale":"2023-11-16T18:23:39.501162+0000","last_undegraded":"2023-11-16T18:23:39.501162+0000","last_fullsized":"2023-11-16T18:23:39.501162+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:25:42.785279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504001+0000","last_change":"2023-11-16T18:22:46.375387+0000","last_active":"2023-11-16T18:23:39.504001+0000","last_peered":"2023-11-16T18:23:39.504001+0000","last_clean":"2023-11-16T18:23:39.504001+0000","last_became_active":"2023-11-16T18:22:46.374886+0000","last_became_peered":"2023-11-16T18:22:46.374886+0000","last_unstale":"2023-11-16T18:23:39.504001+0000","last_undegraded":"2023-11-16T18:23:39.504001+0000","last_fullsized":"2023-11-16T18:23:39.504001+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:18:24.699496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504057+0000","last_change":"2023-11-16T18:22:46.820315+0000","last_active":"2023-11-16T18:23:39.504057+0000","last_peered":"2023-11-16T18:23:39.504057+0000","last_clean":"2023-11-16T18:23:39.504057+0000","last_became_active":"2023-11-16T18:22:46.819592+0000","last_became_peered":"2023-11-16T18:22:46.819592+0000","last_unstale":"2023-11-16T18:23:39.504057+0000","last_undegraded":"2023-11-16T18:23:39.504057+0000","last_fullsized":"2023-11-16T18:23:39.504057+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:43:26.336090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.518963+0000","last_change":"2023-11-16T18:23:39.518963+0000","last_active":"2023-11-16T18:23:39.518963+0000","last_peered":"2023-11-16T18:23:39.518963+0000","last_clean":"2023-11-16T18:23:39.518963+0000","last_became_active":"2023-11-16T18:23:39.513817+0000","last_became_peered":"2023-11-16T18:23:39.513817+0000","last_unstale":"2023-11-16T18:23:39.518963+0000","last_undegraded":"2023-11-16T18:23:39.518963+0000","last_fullsized":"2023-11-16T18:23:39.518963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:16:46.629599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510801+0000","last_change":"2023-11-16T18:22:46.378638+0000","last_active":"2023-11-16T18:23:39.510801+0000","last_peered":"2023-11-16T18:23:39.510801+0000","last_clean":"2023-11-16T18:23:39.510801+0000","last_became_active":"2023-11-16T18:22:46.378341+0000","last_became_peered":"2023-11-16T18:22:46.378341+0000","last_unstale":"2023-11-16T18:23:39.510801+0000","last_undegraded":"2023-11-16T18:23:39.510801+0000","last_fullsized":"2023-11-16T18:23:39.510801+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:36:28.625148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504347+0000","last_change":"2023-11-16T18:22:46.843431+0000","last_active":"2023-11-16T18:23:39.504347+0000","last_peered":"2023-11-16T18:23:39.504347+0000","last_clean":"2023-11-16T18:23:39.504347+0000","last_became_active":"2023-11-16T18:22:46.842992+0000","last_became_peered":"2023-11-16T18:22:46.842992+0000","last_unstale":"2023-11-16T18:23:39.504347+0000","last_undegraded":"2023-11-16T18:23:39.504347+0000","last_fullsized":"2023-11-16T18:23:39.504347+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:23:28.821023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503912+0000","last_change":"2023-11-16T18:22:51.122640+0000","last_active":"2023-11-16T18:23:39.503912+0000","last_peered":"2023-11-16T18:23:39.503912+0000","last_clean":"2023-11-16T18:23:39.503912+0000","last_became_active":"2023-11-16T18:22:51.122006+0000","last_became_peered":"2023-11-16T18:22:51.122006+0000","last_unstale":"2023-11-16T18:23:39.503912+0000","last_undegraded":"2023-11-16T18:23:39.503912+0000","last_fullsized":"2023-11-16T18:23:39.503912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:27:30.406233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504268+0000","last_change":"2023-11-16T18:22:46.384837+0000","last_active":"2023-11-16T18:23:39.504268+0000","last_peered":"2023-11-16T18:23:39.504268+0000","last_clean":"2023-11-16T18:23:39.504268+0000","last_became_active":"2023-11-16T18:22:46.384541+0000","last_became_peered":"2023-11-16T18:22:46.384541+0000","last_unstale":"2023-11-16T18:23:39.504268+0000","last_undegraded":"2023-11-16T18:23:39.504268+0000","last_fullsized":"2023-11-16T18:23:39.504268+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:06:22.863895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502460+0000","last_change":"2023-11-16T18:22:46.839967+0000","last_active":"2023-11-16T18:23:39.502460+0000","last_peered":"2023-11-16T18:23:39.502460+0000","last_clean":"2023-11-16T18:23:39.502460+0000","last_became_active":"2023-11-16T18:22:46.839645+0000","last_became_peered":"2023-11-16T18:22:46.839645+0000","last_unstale":"2023-11-16T18:23:39.502460+0000","last_undegraded":"2023-11-16T18:23:39.502460+0000","last_fullsized":"2023-11-16T18:23:39.502460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:04:34.616063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507240+0000","last_change":"2023-11-16T18:22:51.126548+0000","last_active":"2023-11-16T18:23:39.507240+0000","last_peered":"2023-11-16T18:23:39.507240+0000","last_clean":"2023-11-16T18:23:39.507240+0000","last_became_active":"2023-11-16T18:22:51.126039+0000","last_became_peered":"2023-11-16T18:22:51.126039+0000","last_unstale":"2023-11-16T18:23:39.507240+0000","last_undegraded":"2023-11-16T18:23:39.507240+0000","last_fullsized":"2023-11-16T18:23:39.507240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:22:07.650645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.511343+0000","last_change":"2023-11-16T18:22:46.383947+0000","last_active":"2023-11-16T18:23:39.511343+0000","last_peered":"2023-11-16T18:23:39.511343+0000","last_clean":"2023-11-16T18:23:39.511343+0000","last_became_active":"2023-11-16T18:22:46.383661+0000","last_became_peered":"2023-11-16T18:22:46.383661+0000","last_unstale":"2023-11-16T18:23:39.511343+0000","last_undegraded":"2023-11-16T18:23:39.511343+0000","last_fullsized":"2023-11-16T18:23:39.511343+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:34.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503182+0000","last_change":"2023-11-16T18:22:46.836877+0000","last_active":"2023-11-16T18:23:39.503182+0000","last_peered":"2023-11-16T18:23:39.503182+0000","last_clean":"2023-11-16T18:23:39.503182+0000","last_became_active":"2023-11-16T18:22:46.836477+0000","last_became_peered":"2023-11-16T18:22:46.836477+0000","last_unstale":"2023-11-16T18:23:39.503182+0000","last_undegraded":"2023-11-16T18:23:39.503182+0000","last_fullsized":"2023-11-16T18:23:39.503182+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:38:49.681616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505171+0000","last_change":"2023-11-16T18:22:51.126930+0000","last_active":"2023-11-16T18:23:39.505171+0000","last_peered":"2023-11-16T18:23:39.505171+0000","last_clean":"2023-11-16T18:23:39.505171+0000","last_became_active":"2023-11-16T18:22:51.126409+0000","last_became_peered":"2023-11-16T18:22:51.126409+0000","last_unstale":"2023-11-16T18:23:39.505171+0000","last_undegraded":"2023-11-16T18:23:39.505171+0000","last_fullsized":"2023-11-16T18:23:39.505171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:31:50.012322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508533+0000","last_change":"2023-11-16T18:22:46.380820+0000","last_active":"2023-11-16T18:23:39.508533+0000","last_peered":"2023-11-16T18:23:39.508533+0000","last_clean":"2023-11-16T18:23:39.508533+0000","last_became_active":"2023-11-16T18:22:46.380446+0000","last_became_peered":"2023-11-16T18:22:46.380446+0000","last_unstale":"2023-11-16T18:23:39.508533+0000","last_undegraded":"2023-11-16T18:23:39.508533+0000","last_fullsized":"2023-11-16T18:23:39.508533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:20:07.923022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502955+0000","last_change":"2023-11-16T18:22:46.838153+0000","last_active":"2023-11-16T18:23:39.502955+0000","last_peered":"2023-11-16T18:23:39.502955+0000","last_clean":"2023-11-16T18:23:39.502955+0000","last_became_active":"2023-11-16T18:22:46.837836+0000","last_became_peered":"2023-11-16T18:22:46.837836+0000","last_unstale":"2023-11-16T18:23:39.502955+0000","last_undegraded":"2023-11-16T18:23:39.502955+0000","last_fullsized":"2023-11-16T18:23:39.502955+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:24:42.477381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507608+0000","last_change":"2023-11-16T18:22:51.119993+0000","last_active":"2023-11-16T18:23:39.507608+0000","last_peered":"2023-11-16T18:23:39.507608+0000","last_clean":"2023-11-16T18:23:39.507608+0000","last_became_active":"2023-11-16T18:22:51.119401+0000","last_became_peered":"2023-11-16T18:22:51.119401+0000","last_unstale":"2023-11-16T18:23:39.507608+0000","last_undegraded":"2023-11-16T18:23:39.507608+0000","last_fullsized":"2023-11-16T18:23:39.507608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:19:31.451143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585944+0000","last_change":"2023-11-16T18:22:46.383399+0000","last_active":"2023-11-16T18:23:39.585944+0000","last_peered":"2023-11-16T18:23:39.585944+0000","last_clean":"2023-11-16T18:23:39.585944+0000","last_became_active":"2023-11-16T18:22:46.382938+0000","last_became_peered":"2023-11-16T18:22:46.382938+0000","last_unstale":"2023-11-16T18:23:39.585944+0000","last_undegraded":"2023-11-16T18:23:39.585944+0000","last_fullsized":"2023-11-16T18:23:39.585944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:29.725086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503688+0000","last_change":"2023-11-16T18:22:46.836035+0000","last_active":"2023-11-16T18:23:39.503688+0000","last_peered":"2023-11-16T18:23:39.503688+0000","last_clean":"2023-11-16T18:23:39.503688+0000","last_became_active":"2023-11-16T18:22:46.835709+0000","last_became_peered":"2023-11-16T18:22:46.835709+0000","last_unstale":"2023-11-16T18:23:39.503688+0000","last_undegraded":"2023-11-16T18:23:39.503688+0000","last_fullsized":"2023-11-16T18:23:39.503688+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:32:57.558106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504778+0000","last_change":"2023-11-16T18:22:46.820535+0000","last_active":"2023-11-16T18:23:39.504778+0000","last_peered":"2023-11-16T18:23:39.504778+0000","last_clean":"2023-11-16T18:23:39.504778+0000","last_became_active":"2023-11-16T18:22:46.820009+0000","last_became_peered":"2023-11-16T18:22:46.820009+0000","last_unstale":"2023-11-16T18:23:39.504778+0000","last_undegraded":"2023-11-16T18:23:39.504778+0000","last_fullsized":"2023-11-16T18:23:39.504778+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:50:39.196560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510673+0000","last_change":"2023-11-16T18:22:46.831762+0000","last_active":"2023-11-16T18:23:39.510673+0000","last_peered":"2023-11-16T18:23:39.510673+0000","last_clean":"2023-11-16T18:23:39.510673+0000","last_became_active":"2023-11-16T18:22:46.831429+0000","last_became_peered":"2023-11-16T18:22:46.831429+0000","last_unstale":"2023-11-16T18:23:39.510673+0000","last_undegraded":"2023-11-16T18:23:39.510673+0000","last_fullsized":"2023-11-16T18:23:39.510673+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:15:44.338906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510533+0000","last_change":"2023-11-16T18:22:46.838133+0000","last_active":"2023-11-16T18:23:39.510533+0000","last_peered":"2023-11-16T18:23:39.510533+0000","last_clean":"2023-11-16T18:23:39.510533+0000","last_became_active":"2023-11-16T18:22:46.837845+0000","last_became_peered":"2023-11-16T18:22:46.837845+0000","last_unstale":"2023-11-16T18:23:39.510533+0000","last_undegraded":"2023-11-16T18:23:39.510533+0000","last_fullsized":"2023-11-16T18:23:39.510533+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:13:06.102358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585337+0000","last_change":"2023-11-16T18:23:39.585337+0000","last_active":"2023-11-16T18:23:39.585337+0000","last_peered":"2023-11-16T18:23:39.585337+0000","last_clean":"2023-11-16T18:23:39.585337+0000","last_became_active":"2023-11-16T18:23:39.584874+0000","last_became_peered":"2023-11-16T18:23:39.584874+0000","last_unstale":"2023-11-16T18:23:39.585337+0000","last_undegraded":"2023-11-16T18:23:39.585337+0000","last_fullsized":"2023-11-16T18:23:39.585337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:24:54.928438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585342+0000","last_change":"2023-11-16T18:22:46.823192+0000","last_active":"2023-11-16T18:23:39.585342+0000","last_peered":"2023-11-16T18:23:39.585342+0000","last_clean":"2023-11-16T18:23:39.585342+0000","last_became_active":"2023-11-16T18:22:46.822930+0000","last_became_peered":"2023-11-16T18:22:46.822930+0000","last_unstale":"2023-11-16T18:23:39.585342+0000","last_undegraded":"2023-11-16T18:23:39.585342+0000","last_fullsized":"2023-11-16T18:23:39.585342+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:49:48.831997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505366+0000","last_change":"2023-11-16T18:22:46.825815+0000","last_active":"2023-11-16T18:23:39.505366+0000","last_peered":"2023-11-16T18:23:39.505366+0000","last_clean":"2023-11-16T18:23:39.505366+0000","last_became_active":"2023-11-16T18:22:46.825517+0000","last_became_peered":"2023-11-16T18:22:46.825517+0000","last_unstale":"2023-11-16T18:23:39.505366+0000","last_undegraded":"2023-11-16T18:23:39.505366+0000","last_fullsized":"2023-11-16T18:23:39.505366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:32:02.468288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.511101+0000","last_change":"2023-11-16T18:22:46.826825+0000","last_active":"2023-11-16T18:23:39.511101+0000","last_peered":"2023-11-16T18:23:39.511101+0000","last_clean":"2023-11-16T18:23:39.511101+0000","last_became_active":"2023-11-16T18:22:46.826495+0000","last_became_peered":"2023-11-16T18:22:46.826495+0000","last_unstale":"2023-11-16T18:23:39.511101+0000","last_undegraded":"2023-11-16T18:23:39.511101+0000","last_fullsized":"2023-11-16T18:23:39.511101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:07:42.411162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506174+0000","last_change":"2023-11-16T18:22:46.844090+0000","last_active":"2023-11-16T18:23:39.506174+0000","last_peered":"2023-11-16T18:23:39.506174+0000","last_clean":"2023-11-16T18:23:39.506174+0000","last_became_active":"2023-11-16T18:22:46.843641+0000","last_became_peered":"2023-11-16T18:22:46.843641+0000","last_unstale":"2023-11-16T18:23:39.506174+0000","last_undegraded":"2023-11-16T18:23:39.506174+0000","last_fullsized":"2023-11-16T18:23:39.506174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:00:12.291257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508321+0000","last_change":"2023-11-16T18:22:46.838340+0000","last_active":"2023-11-16T18:23:39.508321+0000","last_peered":"2023-11-16T18:23:39.508321+0000","last_clean":"2023-11-16T18:23:39.508321+0000","last_became_active":"2023-11-16T18:22:46.838039+0000","last_became_peered":"2023-11-16T18:22:46.838039+0000","last_unstale":"2023-11-16T18:23:39.508321+0000","last_undegraded":"2023-11-16T18:23:39.508321+0000","last_fullsized":"2023-11-16T18:23:39.508321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:09:28.607361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504176+0000","last_change":"2023-11-16T18:22:46.841739+0000","last_active":"2023-11-16T18:23:39.504176+0000","last_peered":"2023-11-16T18:23:39.504176+0000","last_clean":"2023-11-16T18:23:39.504176+0000","last_became_active":"2023-11-16T18:22:46.841450+0000","last_became_peered":"2023-11-16T18:22:46.841450+0000","last_unstale":"2023-11-16T18:23:39.504176+0000","last_undegraded":"2023-11-16T18:23:39.504176+0000","last_fullsized":"2023-11-16T18:23:39.504176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:42:53.482129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508013+0000","last_change":"2023-11-16T18:22:46.839060+0000","last_active":"2023-11-16T18:23:39.508013+0000","last_peered":"2023-11-16T18:23:39.508013+0000","last_clean":"2023-11-16T18:23:39.508013+0000","last_became_active":"2023-11-16T18:22:46.838725+0000","last_became_peered":"2023-11-16T18:22:46.838725+0000","last_unstale":"2023-11-16T18:23:39.508013+0000","last_undegraded":"2023-11-16T18:23:39.508013+0000","last_fullsized":"2023-11-16T18:23:39.508013+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:33:35.374697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584469+0000","last_change":"2023-11-16T18:22:46.833439+0000","last_active":"2023-11-16T18:23:39.584469+0000","last_peered":"2023-11-16T18:23:39.584469+0000","last_clean":"2023-11-16T18:23:39.584469+0000","last_became_active":"2023-11-16T18:22:46.833111+0000","last_became_peered":"2023-11-16T18:22:46.833111+0000","last_unstale":"2023-11-16T18:23:39.584469+0000","last_undegraded":"2023-11-16T18:23:39.584469+0000","last_fullsized":"2023-11-16T18:23:39.584469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:10:36.066370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506665+0000","last_change":"2023-11-16T18:22:46.825185+0000","last_active":"2023-11-16T18:23:39.506665+0000","last_peered":"2023-11-16T18:23:39.506665+0000","last_clean":"2023-11-16T18:23:39.506665+0000","last_became_active":"2023-11-16T18:22:46.824866+0000","last_became_peered":"2023-11-16T18:22:46.824866+0000","last_unstale":"2023-11-16T18:23:39.506665+0000","last_undegraded":"2023-11-16T18:23:39.506665+0000","last_fullsized":"2023-11-16T18:23:39.506665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:46:05.263494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":215374,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65767,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":544,"num_read_kb":340,"num_write":158,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":61913,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":197301,"data_compressed":9627,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481826,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6140,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739012,"statfs":{"total":95995035648,"available":95988748288,"internally_reserved":0,"allocated":970752,"data_stored":485806,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.56000000000000005,"15min":0.57499999999999996},"min":{"1min":0.33800000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.66100000000000003,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.66500000000000004,"15min":0.69299999999999995},"min":{"1min":0.379,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":3.0590000000000002,"15min":3.0590000000000002},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63500000000000001,"15min":0.64100000000000001},"min":{"1min":0.41099999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.503,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81599999999999995,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.53900000000000003}]},{"osd":2,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.49299999999999999,"15min":0.48499999999999999},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84999999999999998,"5min":0.874,"15min":0.874},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.52600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.64200000000000002,"15min":0.64700000000000002},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67600000000000005,"15min":0.68000000000000005},"min":{"1min":0.52700000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.86499999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.58299999999999996}]},{"osd":5,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63100000000000001,"15min":0.63500000000000001},"min":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.91100000000000003,"5min":1.339,"15min":1.339},"last":0.629},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59699999999999998,"15min":0.60499999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84499999999999997,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.56399999999999995}]},{"osd":6,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.73599999999999999,"15min":0.754},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.1140000000000001,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.80700000000000005,"15min":0.82899999999999996},"min":{"1min":0.54200000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.196,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.746,"15min":0.75},"min":{"1min":0.53800000000000003,"5min":0.443,"15min":0.443},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.75,"15min":0.752},"min":{"1min":0.56100000000000005,"5min":0.502,"15min":0.502},"max":{"1min":0.97499999999999998,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.72799999999999998}]},{"osd":8,"last update":"Thu Nov 16 18:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.80300000000000005}]},{"osd":9,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.86599999999999999,"15min":0.88300000000000001},"min":{"1min":0.53100000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.161,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.73799999999999999,"15min":0.74099999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.74199999999999999}]},{"osd":10,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80400000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0209999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.755},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.83099999999999996,"15min":0.83699999999999997},"min":{"1min":0.59599999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.032,"5min":1.476,"15min":1.476},"last":0.83599999999999997}]}]},{"osd":10,"up_from":69,"seq":296352743462,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6596,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738556,"statfs":{"total":95995035648,"available":95988281344,"internally_reserved":0,"allocated":1110016,"data_stored":1138555,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.35299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.51800000000000002},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.53700000000000003}]},{"osd":2,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.82899999999999996,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.85699999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.73699999999999999}]},{"osd":3,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.659,"5min":1.659,"15min":1.659},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.72499999999999998,"15min":0.73299999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.629,"15min":0.63},"min":{"1min":0.44,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76800000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.36499999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.85099999999999998,"5min":0.996,"15min":0.996},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.99399999999999999,"5min":1.206,"15min":1.206},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.745},"min":{"1min":0.55200000000000005,"5min":0.502,"15min":0.502},"max":{"1min":0.97799999999999998,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.77400000000000002,"15min":0.77700000000000002},"min":{"1min":0.499,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.57399999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.60699999999999998}]},{"osd":7,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55000000000000004}]},{"osd":8,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.84199999999999997,"15min":0.84999999999999998},"min":{"1min":0.63100000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.98899999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0189999999999999,"5min":1.155,"15min":1.155},"last":0.71499999999999997}]},{"osd":9,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.65600000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.628},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.85699999999999998,"15min":0.85999999999999999},"min":{"1min":0.59799999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.75600000000000001}]},{"osd":11,"last update":"Thu Nov 16 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.63200000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.80400000000000005,"15min":0.80800000000000005},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":0.92400000000000004,"5min":1.034,"15min":1.034},"last":0.79700000000000004}]}]},{"osd":9,"up_from":64,"seq":274877906987,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6340,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":978944,"data_stored":485930,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.433,"15min":0.45500000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54900000000000004,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.371,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56499999999999995,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.56299999999999994}]},{"osd":1,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.64000000000000001,"15min":0.66800000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73599999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.60599999999999998,"15min":0.63},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.75700000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63400000000000001}]},{"osd":2,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.64000000000000001,"15min":0.65800000000000003},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82499999999999996,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61899999999999999,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.80100000000000005,"5min":1.024,"15min":1.024},"last":0.67100000000000004}]},{"osd":3,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.51900000000000002,"15min":0.54900000000000004},"min":{"1min":0.29399999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63200000000000001,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.505,"15min":0.55000000000000004},"min":{"1min":0.29799999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.58599999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.70199999999999996}]},{"osd":4,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.67200000000000004,"15min":0.71299999999999997},"min":{"1min":0.44,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.70799999999999996,"15min":0.747},"min":{"1min":0.377,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85299999999999998,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.73999999999999999}]},{"osd":5,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.69299999999999995,"15min":0.72099999999999997},"min":{"1min":0.45900000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.89700000000000002,"5min":1.357,"15min":1.357},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.72899999999999998,"15min":0.76500000000000001},"min":{"1min":0.51600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.88300000000000001,"5min":1.196,"15min":1.196},"last":0.77300000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.379},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.42699999999999999}]},{"osd":7,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.77800000000000002,"15min":0.83099999999999996},"min":{"1min":0.504,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.81899999999999995,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.73399999999999999,"15min":0.78600000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78900000000000003,"5min":1.157,"15min":1.157},"last":0.85099999999999998}]},{"osd":8,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.749,"15min":0.79800000000000004},"min":{"1min":0.49399999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.85299999999999998,"5min":1.248,"15min":1.248},"last":1.004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.79700000000000004,"15min":0.83999999999999997},"min":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.90500000000000003,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.92200000000000004}]},{"osd":10,"last update":"Thu Nov 16 18:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.76700000000000002,"15min":0.78900000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.879,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.752,"15min":0.76600000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.873,"5min":1.022,"15min":1.022},"last":0.82999999999999996}]},{"osd":11,"last update":"Thu Nov 16 18:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.874},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.88600000000000001}]}]},{"osd":8,"up_from":57,"seq":244813135919,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6460,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738692,"statfs":{"total":95995035648,"available":95988420608,"internally_reserved":0,"allocated":970752,"data_stored":485738,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.42399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.40799999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.63900000000000001}]},{"osd":1,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.59099999999999997,"15min":0.56799999999999995},"min":{"1min":0.45400000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.876,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.626,"15min":0.59599999999999997},"min":{"1min":0.56100000000000005,"5min":0.316,"15min":0.316},"max":{"1min":0.84299999999999997,"5min":0.872,"15min":0.872},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.46100000000000002,"15min":0.44700000000000001},"min":{"1min":0.38500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71099999999999997,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.504,"15min":0.48199999999999998},"min":{"1min":0.36799999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72799999999999998,"5min":1.05,"15min":1.05},"last":0.44800000000000001}]},{"osd":3,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91900000000000004,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.628,"15min":0.628},"min":{"1min":0.48999999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0760000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.70299999999999996}]},{"osd":4,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.621,"15min":0.60499999999999998},"min":{"1min":0.46500000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.5},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.63900000000000001,"15min":0.60999999999999999},"min":{"1min":0.47299999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.61099999999999999}]},{"osd":5,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.56100000000000005,"15min":0.52500000000000002},"min":{"1min":0.44400000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.628},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.55300000000000005,"15min":0.51500000000000001},"min":{"1min":0.53700000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.67800000000000005}]},{"osd":6,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67500000000000004,"15min":0.66300000000000003},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.71299999999999997,"15min":0.69699999999999995},"min":{"1min":0.64600000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.83599999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.71999999999999997,"15min":0.68000000000000005},"min":{"1min":0.64000000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.143,"5min":1.266,"15min":1.266},"last":0.752},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.69899999999999995,"15min":0.67300000000000004},"min":{"1min":0.624,"5min":0.434,"15min":0.434},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.76900000000000002}]},{"osd":9,"last update":"Thu Nov 16 18:24:25 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.72999999999999998,"15min":0.71399999999999997},"min":{"1min":0.56000000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0269999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.751,"15min":0.72799999999999998},"min":{"1min":0.55200000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.86199999999999999}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.72499999999999998,"15min":0.71099999999999997},"min":{"1min":0.60499999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.036,"5min":1.276,"15min":1.276},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.76400000000000001,"15min":0.74399999999999999},"min":{"1min":0.74299999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.71899999999999997}]},{"osd":11,"last update":"Thu Nov 16 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.80700000000000005}]}]},{"osd":7,"up_from":49,"seq":210453397556,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6580,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738572,"statfs":{"total":95995035648,"available":95988297728,"internally_reserved":0,"allocated":1028096,"data_stored":1082370,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.60699999999999998,"15min":0.59499999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81999999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81699999999999995,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.44500000000000001,"15min":0.42799999999999999},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61499999999999999,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.48099999999999998,"15min":0.46600000000000003},"min":{"1min":0.36299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71899999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63100000000000001,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65000000000000002,"15min":0.622},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.876,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85699999999999998,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.439},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.42299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.83599999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47699999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.57799999999999996,"15min":0.55200000000000005},"min":{"1min":0.43099999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.77900000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.372},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.372,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76300000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70099999999999996,"15min":0.67900000000000005},"min":{"1min":0.47099999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89500000000000002,"5min":1.075,"15min":1.075},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.75600000000000001,"15min":0.748},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.94999999999999996,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.55500000000000005}]},{"osd":6,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.71599999999999997,"15min":0.69999999999999996},"min":{"1min":0.52600000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.74399999999999999,"15min":0.73199999999999998},"min":{"1min":0.47799999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.45500000000000002}]},{"osd":8,"last update":"Thu Nov 16 18:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.76000000000000001,"15min":0.745},"min":{"1min":0.60599999999999998,"5min":0.433,"15min":0.433},"max":{"1min":0.98899999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.623},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.78800000000000003,"15min":0.78000000000000003},"min":{"1min":0.57399999999999995,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.018,"5min":1.107,"15min":1.107},"last":0.56299999999999994}]},{"osd":9,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79500000000000004,"15min":0.80400000000000005},"min":{"1min":0.57199999999999995,"5min":0.443,"15min":0.443},"max":{"1min":1.0549999999999999,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.83099999999999996,"15min":0.81699999999999995},"min":{"1min":0.59699999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0369999999999999,"5min":3.4119999999999999,"15min":3.4119999999999999},"last":0.504}]},{"osd":10,"last update":"Thu Nov 16 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.71399999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.99099999999999999,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81000000000000005,"15min":0.80900000000000005},"min":{"1min":0.61199999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.071,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.67400000000000004}]},{"osd":11,"last update":"Thu Nov 16 18:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.59699999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561080,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6384,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738768,"statfs":{"total":95995035648,"available":95988498432,"internally_reserved":0,"allocated":958464,"data_stored":481106,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.39800000000000002,"15min":0.36899999999999999},"min":{"1min":0.30199999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.50600000000000001,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.40500000000000003,"15min":0.38100000000000001},"min":{"1min":0.315,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.44800000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.504}]},{"osd":1,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63,"15min":0.59799999999999998},"min":{"1min":0.44,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77500000000000002,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.53300000000000003},"min":{"1min":0.40799999999999997,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.751,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.47999999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.41599999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71199999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.621,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.754,"5min":1.391,"15min":1.391},"last":0.85399999999999998}]},{"osd":3,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52600000000000002,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61199999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.45500000000000002,"15min":0.437},"min":{"1min":0.33000000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79800000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57399999999999995}]},{"osd":4,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.61399999999999999,"15min":0.60699999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.76700000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.69299999999999995,"15min":0.66500000000000004},"min":{"1min":0.504,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.76100000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60299999999999998,"15min":0.58099999999999996},"min":{"1min":0.40300000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.68400000000000005,"5min":1.004,"15min":1.004},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.64500000000000002,"15min":0.625},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82799999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.81699999999999995}]},{"osd":7,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75800000000000001,"15min":0.75},"min":{"1min":0.505,"5min":0.497,"15min":0.497},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.79100000000000004,"15min":0.77600000000000002},"min":{"1min":0.54600000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.96199999999999997}]},{"osd":8,"last update":"Thu Nov 16 18:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.80300000000000005,"15min":0.81899999999999995},"min":{"1min":0.497,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.95199999999999996,"5min":1.429,"15min":1.429},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.441,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89500000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.78000000000000003}]},{"osd":9,"last update":"Thu Nov 16 18:24:25 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.68000000000000005,"15min":0.73699999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70099999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.87},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.79900000000000004,"15min":0.85699999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.84699999999999998,"5min":1.298,"15min":1.298},"last":1.034}]},{"osd":10,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.85999999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.98599999999999999}]},{"osd":11,"last update":"Thu Nov 16 18:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.88600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7476,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737676,"statfs":{"total":95995035648,"available":95987380224,"internally_reserved":0,"allocated":1028096,"data_stored":543180,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:33 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.52800000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.375},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53300000000000003,"15min":0.53800000000000003},"min":{"1min":0.34200000000000003,"5min":0.31,"15min":0.31},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.35199999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.60299999999999998,"15min":0.54800000000000004},"min":{"1min":0.45000000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.91700000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.61599999999999999,"15min":0.56100000000000005},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.754}]},{"osd":3,"last update":"Thu Nov 16 18:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59199999999999997,"15min":0.56899999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.59799999999999998,"15min":0.57399999999999995},"min":{"1min":0.47899999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.45200000000000001,"15min":0.40799999999999997},"min":{"1min":0.372,"5min":0.252,"15min":0.252},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.47399999999999998,"15min":0.41999999999999998},"min":{"1min":0.39300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80000000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Nov 16 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.624,"15min":0.60699999999999998},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93799999999999994,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.66700000000000004,"15min":0.63500000000000001},"min":{"1min":0.52800000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92600000000000005,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.71199999999999997}]},{"osd":6,"last update":"Thu Nov 16 18:24:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.629,"15min":0.61199999999999999},"min":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.63400000000000001,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.41199999999999998}]},{"osd":7,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.56899999999999995,"15min":0.53900000000000003},"min":{"1min":0.42399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.88200000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.63},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.622,"15min":0.58199999999999996},"min":{"1min":0.46500000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.91200000000000003,"5min":0.998,"15min":0.998},"last":0.52400000000000002}]},{"osd":8,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.69899999999999995,"15min":0.66900000000000004},"min":{"1min":0.48399999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0449999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.69999999999999996,"15min":0.67000000000000004},"min":{"1min":0.61799999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73799999999999999}]},{"osd":9,"last update":"Thu Nov 16 18:24:26 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.73299999999999998,"15min":0.72199999999999998},"min":{"1min":0.52300000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.749,"5min":0.755,"15min":0.749},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.58099999999999996}]},{"osd":10,"last update":"Thu Nov 16 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.77700000000000002,"15min":0.76900000000000002},"min":{"1min":0.57899999999999996,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.80900000000000005,"15min":0.79500000000000004},"min":{"1min":0.66000000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.65700000000000003}]},{"osd":11,"last update":"Thu Nov 16 18:23:48 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.79900000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7300,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737852,"statfs":{"total":95995035648,"available":95987560448,"internally_reserved":0,"allocated":978944,"data_stored":482221,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 18:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.55300000000000005,"15min":0.5},"min":{"1min":0.36799999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.923,"5min":2.923,"15min":2.923},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.56699999999999995,"15min":0.503},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.2109999999999999,"5min":3.2109999999999999,"15min":3.2109999999999999},"last":0.72599999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:23:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60999999999999999,"15min":0.71599999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.93799999999999994,"5min":5.3650000000000002,"15min":5.3650000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.627,"15min":0.67100000000000004},"min":{"1min":0.437,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.8929999999999998,"5min":3.4009999999999998,"15min":3.4009999999999998},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Nov 16 18:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.7810000000000001,"5min":3.7810000000000001,"15min":3.7810000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":3.294,"5min":3.294,"15min":3.294},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.60599999999999998,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.0249999999999999,"5min":3.0249999999999999,"15min":3.0249999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.61299999999999999,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":3.242,"5min":3.242,"15min":3.242},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Nov 16 18:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.48399999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.63200000000000001,"15min":0.59599999999999997},"min":{"1min":0.40699999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.851,"5min":2.851,"15min":2.851},"last":0.64500000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55500000000000005,"15min":0.51900000000000002},"min":{"1min":0.41699999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65700000000000003,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.32800000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.64100000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Nov 16 18:23:54 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.0499999999999998,"5min":3.0499999999999998,"15min":3.0499999999999998},"last":0.77500000000000002}]},{"osd":8,"last update":"Thu Nov 16 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.70299999999999996,"15min":0.72499999999999998},"min":{"1min":0.54200000000000004,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.036,"5min":1.052,"15min":1.052},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.73399999999999999,"15min":0.74299999999999999},"min":{"1min":0.54000000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.9470000000000001,"5min":2.9470000000000001,"15min":2.9470000000000001},"last":0.81699999999999995}]},{"osd":9,"last update":"Thu Nov 16 18:24:28 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.46000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.002,"5min":3.3180000000000001,"15min":3.3180000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.70299999999999996,"15min":0.73099999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.84599999999999997,"5min":2.79,"15min":2.79},"last":0.60599999999999998}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.78600000000000003,"15min":0.78100000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.0009999999999999,"5min":3.0009999999999999,"15min":3.0009999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.77000000000000002,"15min":0.75800000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":3.2679999999999998,"5min":3.2679999999999998,"15min":3.2679999999999998},"last":0.83599999999999997}]},{"osd":11,"last update":"Thu Nov 16 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.70599999999999996},"min":{"1min":0.58699999999999997,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.871},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.751,"15min":0.73499999999999999},"min":{"1min":0.59299999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.9740000000000002,"5min":2.9740000000000002,"15min":2.9740000000000002},"last":0.89200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7484,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737668,"statfs":{"total":95995035648,"available":95987372032,"internally_reserved":0,"allocated":970752,"data_stored":481744,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.54200000000000004,"15min":0.48199999999999998},"min":{"1min":0.34899999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.51800000000000002,"15min":0.46899999999999997},"min":{"1min":0.39500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.44700000000000001}]},{"osd":1,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.57399999999999995,"15min":0.52400000000000002},"min":{"1min":0.42699999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.073,"5min":1.47,"15min":1.47},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.59199999999999997,"15min":0.54800000000000004},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96599999999999997,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.71399999999999997}]},{"osd":3,"last update":"Thu Nov 16 18:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.54400000000000004},"min":{"1min":0.36499999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82999999999999996,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.61499999999999999,"15min":0.55300000000000005},"min":{"1min":0.47399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.89800000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.65200000000000002}]},{"osd":4,"last update":"Thu Nov 16 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.52700000000000002}]},{"osd":5,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.47599999999999998,"15min":0.42499999999999999},"min":{"1min":0.32700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.52500000000000002,"15min":0.48099999999999998},"min":{"1min":0.40300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.02,"5min":1.339,"15min":1.339},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Nov 16 18:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.64800000000000002,"15min":0.61899999999999999},"min":{"1min":0.45500000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.433},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.68799999999999994,"15min":0.63800000000000001},"min":{"1min":0.51300000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.67400000000000004}]},{"osd":7,"last update":"Thu Nov 16 18:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.63200000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:24:10 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.61599999999999999,"15min":0.58199999999999996},"min":{"1min":0.46899999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.622},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.60599999999999998,"15min":0.57799999999999996},"min":{"1min":0.45700000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.97099999999999997,"5min":1.377,"15min":1.377},"last":0.41199999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.54400000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92500000000000004,"5min":1.409,"15min":1.409},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.755,"15min":0.746},"min":{"1min":0.60799999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.59199999999999997}]},{"osd":10,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.73899999999999999,"15min":0.71899999999999997},"min":{"1min":0.60399999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.77200000000000002,"15min":0.76200000000000001},"min":{"1min":0.58799999999999997,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.242,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.66600000000000004}]},{"osd":11,"last update":"Thu Nov 16 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.64300000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182469,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6712,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738440,"statfs":{"total":95995035648,"available":95988162560,"internally_reserved":0,"allocated":966656,"data_stored":481027,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.378,"15min":0.36399999999999999},"min":{"1min":0.21099999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.47899999999999998,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.434},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.39400000000000002,"15min":0.38600000000000001},"min":{"1min":0.26500000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.59499999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Nov 16 18:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55000000000000004,"15min":0.52400000000000002},"min":{"1min":0.35599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.77400000000000002,"5min":1.129,"15min":1.129},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56399999999999995,"15min":0.53800000000000003},"min":{"1min":0.434,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.72399999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59899999999999998,"15min":0.56899999999999995},"min":{"1min":0.42699999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83399999999999996,"5min":1.038,"15min":1.038},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.58199999999999996,"15min":0.56999999999999995},"min":{"1min":0.371,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73399999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63500000000000001,"15min":0.60299999999999998},"min":{"1min":0.498,"5min":0.378,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":0.628},"min":{"1min":0.439,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79800000000000004,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Nov 16 18:23:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.60299999999999998,"15min":0.61299999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70799999999999996,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.63100000000000001,"15min":0.623},"min":{"1min":0.45400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.81399999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.80700000000000005}]},{"osd":6,"last update":"Thu Nov 16 18:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.54200000000000004,"15min":0.49399999999999999},"min":{"1min":0.38600000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.63500000000000001,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.30199999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.72199999999999998,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59499999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.44600000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81399999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69599999999999995,"15min":0.68600000000000005},"min":{"1min":0.46700000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81499999999999995,"5min":1.087,"15min":1.087},"last":0.76600000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:22:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.60799999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.752,"15min":0.81000000000000005},"min":{"1min":0.39000000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79700000000000004,"5min":1.135,"15min":1.135},"last":0.875},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.70099999999999996,"15min":0.747},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.748,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.80000000000000004}]},{"osd":10,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.77000000000000002,"15min":0.78800000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.77600000000000002,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.82699999999999996,"15min":0.85199999999999998},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.81799999999999995,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.90700000000000003}]},{"osd":11,"last update":"Thu Nov 16 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.88700000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953537,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6608,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738544,"statfs":{"total":95995035648,"available":95988269056,"internally_reserved":0,"allocated":991232,"data_stored":546980,"data_compressed":27217,"data_compressed_allocated":36864,"data_compressed_original":106496,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.504,"15min":0.48699999999999999},"min":{"1min":0.39900000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.752,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.48899999999999999,"15min":0.46400000000000002},"min":{"1min":0.41799999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.502}]},{"osd":1,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40600000000000003,"15min":0.377},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.56299999999999994,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.501},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.439,"15min":0.41299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.60699999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.52500000000000002}]},{"osd":2,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59199999999999997,"15min":0.56599999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85399999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.61699999999999999,"15min":0.57499999999999996},"min":{"1min":0.51500000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83599999999999997,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.68200000000000005}]},{"osd":3,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57199999999999995,"15min":0.54000000000000004},"min":{"1min":0.40100000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.03,"5min":0.64800000000000002,"15min":0.56200000000000006},"min":{"1min":0.40300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":10.048,"5min":10.048,"15min":10.048},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Nov 16 18:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.66400000000000003,"15min":0.61799999999999999},"min":{"1min":0.47599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.88300000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66500000000000004,"15min":0.63900000000000001},"min":{"1min":0.48399999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.99299999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.65400000000000003}]},{"osd":6,"last update":"Thu Nov 16 18:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.61599999999999999,"15min":0.59399999999999997},"min":{"1min":0.47799999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84899999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.68000000000000005,"15min":0.64400000000000002},"min":{"1min":0.53900000000000003,"5min":0.432,"15min":0.432},"max":{"1min":0.93200000000000005,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.86599999999999999}]},{"osd":7,"last update":"Thu Nov 16 18:23:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.626,"15min":0.57599999999999996},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95199999999999996,"5min":1.077,"15min":1.077},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56899999999999995,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76900000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63200000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.72499999999999998,"15min":0.72099999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98299999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73599999999999999,"15min":0.73199999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95499999999999996,"5min":1.109,"15min":1.109},"last":0.72699999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.751,"15min":0.76100000000000001},"min":{"1min":0.59099999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.92400000000000004,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80600000000000005,"15min":0.80900000000000005},"min":{"1min":0.65000000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.96999999999999997,"5min":1.117,"15min":1.117},"last":0.94499999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.63400000000000001,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":0.997,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78200000000000003,"15min":0.78500000000000003},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96799999999999997,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.83899999999999997}]},{"osd":11,"last update":"Thu Nov 16 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.88100000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790012,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6716,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738436,"statfs":{"total":95995035648,"available":95988158464,"internally_reserved":0,"allocated":1036288,"data_stored":1077616,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51900000000000002,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.69899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.59399999999999997,"15min":0.54000000000000004},"min":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76100000000000001,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.432}]},{"osd":1,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.54600000000000004,"15min":0.53300000000000003},"min":{"1min":0.41799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.83499999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.55100000000000005,"15min":0.52200000000000002},"min":{"1min":0.34599999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79000000000000004,"5min":0.873,"15min":0.873},"last":0.70599999999999996}]},{"osd":2,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.39400000000000002},"min":{"1min":0.312,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58599999999999997,"5min":0.626,"15min":0.626},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.41599999999999998,"15min":0.38600000000000001},"min":{"1min":0.316,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70399999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.64500000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55200000000000005,"15min":0.54200000000000004},"min":{"1min":0.42399999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.72499999999999998,"5min":0.88,"15min":0.88},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.58999999999999997,"15min":0.57199999999999995},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.85999999999999999,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.58499999999999996,"15min":0.56899999999999995},"min":{"1min":0.47099999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.872,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Nov 16 18:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.46899999999999997,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81100000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.46300000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77500000000000002,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.54600000000000004}]},{"osd":7,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.64000000000000001,"15min":0.629},"min":{"1min":0.48699999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.63800000000000001,"15min":0.625},"min":{"1min":0.44800000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.89600000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.75900000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.41399999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.45200000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.77600000000000002,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66400000000000003}]},{"osd":9,"last update":"Thu Nov 16 18:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66500000000000004,"15min":0.66100000000000003},"min":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.79400000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70299999999999996,"15min":0.70199999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.86899999999999999,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.77300000000000002}]},{"osd":10,"last update":"Thu Nov 16 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.72699999999999998,"15min":0.71599999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.70499999999999996,"15min":0.69299999999999995},"min":{"1min":0.53800000000000003,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93799999999999994,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.79100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34529,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":62141,"data_compressed":9627,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T18:24:38.708 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T18:24:38.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.863+0000 7f6e77601700 1 -- 172.21.15.2:0/2315160143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e701003e0 msgr2=0x7f6e70100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:38.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.863+0000 7f6e77601700 1 --2- 172.21.15.2:0/2315160143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e701003e0 0x7f6e70100830 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009990 tx=0x7f6e64009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.863+0000 7f6e77601700 1 -- 172.21.15.2:0/2315160143 shutdown_connections 2023-11-16T18:24:38.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.863+0000 7f6e77601700 1 --2- 172.21.15.2:0/2315160143 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e700ff1f0 0x7f6e700ff600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.863+0000 7f6e77601700 1 --2- 172.21.15.2:0/2315160143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e70100d70 0x7f6e70107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.863+0000 7f6e77601700 1 --2- 172.21.15.2:0/2315160143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e701003e0 0x7f6e70100830 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.863+0000 7f6e77601700 1 -- 172.21.15.2:0/2315160143 >> 172.21.15.2:0/2315160143 conn(0x7f6e700fa7d0 msgr2=0x7f6e700fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:38.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.863+0000 7f6e77601700 1 -- 172.21.15.2:0/2315160143 shutdown_connections 2023-11-16T18:24:38.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.863+0000 7f6e77601700 1 -- 172.21.15.2:0/2315160143 wait complete. 2023-11-16T18:24:38.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e77601700 1 Processor -- start 2023-11-16T18:24:38.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e77601700 1 -- start start 2023-11-16T18:24:38.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e77601700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e700ff1f0 0x7f6e70198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:38.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e77601700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e701003e0 0x7f6e701987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:38.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e77601700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70100d70 0x7f6e7019f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e77601700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7019fe00 con 0x7f6e70100d70 2023-11-16T18:24:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e77601700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7019ff40 con 0x7f6e700ff1f0 2023-11-16T18:24:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e77601700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e701a00d0 con 0x7f6e701003e0 2023-11-16T18:24:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e75b9e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70100d70 0x7f6e7019f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e75b9e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70100d70 0x7f6e7019f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49728/0 (socket says 172.21.15.2:49728) 2023-11-16T18:24:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e75b9e700 1 -- 172.21.15.2:0/1540498117 learned_addr learned my addr 172.21.15.2:0/1540498117 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e7539d700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e700ff1f0 0x7f6e70198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e74b9c700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e701003e0 0x7f6e701987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e75b9e700 1 -- 172.21.15.2:0/1540498117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e701003e0 msgr2=0x7f6e701987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e75b9e700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e701003e0 0x7f6e701987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e75b9e700 1 -- 172.21.15.2:0/1540498117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e700ff1f0 msgr2=0x7f6e70198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e75b9e700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e700ff1f0 0x7f6e70198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e75b9e700 1 -- 172.21.15.2:0/1540498117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e640095f0 con 0x7f6e70100d70 2023-11-16T18:24:38.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.864+0000 7f6e74b9c700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e701003e0 0x7f6e701987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:38.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.865+0000 7f6e75b9e700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70100d70 0x7f6e7019f830 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6e68007d70 tx=0x7f6e6800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:38.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.865+0000 7f6e627fc700 1 -- 172.21.15.2:0/1540498117 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6800ad40 con 0x7f6e70100d70 2023-11-16T18:24:38.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.865+0000 7f6e77601700 1 -- 172.21.15.2:0/1540498117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e7010ac40 con 0x7f6e70100d70 2023-11-16T18:24:38.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.865+0000 7f6e627fc700 1 -- 172.21.15.2:0/1540498117 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e680128e0 con 0x7f6e70100d70 2023-11-16T18:24:38.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.865+0000 7f6e627fc700 1 -- 172.21.15.2:0/1540498117 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68011780 con 0x7f6e70100d70 2023-11-16T18:24:38.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.865+0000 7f6e77601700 1 -- 172.21.15.2:0/1540498117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7010aab0 con 0x7f6e70100d70 2023-11-16T18:24:38.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.866+0000 7f6e77601700 1 -- 172.21.15.2:0/1540498117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e70067260 con 0x7f6e70100d70 2023-11-16T18:24:38.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.867+0000 7f6e627fc700 1 -- 172.21.15.2:0/1540498117 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f6e68012a50 con 0x7f6e70100d70 2023-11-16T18:24:38.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.867+0000 7f6e627fc700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6e5c0a1150 0x7f6e5c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:38.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.868+0000 7f6e627fc700 1 -- 172.21.15.2:0/1540498117 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f6e680ccfe0 con 0x7f6e70100d70 2023-11-16T18:24:38.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.868+0000 7f6e7539d700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6e5c0a1150 0x7f6e5c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:38.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.868+0000 7f6e7539d700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6e5c0a1150 0x7f6e5c0a3600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6e70100240 tx=0x7f6e6c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:38.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:38.872+0000 7f6e627fc700 1 -- 172.21.15.2:0/1540498117 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6e68096390 con 0x7f6e70100d70 2023-11-16T18:24:39.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.066+0000 7f6e77601700 1 -- 172.21.15.2:0/1540498117 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e701a0cc0 con 0x7f6e5c0a1150 2023-11-16T18:24:39.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.081+0000 7f6e627fc700 1 -- 172.21.15.2:0/1540498117 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426831 (secure 0 0 0) 0x7f6e701a0cc0 con 0x7f6e5c0a1150 2023-11-16T18:24:39.091 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:39.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.085+0000 7f6e77601700 1 -- 172.21.15.2:0/1540498117 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6e5c0a1150 msgr2=0x7f6e5c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:39.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.085+0000 7f6e77601700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6e5c0a1150 0x7f6e5c0a3600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6e70100240 tx=0x7f6e6c008040 comp rx=0 tx=0).stop 2023-11-16T18:24:39.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.085+0000 7f6e77601700 1 -- 172.21.15.2:0/1540498117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70100d70 msgr2=0x7f6e7019f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:39.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.085+0000 7f6e77601700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70100d70 0x7f6e7019f830 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6e68007d70 tx=0x7f6e6800a340 comp rx=0 tx=0).stop 2023-11-16T18:24:39.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.085+0000 7f6e77601700 1 -- 172.21.15.2:0/1540498117 shutdown_connections 2023-11-16T18:24:39.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.085+0000 7f6e77601700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6e5c0a1150 0x7f6e5c0a3600 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.085+0000 7f6e77601700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70100d70 0x7f6e7019f830 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.085+0000 7f6e77601700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e700ff1f0 0x7f6e70198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.085+0000 7f6e77601700 1 --2- 172.21.15.2:0/1540498117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e701003e0 0x7f6e701987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.085+0000 7f6e77601700 1 -- 172.21.15.2:0/1540498117 >> 172.21.15.2:0/1540498117 conn(0x7f6e700fa7d0 msgr2=0x7f6e70068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:39.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.085+0000 7f6e77601700 1 -- 172.21.15.2:0/1540498117 shutdown_connections 2023-11-16T18:24:39.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.085+0000 7f6e77601700 1 -- 172.21.15.2:0/1540498117 wait complete. 2023-11-16T18:24:39.096 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-16T18:24:39.098 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:38 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2962856659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:39.099 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:38 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/519250188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:39.099 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2023-11-16T18:24:39.099 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 8]}]: dispatch 2023-11-16T18:24:39.099 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2023-11-16T18:24:39.099 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:38 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2023-11-16T18:24:39.100 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2023-11-16T18:24:38.814341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656381,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":685,"num_read_kb":464,"num_write":342,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1242772,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":453,"ondisk_log_size":453,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80860,"kb_used_data":11708,"kb_used_omap":0,"kb_used_meta":69056,"kb_avail":1124860964,"statfs":{"total":1151940427776,"available":1151857627136,"internally_reserved":0,"allocated":11988992,"data_stored":7772273,"data_compressed":329214,"data_compressed_allocated":442368,"data_compressed_original":2416640,"omap_allocated":0,"internal_metadata":70713344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,11,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":636,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":37,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.984321"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.859995+0000","last_change":"2023-11-16T18:22:46.829814+0000","last_active":"2023-11-16T18:24:37.859995+0000","last_peered":"2023-11-16T18:24:37.859995+0000","last_clean":"2023-11-16T18:24:37.859995+0000","last_became_active":"2023-11-16T18:22:46.829407+0000","last_became_peered":"2023-11-16T18:22:46.829407+0000","last_unstale":"2023-11-16T18:24:37.859995+0000","last_undegraded":"2023-11-16T18:24:37.859995+0000","last_fullsized":"2023-11-16T18:24:37.859995+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:02:05.303420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505209+0000","last_change":"2023-11-16T18:22:46.818700+0000","last_active":"2023-11-16T18:23:39.505209+0000","last_peered":"2023-11-16T18:23:39.505209+0000","last_clean":"2023-11-16T18:23:39.505209+0000","last_became_active":"2023-11-16T18:22:46.818388+0000","last_became_peered":"2023-11-16T18:22:46.818388+0000","last_unstale":"2023-11-16T18:23:39.505209+0000","last_undegraded":"2023-11-16T18:23:39.505209+0000","last_fullsized":"2023-11-16T18:23:39.505209+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:05:25.862565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504017+0000","last_change":"2023-11-16T18:22:46.827732+0000","last_active":"2023-11-16T18:23:39.504017+0000","last_peered":"2023-11-16T18:23:39.504017+0000","last_clean":"2023-11-16T18:23:39.504017+0000","last_became_active":"2023-11-16T18:22:46.827328+0000","last_became_peered":"2023-11-16T18:22:46.827328+0000","last_unstale":"2023-11-16T18:23:39.504017+0000","last_undegraded":"2023-11-16T18:23:39.504017+0000","last_fullsized":"2023-11-16T18:23:39.504017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:44:05.572178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504758+0000","last_change":"2023-11-16T18:22:46.826837+0000","last_active":"2023-11-16T18:23:39.504758+0000","last_peered":"2023-11-16T18:23:39.504758+0000","last_clean":"2023-11-16T18:23:39.504758+0000","last_became_active":"2023-11-16T18:22:46.826532+0000","last_became_peered":"2023-11-16T18:22:46.826532+0000","last_unstale":"2023-11-16T18:23:39.504758+0000","last_undegraded":"2023-11-16T18:23:39.504758+0000","last_fullsized":"2023-11-16T18:23:39.504758+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:29:24.407020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.842311+0000","last_change":"2023-11-16T18:22:46.841164+0000","last_active":"2023-11-16T18:24:37.842311+0000","last_peered":"2023-11-16T18:24:37.842311+0000","last_clean":"2023-11-16T18:24:37.842311+0000","last_became_active":"2023-11-16T18:22:46.840855+0000","last_became_peered":"2023-11-16T18:22:46.840855+0000","last_unstale":"2023-11-16T18:24:37.842311+0000","last_undegraded":"2023-11-16T18:24:37.842311+0000","last_fullsized":"2023-11-16T18:24:37.842311+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:46.380315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507548+0000","last_change":"2023-11-16T18:22:46.822171+0000","last_active":"2023-11-16T18:23:39.507548+0000","last_peered":"2023-11-16T18:23:39.507548+0000","last_clean":"2023-11-16T18:23:39.507548+0000","last_became_active":"2023-11-16T18:22:46.821860+0000","last_became_peered":"2023-11-16T18:22:46.821860+0000","last_unstale":"2023-11-16T18:23:39.507548+0000","last_undegraded":"2023-11-16T18:23:39.507548+0000","last_fullsized":"2023-11-16T18:23:39.507548+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:28:18.589285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502347+0000","last_change":"2023-11-16T18:22:46.832463+0000","last_active":"2023-11-16T18:23:39.502347+0000","last_peered":"2023-11-16T18:23:39.502347+0000","last_clean":"2023-11-16T18:23:39.502347+0000","last_became_active":"2023-11-16T18:22:46.832141+0000","last_became_peered":"2023-11-16T18:22:46.832141+0000","last_unstale":"2023-11-16T18:23:39.502347+0000","last_undegraded":"2023-11-16T18:23:39.502347+0000","last_fullsized":"2023-11-16T18:23:39.502347+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:12:03.361742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.850564+0000","last_change":"2023-11-16T18:22:46.844239+0000","last_active":"2023-11-16T18:24:37.850564+0000","last_peered":"2023-11-16T18:24:37.850564+0000","last_clean":"2023-11-16T18:24:37.850564+0000","last_became_active":"2023-11-16T18:22:46.843906+0000","last_became_peered":"2023-11-16T18:22:46.843906+0000","last_unstale":"2023-11-16T18:24:37.850564+0000","last_undegraded":"2023-11-16T18:24:37.850564+0000","last_fullsized":"2023-11-16T18:24:37.850564+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:16:15.202242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508804+0000","last_change":"2023-11-16T18:22:46.834625+0000","last_active":"2023-11-16T18:23:39.508804+0000","last_peered":"2023-11-16T18:23:39.508804+0000","last_clean":"2023-11-16T18:23:39.508804+0000","last_became_active":"2023-11-16T18:22:46.834145+0000","last_became_peered":"2023-11-16T18:22:46.834145+0000","last_unstale":"2023-11-16T18:23:39.508804+0000","last_undegraded":"2023-11-16T18:23:39.508804+0000","last_fullsized":"2023-11-16T18:23:39.508804+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:25:16.196016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509061+0000","last_change":"2023-11-16T18:22:46.827569+0000","last_active":"2023-11-16T18:23:39.509061+0000","last_peered":"2023-11-16T18:23:39.509061+0000","last_clean":"2023-11-16T18:23:39.509061+0000","last_became_active":"2023-11-16T18:22:46.827234+0000","last_became_peered":"2023-11-16T18:22:46.827234+0000","last_unstale":"2023-11-16T18:23:39.509061+0000","last_undegraded":"2023-11-16T18:23:39.509061+0000","last_fullsized":"2023-11-16T18:23:39.509061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:38:30.584402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503061+0000","last_change":"2023-11-16T18:22:46.833701+0000","last_active":"2023-11-16T18:23:39.503061+0000","last_peered":"2023-11-16T18:23:39.503061+0000","last_clean":"2023-11-16T18:23:39.503061+0000","last_became_active":"2023-11-16T18:22:46.833333+0000","last_became_peered":"2023-11-16T18:22:46.833333+0000","last_unstale":"2023-11-16T18:23:39.503061+0000","last_undegraded":"2023-11-16T18:23:39.503061+0000","last_fullsized":"2023-11-16T18:23:39.503061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:31:52.429141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504924+0000","last_change":"2023-11-16T18:22:46.823222+0000","last_active":"2023-11-16T18:23:39.504924+0000","last_peered":"2023-11-16T18:23:39.504924+0000","last_clean":"2023-11-16T18:23:39.504924+0000","last_became_active":"2023-11-16T18:22:46.822224+0000","last_became_peered":"2023-11-16T18:22:46.822224+0000","last_unstale":"2023-11-16T18:23:39.504924+0000","last_undegraded":"2023-11-16T18:23:39.504924+0000","last_fullsized":"2023-11-16T18:23:39.504924+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:23:38.463744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.843256+0000","last_change":"2023-11-16T18:22:46.818631+0000","last_active":"2023-11-16T18:24:37.843256+0000","last_peered":"2023-11-16T18:24:37.843256+0000","last_clean":"2023-11-16T18:24:37.843256+0000","last_became_active":"2023-11-16T18:22:46.818305+0000","last_became_peered":"2023-11-16T18:22:46.818305+0000","last_unstale":"2023-11-16T18:24:37.843256+0000","last_undegraded":"2023-11-16T18:24:37.843256+0000","last_fullsized":"2023-11-16T18:24:37.843256+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:06:53.083409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503715+0000","last_change":"2023-11-16T18:22:46.825439+0000","last_active":"2023-11-16T18:23:39.503715+0000","last_peered":"2023-11-16T18:23:39.503715+0000","last_clean":"2023-11-16T18:23:39.503715+0000","last_became_active":"2023-11-16T18:22:46.825062+0000","last_became_peered":"2023-11-16T18:22:46.825062+0000","last_unstale":"2023-11-16T18:23:39.503715+0000","last_undegraded":"2023-11-16T18:23:39.503715+0000","last_fullsized":"2023-11-16T18:23:39.503715+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:32:03.819033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503546+0000","last_change":"2023-11-16T18:22:46.827730+0000","last_active":"2023-11-16T18:23:39.503546+0000","last_peered":"2023-11-16T18:23:39.503546+0000","last_clean":"2023-11-16T18:23:39.503546+0000","last_became_active":"2023-11-16T18:22:46.827416+0000","last_became_peered":"2023-11-16T18:22:46.827416+0000","last_unstale":"2023-11-16T18:23:39.503546+0000","last_undegraded":"2023-11-16T18:23:39.503546+0000","last_fullsized":"2023-11-16T18:23:39.503546+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:18:08.691944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507323+0000","last_change":"2023-11-16T18:22:46.834259+0000","last_active":"2023-11-16T18:23:39.507323+0000","last_peered":"2023-11-16T18:23:39.507323+0000","last_clean":"2023-11-16T18:23:39.507323+0000","last_became_active":"2023-11-16T18:22:46.833775+0000","last_became_peered":"2023-11-16T18:22:46.833775+0000","last_unstale":"2023-11-16T18:23:39.507323+0000","last_undegraded":"2023-11-16T18:23:39.507323+0000","last_fullsized":"2023-11-16T18:23:39.507323+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:02:17.762712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.860025+0000","last_change":"2023-11-16T18:22:46.829475+0000","last_active":"2023-11-16T18:24:37.860025+0000","last_peered":"2023-11-16T18:24:37.860025+0000","last_clean":"2023-11-16T18:24:37.860025+0000","last_became_active":"2023-11-16T18:22:46.829157+0000","last_became_peered":"2023-11-16T18:22:46.829157+0000","last_unstale":"2023-11-16T18:24:37.860025+0000","last_undegraded":"2023-11-16T18:24:37.860025+0000","last_fullsized":"2023-11-16T18:24:37.860025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:29.122041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501933+0000","last_change":"2023-11-16T18:22:46.821716+0000","last_active":"2023-11-16T18:23:39.501933+0000","last_peered":"2023-11-16T18:23:39.501933+0000","last_clean":"2023-11-16T18:23:39.501933+0000","last_became_active":"2023-11-16T18:22:46.821393+0000","last_became_peered":"2023-11-16T18:22:46.821393+0000","last_unstale":"2023-11-16T18:23:39.501933+0000","last_undegraded":"2023-11-16T18:23:39.501933+0000","last_fullsized":"2023-11-16T18:23:39.501933+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:10.453727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510952+0000","last_change":"2023-11-16T18:22:46.837433+0000","last_active":"2023-11-16T18:23:39.510952+0000","last_peered":"2023-11-16T18:23:39.510952+0000","last_clean":"2023-11-16T18:23:39.510952+0000","last_became_active":"2023-11-16T18:22:46.837143+0000","last_became_peered":"2023-11-16T18:22:46.837143+0000","last_unstale":"2023-11-16T18:23:39.510952+0000","last_undegraded":"2023-11-16T18:23:39.510952+0000","last_fullsized":"2023-11-16T18:23:39.510952+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:45:40.959535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510180+0000","last_change":"2023-11-16T18:22:51.123749+0000","last_active":"2023-11-16T18:23:39.510180+0000","last_peered":"2023-11-16T18:23:39.510180+0000","last_clean":"2023-11-16T18:23:39.510180+0000","last_became_active":"2023-11-16T18:22:51.123185+0000","last_became_peered":"2023-11-16T18:22:51.123185+0000","last_unstale":"2023-11-16T18:23:39.510180+0000","last_undegraded":"2023-11-16T18:23:39.510180+0000","last_fullsized":"2023-11-16T18:23:39.510180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:11:31.023936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501272+0000","last_change":"2023-11-16T18:22:46.385440+0000","last_active":"2023-11-16T18:23:39.501272+0000","last_peered":"2023-11-16T18:23:39.501272+0000","last_clean":"2023-11-16T18:23:39.501272+0000","last_became_active":"2023-11-16T18:22:46.385098+0000","last_became_peered":"2023-11-16T18:22:46.385098+0000","last_unstale":"2023-11-16T18:23:39.501272+0000","last_undegraded":"2023-11-16T18:23:39.501272+0000","last_fullsized":"2023-11-16T18:23:39.501272+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:08:49.547450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.846685+0000","last_change":"2023-11-16T18:22:46.842681+0000","last_active":"2023-11-16T18:24:37.846685+0000","last_peered":"2023-11-16T18:24:37.846685+0000","last_clean":"2023-11-16T18:24:37.846685+0000","last_became_active":"2023-11-16T18:22:46.842408+0000","last_became_peered":"2023-11-16T18:22:46.842408+0000","last_unstale":"2023-11-16T18:24:37.846685+0000","last_undegraded":"2023-11-16T18:24:37.846685+0000","last_fullsized":"2023-11-16T18:24:37.846685+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:37:57.050474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503905+0000","last_change":"2023-11-16T18:22:51.123450+0000","last_active":"2023-11-16T18:23:39.503905+0000","last_peered":"2023-11-16T18:23:39.503905+0000","last_clean":"2023-11-16T18:23:39.503905+0000","last_became_active":"2023-11-16T18:22:51.123002+0000","last_became_peered":"2023-11-16T18:22:51.123002+0000","last_unstale":"2023-11-16T18:23:39.503905+0000","last_undegraded":"2023-11-16T18:23:39.503905+0000","last_fullsized":"2023-11-16T18:23:39.503905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:48:43.003189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504416+0000","last_change":"2023-11-16T18:22:46.384768+0000","last_active":"2023-11-16T18:23:39.504416+0000","last_peered":"2023-11-16T18:23:39.504416+0000","last_clean":"2023-11-16T18:23:39.504416+0000","last_became_active":"2023-11-16T18:22:46.384246+0000","last_became_peered":"2023-11-16T18:22:46.384246+0000","last_unstale":"2023-11-16T18:23:39.504416+0000","last_undegraded":"2023-11-16T18:23:39.504416+0000","last_fullsized":"2023-11-16T18:23:39.504416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:20:51.259854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504317+0000","last_change":"2023-11-16T18:22:46.826069+0000","last_active":"2023-11-16T18:23:39.504317+0000","last_peered":"2023-11-16T18:23:39.504317+0000","last_clean":"2023-11-16T18:23:39.504317+0000","last_became_active":"2023-11-16T18:22:46.824676+0000","last_became_peered":"2023-11-16T18:22:46.824676+0000","last_unstale":"2023-11-16T18:23:39.504317+0000","last_undegraded":"2023-11-16T18:23:39.504317+0000","last_fullsized":"2023-11-16T18:23:39.504317+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:36:58.519456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504272+0000","last_change":"2023-11-16T18:22:51.127529+0000","last_active":"2023-11-16T18:23:39.504272+0000","last_peered":"2023-11-16T18:23:39.504272+0000","last_clean":"2023-11-16T18:23:39.504272+0000","last_became_active":"2023-11-16T18:22:51.126748+0000","last_became_peered":"2023-11-16T18:22:51.126748+0000","last_unstale":"2023-11-16T18:23:39.504272+0000","last_undegraded":"2023-11-16T18:23:39.504272+0000","last_fullsized":"2023-11-16T18:23:39.504272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:46:02.931821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505594+0000","last_change":"2023-11-16T18:22:46.385890+0000","last_active":"2023-11-16T18:23:39.505594+0000","last_peered":"2023-11-16T18:23:39.505594+0000","last_clean":"2023-11-16T18:23:39.505594+0000","last_became_active":"2023-11-16T18:22:46.385568+0000","last_became_peered":"2023-11-16T18:22:46.385568+0000","last_unstale":"2023-11-16T18:23:39.505594+0000","last_undegraded":"2023-11-16T18:23:39.505594+0000","last_fullsized":"2023-11-16T18:23:39.505594+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:45:23.088120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508492+0000","last_change":"2023-11-16T18:22:46.824796+0000","last_active":"2023-11-16T18:23:39.508492+0000","last_peered":"2023-11-16T18:23:39.508492+0000","last_clean":"2023-11-16T18:23:39.508492+0000","last_became_active":"2023-11-16T18:22:46.823895+0000","last_became_peered":"2023-11-16T18:22:46.823895+0000","last_unstale":"2023-11-16T18:23:39.508492+0000","last_undegraded":"2023-11-16T18:23:39.508492+0000","last_fullsized":"2023-11-16T18:23:39.508492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:57:28.297178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504555+0000","last_change":"2023-11-16T18:22:51.118714+0000","last_active":"2023-11-16T18:23:39.504555+0000","last_peered":"2023-11-16T18:23:39.504555+0000","last_clean":"2023-11-16T18:23:39.504555+0000","last_became_active":"2023-11-16T18:22:51.118216+0000","last_became_peered":"2023-11-16T18:22:51.118216+0000","last_unstale":"2023-11-16T18:23:39.504555+0000","last_undegraded":"2023-11-16T18:23:39.504555+0000","last_fullsized":"2023-11-16T18:23:39.504555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:27:57.339401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505119+0000","last_change":"2023-11-16T18:22:46.376431+0000","last_active":"2023-11-16T18:23:39.505119+0000","last_peered":"2023-11-16T18:23:39.505119+0000","last_clean":"2023-11-16T18:23:39.505119+0000","last_became_active":"2023-11-16T18:22:46.375914+0000","last_became_peered":"2023-11-16T18:22:46.375914+0000","last_unstale":"2023-11-16T18:23:39.505119+0000","last_undegraded":"2023-11-16T18:23:39.505119+0000","last_fullsized":"2023-11-16T18:23:39.505119+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:31:34.447838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508396+0000","last_change":"2023-11-16T18:22:46.825002+0000","last_active":"2023-11-16T18:23:39.508396+0000","last_peered":"2023-11-16T18:23:39.508396+0000","last_clean":"2023-11-16T18:23:39.508396+0000","last_became_active":"2023-11-16T18:22:46.824012+0000","last_became_peered":"2023-11-16T18:22:46.824012+0000","last_unstale":"2023-11-16T18:23:39.508396+0000","last_undegraded":"2023-11-16T18:23:39.508396+0000","last_fullsized":"2023-11-16T18:23:39.508396+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:04:41.816487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510910+0000","last_change":"2023-11-16T18:22:51.117623+0000","last_active":"2023-11-16T18:23:39.510910+0000","last_peered":"2023-11-16T18:23:39.510910+0000","last_clean":"2023-11-16T18:23:39.510910+0000","last_became_active":"2023-11-16T18:22:51.117183+0000","last_became_peered":"2023-11-16T18:22:51.117183+0000","last_unstale":"2023-11-16T18:23:39.510910+0000","last_undegraded":"2023-11-16T18:23:39.510910+0000","last_fullsized":"2023-11-16T18:23:39.510910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:16:00.063607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:35.602608+0000","last_change":"2023-11-16T18:23:39.664514+0000","last_active":"2023-11-16T18:24:35.602608+0000","last_peered":"2023-11-16T18:24:35.602608+0000","last_clean":"2023-11-16T18:24:35.602608+0000","last_became_active":"2023-11-16T18:23:39.511648+0000","last_became_peered":"2023-11-16T18:23:39.511648+0000","last_unstale":"2023-11-16T18:24:35.602608+0000","last_undegraded":"2023-11-16T18:24:35.602608+0000","last_fullsized":"2023-11-16T18:24:35.602608+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:08:22.914647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1282,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.835629+0000","last_change":"2023-11-16T18:22:46.820073+0000","last_active":"2023-11-16T18:24:37.835629+0000","last_peered":"2023-11-16T18:24:37.835629+0000","last_clean":"2023-11-16T18:24:37.835629+0000","last_became_active":"2023-11-16T18:22:46.819764+0000","last_became_peered":"2023-11-16T18:22:46.819764+0000","last_unstale":"2023-11-16T18:24:37.835629+0000","last_undegraded":"2023-11-16T18:24:37.835629+0000","last_fullsized":"2023-11-16T18:24:37.835629+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:58.766286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510964+0000","last_change":"2023-11-16T18:22:51.116619+0000","last_active":"2023-11-16T18:23:39.510964+0000","last_peered":"2023-11-16T18:23:39.510964+0000","last_clean":"2023-11-16T18:23:39.510964+0000","last_became_active":"2023-11-16T18:22:51.116264+0000","last_became_peered":"2023-11-16T18:22:51.116264+0000","last_unstale":"2023-11-16T18:23:39.510964+0000","last_undegraded":"2023-11-16T18:23:39.510964+0000","last_fullsized":"2023-11-16T18:23:39.510964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:32:46.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507700+0000","last_change":"2023-11-16T18:22:46.389590+0000","last_active":"2023-11-16T18:23:39.507700+0000","last_peered":"2023-11-16T18:23:39.507700+0000","last_clean":"2023-11-16T18:23:39.507700+0000","last_became_active":"2023-11-16T18:22:46.389354+0000","last_became_peered":"2023-11-16T18:22:46.389354+0000","last_unstale":"2023-11-16T18:23:39.507700+0000","last_undegraded":"2023-11-16T18:23:39.507700+0000","last_fullsized":"2023-11-16T18:23:39.507700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:58:58.086957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509671+0000","last_change":"2023-11-16T18:22:46.841106+0000","last_active":"2023-11-16T18:23:39.509671+0000","last_peered":"2023-11-16T18:23:39.509671+0000","last_clean":"2023-11-16T18:23:39.509671+0000","last_became_active":"2023-11-16T18:22:46.840876+0000","last_became_peered":"2023-11-16T18:22:46.840876+0000","last_unstale":"2023-11-16T18:23:39.509671+0000","last_undegraded":"2023-11-16T18:23:39.509671+0000","last_fullsized":"2023-11-16T18:23:39.509671+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:46:23.740026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506599+0000","last_change":"2023-11-16T18:22:51.125813+0000","last_active":"2023-11-16T18:23:39.506599+0000","last_peered":"2023-11-16T18:23:39.506599+0000","last_clean":"2023-11-16T18:23:39.506599+0000","last_became_active":"2023-11-16T18:22:51.125277+0000","last_became_peered":"2023-11-16T18:22:51.125277+0000","last_unstale":"2023-11-16T18:23:39.506599+0000","last_undegraded":"2023-11-16T18:23:39.506599+0000","last_fullsized":"2023-11-16T18:23:39.506599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:02:51.083984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584572+0000","last_change":"2023-11-16T18:22:46.381208+0000","last_active":"2023-11-16T18:23:39.584572+0000","last_peered":"2023-11-16T18:23:39.584572+0000","last_clean":"2023-11-16T18:23:39.584572+0000","last_became_active":"2023-11-16T18:22:46.380351+0000","last_became_peered":"2023-11-16T18:22:46.380351+0000","last_unstale":"2023-11-16T18:23:39.584572+0000","last_undegraded":"2023-11-16T18:23:39.584572+0000","last_fullsized":"2023-11-16T18:23:39.584572+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:27:31.749114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.855370+0000","last_change":"2023-11-16T18:22:46.843043+0000","last_active":"2023-11-16T18:24:37.855370+0000","last_peered":"2023-11-16T18:24:37.855370+0000","last_clean":"2023-11-16T18:24:37.855370+0000","last_became_active":"2023-11-16T18:22:46.841522+0000","last_became_peered":"2023-11-16T18:22:46.841522+0000","last_unstale":"2023-11-16T18:24:37.855370+0000","last_undegraded":"2023-11-16T18:24:37.855370+0000","last_fullsized":"2023-11-16T18:24:37.855370+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:53:29.309989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505013+0000","last_change":"2023-11-16T18:22:51.127729+0000","last_active":"2023-11-16T18:23:39.505013+0000","last_peered":"2023-11-16T18:23:39.505013+0000","last_clean":"2023-11-16T18:23:39.505013+0000","last_became_active":"2023-11-16T18:22:51.127303+0000","last_became_peered":"2023-11-16T18:22:51.127303+0000","last_unstale":"2023-11-16T18:23:39.505013+0000","last_undegraded":"2023-11-16T18:23:39.505013+0000","last_fullsized":"2023-11-16T18:23:39.505013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:16:37.982862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508172+0000","last_change":"2023-11-16T18:22:46.377437+0000","last_active":"2023-11-16T18:23:39.508172+0000","last_peered":"2023-11-16T18:23:39.508172+0000","last_clean":"2023-11-16T18:23:39.508172+0000","last_became_active":"2023-11-16T18:22:46.377120+0000","last_became_peered":"2023-11-16T18:22:46.377120+0000","last_unstale":"2023-11-16T18:23:39.508172+0000","last_undegraded":"2023-11-16T18:23:39.508172+0000","last_fullsized":"2023-11-16T18:23:39.508172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:02:31.673863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500756+0000","last_change":"2023-11-16T18:22:46.837804+0000","last_active":"2023-11-16T18:23:39.500756+0000","last_peered":"2023-11-16T18:23:39.500756+0000","last_clean":"2023-11-16T18:23:39.500756+0000","last_became_active":"2023-11-16T18:22:46.837432+0000","last_became_peered":"2023-11-16T18:22:46.837432+0000","last_unstale":"2023-11-16T18:23:39.500756+0000","last_undegraded":"2023-11-16T18:23:39.500756+0000","last_fullsized":"2023-11-16T18:23:39.500756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:10:14.516300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503848+0000","last_change":"2023-11-16T18:22:51.127546+0000","last_active":"2023-11-16T18:23:39.503848+0000","last_peered":"2023-11-16T18:23:39.503848+0000","last_clean":"2023-11-16T18:23:39.503848+0000","last_became_active":"2023-11-16T18:22:51.127133+0000","last_became_peered":"2023-11-16T18:22:51.127133+0000","last_unstale":"2023-11-16T18:23:39.503848+0000","last_undegraded":"2023-11-16T18:23:39.503848+0000","last_fullsized":"2023-11-16T18:23:39.503848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:02:33.244659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":117,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:33.883048+0000","last_change":"2023-11-16T18:22:46.386309+0000","last_active":"2023-11-16T18:24:33.883048+0000","last_peered":"2023-11-16T18:24:33.883048+0000","last_clean":"2023-11-16T18:24:33.883048+0000","last_became_active":"2023-11-16T18:22:46.385960+0000","last_became_peered":"2023-11-16T18:22:46.385960+0000","last_unstale":"2023-11-16T18:24:33.883048+0000","last_undegraded":"2023-11-16T18:24:33.883048+0000","last_fullsized":"2023-11-16T18:24:33.883048+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:46:45.743666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508584+0000","last_change":"2023-11-16T18:22:46.827211+0000","last_active":"2023-11-16T18:23:39.508584+0000","last_peered":"2023-11-16T18:23:39.508584+0000","last_clean":"2023-11-16T18:23:39.508584+0000","last_became_active":"2023-11-16T18:22:46.826396+0000","last_became_peered":"2023-11-16T18:22:46.826396+0000","last_unstale":"2023-11-16T18:23:39.508584+0000","last_undegraded":"2023-11-16T18:23:39.508584+0000","last_fullsized":"2023-11-16T18:23:39.508584+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:58:36.202010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.513065+0000","last_change":"2023-11-16T18:23:39.512554+0000","last_active":"2023-11-16T18:23:39.513065+0000","last_peered":"2023-11-16T18:23:39.513065+0000","last_clean":"2023-11-16T18:23:39.513065+0000","last_became_active":"2023-11-16T18:23:39.512118+0000","last_became_peered":"2023-11-16T18:23:39.512118+0000","last_unstale":"2023-11-16T18:23:39.513065+0000","last_undegraded":"2023-11-16T18:23:39.513065+0000","last_fullsized":"2023-11-16T18:23:39.513065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:55:33.269875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503214+0000","last_change":"2023-11-16T18:22:46.384680+0000","last_active":"2023-11-16T18:23:39.503214+0000","last_peered":"2023-11-16T18:23:39.503214+0000","last_clean":"2023-11-16T18:23:39.503214+0000","last_became_active":"2023-11-16T18:22:46.384290+0000","last_became_peered":"2023-11-16T18:22:46.384290+0000","last_unstale":"2023-11-16T18:23:39.503214+0000","last_undegraded":"2023-11-16T18:23:39.503214+0000","last_fullsized":"2023-11-16T18:23:39.503214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:05:40.782246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504447+0000","last_change":"2023-11-16T18:22:46.834726+0000","last_active":"2023-11-16T18:23:39.504447+0000","last_peered":"2023-11-16T18:23:39.504447+0000","last_clean":"2023-11-16T18:23:39.504447+0000","last_became_active":"2023-11-16T18:22:46.834424+0000","last_became_peered":"2023-11-16T18:22:46.834424+0000","last_unstale":"2023-11-16T18:23:39.504447+0000","last_undegraded":"2023-11-16T18:23:39.504447+0000","last_fullsized":"2023-11-16T18:23:39.504447+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:52:11.738947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.513064+0000","last_change":"2023-11-16T18:23:39.513064+0000","last_active":"2023-11-16T18:23:39.513064+0000","last_peered":"2023-11-16T18:23:39.513064+0000","last_clean":"2023-11-16T18:23:39.513064+0000","last_became_active":"2023-11-16T18:23:39.512606+0000","last_became_peered":"2023-11-16T18:23:39.512606+0000","last_unstale":"2023-11-16T18:23:39.513064+0000","last_undegraded":"2023-11-16T18:23:39.513064+0000","last_fullsized":"2023-11-16T18:23:39.513064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:08:18.327854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508258+0000","last_change":"2023-11-16T18:22:46.380804+0000","last_active":"2023-11-16T18:23:39.508258+0000","last_peered":"2023-11-16T18:23:39.508258+0000","last_clean":"2023-11-16T18:23:39.508258+0000","last_became_active":"2023-11-16T18:22:46.380477+0000","last_became_peered":"2023-11-16T18:22:46.380477+0000","last_unstale":"2023-11-16T18:23:39.508258+0000","last_undegraded":"2023-11-16T18:23:39.508258+0000","last_fullsized":"2023-11-16T18:23:39.508258+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:17:45.128995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.586650+0000","last_change":"2023-11-16T18:23:39.586650+0000","last_active":"2023-11-16T18:23:39.586650+0000","last_peered":"2023-11-16T18:23:39.586650+0000","last_clean":"2023-11-16T18:23:39.586650+0000","last_became_active":"2023-11-16T18:23:39.586241+0000","last_became_peered":"2023-11-16T18:23:39.586241+0000","last_unstale":"2023-11-16T18:23:39.586650+0000","last_undegraded":"2023-11-16T18:23:39.586650+0000","last_fullsized":"2023-11-16T18:23:39.586650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:15:14.907282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500754+0000","last_change":"2023-11-16T18:22:51.119711+0000","last_active":"2023-11-16T18:23:39.500754+0000","last_peered":"2023-11-16T18:23:39.500754+0000","last_clean":"2023-11-16T18:23:39.500754+0000","last_became_active":"2023-11-16T18:22:51.119304+0000","last_became_peered":"2023-11-16T18:22:51.119304+0000","last_unstale":"2023-11-16T18:23:39.500754+0000","last_undegraded":"2023-11-16T18:23:39.500754+0000","last_fullsized":"2023-11-16T18:23:39.500754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:37:34.450684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":260,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:34.600234+0000","last_change":"2023-11-16T18:23:39.671486+0000","last_active":"2023-11-16T18:24:34.600234+0000","last_peered":"2023-11-16T18:24:34.600234+0000","last_clean":"2023-11-16T18:24:34.600234+0000","last_became_active":"2023-11-16T18:23:39.513413+0000","last_became_peered":"2023-11-16T18:23:39.513413+0000","last_unstale":"2023-11-16T18:24:34.600234+0000","last_undegraded":"2023-11-16T18:24:34.600234+0000","last_fullsized":"2023-11-16T18:24:34.600234+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:07:37.199058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61155,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":178,"num_read_kb":147,"num_write":34,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60837,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.845612+0000","last_change":"2023-11-16T18:22:46.833537+0000","last_active":"2023-11-16T18:24:37.845612+0000","last_peered":"2023-11-16T18:24:37.845612+0000","last_clean":"2023-11-16T18:24:37.845612+0000","last_became_active":"2023-11-16T18:22:46.833217+0000","last_became_peered":"2023-11-16T18:22:46.833217+0000","last_unstale":"2023-11-16T18:24:37.845612+0000","last_undegraded":"2023-11-16T18:24:37.845612+0000","last_fullsized":"2023-11-16T18:24:37.845612+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:54:32.987198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585712+0000","last_change":"2023-11-16T18:23:39.585247+0000","last_active":"2023-11-16T18:23:39.585712+0000","last_peered":"2023-11-16T18:23:39.585712+0000","last_clean":"2023-11-16T18:23:39.585712+0000","last_became_active":"2023-11-16T18:23:39.584802+0000","last_became_peered":"2023-11-16T18:23:39.584802+0000","last_unstale":"2023-11-16T18:23:39.585712+0000","last_undegraded":"2023-11-16T18:23:39.585712+0000","last_fullsized":"2023-11-16T18:23:39.585712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:45:43.860286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509671+0000","last_change":"2023-11-16T18:22:46.382812+0000","last_active":"2023-11-16T18:23:39.509671+0000","last_peered":"2023-11-16T18:23:39.509671+0000","last_clean":"2023-11-16T18:23:39.509671+0000","last_became_active":"2023-11-16T18:22:46.382499+0000","last_became_peered":"2023-11-16T18:22:46.382499+0000","last_unstale":"2023-11-16T18:23:39.509671+0000","last_undegraded":"2023-11-16T18:23:39.509671+0000","last_fullsized":"2023-11-16T18:23:39.509671+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:48:39.263999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.852503+0000","last_change":"2023-11-16T18:22:46.840917+0000","last_active":"2023-11-16T18:24:37.852503+0000","last_peered":"2023-11-16T18:24:37.852503+0000","last_clean":"2023-11-16T18:24:37.852503+0000","last_became_active":"2023-11-16T18:22:46.840579+0000","last_became_peered":"2023-11-16T18:22:46.840579+0000","last_unstale":"2023-11-16T18:24:37.852503+0000","last_undegraded":"2023-11-16T18:24:37.852503+0000","last_fullsized":"2023-11-16T18:24:37.852503+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:05:17.920819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506477+0000","last_change":"2023-11-16T18:22:51.117852+0000","last_active":"2023-11-16T18:23:39.506477+0000","last_peered":"2023-11-16T18:23:39.506477+0000","last_clean":"2023-11-16T18:23:39.506477+0000","last_became_active":"2023-11-16T18:22:51.117390+0000","last_became_peered":"2023-11-16T18:22:51.117390+0000","last_unstale":"2023-11-16T18:23:39.506477+0000","last_undegraded":"2023-11-16T18:23:39.506477+0000","last_fullsized":"2023-11-16T18:23:39.506477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:11:37.167591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508136+0000","last_change":"2023-11-16T18:22:46.385137+0000","last_active":"2023-11-16T18:23:39.508136+0000","last_peered":"2023-11-16T18:23:39.508136+0000","last_clean":"2023-11-16T18:23:39.508136+0000","last_became_active":"2023-11-16T18:22:46.384636+0000","last_became_peered":"2023-11-16T18:22:46.384636+0000","last_unstale":"2023-11-16T18:23:39.508136+0000","last_undegraded":"2023-11-16T18:23:39.508136+0000","last_fullsized":"2023-11-16T18:23:39.508136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:11:10.885960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502619+0000","last_change":"2023-11-16T18:22:46.834786+0000","last_active":"2023-11-16T18:23:39.502619+0000","last_peered":"2023-11-16T18:23:39.502619+0000","last_clean":"2023-11-16T18:23:39.502619+0000","last_became_active":"2023-11-16T18:22:46.834338+0000","last_became_peered":"2023-11-16T18:22:46.834338+0000","last_unstale":"2023-11-16T18:23:39.502619+0000","last_undegraded":"2023-11-16T18:23:39.502619+0000","last_fullsized":"2023-11-16T18:23:39.502619+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:52:01.046978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500921+0000","last_change":"2023-11-16T18:22:51.124667+0000","last_active":"2023-11-16T18:23:39.500921+0000","last_peered":"2023-11-16T18:23:39.500921+0000","last_clean":"2023-11-16T18:23:39.500921+0000","last_became_active":"2023-11-16T18:22:51.124211+0000","last_became_peered":"2023-11-16T18:22:51.124211+0000","last_unstale":"2023-11-16T18:23:39.500921+0000","last_undegraded":"2023-11-16T18:23:39.500921+0000","last_fullsized":"2023-11-16T18:23:39.500921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:22:02.360209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":121,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:34.054214+0000","last_change":"2023-11-16T18:22:46.380829+0000","last_active":"2023-11-16T18:24:34.054214+0000","last_peered":"2023-11-16T18:24:34.054214+0000","last_clean":"2023-11-16T18:24:34.054214+0000","last_became_active":"2023-11-16T18:22:46.380466+0000","last_became_peered":"2023-11-16T18:22:46.380466+0000","last_unstale":"2023-11-16T18:24:34.054214+0000","last_undegraded":"2023-11-16T18:24:34.054214+0000","last_fullsized":"2023-11-16T18:24:34.054214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:22:52.401291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502257+0000","last_change":"2023-11-16T18:22:46.835900+0000","last_active":"2023-11-16T18:23:39.502257+0000","last_peered":"2023-11-16T18:23:39.502257+0000","last_clean":"2023-11-16T18:23:39.502257+0000","last_became_active":"2023-11-16T18:22:46.835438+0000","last_became_peered":"2023-11-16T18:22:46.835438+0000","last_unstale":"2023-11-16T18:23:39.502257+0000","last_undegraded":"2023-11-16T18:23:39.502257+0000","last_fullsized":"2023-11-16T18:23:39.502257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:36:23.733440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585779+0000","last_change":"2023-11-16T18:22:51.120005+0000","last_active":"2023-11-16T18:23:39.585779+0000","last_peered":"2023-11-16T18:23:39.585779+0000","last_clean":"2023-11-16T18:23:39.585779+0000","last_became_active":"2023-11-16T18:22:51.119540+0000","last_became_peered":"2023-11-16T18:22:51.119540+0000","last_unstale":"2023-11-16T18:23:39.585779+0000","last_undegraded":"2023-11-16T18:23:39.585779+0000","last_fullsized":"2023-11-16T18:23:39.585779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:33:29.490015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:41.897854+0000","last_change":"2023-11-16T18:23:41.897338+0000","last_active":"2023-11-16T18:23:41.897854+0000","last_peered":"2023-11-16T18:23:41.897854+0000","last_clean":"2023-11-16T18:23:41.897854+0000","last_became_active":"2023-11-16T18:23:39.512018+0000","last_became_peered":"2023-11-16T18:23:39.512018+0000","last_unstale":"2023-11-16T18:23:41.897854+0000","last_undegraded":"2023-11-16T18:23:41.897854+0000","last_fullsized":"2023-11-16T18:23:41.897854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:53:22.707297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507902+0000","last_change":"2023-11-16T18:22:46.823838+0000","last_active":"2023-11-16T18:23:39.507902+0000","last_peered":"2023-11-16T18:23:39.507902+0000","last_clean":"2023-11-16T18:23:39.507902+0000","last_became_active":"2023-11-16T18:22:46.823545+0000","last_became_peered":"2023-11-16T18:22:46.823545+0000","last_unstale":"2023-11-16T18:23:39.507902+0000","last_undegraded":"2023-11-16T18:23:39.507902+0000","last_fullsized":"2023-11-16T18:23:39.507902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:11:32.214804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507865+0000","last_change":"2023-11-16T18:22:51.127371+0000","last_active":"2023-11-16T18:23:39.507865+0000","last_peered":"2023-11-16T18:23:39.507865+0000","last_clean":"2023-11-16T18:23:39.507865+0000","last_became_active":"2023-11-16T18:22:51.126951+0000","last_became_peered":"2023-11-16T18:22:51.126951+0000","last_unstale":"2023-11-16T18:23:39.507865+0000","last_undegraded":"2023-11-16T18:23:39.507865+0000","last_fullsized":"2023-11-16T18:23:39.507865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:55.418916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504323+0000","last_change":"2023-11-16T18:22:46.376561+0000","last_active":"2023-11-16T18:23:39.504323+0000","last_peered":"2023-11-16T18:23:39.504323+0000","last_clean":"2023-11-16T18:23:39.504323+0000","last_became_active":"2023-11-16T18:22:46.375426+0000","last_became_peered":"2023-11-16T18:22:46.375426+0000","last_unstale":"2023-11-16T18:23:39.504323+0000","last_undegraded":"2023-11-16T18:23:39.504323+0000","last_fullsized":"2023-11-16T18:23:39.504323+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:29:00.383105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.855455+0000","last_change":"2023-11-16T18:22:46.826657+0000","last_active":"2023-11-16T18:24:37.855455+0000","last_peered":"2023-11-16T18:24:37.855455+0000","last_clean":"2023-11-16T18:24:37.855455+0000","last_became_active":"2023-11-16T18:22:46.826126+0000","last_became_peered":"2023-11-16T18:22:46.826126+0000","last_unstale":"2023-11-16T18:24:37.855455+0000","last_undegraded":"2023-11-16T18:24:37.855455+0000","last_fullsized":"2023-11-16T18:24:37.855455+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:13:34.794628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504904+0000","last_change":"2023-11-16T18:22:51.127949+0000","last_active":"2023-11-16T18:23:39.504904+0000","last_peered":"2023-11-16T18:23:39.504904+0000","last_clean":"2023-11-16T18:23:39.504904+0000","last_became_active":"2023-11-16T18:22:51.127388+0000","last_became_peered":"2023-11-16T18:22:51.127388+0000","last_unstale":"2023-11-16T18:23:39.504904+0000","last_undegraded":"2023-11-16T18:23:39.504904+0000","last_fullsized":"2023-11-16T18:23:39.504904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:37:30.745978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500985+0000","last_change":"2023-11-16T18:22:46.378568+0000","last_active":"2023-11-16T18:23:39.500985+0000","last_peered":"2023-11-16T18:23:39.500985+0000","last_clean":"2023-11-16T18:23:39.500985+0000","last_became_active":"2023-11-16T18:22:46.378018+0000","last_became_peered":"2023-11-16T18:22:46.378018+0000","last_unstale":"2023-11-16T18:23:39.500985+0000","last_undegraded":"2023-11-16T18:23:39.500985+0000","last_fullsized":"2023-11-16T18:23:39.500985+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:05:59.328232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509189+0000","last_change":"2023-11-16T18:22:46.841129+0000","last_active":"2023-11-16T18:23:39.509189+0000","last_peered":"2023-11-16T18:23:39.509189+0000","last_clean":"2023-11-16T18:23:39.509189+0000","last_became_active":"2023-11-16T18:22:46.840780+0000","last_became_peered":"2023-11-16T18:22:46.840780+0000","last_unstale":"2023-11-16T18:23:39.509189+0000","last_undegraded":"2023-11-16T18:23:39.509189+0000","last_fullsized":"2023-11-16T18:23:39.509189+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:09:39.912158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585594+0000","last_change":"2023-11-16T18:22:51.126221+0000","last_active":"2023-11-16T18:23:39.585594+0000","last_peered":"2023-11-16T18:23:39.585594+0000","last_clean":"2023-11-16T18:23:39.585594+0000","last_became_active":"2023-11-16T18:22:51.125770+0000","last_became_peered":"2023-11-16T18:22:51.125770+0000","last_unstale":"2023-11-16T18:23:39.585594+0000","last_undegraded":"2023-11-16T18:23:39.585594+0000","last_fullsized":"2023-11-16T18:23:39.585594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:35:06.867124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502402+0000","last_change":"2023-11-16T18:22:46.384872+0000","last_active":"2023-11-16T18:23:39.502402+0000","last_peered":"2023-11-16T18:23:39.502402+0000","last_clean":"2023-11-16T18:23:39.502402+0000","last_became_active":"2023-11-16T18:22:46.384499+0000","last_became_peered":"2023-11-16T18:22:46.384499+0000","last_unstale":"2023-11-16T18:23:39.502402+0000","last_undegraded":"2023-11-16T18:23:39.502402+0000","last_fullsized":"2023-11-16T18:23:39.502402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:26:53.535471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507989+0000","last_change":"2023-11-16T18:22:46.822574+0000","last_active":"2023-11-16T18:23:39.507989+0000","last_peered":"2023-11-16T18:23:39.507989+0000","last_clean":"2023-11-16T18:23:39.507989+0000","last_became_active":"2023-11-16T18:22:46.822230+0000","last_became_peered":"2023-11-16T18:22:46.822230+0000","last_unstale":"2023-11-16T18:23:39.507989+0000","last_undegraded":"2023-11-16T18:23:39.507989+0000","last_fullsized":"2023-11-16T18:23:39.507989+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:16:44.677701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585706+0000","last_change":"2023-11-16T18:22:51.117838+0000","last_active":"2023-11-16T18:23:39.585706+0000","last_peered":"2023-11-16T18:23:39.585706+0000","last_clean":"2023-11-16T18:23:39.585706+0000","last_became_active":"2023-11-16T18:22:51.117275+0000","last_became_peered":"2023-11-16T18:22:51.117275+0000","last_unstale":"2023-11-16T18:23:39.585706+0000","last_undegraded":"2023-11-16T18:23:39.585706+0000","last_fullsized":"2023-11-16T18:23:39.585706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:42:14.482945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508453+0000","last_change":"2023-11-16T18:22:46.827499+0000","last_active":"2023-11-16T18:23:39.508453+0000","last_peered":"2023-11-16T18:23:39.508453+0000","last_clean":"2023-11-16T18:23:39.508453+0000","last_became_active":"2023-11-16T18:22:46.827166+0000","last_became_peered":"2023-11-16T18:22:46.827166+0000","last_unstale":"2023-11-16T18:23:39.508453+0000","last_undegraded":"2023-11-16T18:23:39.508453+0000","last_fullsized":"2023-11-16T18:23:39.508453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:23:43.219136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":122,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.807927+0000","last_change":"2023-11-16T18:22:46.381397+0000","last_active":"2023-11-16T18:24:37.807927+0000","last_peered":"2023-11-16T18:24:37.807927+0000","last_clean":"2023-11-16T18:24:37.807927+0000","last_became_active":"2023-11-16T18:22:46.380939+0000","last_became_peered":"2023-11-16T18:22:46.380939+0000","last_unstale":"2023-11-16T18:24:37.807927+0000","last_undegraded":"2023-11-16T18:24:37.807927+0000","last_fullsized":"2023-11-16T18:24:37.807927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:44:30.574771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":91,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502915+0000","last_change":"2023-11-16T18:22:51.125044+0000","last_active":"2023-11-16T18:23:39.502915+0000","last_peered":"2023-11-16T18:23:39.502915+0000","last_clean":"2023-11-16T18:23:39.502915+0000","last_became_active":"2023-11-16T18:22:51.124422+0000","last_became_peered":"2023-11-16T18:22:51.124422+0000","last_unstale":"2023-11-16T18:23:39.502915+0000","last_undegraded":"2023-11-16T18:23:39.502915+0000","last_fullsized":"2023-11-16T18:23:39.502915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:11.651599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.858524+0000","last_change":"2023-11-16T18:22:46.841500+0000","last_active":"2023-11-16T18:24:37.858524+0000","last_peered":"2023-11-16T18:24:37.858524+0000","last_clean":"2023-11-16T18:24:37.858524+0000","last_became_active":"2023-11-16T18:22:46.841186+0000","last_became_peered":"2023-11-16T18:22:46.841186+0000","last_unstale":"2023-11-16T18:24:37.858524+0000","last_undegraded":"2023-11-16T18:24:37.858524+0000","last_fullsized":"2023-11-16T18:24:37.858524+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:38:25.407837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504887+0000","last_change":"2023-11-16T18:22:46.380988+0000","last_active":"2023-11-16T18:23:39.504887+0000","last_peered":"2023-11-16T18:23:39.504887+0000","last_clean":"2023-11-16T18:23:39.504887+0000","last_became_active":"2023-11-16T18:22:46.380616+0000","last_became_peered":"2023-11-16T18:22:46.380616+0000","last_unstale":"2023-11-16T18:23:39.504887+0000","last_undegraded":"2023-11-16T18:23:39.504887+0000","last_fullsized":"2023-11-16T18:23:39.504887+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:24:26.911536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502037+0000","last_change":"2023-11-16T18:21:44.282706+0000","last_active":"2023-11-16T18:23:39.502037+0000","last_peered":"2023-11-16T18:23:39.502037+0000","last_clean":"2023-11-16T18:23:39.502037+0000","last_became_active":"2023-11-16T18:21:44.282201+0000","last_became_peered":"2023-11-16T18:21:44.282201+0000","last_unstale":"2023-11-16T18:23:39.502037+0000","last_undegraded":"2023-11-16T18:23:39.502037+0000","last_fullsized":"2023-11-16T18:23:39.502037+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_clean_scrub_stamp":"2023-11-16T18:18:46.711191+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:41:17.702365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508868+0000","last_change":"2023-11-16T18:22:51.123051+0000","last_active":"2023-11-16T18:23:39.508868+0000","last_peered":"2023-11-16T18:23:39.508868+0000","last_clean":"2023-11-16T18:23:39.508868+0000","last_became_active":"2023-11-16T18:22:51.122635+0000","last_became_peered":"2023-11-16T18:22:51.122635+0000","last_unstale":"2023-11-16T18:23:39.508868+0000","last_undegraded":"2023-11-16T18:23:39.508868+0000","last_fullsized":"2023-11-16T18:23:39.508868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:59:09.977278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.852848+0000","last_change":"2023-11-16T18:22:46.822625+0000","last_active":"2023-11-16T18:24:37.852848+0000","last_peered":"2023-11-16T18:24:37.852848+0000","last_clean":"2023-11-16T18:24:37.852848+0000","last_became_active":"2023-11-16T18:22:46.822296+0000","last_became_peered":"2023-11-16T18:22:46.822296+0000","last_unstale":"2023-11-16T18:24:37.852848+0000","last_undegraded":"2023-11-16T18:24:37.852848+0000","last_fullsized":"2023-11-16T18:24:37.852848+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:43:40.832756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502155+0000","last_change":"2023-11-16T18:22:46.380445+0000","last_active":"2023-11-16T18:23:39.502155+0000","last_peered":"2023-11-16T18:23:39.502155+0000","last_clean":"2023-11-16T18:23:39.502155+0000","last_became_active":"2023-11-16T18:22:46.380113+0000","last_became_peered":"2023-11-16T18:22:46.380113+0000","last_unstale":"2023-11-16T18:23:39.502155+0000","last_undegraded":"2023-11-16T18:23:39.502155+0000","last_fullsized":"2023-11-16T18:23:39.502155+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:17:48.769928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504943+0000","last_change":"2023-11-16T18:22:51.118457+0000","last_active":"2023-11-16T18:23:39.504943+0000","last_peered":"2023-11-16T18:23:39.504943+0000","last_clean":"2023-11-16T18:23:39.504943+0000","last_became_active":"2023-11-16T18:22:51.118013+0000","last_became_peered":"2023-11-16T18:22:51.118013+0000","last_unstale":"2023-11-16T18:23:39.504943+0000","last_undegraded":"2023-11-16T18:23:39.504943+0000","last_fullsized":"2023-11-16T18:23:39.504943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:11.432801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506267+0000","last_change":"2023-11-16T18:22:46.384799+0000","last_active":"2023-11-16T18:23:39.506267+0000","last_peered":"2023-11-16T18:23:39.506267+0000","last_clean":"2023-11-16T18:23:39.506267+0000","last_became_active":"2023-11-16T18:22:46.384482+0000","last_became_peered":"2023-11-16T18:22:46.384482+0000","last_unstale":"2023-11-16T18:23:39.506267+0000","last_undegraded":"2023-11-16T18:23:39.506267+0000","last_fullsized":"2023-11-16T18:23:39.506267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:42:48.220989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.839008+0000","last_change":"2023-11-16T18:22:46.829541+0000","last_active":"2023-11-16T18:24:37.839008+0000","last_peered":"2023-11-16T18:24:37.839008+0000","last_clean":"2023-11-16T18:24:37.839008+0000","last_became_active":"2023-11-16T18:22:46.829006+0000","last_became_peered":"2023-11-16T18:22:46.829006+0000","last_unstale":"2023-11-16T18:24:37.839008+0000","last_undegraded":"2023-11-16T18:24:37.839008+0000","last_fullsized":"2023-11-16T18:24:37.839008+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:13:02.956336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504640+0000","last_change":"2023-11-16T18:22:51.122751+0000","last_active":"2023-11-16T18:23:39.504640+0000","last_peered":"2023-11-16T18:23:39.504640+0000","last_clean":"2023-11-16T18:23:39.504640+0000","last_became_active":"2023-11-16T18:22:51.122423+0000","last_became_peered":"2023-11-16T18:22:51.122423+0000","last_unstale":"2023-11-16T18:23:39.504640+0000","last_undegraded":"2023-11-16T18:23:39.504640+0000","last_fullsized":"2023-11-16T18:23:39.504640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:27:49.338555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'5","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.516672+0000","last_change":"2023-11-16T18:23:39.516672+0000","last_active":"2023-11-16T18:23:39.516672+0000","last_peered":"2023-11-16T18:23:39.516672+0000","last_clean":"2023-11-16T18:23:39.516672+0000","last_became_active":"2023-11-16T18:23:39.509087+0000","last_became_peered":"2023-11-16T18:23:39.509087+0000","last_unstale":"2023-11-16T18:23:39.516672+0000","last_undegraded":"2023-11-16T18:23:39.516672+0000","last_fullsized":"2023-11-16T18:23:39.516672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:57:28.479767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.850681+0000","last_change":"2023-11-16T18:22:46.843802+0000","last_active":"2023-11-16T18:24:37.850681+0000","last_peered":"2023-11-16T18:24:37.850681+0000","last_clean":"2023-11-16T18:24:37.850681+0000","last_became_active":"2023-11-16T18:22:46.842210+0000","last_became_peered":"2023-11-16T18:22:46.842210+0000","last_unstale":"2023-11-16T18:24:37.850681+0000","last_undegraded":"2023-11-16T18:24:37.850681+0000","last_fullsized":"2023-11-16T18:24:37.850681+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:20:02.893140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509951+0000","last_change":"2023-11-16T18:22:51.124632+0000","last_active":"2023-11-16T18:23:39.509951+0000","last_peered":"2023-11-16T18:23:39.509951+0000","last_clean":"2023-11-16T18:23:39.509951+0000","last_became_active":"2023-11-16T18:22:51.124220+0000","last_became_peered":"2023-11-16T18:22:51.124220+0000","last_unstale":"2023-11-16T18:23:39.509951+0000","last_undegraded":"2023-11-16T18:23:39.509951+0000","last_fullsized":"2023-11-16T18:23:39.509951+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:51:57.981627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505140+0000","last_change":"2023-11-16T18:22:46.381456+0000","last_active":"2023-11-16T18:23:39.505140+0000","last_peered":"2023-11-16T18:23:39.505140+0000","last_clean":"2023-11-16T18:23:39.505140+0000","last_became_active":"2023-11-16T18:22:46.381137+0000","last_became_peered":"2023-11-16T18:22:46.381137+0000","last_unstale":"2023-11-16T18:23:39.505140+0000","last_undegraded":"2023-11-16T18:23:39.505140+0000","last_fullsized":"2023-11-16T18:23:39.505140+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:34:36.759190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505363+0000","last_change":"2023-11-16T18:22:46.838249+0000","last_active":"2023-11-16T18:23:39.505363+0000","last_peered":"2023-11-16T18:23:39.505363+0000","last_clean":"2023-11-16T18:23:39.505363+0000","last_became_active":"2023-11-16T18:22:46.837925+0000","last_became_peered":"2023-11-16T18:22:46.837925+0000","last_unstale":"2023-11-16T18:23:39.505363+0000","last_undegraded":"2023-11-16T18:23:39.505363+0000","last_fullsized":"2023-11-16T18:23:39.505363+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:02:17.287183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504877+0000","last_change":"2023-11-16T18:22:51.123687+0000","last_active":"2023-11-16T18:23:39.504877+0000","last_peered":"2023-11-16T18:23:39.504877+0000","last_clean":"2023-11-16T18:23:39.504877+0000","last_became_active":"2023-11-16T18:22:51.123209+0000","last_became_peered":"2023-11-16T18:22:51.123209+0000","last_unstale":"2023-11-16T18:23:39.504877+0000","last_undegraded":"2023-11-16T18:23:39.504877+0000","last_fullsized":"2023-11-16T18:23:39.504877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:09:15.998485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508037+0000","last_change":"2023-11-16T18:22:46.378038+0000","last_active":"2023-11-16T18:23:39.508037+0000","last_peered":"2023-11-16T18:23:39.508037+0000","last_clean":"2023-11-16T18:23:39.508037+0000","last_became_active":"2023-11-16T18:22:46.377680+0000","last_became_peered":"2023-11-16T18:22:46.377680+0000","last_unstale":"2023-11-16T18:23:39.508037+0000","last_undegraded":"2023-11-16T18:23:39.508037+0000","last_fullsized":"2023-11-16T18:23:39.508037+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:31.194156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.818543+0000","last_change":"2023-11-16T18:22:46.821888+0000","last_active":"2023-11-16T18:24:37.818543+0000","last_peered":"2023-11-16T18:24:37.818543+0000","last_clean":"2023-11-16T18:24:37.818543+0000","last_became_active":"2023-11-16T18:22:46.821446+0000","last_became_peered":"2023-11-16T18:22:46.821446+0000","last_unstale":"2023-11-16T18:24:37.818543+0000","last_undegraded":"2023-11-16T18:24:37.818543+0000","last_fullsized":"2023-11-16T18:24:37.818543+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:11:30.632655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501162+0000","last_change":"2023-11-16T18:22:51.119717+0000","last_active":"2023-11-16T18:23:39.501162+0000","last_peered":"2023-11-16T18:23:39.501162+0000","last_clean":"2023-11-16T18:23:39.501162+0000","last_became_active":"2023-11-16T18:22:51.119256+0000","last_became_peered":"2023-11-16T18:22:51.119256+0000","last_unstale":"2023-11-16T18:23:39.501162+0000","last_undegraded":"2023-11-16T18:23:39.501162+0000","last_fullsized":"2023-11-16T18:23:39.501162+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:25:42.785279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504001+0000","last_change":"2023-11-16T18:22:46.375387+0000","last_active":"2023-11-16T18:23:39.504001+0000","last_peered":"2023-11-16T18:23:39.504001+0000","last_clean":"2023-11-16T18:23:39.504001+0000","last_became_active":"2023-11-16T18:22:46.374886+0000","last_became_peered":"2023-11-16T18:22:46.374886+0000","last_unstale":"2023-11-16T18:23:39.504001+0000","last_undegraded":"2023-11-16T18:23:39.504001+0000","last_fullsized":"2023-11-16T18:23:39.504001+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:18:24.699496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.854268+0000","last_change":"2023-11-16T18:22:46.820315+0000","last_active":"2023-11-16T18:24:37.854268+0000","last_peered":"2023-11-16T18:24:37.854268+0000","last_clean":"2023-11-16T18:24:37.854268+0000","last_became_active":"2023-11-16T18:22:46.819592+0000","last_became_peered":"2023-11-16T18:22:46.819592+0000","last_unstale":"2023-11-16T18:24:37.854268+0000","last_undegraded":"2023-11-16T18:24:37.854268+0000","last_fullsized":"2023-11-16T18:24:37.854268+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:43:26.336090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.518963+0000","last_change":"2023-11-16T18:23:39.518963+0000","last_active":"2023-11-16T18:23:39.518963+0000","last_peered":"2023-11-16T18:23:39.518963+0000","last_clean":"2023-11-16T18:23:39.518963+0000","last_became_active":"2023-11-16T18:23:39.513817+0000","last_became_peered":"2023-11-16T18:23:39.513817+0000","last_unstale":"2023-11-16T18:23:39.518963+0000","last_undegraded":"2023-11-16T18:23:39.518963+0000","last_fullsized":"2023-11-16T18:23:39.518963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:16:46.629599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510801+0000","last_change":"2023-11-16T18:22:46.378638+0000","last_active":"2023-11-16T18:23:39.510801+0000","last_peered":"2023-11-16T18:23:39.510801+0000","last_clean":"2023-11-16T18:23:39.510801+0000","last_became_active":"2023-11-16T18:22:46.378341+0000","last_became_peered":"2023-11-16T18:22:46.378341+0000","last_unstale":"2023-11-16T18:23:39.510801+0000","last_undegraded":"2023-11-16T18:23:39.510801+0000","last_fullsized":"2023-11-16T18:23:39.510801+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:36:28.625148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.857732+0000","last_change":"2023-11-16T18:22:46.843431+0000","last_active":"2023-11-16T18:24:37.857732+0000","last_peered":"2023-11-16T18:24:37.857732+0000","last_clean":"2023-11-16T18:24:37.857732+0000","last_became_active":"2023-11-16T18:22:46.842992+0000","last_became_peered":"2023-11-16T18:22:46.842992+0000","last_unstale":"2023-11-16T18:24:37.857732+0000","last_undegraded":"2023-11-16T18:24:37.857732+0000","last_fullsized":"2023-11-16T18:24:37.857732+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:23:28.821023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503912+0000","last_change":"2023-11-16T18:22:51.122640+0000","last_active":"2023-11-16T18:23:39.503912+0000","last_peered":"2023-11-16T18:23:39.503912+0000","last_clean":"2023-11-16T18:23:39.503912+0000","last_became_active":"2023-11-16T18:22:51.122006+0000","last_became_peered":"2023-11-16T18:22:51.122006+0000","last_unstale":"2023-11-16T18:23:39.503912+0000","last_undegraded":"2023-11-16T18:23:39.503912+0000","last_fullsized":"2023-11-16T18:23:39.503912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:27:30.406233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504268+0000","last_change":"2023-11-16T18:22:46.384837+0000","last_active":"2023-11-16T18:23:39.504268+0000","last_peered":"2023-11-16T18:23:39.504268+0000","last_clean":"2023-11-16T18:23:39.504268+0000","last_became_active":"2023-11-16T18:22:46.384541+0000","last_became_peered":"2023-11-16T18:22:46.384541+0000","last_unstale":"2023-11-16T18:23:39.504268+0000","last_undegraded":"2023-11-16T18:23:39.504268+0000","last_fullsized":"2023-11-16T18:23:39.504268+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:06:22.863895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502460+0000","last_change":"2023-11-16T18:22:46.839967+0000","last_active":"2023-11-16T18:23:39.502460+0000","last_peered":"2023-11-16T18:23:39.502460+0000","last_clean":"2023-11-16T18:23:39.502460+0000","last_became_active":"2023-11-16T18:22:46.839645+0000","last_became_peered":"2023-11-16T18:22:46.839645+0000","last_unstale":"2023-11-16T18:23:39.502460+0000","last_undegraded":"2023-11-16T18:23:39.502460+0000","last_fullsized":"2023-11-16T18:23:39.502460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:04:34.616063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507240+0000","last_change":"2023-11-16T18:22:51.126548+0000","last_active":"2023-11-16T18:23:39.507240+0000","last_peered":"2023-11-16T18:23:39.507240+0000","last_clean":"2023-11-16T18:23:39.507240+0000","last_became_active":"2023-11-16T18:22:51.126039+0000","last_became_peered":"2023-11-16T18:22:51.126039+0000","last_unstale":"2023-11-16T18:23:39.507240+0000","last_undegraded":"2023-11-16T18:23:39.507240+0000","last_fullsized":"2023-11-16T18:23:39.507240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:22:07.650645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.511343+0000","last_change":"2023-11-16T18:22:46.383947+0000","last_active":"2023-11-16T18:23:39.511343+0000","last_peered":"2023-11-16T18:23:39.511343+0000","last_clean":"2023-11-16T18:23:39.511343+0000","last_became_active":"2023-11-16T18:22:46.383661+0000","last_became_peered":"2023-11-16T18:22:46.383661+0000","last_unstale":"2023-11-16T18:23:39.511343+0000","last_undegraded":"2023-11-16T18:23:39.511343+0000","last_fullsized":"2023-11-16T18:23:39.511343+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:34.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503182+0000","last_change":"2023-11-16T18:22:46.836877+0000","last_active":"2023-11-16T18:23:39.503182+0000","last_peered":"2023-11-16T18:23:39.503182+0000","last_clean":"2023-11-16T18:23:39.503182+0000","last_became_active":"2023-11-16T18:22:46.836477+0000","last_became_peered":"2023-11-16T18:22:46.836477+0000","last_unstale":"2023-11-16T18:23:39.503182+0000","last_undegraded":"2023-11-16T18:23:39.503182+0000","last_fullsized":"2023-11-16T18:23:39.503182+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:38:49.681616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505171+0000","last_change":"2023-11-16T18:22:51.126930+0000","last_active":"2023-11-16T18:23:39.505171+0000","last_peered":"2023-11-16T18:23:39.505171+0000","last_clean":"2023-11-16T18:23:39.505171+0000","last_became_active":"2023-11-16T18:22:51.126409+0000","last_became_peered":"2023-11-16T18:22:51.126409+0000","last_unstale":"2023-11-16T18:23:39.505171+0000","last_undegraded":"2023-11-16T18:23:39.505171+0000","last_fullsized":"2023-11-16T18:23:39.505171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:31:50.012322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508533+0000","last_change":"2023-11-16T18:22:46.380820+0000","last_active":"2023-11-16T18:23:39.508533+0000","last_peered":"2023-11-16T18:23:39.508533+0000","last_clean":"2023-11-16T18:23:39.508533+0000","last_became_active":"2023-11-16T18:22:46.380446+0000","last_became_peered":"2023-11-16T18:22:46.380446+0000","last_unstale":"2023-11-16T18:23:39.508533+0000","last_undegraded":"2023-11-16T18:23:39.508533+0000","last_fullsized":"2023-11-16T18:23:39.508533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:20:07.923022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502955+0000","last_change":"2023-11-16T18:22:46.838153+0000","last_active":"2023-11-16T18:23:39.502955+0000","last_peered":"2023-11-16T18:23:39.502955+0000","last_clean":"2023-11-16T18:23:39.502955+0000","last_became_active":"2023-11-16T18:22:46.837836+0000","last_became_peered":"2023-11-16T18:22:46.837836+0000","last_unstale":"2023-11-16T18:23:39.502955+0000","last_undegraded":"2023-11-16T18:23:39.502955+0000","last_fullsized":"2023-11-16T18:23:39.502955+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:24:42.477381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507608+0000","last_change":"2023-11-16T18:22:51.119993+0000","last_active":"2023-11-16T18:23:39.507608+0000","last_peered":"2023-11-16T18:23:39.507608+0000","last_clean":"2023-11-16T18:23:39.507608+0000","last_became_active":"2023-11-16T18:22:51.119401+0000","last_became_peered":"2023-11-16T18:22:51.119401+0000","last_unstale":"2023-11-16T18:23:39.507608+0000","last_undegraded":"2023-11-16T18:23:39.507608+0000","last_fullsized":"2023-11-16T18:23:39.507608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:19:31.451143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585944+0000","last_change":"2023-11-16T18:22:46.383399+0000","last_active":"2023-11-16T18:23:39.585944+0000","last_peered":"2023-11-16T18:23:39.585944+0000","last_clean":"2023-11-16T18:23:39.585944+0000","last_became_active":"2023-11-16T18:22:46.382938+0000","last_became_peered":"2023-11-16T18:22:46.382938+0000","last_unstale":"2023-11-16T18:23:39.585944+0000","last_undegraded":"2023-11-16T18:23:39.585944+0000","last_fullsized":"2023-11-16T18:23:39.585944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:29.725086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503688+0000","last_change":"2023-11-16T18:22:46.836035+0000","last_active":"2023-11-16T18:23:39.503688+0000","last_peered":"2023-11-16T18:23:39.503688+0000","last_clean":"2023-11-16T18:23:39.503688+0000","last_became_active":"2023-11-16T18:22:46.835709+0000","last_became_peered":"2023-11-16T18:22:46.835709+0000","last_unstale":"2023-11-16T18:23:39.503688+0000","last_undegraded":"2023-11-16T18:23:39.503688+0000","last_fullsized":"2023-11-16T18:23:39.503688+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:32:57.558106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.848398+0000","last_change":"2023-11-16T18:22:46.820535+0000","last_active":"2023-11-16T18:24:37.848398+0000","last_peered":"2023-11-16T18:24:37.848398+0000","last_clean":"2023-11-16T18:24:37.848398+0000","last_became_active":"2023-11-16T18:22:46.820009+0000","last_became_peered":"2023-11-16T18:22:46.820009+0000","last_unstale":"2023-11-16T18:24:37.848398+0000","last_undegraded":"2023-11-16T18:24:37.848398+0000","last_fullsized":"2023-11-16T18:24:37.848398+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:50:39.196560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510673+0000","last_change":"2023-11-16T18:22:46.831762+0000","last_active":"2023-11-16T18:23:39.510673+0000","last_peered":"2023-11-16T18:23:39.510673+0000","last_clean":"2023-11-16T18:23:39.510673+0000","last_became_active":"2023-11-16T18:22:46.831429+0000","last_became_peered":"2023-11-16T18:22:46.831429+0000","last_unstale":"2023-11-16T18:23:39.510673+0000","last_undegraded":"2023-11-16T18:23:39.510673+0000","last_fullsized":"2023-11-16T18:23:39.510673+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:15:44.338906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510533+0000","last_change":"2023-11-16T18:22:46.838133+0000","last_active":"2023-11-16T18:23:39.510533+0000","last_peered":"2023-11-16T18:23:39.510533+0000","last_clean":"2023-11-16T18:23:39.510533+0000","last_became_active":"2023-11-16T18:22:46.837845+0000","last_became_peered":"2023-11-16T18:22:46.837845+0000","last_unstale":"2023-11-16T18:23:39.510533+0000","last_undegraded":"2023-11-16T18:23:39.510533+0000","last_fullsized":"2023-11-16T18:23:39.510533+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:13:06.102358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585337+0000","last_change":"2023-11-16T18:23:39.585337+0000","last_active":"2023-11-16T18:23:39.585337+0000","last_peered":"2023-11-16T18:23:39.585337+0000","last_clean":"2023-11-16T18:23:39.585337+0000","last_became_active":"2023-11-16T18:23:39.584874+0000","last_became_peered":"2023-11-16T18:23:39.584874+0000","last_unstale":"2023-11-16T18:23:39.585337+0000","last_undegraded":"2023-11-16T18:23:39.585337+0000","last_fullsized":"2023-11-16T18:23:39.585337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:24:54.928438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.842092+0000","last_change":"2023-11-16T18:22:46.823192+0000","last_active":"2023-11-16T18:24:37.842092+0000","last_peered":"2023-11-16T18:24:37.842092+0000","last_clean":"2023-11-16T18:24:37.842092+0000","last_became_active":"2023-11-16T18:22:46.822930+0000","last_became_peered":"2023-11-16T18:22:46.822930+0000","last_unstale":"2023-11-16T18:24:37.842092+0000","last_undegraded":"2023-11-16T18:24:37.842092+0000","last_fullsized":"2023-11-16T18:24:37.842092+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:49:48.831997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.848926+0000","last_change":"2023-11-16T18:22:46.825815+0000","last_active":"2023-11-16T18:24:37.848926+0000","last_peered":"2023-11-16T18:24:37.848926+0000","last_clean":"2023-11-16T18:24:37.848926+0000","last_became_active":"2023-11-16T18:22:46.825517+0000","last_became_peered":"2023-11-16T18:22:46.825517+0000","last_unstale":"2023-11-16T18:24:37.848926+0000","last_undegraded":"2023-11-16T18:24:37.848926+0000","last_fullsized":"2023-11-16T18:24:37.848926+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:32:02.468288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.511101+0000","last_change":"2023-11-16T18:22:46.826825+0000","last_active":"2023-11-16T18:23:39.511101+0000","last_peered":"2023-11-16T18:23:39.511101+0000","last_clean":"2023-11-16T18:23:39.511101+0000","last_became_active":"2023-11-16T18:22:46.826495+0000","last_became_peered":"2023-11-16T18:22:46.826495+0000","last_unstale":"2023-11-16T18:23:39.511101+0000","last_undegraded":"2023-11-16T18:23:39.511101+0000","last_fullsized":"2023-11-16T18:23:39.511101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:07:42.411162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.841815+0000","last_change":"2023-11-16T18:22:46.844090+0000","last_active":"2023-11-16T18:24:37.841815+0000","last_peered":"2023-11-16T18:24:37.841815+0000","last_clean":"2023-11-16T18:24:37.841815+0000","last_became_active":"2023-11-16T18:22:46.843641+0000","last_became_peered":"2023-11-16T18:22:46.843641+0000","last_unstale":"2023-11-16T18:24:37.841815+0000","last_undegraded":"2023-11-16T18:24:37.841815+0000","last_fullsized":"2023-11-16T18:24:37.841815+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:00:12.291257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508321+0000","last_change":"2023-11-16T18:22:46.838340+0000","last_active":"2023-11-16T18:23:39.508321+0000","last_peered":"2023-11-16T18:23:39.508321+0000","last_clean":"2023-11-16T18:23:39.508321+0000","last_became_active":"2023-11-16T18:22:46.838039+0000","last_became_peered":"2023-11-16T18:22:46.838039+0000","last_unstale":"2023-11-16T18:23:39.508321+0000","last_undegraded":"2023-11-16T18:23:39.508321+0000","last_fullsized":"2023-11-16T18:23:39.508321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:09:28.607361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.853480+0000","last_change":"2023-11-16T18:22:46.841739+0000","last_active":"2023-11-16T18:24:37.853480+0000","last_peered":"2023-11-16T18:24:37.853480+0000","last_clean":"2023-11-16T18:24:37.853480+0000","last_became_active":"2023-11-16T18:22:46.841450+0000","last_became_peered":"2023-11-16T18:22:46.841450+0000","last_unstale":"2023-11-16T18:24:37.853480+0000","last_undegraded":"2023-11-16T18:24:37.853480+0000","last_fullsized":"2023-11-16T18:24:37.853480+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:42:53.482129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508013+0000","last_change":"2023-11-16T18:22:46.839060+0000","last_active":"2023-11-16T18:23:39.508013+0000","last_peered":"2023-11-16T18:23:39.508013+0000","last_clean":"2023-11-16T18:23:39.508013+0000","last_became_active":"2023-11-16T18:22:46.838725+0000","last_became_peered":"2023-11-16T18:22:46.838725+0000","last_unstale":"2023-11-16T18:23:39.508013+0000","last_undegraded":"2023-11-16T18:23:39.508013+0000","last_fullsized":"2023-11-16T18:23:39.508013+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:33:35.374697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.853570+0000","last_change":"2023-11-16T18:22:46.833439+0000","last_active":"2023-11-16T18:24:37.853570+0000","last_peered":"2023-11-16T18:24:37.853570+0000","last_clean":"2023-11-16T18:24:37.853570+0000","last_became_active":"2023-11-16T18:22:46.833111+0000","last_became_peered":"2023-11-16T18:22:46.833111+0000","last_unstale":"2023-11-16T18:24:37.853570+0000","last_undegraded":"2023-11-16T18:24:37.853570+0000","last_fullsized":"2023-11-16T18:24:37.853570+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:10:36.066370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.848188+0000","last_change":"2023-11-16T18:22:46.825185+0000","last_active":"2023-11-16T18:24:37.848188+0000","last_peered":"2023-11-16T18:24:37.848188+0000","last_clean":"2023-11-16T18:24:37.848188+0000","last_became_active":"2023-11-16T18:22:46.824866+0000","last_became_peered":"2023-11-16T18:22:46.824866+0000","last_unstale":"2023-11-16T18:24:37.848188+0000","last_undegraded":"2023-11-16T18:24:37.848188+0000","last_fullsized":"2023-11-16T18:24:37.848188+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:46:05.263494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":173,"ondisk_log_size":173,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":215374,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65767,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":549,"num_read_kb":345,"num_write":159,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":61913,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":197301,"data_compressed":9627,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481827,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6140,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739012,"statfs":{"total":95995035648,"available":95988748288,"internally_reserved":0,"allocated":970752,"data_stored":485806,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.56000000000000005,"15min":0.57499999999999996},"min":{"1min":0.33800000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.66100000000000003,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.66500000000000004,"15min":0.69299999999999995},"min":{"1min":0.379,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":3.0590000000000002,"15min":3.0590000000000002},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63500000000000001,"15min":0.64100000000000001},"min":{"1min":0.41099999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.503,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81599999999999995,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.53900000000000003}]},{"osd":2,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.49299999999999999,"15min":0.48499999999999999},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84999999999999998,"5min":0.874,"15min":0.874},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.52600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.64200000000000002,"15min":0.64700000000000002},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67600000000000005,"15min":0.68000000000000005},"min":{"1min":0.52700000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.86499999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.58299999999999996}]},{"osd":5,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63100000000000001,"15min":0.63500000000000001},"min":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.91100000000000003,"5min":1.339,"15min":1.339},"last":0.629},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59699999999999998,"15min":0.60499999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84499999999999997,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.56399999999999995}]},{"osd":6,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.73599999999999999,"15min":0.754},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.1140000000000001,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.80700000000000005,"15min":0.82899999999999996},"min":{"1min":0.54200000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.196,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.746,"15min":0.75},"min":{"1min":0.53800000000000003,"5min":0.443,"15min":0.443},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.75,"15min":0.752},"min":{"1min":0.56100000000000005,"5min":0.502,"15min":0.502},"max":{"1min":0.97499999999999998,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.72799999999999998}]},{"osd":8,"last update":"Thu Nov 16 18:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.80300000000000005}]},{"osd":9,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.86599999999999999,"15min":0.88300000000000001},"min":{"1min":0.53100000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.161,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.73799999999999999,"15min":0.74099999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.74199999999999999}]},{"osd":10,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80400000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0209999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.755},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.83099999999999996,"15min":0.83699999999999997},"min":{"1min":0.59599999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.032,"5min":1.476,"15min":1.476},"last":0.83599999999999997}]}]},{"osd":10,"up_from":69,"seq":296352743462,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6596,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738556,"statfs":{"total":95995035648,"available":95988281344,"internally_reserved":0,"allocated":1110016,"data_stored":1138555,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.35299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.51800000000000002},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.53700000000000003}]},{"osd":2,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.82899999999999996,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.85699999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.73699999999999999}]},{"osd":3,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.659,"5min":1.659,"15min":1.659},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.72499999999999998,"15min":0.73299999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.629,"15min":0.63},"min":{"1min":0.44,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76800000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.36499999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.85099999999999998,"5min":0.996,"15min":0.996},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.99399999999999999,"5min":1.206,"15min":1.206},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.745},"min":{"1min":0.55200000000000005,"5min":0.502,"15min":0.502},"max":{"1min":0.97799999999999998,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.77400000000000002,"15min":0.77700000000000002},"min":{"1min":0.499,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.57399999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.60699999999999998}]},{"osd":7,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55000000000000004}]},{"osd":8,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.84199999999999997,"15min":0.84999999999999998},"min":{"1min":0.63100000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.98899999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0189999999999999,"5min":1.155,"15min":1.155},"last":0.71499999999999997}]},{"osd":9,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.65600000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.628},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.85699999999999998,"15min":0.85999999999999999},"min":{"1min":0.59799999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.75600000000000001}]},{"osd":11,"last update":"Thu Nov 16 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.63200000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.80400000000000005,"15min":0.80800000000000005},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":0.92400000000000004,"5min":1.034,"15min":1.034},"last":0.79700000000000004}]}]},{"osd":9,"up_from":64,"seq":274877906987,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6340,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":978944,"data_stored":485930,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.433,"15min":0.45500000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54900000000000004,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.371,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56499999999999995,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.56299999999999994}]},{"osd":1,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.64000000000000001,"15min":0.66800000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73599999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.60599999999999998,"15min":0.63},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.75700000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63400000000000001}]},{"osd":2,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.64000000000000001,"15min":0.65800000000000003},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82499999999999996,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61899999999999999,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.80100000000000005,"5min":1.024,"15min":1.024},"last":0.67100000000000004}]},{"osd":3,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.51900000000000002,"15min":0.54900000000000004},"min":{"1min":0.29399999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63200000000000001,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.505,"15min":0.55000000000000004},"min":{"1min":0.29799999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.58599999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.70199999999999996}]},{"osd":4,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.67200000000000004,"15min":0.71299999999999997},"min":{"1min":0.44,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.70799999999999996,"15min":0.747},"min":{"1min":0.377,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85299999999999998,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.73999999999999999}]},{"osd":5,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.69299999999999995,"15min":0.72099999999999997},"min":{"1min":0.45900000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.89700000000000002,"5min":1.357,"15min":1.357},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.72899999999999998,"15min":0.76500000000000001},"min":{"1min":0.51600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.88300000000000001,"5min":1.196,"15min":1.196},"last":0.77300000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.379},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.42699999999999999}]},{"osd":7,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.77800000000000002,"15min":0.83099999999999996},"min":{"1min":0.504,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.81899999999999995,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.73399999999999999,"15min":0.78600000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78900000000000003,"5min":1.157,"15min":1.157},"last":0.85099999999999998}]},{"osd":8,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.749,"15min":0.79800000000000004},"min":{"1min":0.49399999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.85299999999999998,"5min":1.248,"15min":1.248},"last":1.004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.79700000000000004,"15min":0.83999999999999997},"min":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.90500000000000003,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.92200000000000004}]},{"osd":10,"last update":"Thu Nov 16 18:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.76700000000000002,"15min":0.78900000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.879,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.752,"15min":0.76600000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.873,"5min":1.022,"15min":1.022},"last":0.82999999999999996}]},{"osd":11,"last update":"Thu Nov 16 18:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.874},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.88600000000000001}]}]},{"osd":8,"up_from":57,"seq":244813135920,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6460,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738692,"statfs":{"total":95995035648,"available":95988420608,"internally_reserved":0,"allocated":970752,"data_stored":485738,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,11,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.42399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":6.899},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.40799999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":5.3760000000000003}]},{"osd":1,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.59099999999999997,"15min":0.56799999999999995},"min":{"1min":0.45400000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.876,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":8.3309999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.626,"15min":0.59599999999999997},"min":{"1min":0.56100000000000005,"5min":0.316,"15min":0.316},"max":{"1min":0.84299999999999997,"5min":0.872,"15min":0.872},"last":0.97399999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.46100000000000002,"15min":0.44700000000000001},"min":{"1min":0.38500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71099999999999997,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.504,"15min":0.48199999999999998},"min":{"1min":0.36799999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72799999999999998,"5min":1.05,"15min":1.05},"last":0.998}]},{"osd":3,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91900000000000004,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":6.6440000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.628,"15min":0.628},"min":{"1min":0.48999999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0760000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.82799999999999996}]},{"osd":4,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.621,"15min":0.60499999999999998},"min":{"1min":0.46500000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":20.532},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.63900000000000001,"15min":0.60999999999999999},"min":{"1min":0.47299999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":14.776999999999999}]},{"osd":5,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.56100000000000005,"15min":0.52500000000000002},"min":{"1min":0.44400000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.55300000000000005,"15min":0.51500000000000001},"min":{"1min":0.53700000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.44900000000000001}]},{"osd":6,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67500000000000004,"15min":0.66300000000000003},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.71299999999999997,"15min":0.69699999999999995},"min":{"1min":0.64600000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.71999999999999997,"15min":0.68000000000000005},"min":{"1min":0.64000000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.143,"5min":1.266,"15min":1.266},"last":2.2400000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.69899999999999995,"15min":0.67300000000000004},"min":{"1min":0.624,"5min":0.434,"15min":0.434},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":2.3479999999999999}]},{"osd":9,"last update":"Thu Nov 16 18:24:25 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.72999999999999998,"15min":0.71399999999999997},"min":{"1min":0.56000000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0269999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.751,"15min":0.72799999999999998},"min":{"1min":0.55200000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":10.176}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.72499999999999998,"15min":0.71099999999999997},"min":{"1min":0.60499999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.036,"5min":1.276,"15min":1.276},"last":3.9489999999999998},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.76400000000000001,"15min":0.74399999999999999},"min":{"1min":0.74299999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":3.5649999999999999}]},{"osd":11,"last update":"Thu Nov 16 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":8.0890000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":4.6399999999999997}]}]},{"osd":7,"up_from":49,"seq":210453397556,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6580,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738572,"statfs":{"total":95995035648,"available":95988297728,"internally_reserved":0,"allocated":1028096,"data_stored":1082370,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.60699999999999998,"15min":0.59499999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81999999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81699999999999995,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.44500000000000001,"15min":0.42799999999999999},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61499999999999999,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.48099999999999998,"15min":0.46600000000000003},"min":{"1min":0.36299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71899999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63100000000000001,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65000000000000002,"15min":0.622},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.876,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85699999999999998,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.439},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.42299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.83599999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47699999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.57799999999999996,"15min":0.55200000000000005},"min":{"1min":0.43099999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.77900000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.372},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.372,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76300000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70099999999999996,"15min":0.67900000000000005},"min":{"1min":0.47099999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89500000000000002,"5min":1.075,"15min":1.075},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.75600000000000001,"15min":0.748},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.94999999999999996,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.55500000000000005}]},{"osd":6,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.71599999999999997,"15min":0.69999999999999996},"min":{"1min":0.52600000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.74399999999999999,"15min":0.73199999999999998},"min":{"1min":0.47799999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.45500000000000002}]},{"osd":8,"last update":"Thu Nov 16 18:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.76000000000000001,"15min":0.745},"min":{"1min":0.60599999999999998,"5min":0.433,"15min":0.433},"max":{"1min":0.98899999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.623},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.78800000000000003,"15min":0.78000000000000003},"min":{"1min":0.57399999999999995,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.018,"5min":1.107,"15min":1.107},"last":0.56299999999999994}]},{"osd":9,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79500000000000004,"15min":0.80400000000000005},"min":{"1min":0.57199999999999995,"5min":0.443,"15min":0.443},"max":{"1min":1.0549999999999999,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.83099999999999996,"15min":0.81699999999999995},"min":{"1min":0.59699999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0369999999999999,"5min":3.4119999999999999,"15min":3.4119999999999999},"last":0.504}]},{"osd":10,"last update":"Thu Nov 16 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.71399999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.99099999999999999,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81000000000000005,"15min":0.80900000000000005},"min":{"1min":0.61199999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.071,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.67400000000000004}]},{"osd":11,"last update":"Thu Nov 16 18:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.59699999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561081,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6384,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738768,"statfs":{"total":95995035648,"available":95988498432,"internally_reserved":0,"allocated":958464,"data_stored":481106,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.39800000000000002,"15min":0.36899999999999999},"min":{"1min":0.30199999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.50600000000000001,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.434},{"interface":"front","average":{"1min":0.374,"5min":0.40500000000000003,"15min":0.38100000000000001},"min":{"1min":0.315,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.44800000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63,"15min":0.59799999999999998},"min":{"1min":0.44,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77500000000000002,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.53300000000000003},"min":{"1min":0.40799999999999997,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.751,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.754}]},{"osd":2,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.41599999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71199999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.621,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.754,"5min":1.391,"15min":1.391},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52600000000000002,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61199999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.45500000000000002,"15min":0.437},"min":{"1min":0.33000000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79800000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.377}]},{"osd":4,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.61399999999999999,"15min":0.60699999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.76700000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.69299999999999995,"15min":0.66500000000000004},"min":{"1min":0.504,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.76100000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60299999999999998,"15min":0.58099999999999996},"min":{"1min":0.40300000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.68400000000000005,"5min":1.004,"15min":1.004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.64500000000000002,"15min":0.625},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82799999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75800000000000001,"15min":0.75},"min":{"1min":0.505,"5min":0.497,"15min":0.497},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.79100000000000004,"15min":0.77600000000000002},"min":{"1min":0.54600000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.83099999999999996}]},{"osd":8,"last update":"Thu Nov 16 18:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.80300000000000005,"15min":0.81899999999999995},"min":{"1min":0.497,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.95199999999999996,"5min":1.429,"15min":1.429},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.441,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89500000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.52200000000000002}]},{"osd":9,"last update":"Thu Nov 16 18:24:25 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.68000000000000005,"15min":0.73699999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70099999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.625},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.79900000000000004,"15min":0.85699999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.84699999999999998,"5min":1.298,"15min":1.298},"last":0.61099999999999999}]},{"osd":10,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.85999999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.70899999999999996}]},{"osd":11,"last update":"Thu Nov 16 18:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.65200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7476,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737676,"statfs":{"total":95995035648,"available":95987380224,"internally_reserved":0,"allocated":1028096,"data_stored":543180,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:33 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.52800000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.375},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53300000000000003,"15min":0.53800000000000003},"min":{"1min":0.34200000000000003,"5min":0.31,"15min":0.31},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.35199999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.60299999999999998,"15min":0.54800000000000004},"min":{"1min":0.45000000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.91700000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.61599999999999999,"15min":0.56100000000000005},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.754}]},{"osd":3,"last update":"Thu Nov 16 18:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59199999999999997,"15min":0.56899999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.59799999999999998,"15min":0.57399999999999995},"min":{"1min":0.47899999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.45200000000000001,"15min":0.40799999999999997},"min":{"1min":0.372,"5min":0.252,"15min":0.252},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.47399999999999998,"15min":0.41999999999999998},"min":{"1min":0.39300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80000000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Nov 16 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.624,"15min":0.60699999999999998},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93799999999999994,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.66700000000000004,"15min":0.63500000000000001},"min":{"1min":0.52800000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92600000000000005,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.71199999999999997}]},{"osd":6,"last update":"Thu Nov 16 18:24:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.629,"15min":0.61199999999999999},"min":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.63400000000000001,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.41199999999999998}]},{"osd":7,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.56899999999999995,"15min":0.53900000000000003},"min":{"1min":0.42399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.88200000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.63},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.622,"15min":0.58199999999999996},"min":{"1min":0.46500000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.91200000000000003,"5min":0.998,"15min":0.998},"last":0.52400000000000002}]},{"osd":8,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.69899999999999995,"15min":0.66900000000000004},"min":{"1min":0.48399999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0449999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.69999999999999996,"15min":0.67000000000000004},"min":{"1min":0.61799999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73799999999999999}]},{"osd":9,"last update":"Thu Nov 16 18:24:26 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.73299999999999998,"15min":0.72199999999999998},"min":{"1min":0.52300000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.749,"5min":0.755,"15min":0.749},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.58099999999999996}]},{"osd":10,"last update":"Thu Nov 16 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.77700000000000002,"15min":0.76900000000000002},"min":{"1min":0.57899999999999996,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.80900000000000005,"15min":0.79500000000000004},"min":{"1min":0.66000000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.65700000000000003}]},{"osd":11,"last update":"Thu Nov 16 18:23:48 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.79900000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7300,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737852,"statfs":{"total":95995035648,"available":95987560448,"internally_reserved":0,"allocated":978944,"data_stored":482221,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 18:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.55300000000000005,"15min":0.5},"min":{"1min":0.36799999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.923,"5min":2.923,"15min":2.923},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.56699999999999995,"15min":0.503},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.2109999999999999,"5min":3.2109999999999999,"15min":3.2109999999999999},"last":0.72599999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:23:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60999999999999999,"15min":0.71599999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.93799999999999994,"5min":5.3650000000000002,"15min":5.3650000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.627,"15min":0.67100000000000004},"min":{"1min":0.437,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.8929999999999998,"5min":3.4009999999999998,"15min":3.4009999999999998},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Nov 16 18:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.7810000000000001,"5min":3.7810000000000001,"15min":3.7810000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":3.294,"5min":3.294,"15min":3.294},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.60599999999999998,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.0249999999999999,"5min":3.0249999999999999,"15min":3.0249999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.61299999999999999,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":3.242,"5min":3.242,"15min":3.242},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Nov 16 18:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.48399999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.63200000000000001,"15min":0.59599999999999997},"min":{"1min":0.40699999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.851,"5min":2.851,"15min":2.851},"last":0.64500000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55500000000000005,"15min":0.51900000000000002},"min":{"1min":0.41699999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65700000000000003,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.32800000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.64100000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Nov 16 18:23:54 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.0499999999999998,"5min":3.0499999999999998,"15min":3.0499999999999998},"last":0.77500000000000002}]},{"osd":8,"last update":"Thu Nov 16 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.70299999999999996,"15min":0.72499999999999998},"min":{"1min":0.54200000000000004,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.036,"5min":1.052,"15min":1.052},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.73399999999999999,"15min":0.74299999999999999},"min":{"1min":0.54000000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.9470000000000001,"5min":2.9470000000000001,"15min":2.9470000000000001},"last":0.81699999999999995}]},{"osd":9,"last update":"Thu Nov 16 18:24:28 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.46000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.002,"5min":3.3180000000000001,"15min":3.3180000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.70299999999999996,"15min":0.73099999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.84599999999999997,"5min":2.79,"15min":2.79},"last":0.60599999999999998}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.78600000000000003,"15min":0.78100000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.0009999999999999,"5min":3.0009999999999999,"15min":3.0009999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.77000000000000002,"15min":0.75800000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":3.2679999999999998,"5min":3.2679999999999998,"15min":3.2679999999999998},"last":0.83599999999999997}]},{"osd":11,"last update":"Thu Nov 16 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.70599999999999996},"min":{"1min":0.58699999999999997,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.871},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.751,"15min":0.73499999999999999},"min":{"1min":0.59299999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.9740000000000002,"5min":2.9740000000000002,"15min":2.9740000000000002},"last":0.89200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7484,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737668,"statfs":{"total":95995035648,"available":95987372032,"internally_reserved":0,"allocated":970752,"data_stored":481744,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.54200000000000004,"15min":0.48199999999999998},"min":{"1min":0.34899999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.51800000000000002,"15min":0.46899999999999997},"min":{"1min":0.39500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.44700000000000001}]},{"osd":1,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.57399999999999995,"15min":0.52400000000000002},"min":{"1min":0.42699999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.073,"5min":1.47,"15min":1.47},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.59199999999999997,"15min":0.54800000000000004},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96599999999999997,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.71399999999999997}]},{"osd":3,"last update":"Thu Nov 16 18:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.54400000000000004},"min":{"1min":0.36499999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82999999999999996,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.61499999999999999,"15min":0.55300000000000005},"min":{"1min":0.47399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.89800000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.65200000000000002}]},{"osd":4,"last update":"Thu Nov 16 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.52700000000000002}]},{"osd":5,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.47599999999999998,"15min":0.42499999999999999},"min":{"1min":0.32700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.52500000000000002,"15min":0.48099999999999998},"min":{"1min":0.40300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.02,"5min":1.339,"15min":1.339},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Nov 16 18:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.64800000000000002,"15min":0.61899999999999999},"min":{"1min":0.45500000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.433},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.68799999999999994,"15min":0.63800000000000001},"min":{"1min":0.51300000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.67400000000000004}]},{"osd":7,"last update":"Thu Nov 16 18:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.63200000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:24:10 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.61599999999999999,"15min":0.58199999999999996},"min":{"1min":0.46899999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.622},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.60599999999999998,"15min":0.57799999999999996},"min":{"1min":0.45700000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.97099999999999997,"5min":1.377,"15min":1.377},"last":0.41199999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.54400000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92500000000000004,"5min":1.409,"15min":1.409},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.755,"15min":0.746},"min":{"1min":0.60799999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.59199999999999997}]},{"osd":10,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.73899999999999999,"15min":0.71899999999999997},"min":{"1min":0.60399999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.77200000000000002,"15min":0.76200000000000001},"min":{"1min":0.58799999999999997,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.242,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.66600000000000004}]},{"osd":11,"last update":"Thu Nov 16 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.64300000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6776,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738376,"statfs":{"total":95995035648,"available":95988097024,"internally_reserved":0,"allocated":966656,"data_stored":481027,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.378,"15min":0.36399999999999999},"min":{"1min":0.21099999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.47899999999999998,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.39400000000000002,"15min":0.38600000000000001},"min":{"1min":0.26500000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.59499999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.68500000000000005}]},{"osd":1,"last update":"Thu Nov 16 18:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55000000000000004,"15min":0.52400000000000002},"min":{"1min":0.35599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.77400000000000002,"5min":1.129,"15min":1.129},"last":0.495},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56399999999999995,"15min":0.53800000000000003},"min":{"1min":0.434,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.54800000000000004}]},{"osd":2,"last update":"Thu Nov 16 18:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59899999999999998,"15min":0.56899999999999995},"min":{"1min":0.42699999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83399999999999996,"5min":1.038,"15min":1.038},"last":0.628},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.58199999999999996,"15min":0.56999999999999995},"min":{"1min":0.371,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73399999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.53000000000000003}]},{"osd":4,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63500000000000001,"15min":0.60299999999999998},"min":{"1min":0.498,"5min":0.378,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":0.628},"min":{"1min":0.439,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79800000000000004,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Thu Nov 16 18:23:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.60299999999999998,"15min":0.61299999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70799999999999996,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.63100000000000001,"15min":0.623},"min":{"1min":0.45400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.81399999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.58699999999999997}]},{"osd":6,"last update":"Thu Nov 16 18:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.54200000000000004,"15min":0.49399999999999999},"min":{"1min":0.38600000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.63500000000000001,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.30199999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.72199999999999998,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.46300000000000002}]},{"osd":7,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.44600000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81399999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69599999999999995,"15min":0.68600000000000005},"min":{"1min":0.46700000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81499999999999995,"5min":1.087,"15min":1.087},"last":0.56699999999999995}]},{"osd":8,"last update":"Thu Nov 16 18:22:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.60799999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.752,"15min":0.81000000000000005},"min":{"1min":0.39000000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79700000000000004,"5min":1.135,"15min":1.135},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.70099999999999996,"15min":0.747},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.748,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.63600000000000001}]},{"osd":10,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.77000000000000002,"15min":0.78800000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.77600000000000002,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.82699999999999996,"15min":0.85199999999999998},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.81799999999999995,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.67000000000000004}]},{"osd":11,"last update":"Thu Nov 16 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.69899999999999995}]}]},{"osd":4,"up_from":32,"seq":137438953537,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6608,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738544,"statfs":{"total":95995035648,"available":95988269056,"internally_reserved":0,"allocated":991232,"data_stored":546980,"data_compressed":27217,"data_compressed_allocated":36864,"data_compressed_original":106496,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.504,"15min":0.48699999999999999},"min":{"1min":0.39900000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.752,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.48899999999999999,"15min":0.46400000000000002},"min":{"1min":0.41799999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.502}]},{"osd":1,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40600000000000003,"15min":0.377},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.56299999999999994,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.501},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.439,"15min":0.41299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.60699999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.52500000000000002}]},{"osd":2,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59199999999999997,"15min":0.56599999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85399999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.61699999999999999,"15min":0.57499999999999996},"min":{"1min":0.51500000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83599999999999997,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.68200000000000005}]},{"osd":3,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57199999999999995,"15min":0.54000000000000004},"min":{"1min":0.40100000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.03,"5min":0.64800000000000002,"15min":0.56200000000000006},"min":{"1min":0.40300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":10.048,"5min":10.048,"15min":10.048},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Nov 16 18:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.66400000000000003,"15min":0.61799999999999999},"min":{"1min":0.47599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.88300000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66500000000000004,"15min":0.63900000000000001},"min":{"1min":0.48399999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.99299999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.65400000000000003}]},{"osd":6,"last update":"Thu Nov 16 18:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.61599999999999999,"15min":0.59399999999999997},"min":{"1min":0.47799999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84899999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.68000000000000005,"15min":0.64400000000000002},"min":{"1min":0.53900000000000003,"5min":0.432,"15min":0.432},"max":{"1min":0.93200000000000005,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.86599999999999999}]},{"osd":7,"last update":"Thu Nov 16 18:23:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.626,"15min":0.57599999999999996},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95199999999999996,"5min":1.077,"15min":1.077},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56899999999999995,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76900000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63200000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.72499999999999998,"15min":0.72099999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98299999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73599999999999999,"15min":0.73199999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95499999999999996,"5min":1.109,"15min":1.109},"last":0.72699999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.751,"15min":0.76100000000000001},"min":{"1min":0.59099999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.92400000000000004,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80600000000000005,"15min":0.80900000000000005},"min":{"1min":0.65000000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.96999999999999997,"5min":1.117,"15min":1.117},"last":0.94499999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.63400000000000001,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":0.997,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78200000000000003,"15min":0.78500000000000003},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96799999999999997,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.83899999999999997}]},{"osd":11,"last update":"Thu Nov 16 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.88100000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790012,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6716,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738436,"statfs":{"total":95995035648,"available":95988158464,"internally_reserved":0,"allocated":1036288,"data_stored":1077616,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51900000000000002,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.69899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.59399999999999997,"15min":0.54000000000000004},"min":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76100000000000001,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.432}]},{"osd":1,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.54600000000000004,"15min":0.53300000000000003},"min":{"1min":0.41799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.83499999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.55100000000000005,"15min":0.52200000000000002},"min":{"1min":0.34599999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79000000000000004,"5min":0.873,"15min":0.873},"last":0.70599999999999996}]},{"osd":2,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.39400000000000002},"min":{"1min":0.312,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58599999999999997,"5min":0.626,"15min":0.626},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.41599999999999998,"15min":0.38600000000000001},"min":{"1min":0.316,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70399999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.64500000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55200000000000005,"15min":0.54200000000000004},"min":{"1min":0.42399999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.72499999999999998,"5min":0.88,"15min":0.88},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.58999999999999997,"15min":0.57199999999999995},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.85999999999999999,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.58499999999999996,"15min":0.56899999999999995},"min":{"1min":0.47099999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.872,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Nov 16 18:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.46899999999999997,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81100000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.46300000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77500000000000002,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.54600000000000004}]},{"osd":7,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.64000000000000001,"15min":0.629},"min":{"1min":0.48699999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.63800000000000001,"15min":0.625},"min":{"1min":0.44800000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.89600000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.75900000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.41399999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.45200000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.77600000000000002,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66400000000000003}]},{"osd":9,"last update":"Thu Nov 16 18:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66500000000000004,"15min":0.66100000000000003},"min":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.79400000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70299999999999996,"15min":0.70199999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.86899999999999999,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.77300000000000002}]},{"osd":10,"last update":"Thu Nov 16 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.72699999999999998,"15min":0.71599999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.70499999999999996,"15min":0.69299999999999995},"min":{"1min":0.53800000000000003,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93799999999999994,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.79100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34529,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":62141,"data_compressed":9627,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T18:24:39.108 INFO:tasks.fwd_scrub.ceph_manager:clean! 2023-11-16T18:24:39.108 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T18:24:39.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:38 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2962856659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:39.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:38 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/519250188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:39.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2023-11-16T18:24:39.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 8]}]: dispatch 2023-11-16T18:24:39.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2023-11-16T18:24:39.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:38 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2023-11-16T18:24:39.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.265+0000 7fd2b35a1700 1 -- 172.21.15.2:0/2321734565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2ac1002f0 msgr2=0x7fd2ac100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:39.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.265+0000 7fd2b35a1700 1 --2- 172.21.15.2:0/2321734565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2ac1002f0 0x7fd2ac100700 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009990 tx=0x7fd2a0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.265+0000 7fd2b35a1700 1 -- 172.21.15.2:0/2321734565 shutdown_connections 2023-11-16T18:24:39.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.265+0000 7fd2b35a1700 1 --2- 172.21.15.2:0/2321734565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2ac101e70 0x7fd2ac108730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.265+0000 7fd2b35a1700 1 --2- 172.21.15.2:0/2321734565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2ac1002f0 0x7fd2ac100700 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.265+0000 7fd2b35a1700 1 --2- 172.21.15.2:0/2321734565 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac1014e0 0x7fd2ac101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.265+0000 7fd2b35a1700 1 -- 172.21.15.2:0/2321734565 >> 172.21.15.2:0/2321734565 conn(0x7fd2ac0fb930 msgr2=0x7fd2ac0fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:39.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.265+0000 7fd2b35a1700 1 -- 172.21.15.2:0/2321734565 shutdown_connections 2023-11-16T18:24:39.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.265+0000 7fd2b35a1700 1 -- 172.21.15.2:0/2321734565 wait complete. 2023-11-16T18:24:39.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.265+0000 7fd2b35a1700 1 Processor -- start 2023-11-16T18:24:39.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.265+0000 7fd2b35a1700 1 -- start start 2023-11-16T18:24:39.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b35a1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac1002f0 0x7fd2ac1960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:39.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b35a1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2ac1014e0 0x7fd2ac1965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:39.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b35a1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2ac101e70 0x7fd2ac19d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:39.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b35a1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac19dc30 con 0x7fd2ac1014e0 2023-11-16T18:24:39.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b35a1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac19dd70 con 0x7fd2ac101e70 2023-11-16T18:24:39.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b35a1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac19df00 con 0x7fd2ac1002f0 2023-11-16T18:24:39.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b0b3c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2ac1014e0 0x7fd2ac1965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:39.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b0b3c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2ac1014e0 0x7fd2ac1965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49740/0 (socket says 172.21.15.2:49740) 2023-11-16T18:24:39.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b0b3c700 1 -- 172.21.15.2:0/3755805153 learned_addr learned my addr 172.21.15.2:0/3755805153 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:39.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b133d700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac1002f0 0x7fd2ac1960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:39.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b1b3e700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2ac101e70 0x7fd2ac19d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:39.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b0b3c700 1 -- 172.21.15.2:0/3755805153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac1002f0 msgr2=0x7fd2ac1960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:39.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b0b3c700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac1002f0 0x7fd2ac1960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b0b3c700 1 -- 172.21.15.2:0/3755805153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2ac101e70 msgr2=0x7fd2ac19d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:39.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b0b3c700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2ac101e70 0x7fd2ac19d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b0b3c700 1 -- 172.21.15.2:0/3755805153 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a00095f0 con 0x7fd2ac1014e0 2023-11-16T18:24:39.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b0b3c700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2ac1014e0 0x7fd2ac1965f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd2a800d980 tx=0x7fd2a800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:39.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd29e7fc700 1 -- 172.21.15.2:0/3755805153 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8004a30 con 0x7fd2ac1014e0 2023-11-16T18:24:39.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd2b35a1700 1 -- 172.21.15.2:0/3755805153 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ac10bce0 con 0x7fd2ac1014e0 2023-11-16T18:24:39.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd29e7fc700 1 -- 172.21.15.2:0/3755805153 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd2a80054e0 con 0x7fd2ac1014e0 2023-11-16T18:24:39.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.266+0000 7fd29e7fc700 1 -- 172.21.15.2:0/3755805153 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8009c90 con 0x7fd2ac1014e0 2023-11-16T18:24:39.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.267+0000 7fd2b35a1700 1 -- 172.21.15.2:0/3755805153 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ac10bb50 con 0x7fd2ac1014e0 2023-11-16T18:24:39.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.268+0000 7fd2b35a1700 1 -- 172.21.15.2:0/3755805153 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ac107830 con 0x7fd2ac1014e0 2023-11-16T18:24:39.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.269+0000 7fd29e7fc700 1 -- 172.21.15.2:0/3755805153 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fd2a8004c20 con 0x7fd2ac1014e0 2023-11-16T18:24:39.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.269+0000 7fd29e7fc700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd2980a1150 0x7fd2980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:39.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.269+0000 7fd2b133d700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd2980a1150 0x7fd2980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:39.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.269+0000 7fd29e7fc700 1 -- 172.21.15.2:0/3755805153 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fd2a80ccbb0 con 0x7fd2ac1014e0 2023-11-16T18:24:39.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.270+0000 7fd2b133d700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd2980a1150 0x7fd2980a3600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd2a000b5c0 tx=0x7fd2a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:39.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.273+0000 7fd29e7fc700 1 -- 172.21.15.2:0/3755805153 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd2a8095f40 con 0x7fd2ac1014e0 2023-11-16T18:24:39.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:38 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2962856659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:39.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:38 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/519250188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:39.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2023-11-16T18:24:39.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 8]}]: dispatch 2023-11-16T18:24:39.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2023-11-16T18:24:39.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:38 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2023-11-16T18:24:39.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.470+0000 7fd2b35a1700 1 -- 172.21.15.2:0/3755805153 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2ac061c40 con 0x7fd2980a1150 2023-11-16T18:24:39.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.485+0000 7fd29e7fc700 1 -- 172.21.15.2:0/3755805153 <== mgr.14152 v2:172.21.15.2:6800/3369431713 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426831 (secure 0 0 0) 0x7fd2ac061c40 con 0x7fd2980a1150 2023-11-16T18:24:39.492 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:39.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.489+0000 7fd2b35a1700 1 -- 172.21.15.2:0/3755805153 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd2980a1150 msgr2=0x7fd2980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:39.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.489+0000 7fd2b35a1700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd2980a1150 0x7fd2980a3600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd2a000b5c0 tx=0x7fd2a0005fb0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.489+0000 7fd2b35a1700 1 -- 172.21.15.2:0/3755805153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2ac1014e0 msgr2=0x7fd2ac1965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:39.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.489+0000 7fd2b35a1700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2ac1014e0 0x7fd2ac1965f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd2a800d980 tx=0x7fd2a800dc90 comp rx=0 tx=0).stop 2023-11-16T18:24:39.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.489+0000 7fd2b35a1700 1 -- 172.21.15.2:0/3755805153 shutdown_connections 2023-11-16T18:24:39.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.489+0000 7fd2b35a1700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fd2980a1150 0x7fd2980a3600 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.489+0000 7fd2b35a1700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac1002f0 0x7fd2ac1960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.489+0000 7fd2b35a1700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2ac101e70 0x7fd2ac19d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.489+0000 7fd2b35a1700 1 --2- 172.21.15.2:0/3755805153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2ac1014e0 0x7fd2ac1965f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.497 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.489+0000 7fd2b35a1700 1 -- 172.21.15.2:0/3755805153 >> 172.21.15.2:0/3755805153 conn(0x7fd2ac0fb930 msgr2=0x7fd2ac104970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:39.497 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.489+0000 7fd2b35a1700 1 -- 172.21.15.2:0/3755805153 shutdown_connections 2023-11-16T18:24:39.497 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.489+0000 7fd2b35a1700 1 -- 172.21.15.2:0/3755805153 wait complete. 2023-11-16T18:24:39.497 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-16T18:24:39.504 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2023-11-16T18:24:38.814341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656381,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":685,"num_read_kb":464,"num_write":342,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1242772,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":453,"ondisk_log_size":453,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80860,"kb_used_data":11708,"kb_used_omap":0,"kb_used_meta":69056,"kb_avail":1124860964,"statfs":{"total":1151940427776,"available":1151857627136,"internally_reserved":0,"allocated":11988992,"data_stored":7772273,"data_compressed":329214,"data_compressed_allocated":442368,"data_compressed_original":2416640,"omap_allocated":0,"internal_metadata":70713344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,11,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":636,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":37,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.984321"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.859995+0000","last_change":"2023-11-16T18:22:46.829814+0000","last_active":"2023-11-16T18:24:37.859995+0000","last_peered":"2023-11-16T18:24:37.859995+0000","last_clean":"2023-11-16T18:24:37.859995+0000","last_became_active":"2023-11-16T18:22:46.829407+0000","last_became_peered":"2023-11-16T18:22:46.829407+0000","last_unstale":"2023-11-16T18:24:37.859995+0000","last_undegraded":"2023-11-16T18:24:37.859995+0000","last_fullsized":"2023-11-16T18:24:37.859995+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:02:05.303420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505209+0000","last_change":"2023-11-16T18:22:46.818700+0000","last_active":"2023-11-16T18:23:39.505209+0000","last_peered":"2023-11-16T18:23:39.505209+0000","last_clean":"2023-11-16T18:23:39.505209+0000","last_became_active":"2023-11-16T18:22:46.818388+0000","last_became_peered":"2023-11-16T18:22:46.818388+0000","last_unstale":"2023-11-16T18:23:39.505209+0000","last_undegraded":"2023-11-16T18:23:39.505209+0000","last_fullsized":"2023-11-16T18:23:39.505209+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:05:25.862565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504017+0000","last_change":"2023-11-16T18:22:46.827732+0000","last_active":"2023-11-16T18:23:39.504017+0000","last_peered":"2023-11-16T18:23:39.504017+0000","last_clean":"2023-11-16T18:23:39.504017+0000","last_became_active":"2023-11-16T18:22:46.827328+0000","last_became_peered":"2023-11-16T18:22:46.827328+0000","last_unstale":"2023-11-16T18:23:39.504017+0000","last_undegraded":"2023-11-16T18:23:39.504017+0000","last_fullsized":"2023-11-16T18:23:39.504017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:44:05.572178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504758+0000","last_change":"2023-11-16T18:22:46.826837+0000","last_active":"2023-11-16T18:23:39.504758+0000","last_peered":"2023-11-16T18:23:39.504758+0000","last_clean":"2023-11-16T18:23:39.504758+0000","last_became_active":"2023-11-16T18:22:46.826532+0000","last_became_peered":"2023-11-16T18:22:46.826532+0000","last_unstale":"2023-11-16T18:23:39.504758+0000","last_undegraded":"2023-11-16T18:23:39.504758+0000","last_fullsized":"2023-11-16T18:23:39.504758+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:29:24.407020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.842311+0000","last_change":"2023-11-16T18:22:46.841164+0000","last_active":"2023-11-16T18:24:37.842311+0000","last_peered":"2023-11-16T18:24:37.842311+0000","last_clean":"2023-11-16T18:24:37.842311+0000","last_became_active":"2023-11-16T18:22:46.840855+0000","last_became_peered":"2023-11-16T18:22:46.840855+0000","last_unstale":"2023-11-16T18:24:37.842311+0000","last_undegraded":"2023-11-16T18:24:37.842311+0000","last_fullsized":"2023-11-16T18:24:37.842311+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:46.380315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507548+0000","last_change":"2023-11-16T18:22:46.822171+0000","last_active":"2023-11-16T18:23:39.507548+0000","last_peered":"2023-11-16T18:23:39.507548+0000","last_clean":"2023-11-16T18:23:39.507548+0000","last_became_active":"2023-11-16T18:22:46.821860+0000","last_became_peered":"2023-11-16T18:22:46.821860+0000","last_unstale":"2023-11-16T18:23:39.507548+0000","last_undegraded":"2023-11-16T18:23:39.507548+0000","last_fullsized":"2023-11-16T18:23:39.507548+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:28:18.589285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502347+0000","last_change":"2023-11-16T18:22:46.832463+0000","last_active":"2023-11-16T18:23:39.502347+0000","last_peered":"2023-11-16T18:23:39.502347+0000","last_clean":"2023-11-16T18:23:39.502347+0000","last_became_active":"2023-11-16T18:22:46.832141+0000","last_became_peered":"2023-11-16T18:22:46.832141+0000","last_unstale":"2023-11-16T18:23:39.502347+0000","last_undegraded":"2023-11-16T18:23:39.502347+0000","last_fullsized":"2023-11-16T18:23:39.502347+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:12:03.361742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.850564+0000","last_change":"2023-11-16T18:22:46.844239+0000","last_active":"2023-11-16T18:24:37.850564+0000","last_peered":"2023-11-16T18:24:37.850564+0000","last_clean":"2023-11-16T18:24:37.850564+0000","last_became_active":"2023-11-16T18:22:46.843906+0000","last_became_peered":"2023-11-16T18:22:46.843906+0000","last_unstale":"2023-11-16T18:24:37.850564+0000","last_undegraded":"2023-11-16T18:24:37.850564+0000","last_fullsized":"2023-11-16T18:24:37.850564+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:16:15.202242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508804+0000","last_change":"2023-11-16T18:22:46.834625+0000","last_active":"2023-11-16T18:23:39.508804+0000","last_peered":"2023-11-16T18:23:39.508804+0000","last_clean":"2023-11-16T18:23:39.508804+0000","last_became_active":"2023-11-16T18:22:46.834145+0000","last_became_peered":"2023-11-16T18:22:46.834145+0000","last_unstale":"2023-11-16T18:23:39.508804+0000","last_undegraded":"2023-11-16T18:23:39.508804+0000","last_fullsized":"2023-11-16T18:23:39.508804+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:25:16.196016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509061+0000","last_change":"2023-11-16T18:22:46.827569+0000","last_active":"2023-11-16T18:23:39.509061+0000","last_peered":"2023-11-16T18:23:39.509061+0000","last_clean":"2023-11-16T18:23:39.509061+0000","last_became_active":"2023-11-16T18:22:46.827234+0000","last_became_peered":"2023-11-16T18:22:46.827234+0000","last_unstale":"2023-11-16T18:23:39.509061+0000","last_undegraded":"2023-11-16T18:23:39.509061+0000","last_fullsized":"2023-11-16T18:23:39.509061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:38:30.584402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503061+0000","last_change":"2023-11-16T18:22:46.833701+0000","last_active":"2023-11-16T18:23:39.503061+0000","last_peered":"2023-11-16T18:23:39.503061+0000","last_clean":"2023-11-16T18:23:39.503061+0000","last_became_active":"2023-11-16T18:22:46.833333+0000","last_became_peered":"2023-11-16T18:22:46.833333+0000","last_unstale":"2023-11-16T18:23:39.503061+0000","last_undegraded":"2023-11-16T18:23:39.503061+0000","last_fullsized":"2023-11-16T18:23:39.503061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:31:52.429141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504924+0000","last_change":"2023-11-16T18:22:46.823222+0000","last_active":"2023-11-16T18:23:39.504924+0000","last_peered":"2023-11-16T18:23:39.504924+0000","last_clean":"2023-11-16T18:23:39.504924+0000","last_became_active":"2023-11-16T18:22:46.822224+0000","last_became_peered":"2023-11-16T18:22:46.822224+0000","last_unstale":"2023-11-16T18:23:39.504924+0000","last_undegraded":"2023-11-16T18:23:39.504924+0000","last_fullsized":"2023-11-16T18:23:39.504924+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:23:38.463744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.843256+0000","last_change":"2023-11-16T18:22:46.818631+0000","last_active":"2023-11-16T18:24:37.843256+0000","last_peered":"2023-11-16T18:24:37.843256+0000","last_clean":"2023-11-16T18:24:37.843256+0000","last_became_active":"2023-11-16T18:22:46.818305+0000","last_became_peered":"2023-11-16T18:22:46.818305+0000","last_unstale":"2023-11-16T18:24:37.843256+0000","last_undegraded":"2023-11-16T18:24:37.843256+0000","last_fullsized":"2023-11-16T18:24:37.843256+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:06:53.083409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503715+0000","last_change":"2023-11-16T18:22:46.825439+0000","last_active":"2023-11-16T18:23:39.503715+0000","last_peered":"2023-11-16T18:23:39.503715+0000","last_clean":"2023-11-16T18:23:39.503715+0000","last_became_active":"2023-11-16T18:22:46.825062+0000","last_became_peered":"2023-11-16T18:22:46.825062+0000","last_unstale":"2023-11-16T18:23:39.503715+0000","last_undegraded":"2023-11-16T18:23:39.503715+0000","last_fullsized":"2023-11-16T18:23:39.503715+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:32:03.819033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503546+0000","last_change":"2023-11-16T18:22:46.827730+0000","last_active":"2023-11-16T18:23:39.503546+0000","last_peered":"2023-11-16T18:23:39.503546+0000","last_clean":"2023-11-16T18:23:39.503546+0000","last_became_active":"2023-11-16T18:22:46.827416+0000","last_became_peered":"2023-11-16T18:22:46.827416+0000","last_unstale":"2023-11-16T18:23:39.503546+0000","last_undegraded":"2023-11-16T18:23:39.503546+0000","last_fullsized":"2023-11-16T18:23:39.503546+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:18:08.691944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507323+0000","last_change":"2023-11-16T18:22:46.834259+0000","last_active":"2023-11-16T18:23:39.507323+0000","last_peered":"2023-11-16T18:23:39.507323+0000","last_clean":"2023-11-16T18:23:39.507323+0000","last_became_active":"2023-11-16T18:22:46.833775+0000","last_became_peered":"2023-11-16T18:22:46.833775+0000","last_unstale":"2023-11-16T18:23:39.507323+0000","last_undegraded":"2023-11-16T18:23:39.507323+0000","last_fullsized":"2023-11-16T18:23:39.507323+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:02:17.762712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.860025+0000","last_change":"2023-11-16T18:22:46.829475+0000","last_active":"2023-11-16T18:24:37.860025+0000","last_peered":"2023-11-16T18:24:37.860025+0000","last_clean":"2023-11-16T18:24:37.860025+0000","last_became_active":"2023-11-16T18:22:46.829157+0000","last_became_peered":"2023-11-16T18:22:46.829157+0000","last_unstale":"2023-11-16T18:24:37.860025+0000","last_undegraded":"2023-11-16T18:24:37.860025+0000","last_fullsized":"2023-11-16T18:24:37.860025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:29.122041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501933+0000","last_change":"2023-11-16T18:22:46.821716+0000","last_active":"2023-11-16T18:23:39.501933+0000","last_peered":"2023-11-16T18:23:39.501933+0000","last_clean":"2023-11-16T18:23:39.501933+0000","last_became_active":"2023-11-16T18:22:46.821393+0000","last_became_peered":"2023-11-16T18:22:46.821393+0000","last_unstale":"2023-11-16T18:23:39.501933+0000","last_undegraded":"2023-11-16T18:23:39.501933+0000","last_fullsized":"2023-11-16T18:23:39.501933+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:10.453727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510952+0000","last_change":"2023-11-16T18:22:46.837433+0000","last_active":"2023-11-16T18:23:39.510952+0000","last_peered":"2023-11-16T18:23:39.510952+0000","last_clean":"2023-11-16T18:23:39.510952+0000","last_became_active":"2023-11-16T18:22:46.837143+0000","last_became_peered":"2023-11-16T18:22:46.837143+0000","last_unstale":"2023-11-16T18:23:39.510952+0000","last_undegraded":"2023-11-16T18:23:39.510952+0000","last_fullsized":"2023-11-16T18:23:39.510952+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:45:40.959535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510180+0000","last_change":"2023-11-16T18:22:51.123749+0000","last_active":"2023-11-16T18:23:39.510180+0000","last_peered":"2023-11-16T18:23:39.510180+0000","last_clean":"2023-11-16T18:23:39.510180+0000","last_became_active":"2023-11-16T18:22:51.123185+0000","last_became_peered":"2023-11-16T18:22:51.123185+0000","last_unstale":"2023-11-16T18:23:39.510180+0000","last_undegraded":"2023-11-16T18:23:39.510180+0000","last_fullsized":"2023-11-16T18:23:39.510180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:11:31.023936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501272+0000","last_change":"2023-11-16T18:22:46.385440+0000","last_active":"2023-11-16T18:23:39.501272+0000","last_peered":"2023-11-16T18:23:39.501272+0000","last_clean":"2023-11-16T18:23:39.501272+0000","last_became_active":"2023-11-16T18:22:46.385098+0000","last_became_peered":"2023-11-16T18:22:46.385098+0000","last_unstale":"2023-11-16T18:23:39.501272+0000","last_undegraded":"2023-11-16T18:23:39.501272+0000","last_fullsized":"2023-11-16T18:23:39.501272+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:08:49.547450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.846685+0000","last_change":"2023-11-16T18:22:46.842681+0000","last_active":"2023-11-16T18:24:37.846685+0000","last_peered":"2023-11-16T18:24:37.846685+0000","last_clean":"2023-11-16T18:24:37.846685+0000","last_became_active":"2023-11-16T18:22:46.842408+0000","last_became_peered":"2023-11-16T18:22:46.842408+0000","last_unstale":"2023-11-16T18:24:37.846685+0000","last_undegraded":"2023-11-16T18:24:37.846685+0000","last_fullsized":"2023-11-16T18:24:37.846685+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:37:57.050474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503905+0000","last_change":"2023-11-16T18:22:51.123450+0000","last_active":"2023-11-16T18:23:39.503905+0000","last_peered":"2023-11-16T18:23:39.503905+0000","last_clean":"2023-11-16T18:23:39.503905+0000","last_became_active":"2023-11-16T18:22:51.123002+0000","last_became_peered":"2023-11-16T18:22:51.123002+0000","last_unstale":"2023-11-16T18:23:39.503905+0000","last_undegraded":"2023-11-16T18:23:39.503905+0000","last_fullsized":"2023-11-16T18:23:39.503905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:48:43.003189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504416+0000","last_change":"2023-11-16T18:22:46.384768+0000","last_active":"2023-11-16T18:23:39.504416+0000","last_peered":"2023-11-16T18:23:39.504416+0000","last_clean":"2023-11-16T18:23:39.504416+0000","last_became_active":"2023-11-16T18:22:46.384246+0000","last_became_peered":"2023-11-16T18:22:46.384246+0000","last_unstale":"2023-11-16T18:23:39.504416+0000","last_undegraded":"2023-11-16T18:23:39.504416+0000","last_fullsized":"2023-11-16T18:23:39.504416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:20:51.259854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504317+0000","last_change":"2023-11-16T18:22:46.826069+0000","last_active":"2023-11-16T18:23:39.504317+0000","last_peered":"2023-11-16T18:23:39.504317+0000","last_clean":"2023-11-16T18:23:39.504317+0000","last_became_active":"2023-11-16T18:22:46.824676+0000","last_became_peered":"2023-11-16T18:22:46.824676+0000","last_unstale":"2023-11-16T18:23:39.504317+0000","last_undegraded":"2023-11-16T18:23:39.504317+0000","last_fullsized":"2023-11-16T18:23:39.504317+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:36:58.519456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504272+0000","last_change":"2023-11-16T18:22:51.127529+0000","last_active":"2023-11-16T18:23:39.504272+0000","last_peered":"2023-11-16T18:23:39.504272+0000","last_clean":"2023-11-16T18:23:39.504272+0000","last_became_active":"2023-11-16T18:22:51.126748+0000","last_became_peered":"2023-11-16T18:22:51.126748+0000","last_unstale":"2023-11-16T18:23:39.504272+0000","last_undegraded":"2023-11-16T18:23:39.504272+0000","last_fullsized":"2023-11-16T18:23:39.504272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:46:02.931821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505594+0000","last_change":"2023-11-16T18:22:46.385890+0000","last_active":"2023-11-16T18:23:39.505594+0000","last_peered":"2023-11-16T18:23:39.505594+0000","last_clean":"2023-11-16T18:23:39.505594+0000","last_became_active":"2023-11-16T18:22:46.385568+0000","last_became_peered":"2023-11-16T18:22:46.385568+0000","last_unstale":"2023-11-16T18:23:39.505594+0000","last_undegraded":"2023-11-16T18:23:39.505594+0000","last_fullsized":"2023-11-16T18:23:39.505594+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:45:23.088120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508492+0000","last_change":"2023-11-16T18:22:46.824796+0000","last_active":"2023-11-16T18:23:39.508492+0000","last_peered":"2023-11-16T18:23:39.508492+0000","last_clean":"2023-11-16T18:23:39.508492+0000","last_became_active":"2023-11-16T18:22:46.823895+0000","last_became_peered":"2023-11-16T18:22:46.823895+0000","last_unstale":"2023-11-16T18:23:39.508492+0000","last_undegraded":"2023-11-16T18:23:39.508492+0000","last_fullsized":"2023-11-16T18:23:39.508492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:57:28.297178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504555+0000","last_change":"2023-11-16T18:22:51.118714+0000","last_active":"2023-11-16T18:23:39.504555+0000","last_peered":"2023-11-16T18:23:39.504555+0000","last_clean":"2023-11-16T18:23:39.504555+0000","last_became_active":"2023-11-16T18:22:51.118216+0000","last_became_peered":"2023-11-16T18:22:51.118216+0000","last_unstale":"2023-11-16T18:23:39.504555+0000","last_undegraded":"2023-11-16T18:23:39.504555+0000","last_fullsized":"2023-11-16T18:23:39.504555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:27:57.339401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505119+0000","last_change":"2023-11-16T18:22:46.376431+0000","last_active":"2023-11-16T18:23:39.505119+0000","last_peered":"2023-11-16T18:23:39.505119+0000","last_clean":"2023-11-16T18:23:39.505119+0000","last_became_active":"2023-11-16T18:22:46.375914+0000","last_became_peered":"2023-11-16T18:22:46.375914+0000","last_unstale":"2023-11-16T18:23:39.505119+0000","last_undegraded":"2023-11-16T18:23:39.505119+0000","last_fullsized":"2023-11-16T18:23:39.505119+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:31:34.447838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508396+0000","last_change":"2023-11-16T18:22:46.825002+0000","last_active":"2023-11-16T18:23:39.508396+0000","last_peered":"2023-11-16T18:23:39.508396+0000","last_clean":"2023-11-16T18:23:39.508396+0000","last_became_active":"2023-11-16T18:22:46.824012+0000","last_became_peered":"2023-11-16T18:22:46.824012+0000","last_unstale":"2023-11-16T18:23:39.508396+0000","last_undegraded":"2023-11-16T18:23:39.508396+0000","last_fullsized":"2023-11-16T18:23:39.508396+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:04:41.816487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510910+0000","last_change":"2023-11-16T18:22:51.117623+0000","last_active":"2023-11-16T18:23:39.510910+0000","last_peered":"2023-11-16T18:23:39.510910+0000","last_clean":"2023-11-16T18:23:39.510910+0000","last_became_active":"2023-11-16T18:22:51.117183+0000","last_became_peered":"2023-11-16T18:22:51.117183+0000","last_unstale":"2023-11-16T18:23:39.510910+0000","last_undegraded":"2023-11-16T18:23:39.510910+0000","last_fullsized":"2023-11-16T18:23:39.510910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:16:00.063607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:35.602608+0000","last_change":"2023-11-16T18:23:39.664514+0000","last_active":"2023-11-16T18:24:35.602608+0000","last_peered":"2023-11-16T18:24:35.602608+0000","last_clean":"2023-11-16T18:24:35.602608+0000","last_became_active":"2023-11-16T18:23:39.511648+0000","last_became_peered":"2023-11-16T18:23:39.511648+0000","last_unstale":"2023-11-16T18:24:35.602608+0000","last_undegraded":"2023-11-16T18:24:35.602608+0000","last_fullsized":"2023-11-16T18:24:35.602608+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:08:22.914647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1282,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.835629+0000","last_change":"2023-11-16T18:22:46.820073+0000","last_active":"2023-11-16T18:24:37.835629+0000","last_peered":"2023-11-16T18:24:37.835629+0000","last_clean":"2023-11-16T18:24:37.835629+0000","last_became_active":"2023-11-16T18:22:46.819764+0000","last_became_peered":"2023-11-16T18:22:46.819764+0000","last_unstale":"2023-11-16T18:24:37.835629+0000","last_undegraded":"2023-11-16T18:24:37.835629+0000","last_fullsized":"2023-11-16T18:24:37.835629+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:58.766286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510964+0000","last_change":"2023-11-16T18:22:51.116619+0000","last_active":"2023-11-16T18:23:39.510964+0000","last_peered":"2023-11-16T18:23:39.510964+0000","last_clean":"2023-11-16T18:23:39.510964+0000","last_became_active":"2023-11-16T18:22:51.116264+0000","last_became_peered":"2023-11-16T18:22:51.116264+0000","last_unstale":"2023-11-16T18:23:39.510964+0000","last_undegraded":"2023-11-16T18:23:39.510964+0000","last_fullsized":"2023-11-16T18:23:39.510964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:32:46.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507700+0000","last_change":"2023-11-16T18:22:46.389590+0000","last_active":"2023-11-16T18:23:39.507700+0000","last_peered":"2023-11-16T18:23:39.507700+0000","last_clean":"2023-11-16T18:23:39.507700+0000","last_became_active":"2023-11-16T18:22:46.389354+0000","last_became_peered":"2023-11-16T18:22:46.389354+0000","last_unstale":"2023-11-16T18:23:39.507700+0000","last_undegraded":"2023-11-16T18:23:39.507700+0000","last_fullsized":"2023-11-16T18:23:39.507700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:58:58.086957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509671+0000","last_change":"2023-11-16T18:22:46.841106+0000","last_active":"2023-11-16T18:23:39.509671+0000","last_peered":"2023-11-16T18:23:39.509671+0000","last_clean":"2023-11-16T18:23:39.509671+0000","last_became_active":"2023-11-16T18:22:46.840876+0000","last_became_peered":"2023-11-16T18:22:46.840876+0000","last_unstale":"2023-11-16T18:23:39.509671+0000","last_undegraded":"2023-11-16T18:23:39.509671+0000","last_fullsized":"2023-11-16T18:23:39.509671+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:46:23.740026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506599+0000","last_change":"2023-11-16T18:22:51.125813+0000","last_active":"2023-11-16T18:23:39.506599+0000","last_peered":"2023-11-16T18:23:39.506599+0000","last_clean":"2023-11-16T18:23:39.506599+0000","last_became_active":"2023-11-16T18:22:51.125277+0000","last_became_peered":"2023-11-16T18:22:51.125277+0000","last_unstale":"2023-11-16T18:23:39.506599+0000","last_undegraded":"2023-11-16T18:23:39.506599+0000","last_fullsized":"2023-11-16T18:23:39.506599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:02:51.083984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.584572+0000","last_change":"2023-11-16T18:22:46.381208+0000","last_active":"2023-11-16T18:23:39.584572+0000","last_peered":"2023-11-16T18:23:39.584572+0000","last_clean":"2023-11-16T18:23:39.584572+0000","last_became_active":"2023-11-16T18:22:46.380351+0000","last_became_peered":"2023-11-16T18:22:46.380351+0000","last_unstale":"2023-11-16T18:23:39.584572+0000","last_undegraded":"2023-11-16T18:23:39.584572+0000","last_fullsized":"2023-11-16T18:23:39.584572+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:27:31.749114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.855370+0000","last_change":"2023-11-16T18:22:46.843043+0000","last_active":"2023-11-16T18:24:37.855370+0000","last_peered":"2023-11-16T18:24:37.855370+0000","last_clean":"2023-11-16T18:24:37.855370+0000","last_became_active":"2023-11-16T18:22:46.841522+0000","last_became_peered":"2023-11-16T18:22:46.841522+0000","last_unstale":"2023-11-16T18:24:37.855370+0000","last_undegraded":"2023-11-16T18:24:37.855370+0000","last_fullsized":"2023-11-16T18:24:37.855370+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:53:29.309989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505013+0000","last_change":"2023-11-16T18:22:51.127729+0000","last_active":"2023-11-16T18:23:39.505013+0000","last_peered":"2023-11-16T18:23:39.505013+0000","last_clean":"2023-11-16T18:23:39.505013+0000","last_became_active":"2023-11-16T18:22:51.127303+0000","last_became_peered":"2023-11-16T18:22:51.127303+0000","last_unstale":"2023-11-16T18:23:39.505013+0000","last_undegraded":"2023-11-16T18:23:39.505013+0000","last_fullsized":"2023-11-16T18:23:39.505013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:16:37.982862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508172+0000","last_change":"2023-11-16T18:22:46.377437+0000","last_active":"2023-11-16T18:23:39.508172+0000","last_peered":"2023-11-16T18:23:39.508172+0000","last_clean":"2023-11-16T18:23:39.508172+0000","last_became_active":"2023-11-16T18:22:46.377120+0000","last_became_peered":"2023-11-16T18:22:46.377120+0000","last_unstale":"2023-11-16T18:23:39.508172+0000","last_undegraded":"2023-11-16T18:23:39.508172+0000","last_fullsized":"2023-11-16T18:23:39.508172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:02:31.673863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500756+0000","last_change":"2023-11-16T18:22:46.837804+0000","last_active":"2023-11-16T18:23:39.500756+0000","last_peered":"2023-11-16T18:23:39.500756+0000","last_clean":"2023-11-16T18:23:39.500756+0000","last_became_active":"2023-11-16T18:22:46.837432+0000","last_became_peered":"2023-11-16T18:22:46.837432+0000","last_unstale":"2023-11-16T18:23:39.500756+0000","last_undegraded":"2023-11-16T18:23:39.500756+0000","last_fullsized":"2023-11-16T18:23:39.500756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:10:14.516300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503848+0000","last_change":"2023-11-16T18:22:51.127546+0000","last_active":"2023-11-16T18:23:39.503848+0000","last_peered":"2023-11-16T18:23:39.503848+0000","last_clean":"2023-11-16T18:23:39.503848+0000","last_became_active":"2023-11-16T18:22:51.127133+0000","last_became_peered":"2023-11-16T18:22:51.127133+0000","last_unstale":"2023-11-16T18:23:39.503848+0000","last_undegraded":"2023-11-16T18:23:39.503848+0000","last_fullsized":"2023-11-16T18:23:39.503848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:02:33.244659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":117,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:33.883048+0000","last_change":"2023-11-16T18:22:46.386309+0000","last_active":"2023-11-16T18:24:33.883048+0000","last_peered":"2023-11-16T18:24:33.883048+0000","last_clean":"2023-11-16T18:24:33.883048+0000","last_became_active":"2023-11-16T18:22:46.385960+0000","last_became_peered":"2023-11-16T18:22:46.385960+0000","last_unstale":"2023-11-16T18:24:33.883048+0000","last_undegraded":"2023-11-16T18:24:33.883048+0000","last_fullsized":"2023-11-16T18:24:33.883048+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:46:45.743666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508584+0000","last_change":"2023-11-16T18:22:46.827211+0000","last_active":"2023-11-16T18:23:39.508584+0000","last_peered":"2023-11-16T18:23:39.508584+0000","last_clean":"2023-11-16T18:23:39.508584+0000","last_became_active":"2023-11-16T18:22:46.826396+0000","last_became_peered":"2023-11-16T18:22:46.826396+0000","last_unstale":"2023-11-16T18:23:39.508584+0000","last_undegraded":"2023-11-16T18:23:39.508584+0000","last_fullsized":"2023-11-16T18:23:39.508584+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:58:36.202010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.513065+0000","last_change":"2023-11-16T18:23:39.512554+0000","last_active":"2023-11-16T18:23:39.513065+0000","last_peered":"2023-11-16T18:23:39.513065+0000","last_clean":"2023-11-16T18:23:39.513065+0000","last_became_active":"2023-11-16T18:23:39.512118+0000","last_became_peered":"2023-11-16T18:23:39.512118+0000","last_unstale":"2023-11-16T18:23:39.513065+0000","last_undegraded":"2023-11-16T18:23:39.513065+0000","last_fullsized":"2023-11-16T18:23:39.513065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:55:33.269875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503214+0000","last_change":"2023-11-16T18:22:46.384680+0000","last_active":"2023-11-16T18:23:39.503214+0000","last_peered":"2023-11-16T18:23:39.503214+0000","last_clean":"2023-11-16T18:23:39.503214+0000","last_became_active":"2023-11-16T18:22:46.384290+0000","last_became_peered":"2023-11-16T18:22:46.384290+0000","last_unstale":"2023-11-16T18:23:39.503214+0000","last_undegraded":"2023-11-16T18:23:39.503214+0000","last_fullsized":"2023-11-16T18:23:39.503214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:05:40.782246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504447+0000","last_change":"2023-11-16T18:22:46.834726+0000","last_active":"2023-11-16T18:23:39.504447+0000","last_peered":"2023-11-16T18:23:39.504447+0000","last_clean":"2023-11-16T18:23:39.504447+0000","last_became_active":"2023-11-16T18:22:46.834424+0000","last_became_peered":"2023-11-16T18:22:46.834424+0000","last_unstale":"2023-11-16T18:23:39.504447+0000","last_undegraded":"2023-11-16T18:23:39.504447+0000","last_fullsized":"2023-11-16T18:23:39.504447+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:52:11.738947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.513064+0000","last_change":"2023-11-16T18:23:39.513064+0000","last_active":"2023-11-16T18:23:39.513064+0000","last_peered":"2023-11-16T18:23:39.513064+0000","last_clean":"2023-11-16T18:23:39.513064+0000","last_became_active":"2023-11-16T18:23:39.512606+0000","last_became_peered":"2023-11-16T18:23:39.512606+0000","last_unstale":"2023-11-16T18:23:39.513064+0000","last_undegraded":"2023-11-16T18:23:39.513064+0000","last_fullsized":"2023-11-16T18:23:39.513064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:08:18.327854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508258+0000","last_change":"2023-11-16T18:22:46.380804+0000","last_active":"2023-11-16T18:23:39.508258+0000","last_peered":"2023-11-16T18:23:39.508258+0000","last_clean":"2023-11-16T18:23:39.508258+0000","last_became_active":"2023-11-16T18:22:46.380477+0000","last_became_peered":"2023-11-16T18:22:46.380477+0000","last_unstale":"2023-11-16T18:23:39.508258+0000","last_undegraded":"2023-11-16T18:23:39.508258+0000","last_fullsized":"2023-11-16T18:23:39.508258+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:17:45.128995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.586650+0000","last_change":"2023-11-16T18:23:39.586650+0000","last_active":"2023-11-16T18:23:39.586650+0000","last_peered":"2023-11-16T18:23:39.586650+0000","last_clean":"2023-11-16T18:23:39.586650+0000","last_became_active":"2023-11-16T18:23:39.586241+0000","last_became_peered":"2023-11-16T18:23:39.586241+0000","last_unstale":"2023-11-16T18:23:39.586650+0000","last_undegraded":"2023-11-16T18:23:39.586650+0000","last_fullsized":"2023-11-16T18:23:39.586650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:15:14.907282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500754+0000","last_change":"2023-11-16T18:22:51.119711+0000","last_active":"2023-11-16T18:23:39.500754+0000","last_peered":"2023-11-16T18:23:39.500754+0000","last_clean":"2023-11-16T18:23:39.500754+0000","last_became_active":"2023-11-16T18:22:51.119304+0000","last_became_peered":"2023-11-16T18:22:51.119304+0000","last_unstale":"2023-11-16T18:23:39.500754+0000","last_undegraded":"2023-11-16T18:23:39.500754+0000","last_fullsized":"2023-11-16T18:23:39.500754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:37:34.450684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":260,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:34.600234+0000","last_change":"2023-11-16T18:23:39.671486+0000","last_active":"2023-11-16T18:24:34.600234+0000","last_peered":"2023-11-16T18:24:34.600234+0000","last_clean":"2023-11-16T18:24:34.600234+0000","last_became_active":"2023-11-16T18:23:39.513413+0000","last_became_peered":"2023-11-16T18:23:39.513413+0000","last_unstale":"2023-11-16T18:24:34.600234+0000","last_undegraded":"2023-11-16T18:24:34.600234+0000","last_fullsized":"2023-11-16T18:24:34.600234+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:07:37.199058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61155,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":178,"num_read_kb":147,"num_write":34,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60837,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.845612+0000","last_change":"2023-11-16T18:22:46.833537+0000","last_active":"2023-11-16T18:24:37.845612+0000","last_peered":"2023-11-16T18:24:37.845612+0000","last_clean":"2023-11-16T18:24:37.845612+0000","last_became_active":"2023-11-16T18:22:46.833217+0000","last_became_peered":"2023-11-16T18:22:46.833217+0000","last_unstale":"2023-11-16T18:24:37.845612+0000","last_undegraded":"2023-11-16T18:24:37.845612+0000","last_fullsized":"2023-11-16T18:24:37.845612+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:54:32.987198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585712+0000","last_change":"2023-11-16T18:23:39.585247+0000","last_active":"2023-11-16T18:23:39.585712+0000","last_peered":"2023-11-16T18:23:39.585712+0000","last_clean":"2023-11-16T18:23:39.585712+0000","last_became_active":"2023-11-16T18:23:39.584802+0000","last_became_peered":"2023-11-16T18:23:39.584802+0000","last_unstale":"2023-11-16T18:23:39.585712+0000","last_undegraded":"2023-11-16T18:23:39.585712+0000","last_fullsized":"2023-11-16T18:23:39.585712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:45:43.860286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509671+0000","last_change":"2023-11-16T18:22:46.382812+0000","last_active":"2023-11-16T18:23:39.509671+0000","last_peered":"2023-11-16T18:23:39.509671+0000","last_clean":"2023-11-16T18:23:39.509671+0000","last_became_active":"2023-11-16T18:22:46.382499+0000","last_became_peered":"2023-11-16T18:22:46.382499+0000","last_unstale":"2023-11-16T18:23:39.509671+0000","last_undegraded":"2023-11-16T18:23:39.509671+0000","last_fullsized":"2023-11-16T18:23:39.509671+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:48:39.263999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.852503+0000","last_change":"2023-11-16T18:22:46.840917+0000","last_active":"2023-11-16T18:24:37.852503+0000","last_peered":"2023-11-16T18:24:37.852503+0000","last_clean":"2023-11-16T18:24:37.852503+0000","last_became_active":"2023-11-16T18:22:46.840579+0000","last_became_peered":"2023-11-16T18:22:46.840579+0000","last_unstale":"2023-11-16T18:24:37.852503+0000","last_undegraded":"2023-11-16T18:24:37.852503+0000","last_fullsized":"2023-11-16T18:24:37.852503+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:05:17.920819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506477+0000","last_change":"2023-11-16T18:22:51.117852+0000","last_active":"2023-11-16T18:23:39.506477+0000","last_peered":"2023-11-16T18:23:39.506477+0000","last_clean":"2023-11-16T18:23:39.506477+0000","last_became_active":"2023-11-16T18:22:51.117390+0000","last_became_peered":"2023-11-16T18:22:51.117390+0000","last_unstale":"2023-11-16T18:23:39.506477+0000","last_undegraded":"2023-11-16T18:23:39.506477+0000","last_fullsized":"2023-11-16T18:23:39.506477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:11:37.167591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508136+0000","last_change":"2023-11-16T18:22:46.385137+0000","last_active":"2023-11-16T18:23:39.508136+0000","last_peered":"2023-11-16T18:23:39.508136+0000","last_clean":"2023-11-16T18:23:39.508136+0000","last_became_active":"2023-11-16T18:22:46.384636+0000","last_became_peered":"2023-11-16T18:22:46.384636+0000","last_unstale":"2023-11-16T18:23:39.508136+0000","last_undegraded":"2023-11-16T18:23:39.508136+0000","last_fullsized":"2023-11-16T18:23:39.508136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:11:10.885960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502619+0000","last_change":"2023-11-16T18:22:46.834786+0000","last_active":"2023-11-16T18:23:39.502619+0000","last_peered":"2023-11-16T18:23:39.502619+0000","last_clean":"2023-11-16T18:23:39.502619+0000","last_became_active":"2023-11-16T18:22:46.834338+0000","last_became_peered":"2023-11-16T18:22:46.834338+0000","last_unstale":"2023-11-16T18:23:39.502619+0000","last_undegraded":"2023-11-16T18:23:39.502619+0000","last_fullsized":"2023-11-16T18:23:39.502619+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:52:01.046978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500921+0000","last_change":"2023-11-16T18:22:51.124667+0000","last_active":"2023-11-16T18:23:39.500921+0000","last_peered":"2023-11-16T18:23:39.500921+0000","last_clean":"2023-11-16T18:23:39.500921+0000","last_became_active":"2023-11-16T18:22:51.124211+0000","last_became_peered":"2023-11-16T18:22:51.124211+0000","last_unstale":"2023-11-16T18:23:39.500921+0000","last_undegraded":"2023-11-16T18:23:39.500921+0000","last_fullsized":"2023-11-16T18:23:39.500921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:22:02.360209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":121,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:34.054214+0000","last_change":"2023-11-16T18:22:46.380829+0000","last_active":"2023-11-16T18:24:34.054214+0000","last_peered":"2023-11-16T18:24:34.054214+0000","last_clean":"2023-11-16T18:24:34.054214+0000","last_became_active":"2023-11-16T18:22:46.380466+0000","last_became_peered":"2023-11-16T18:22:46.380466+0000","last_unstale":"2023-11-16T18:24:34.054214+0000","last_undegraded":"2023-11-16T18:24:34.054214+0000","last_fullsized":"2023-11-16T18:24:34.054214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:22:52.401291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502257+0000","last_change":"2023-11-16T18:22:46.835900+0000","last_active":"2023-11-16T18:23:39.502257+0000","last_peered":"2023-11-16T18:23:39.502257+0000","last_clean":"2023-11-16T18:23:39.502257+0000","last_became_active":"2023-11-16T18:22:46.835438+0000","last_became_peered":"2023-11-16T18:22:46.835438+0000","last_unstale":"2023-11-16T18:23:39.502257+0000","last_undegraded":"2023-11-16T18:23:39.502257+0000","last_fullsized":"2023-11-16T18:23:39.502257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:36:23.733440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585779+0000","last_change":"2023-11-16T18:22:51.120005+0000","last_active":"2023-11-16T18:23:39.585779+0000","last_peered":"2023-11-16T18:23:39.585779+0000","last_clean":"2023-11-16T18:23:39.585779+0000","last_became_active":"2023-11-16T18:22:51.119540+0000","last_became_peered":"2023-11-16T18:22:51.119540+0000","last_unstale":"2023-11-16T18:23:39.585779+0000","last_undegraded":"2023-11-16T18:23:39.585779+0000","last_fullsized":"2023-11-16T18:23:39.585779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:33:29.490015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:41.897854+0000","last_change":"2023-11-16T18:23:41.897338+0000","last_active":"2023-11-16T18:23:41.897854+0000","last_peered":"2023-11-16T18:23:41.897854+0000","last_clean":"2023-11-16T18:23:41.897854+0000","last_became_active":"2023-11-16T18:23:39.512018+0000","last_became_peered":"2023-11-16T18:23:39.512018+0000","last_unstale":"2023-11-16T18:23:41.897854+0000","last_undegraded":"2023-11-16T18:23:41.897854+0000","last_fullsized":"2023-11-16T18:23:41.897854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:53:22.707297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507902+0000","last_change":"2023-11-16T18:22:46.823838+0000","last_active":"2023-11-16T18:23:39.507902+0000","last_peered":"2023-11-16T18:23:39.507902+0000","last_clean":"2023-11-16T18:23:39.507902+0000","last_became_active":"2023-11-16T18:22:46.823545+0000","last_became_peered":"2023-11-16T18:22:46.823545+0000","last_unstale":"2023-11-16T18:23:39.507902+0000","last_undegraded":"2023-11-16T18:23:39.507902+0000","last_fullsized":"2023-11-16T18:23:39.507902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:11:32.214804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507865+0000","last_change":"2023-11-16T18:22:51.127371+0000","last_active":"2023-11-16T18:23:39.507865+0000","last_peered":"2023-11-16T18:23:39.507865+0000","last_clean":"2023-11-16T18:23:39.507865+0000","last_became_active":"2023-11-16T18:22:51.126951+0000","last_became_peered":"2023-11-16T18:22:51.126951+0000","last_unstale":"2023-11-16T18:23:39.507865+0000","last_undegraded":"2023-11-16T18:23:39.507865+0000","last_fullsized":"2023-11-16T18:23:39.507865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:55.418916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504323+0000","last_change":"2023-11-16T18:22:46.376561+0000","last_active":"2023-11-16T18:23:39.504323+0000","last_peered":"2023-11-16T18:23:39.504323+0000","last_clean":"2023-11-16T18:23:39.504323+0000","last_became_active":"2023-11-16T18:22:46.375426+0000","last_became_peered":"2023-11-16T18:22:46.375426+0000","last_unstale":"2023-11-16T18:23:39.504323+0000","last_undegraded":"2023-11-16T18:23:39.504323+0000","last_fullsized":"2023-11-16T18:23:39.504323+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:29:00.383105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.855455+0000","last_change":"2023-11-16T18:22:46.826657+0000","last_active":"2023-11-16T18:24:37.855455+0000","last_peered":"2023-11-16T18:24:37.855455+0000","last_clean":"2023-11-16T18:24:37.855455+0000","last_became_active":"2023-11-16T18:22:46.826126+0000","last_became_peered":"2023-11-16T18:22:46.826126+0000","last_unstale":"2023-11-16T18:24:37.855455+0000","last_undegraded":"2023-11-16T18:24:37.855455+0000","last_fullsized":"2023-11-16T18:24:37.855455+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:13:34.794628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504904+0000","last_change":"2023-11-16T18:22:51.127949+0000","last_active":"2023-11-16T18:23:39.504904+0000","last_peered":"2023-11-16T18:23:39.504904+0000","last_clean":"2023-11-16T18:23:39.504904+0000","last_became_active":"2023-11-16T18:22:51.127388+0000","last_became_peered":"2023-11-16T18:22:51.127388+0000","last_unstale":"2023-11-16T18:23:39.504904+0000","last_undegraded":"2023-11-16T18:23:39.504904+0000","last_fullsized":"2023-11-16T18:23:39.504904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:37:30.745978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.500985+0000","last_change":"2023-11-16T18:22:46.378568+0000","last_active":"2023-11-16T18:23:39.500985+0000","last_peered":"2023-11-16T18:23:39.500985+0000","last_clean":"2023-11-16T18:23:39.500985+0000","last_became_active":"2023-11-16T18:22:46.378018+0000","last_became_peered":"2023-11-16T18:22:46.378018+0000","last_unstale":"2023-11-16T18:23:39.500985+0000","last_undegraded":"2023-11-16T18:23:39.500985+0000","last_fullsized":"2023-11-16T18:23:39.500985+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:05:59.328232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509189+0000","last_change":"2023-11-16T18:22:46.841129+0000","last_active":"2023-11-16T18:23:39.509189+0000","last_peered":"2023-11-16T18:23:39.509189+0000","last_clean":"2023-11-16T18:23:39.509189+0000","last_became_active":"2023-11-16T18:22:46.840780+0000","last_became_peered":"2023-11-16T18:22:46.840780+0000","last_unstale":"2023-11-16T18:23:39.509189+0000","last_undegraded":"2023-11-16T18:23:39.509189+0000","last_fullsized":"2023-11-16T18:23:39.509189+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:09:39.912158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585594+0000","last_change":"2023-11-16T18:22:51.126221+0000","last_active":"2023-11-16T18:23:39.585594+0000","last_peered":"2023-11-16T18:23:39.585594+0000","last_clean":"2023-11-16T18:23:39.585594+0000","last_became_active":"2023-11-16T18:22:51.125770+0000","last_became_peered":"2023-11-16T18:22:51.125770+0000","last_unstale":"2023-11-16T18:23:39.585594+0000","last_undegraded":"2023-11-16T18:23:39.585594+0000","last_fullsized":"2023-11-16T18:23:39.585594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:35:06.867124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502402+0000","last_change":"2023-11-16T18:22:46.384872+0000","last_active":"2023-11-16T18:23:39.502402+0000","last_peered":"2023-11-16T18:23:39.502402+0000","last_clean":"2023-11-16T18:23:39.502402+0000","last_became_active":"2023-11-16T18:22:46.384499+0000","last_became_peered":"2023-11-16T18:22:46.384499+0000","last_unstale":"2023-11-16T18:23:39.502402+0000","last_undegraded":"2023-11-16T18:23:39.502402+0000","last_fullsized":"2023-11-16T18:23:39.502402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:26:53.535471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507989+0000","last_change":"2023-11-16T18:22:46.822574+0000","last_active":"2023-11-16T18:23:39.507989+0000","last_peered":"2023-11-16T18:23:39.507989+0000","last_clean":"2023-11-16T18:23:39.507989+0000","last_became_active":"2023-11-16T18:22:46.822230+0000","last_became_peered":"2023-11-16T18:22:46.822230+0000","last_unstale":"2023-11-16T18:23:39.507989+0000","last_undegraded":"2023-11-16T18:23:39.507989+0000","last_fullsized":"2023-11-16T18:23:39.507989+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:16:44.677701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585706+0000","last_change":"2023-11-16T18:22:51.117838+0000","last_active":"2023-11-16T18:23:39.585706+0000","last_peered":"2023-11-16T18:23:39.585706+0000","last_clean":"2023-11-16T18:23:39.585706+0000","last_became_active":"2023-11-16T18:22:51.117275+0000","last_became_peered":"2023-11-16T18:22:51.117275+0000","last_unstale":"2023-11-16T18:23:39.585706+0000","last_undegraded":"2023-11-16T18:23:39.585706+0000","last_fullsized":"2023-11-16T18:23:39.585706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:42:14.482945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508453+0000","last_change":"2023-11-16T18:22:46.827499+0000","last_active":"2023-11-16T18:23:39.508453+0000","last_peered":"2023-11-16T18:23:39.508453+0000","last_clean":"2023-11-16T18:23:39.508453+0000","last_became_active":"2023-11-16T18:22:46.827166+0000","last_became_peered":"2023-11-16T18:22:46.827166+0000","last_unstale":"2023-11-16T18:23:39.508453+0000","last_undegraded":"2023-11-16T18:23:39.508453+0000","last_fullsized":"2023-11-16T18:23:39.508453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:23:43.219136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":122,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.807927+0000","last_change":"2023-11-16T18:22:46.381397+0000","last_active":"2023-11-16T18:24:37.807927+0000","last_peered":"2023-11-16T18:24:37.807927+0000","last_clean":"2023-11-16T18:24:37.807927+0000","last_became_active":"2023-11-16T18:22:46.380939+0000","last_became_peered":"2023-11-16T18:22:46.380939+0000","last_unstale":"2023-11-16T18:24:37.807927+0000","last_undegraded":"2023-11-16T18:24:37.807927+0000","last_fullsized":"2023-11-16T18:24:37.807927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:44:30.574771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":91,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502915+0000","last_change":"2023-11-16T18:22:51.125044+0000","last_active":"2023-11-16T18:23:39.502915+0000","last_peered":"2023-11-16T18:23:39.502915+0000","last_clean":"2023-11-16T18:23:39.502915+0000","last_became_active":"2023-11-16T18:22:51.124422+0000","last_became_peered":"2023-11-16T18:22:51.124422+0000","last_unstale":"2023-11-16T18:23:39.502915+0000","last_undegraded":"2023-11-16T18:23:39.502915+0000","last_fullsized":"2023-11-16T18:23:39.502915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:11.651599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.858524+0000","last_change":"2023-11-16T18:22:46.841500+0000","last_active":"2023-11-16T18:24:37.858524+0000","last_peered":"2023-11-16T18:24:37.858524+0000","last_clean":"2023-11-16T18:24:37.858524+0000","last_became_active":"2023-11-16T18:22:46.841186+0000","last_became_peered":"2023-11-16T18:22:46.841186+0000","last_unstale":"2023-11-16T18:24:37.858524+0000","last_undegraded":"2023-11-16T18:24:37.858524+0000","last_fullsized":"2023-11-16T18:24:37.858524+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:38:25.407837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504887+0000","last_change":"2023-11-16T18:22:46.380988+0000","last_active":"2023-11-16T18:23:39.504887+0000","last_peered":"2023-11-16T18:23:39.504887+0000","last_clean":"2023-11-16T18:23:39.504887+0000","last_became_active":"2023-11-16T18:22:46.380616+0000","last_became_peered":"2023-11-16T18:22:46.380616+0000","last_unstale":"2023-11-16T18:23:39.504887+0000","last_undegraded":"2023-11-16T18:23:39.504887+0000","last_fullsized":"2023-11-16T18:23:39.504887+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:24:26.911536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502037+0000","last_change":"2023-11-16T18:21:44.282706+0000","last_active":"2023-11-16T18:23:39.502037+0000","last_peered":"2023-11-16T18:23:39.502037+0000","last_clean":"2023-11-16T18:23:39.502037+0000","last_became_active":"2023-11-16T18:21:44.282201+0000","last_became_peered":"2023-11-16T18:21:44.282201+0000","last_unstale":"2023-11-16T18:23:39.502037+0000","last_undegraded":"2023-11-16T18:23:39.502037+0000","last_fullsized":"2023-11-16T18:23:39.502037+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:18:46.711191+0000","last_clean_scrub_stamp":"2023-11-16T18:18:46.711191+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:41:17.702365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508868+0000","last_change":"2023-11-16T18:22:51.123051+0000","last_active":"2023-11-16T18:23:39.508868+0000","last_peered":"2023-11-16T18:23:39.508868+0000","last_clean":"2023-11-16T18:23:39.508868+0000","last_became_active":"2023-11-16T18:22:51.122635+0000","last_became_peered":"2023-11-16T18:22:51.122635+0000","last_unstale":"2023-11-16T18:23:39.508868+0000","last_undegraded":"2023-11-16T18:23:39.508868+0000","last_fullsized":"2023-11-16T18:23:39.508868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:59:09.977278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.852848+0000","last_change":"2023-11-16T18:22:46.822625+0000","last_active":"2023-11-16T18:24:37.852848+0000","last_peered":"2023-11-16T18:24:37.852848+0000","last_clean":"2023-11-16T18:24:37.852848+0000","last_became_active":"2023-11-16T18:22:46.822296+0000","last_became_peered":"2023-11-16T18:22:46.822296+0000","last_unstale":"2023-11-16T18:24:37.852848+0000","last_undegraded":"2023-11-16T18:24:37.852848+0000","last_fullsized":"2023-11-16T18:24:37.852848+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:43:40.832756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502155+0000","last_change":"2023-11-16T18:22:46.380445+0000","last_active":"2023-11-16T18:23:39.502155+0000","last_peered":"2023-11-16T18:23:39.502155+0000","last_clean":"2023-11-16T18:23:39.502155+0000","last_became_active":"2023-11-16T18:22:46.380113+0000","last_became_peered":"2023-11-16T18:22:46.380113+0000","last_unstale":"2023-11-16T18:23:39.502155+0000","last_undegraded":"2023-11-16T18:23:39.502155+0000","last_fullsized":"2023-11-16T18:23:39.502155+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:17:48.769928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504943+0000","last_change":"2023-11-16T18:22:51.118457+0000","last_active":"2023-11-16T18:23:39.504943+0000","last_peered":"2023-11-16T18:23:39.504943+0000","last_clean":"2023-11-16T18:23:39.504943+0000","last_became_active":"2023-11-16T18:22:51.118013+0000","last_became_peered":"2023-11-16T18:22:51.118013+0000","last_unstale":"2023-11-16T18:23:39.504943+0000","last_undegraded":"2023-11-16T18:23:39.504943+0000","last_fullsized":"2023-11-16T18:23:39.504943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:11.432801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.506267+0000","last_change":"2023-11-16T18:22:46.384799+0000","last_active":"2023-11-16T18:23:39.506267+0000","last_peered":"2023-11-16T18:23:39.506267+0000","last_clean":"2023-11-16T18:23:39.506267+0000","last_became_active":"2023-11-16T18:22:46.384482+0000","last_became_peered":"2023-11-16T18:22:46.384482+0000","last_unstale":"2023-11-16T18:23:39.506267+0000","last_undegraded":"2023-11-16T18:23:39.506267+0000","last_fullsized":"2023-11-16T18:23:39.506267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:42:48.220989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.839008+0000","last_change":"2023-11-16T18:22:46.829541+0000","last_active":"2023-11-16T18:24:37.839008+0000","last_peered":"2023-11-16T18:24:37.839008+0000","last_clean":"2023-11-16T18:24:37.839008+0000","last_became_active":"2023-11-16T18:22:46.829006+0000","last_became_peered":"2023-11-16T18:22:46.829006+0000","last_unstale":"2023-11-16T18:24:37.839008+0000","last_undegraded":"2023-11-16T18:24:37.839008+0000","last_fullsized":"2023-11-16T18:24:37.839008+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:13:02.956336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504640+0000","last_change":"2023-11-16T18:22:51.122751+0000","last_active":"2023-11-16T18:23:39.504640+0000","last_peered":"2023-11-16T18:23:39.504640+0000","last_clean":"2023-11-16T18:23:39.504640+0000","last_became_active":"2023-11-16T18:22:51.122423+0000","last_became_peered":"2023-11-16T18:22:51.122423+0000","last_unstale":"2023-11-16T18:23:39.504640+0000","last_undegraded":"2023-11-16T18:23:39.504640+0000","last_fullsized":"2023-11-16T18:23:39.504640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:27:49.338555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'5","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.516672+0000","last_change":"2023-11-16T18:23:39.516672+0000","last_active":"2023-11-16T18:23:39.516672+0000","last_peered":"2023-11-16T18:23:39.516672+0000","last_clean":"2023-11-16T18:23:39.516672+0000","last_became_active":"2023-11-16T18:23:39.509087+0000","last_became_peered":"2023-11-16T18:23:39.509087+0000","last_unstale":"2023-11-16T18:23:39.516672+0000","last_undegraded":"2023-11-16T18:23:39.516672+0000","last_fullsized":"2023-11-16T18:23:39.516672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:57:28.479767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.850681+0000","last_change":"2023-11-16T18:22:46.843802+0000","last_active":"2023-11-16T18:24:37.850681+0000","last_peered":"2023-11-16T18:24:37.850681+0000","last_clean":"2023-11-16T18:24:37.850681+0000","last_became_active":"2023-11-16T18:22:46.842210+0000","last_became_peered":"2023-11-16T18:22:46.842210+0000","last_unstale":"2023-11-16T18:24:37.850681+0000","last_undegraded":"2023-11-16T18:24:37.850681+0000","last_fullsized":"2023-11-16T18:24:37.850681+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:20:02.893140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.509951+0000","last_change":"2023-11-16T18:22:51.124632+0000","last_active":"2023-11-16T18:23:39.509951+0000","last_peered":"2023-11-16T18:23:39.509951+0000","last_clean":"2023-11-16T18:23:39.509951+0000","last_became_active":"2023-11-16T18:22:51.124220+0000","last_became_peered":"2023-11-16T18:22:51.124220+0000","last_unstale":"2023-11-16T18:23:39.509951+0000","last_undegraded":"2023-11-16T18:23:39.509951+0000","last_fullsized":"2023-11-16T18:23:39.509951+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:51:57.981627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505140+0000","last_change":"2023-11-16T18:22:46.381456+0000","last_active":"2023-11-16T18:23:39.505140+0000","last_peered":"2023-11-16T18:23:39.505140+0000","last_clean":"2023-11-16T18:23:39.505140+0000","last_became_active":"2023-11-16T18:22:46.381137+0000","last_became_peered":"2023-11-16T18:22:46.381137+0000","last_unstale":"2023-11-16T18:23:39.505140+0000","last_undegraded":"2023-11-16T18:23:39.505140+0000","last_fullsized":"2023-11-16T18:23:39.505140+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:34:36.759190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505363+0000","last_change":"2023-11-16T18:22:46.838249+0000","last_active":"2023-11-16T18:23:39.505363+0000","last_peered":"2023-11-16T18:23:39.505363+0000","last_clean":"2023-11-16T18:23:39.505363+0000","last_became_active":"2023-11-16T18:22:46.837925+0000","last_became_peered":"2023-11-16T18:22:46.837925+0000","last_unstale":"2023-11-16T18:23:39.505363+0000","last_undegraded":"2023-11-16T18:23:39.505363+0000","last_fullsized":"2023-11-16T18:23:39.505363+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:02:17.287183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504877+0000","last_change":"2023-11-16T18:22:51.123687+0000","last_active":"2023-11-16T18:23:39.504877+0000","last_peered":"2023-11-16T18:23:39.504877+0000","last_clean":"2023-11-16T18:23:39.504877+0000","last_became_active":"2023-11-16T18:22:51.123209+0000","last_became_peered":"2023-11-16T18:22:51.123209+0000","last_unstale":"2023-11-16T18:23:39.504877+0000","last_undegraded":"2023-11-16T18:23:39.504877+0000","last_fullsized":"2023-11-16T18:23:39.504877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:09:15.998485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508037+0000","last_change":"2023-11-16T18:22:46.378038+0000","last_active":"2023-11-16T18:23:39.508037+0000","last_peered":"2023-11-16T18:23:39.508037+0000","last_clean":"2023-11-16T18:23:39.508037+0000","last_became_active":"2023-11-16T18:22:46.377680+0000","last_became_peered":"2023-11-16T18:22:46.377680+0000","last_unstale":"2023-11-16T18:23:39.508037+0000","last_undegraded":"2023-11-16T18:23:39.508037+0000","last_fullsized":"2023-11-16T18:23:39.508037+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:31.194156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.818543+0000","last_change":"2023-11-16T18:22:46.821888+0000","last_active":"2023-11-16T18:24:37.818543+0000","last_peered":"2023-11-16T18:24:37.818543+0000","last_clean":"2023-11-16T18:24:37.818543+0000","last_became_active":"2023-11-16T18:22:46.821446+0000","last_became_peered":"2023-11-16T18:22:46.821446+0000","last_unstale":"2023-11-16T18:24:37.818543+0000","last_undegraded":"2023-11-16T18:24:37.818543+0000","last_fullsized":"2023-11-16T18:24:37.818543+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:11:30.632655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.501162+0000","last_change":"2023-11-16T18:22:51.119717+0000","last_active":"2023-11-16T18:23:39.501162+0000","last_peered":"2023-11-16T18:23:39.501162+0000","last_clean":"2023-11-16T18:23:39.501162+0000","last_became_active":"2023-11-16T18:22:51.119256+0000","last_became_peered":"2023-11-16T18:22:51.119256+0000","last_unstale":"2023-11-16T18:23:39.501162+0000","last_undegraded":"2023-11-16T18:23:39.501162+0000","last_fullsized":"2023-11-16T18:23:39.501162+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:25:42.785279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504001+0000","last_change":"2023-11-16T18:22:46.375387+0000","last_active":"2023-11-16T18:23:39.504001+0000","last_peered":"2023-11-16T18:23:39.504001+0000","last_clean":"2023-11-16T18:23:39.504001+0000","last_became_active":"2023-11-16T18:22:46.374886+0000","last_became_peered":"2023-11-16T18:22:46.374886+0000","last_unstale":"2023-11-16T18:23:39.504001+0000","last_undegraded":"2023-11-16T18:23:39.504001+0000","last_fullsized":"2023-11-16T18:23:39.504001+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:18:24.699496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.854268+0000","last_change":"2023-11-16T18:22:46.820315+0000","last_active":"2023-11-16T18:24:37.854268+0000","last_peered":"2023-11-16T18:24:37.854268+0000","last_clean":"2023-11-16T18:24:37.854268+0000","last_became_active":"2023-11-16T18:22:46.819592+0000","last_became_peered":"2023-11-16T18:22:46.819592+0000","last_unstale":"2023-11-16T18:24:37.854268+0000","last_undegraded":"2023-11-16T18:24:37.854268+0000","last_fullsized":"2023-11-16T18:24:37.854268+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:43:26.336090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.518963+0000","last_change":"2023-11-16T18:23:39.518963+0000","last_active":"2023-11-16T18:23:39.518963+0000","last_peered":"2023-11-16T18:23:39.518963+0000","last_clean":"2023-11-16T18:23:39.518963+0000","last_became_active":"2023-11-16T18:23:39.513817+0000","last_became_peered":"2023-11-16T18:23:39.513817+0000","last_unstale":"2023-11-16T18:23:39.518963+0000","last_undegraded":"2023-11-16T18:23:39.518963+0000","last_fullsized":"2023-11-16T18:23:39.518963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:16:46.629599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510801+0000","last_change":"2023-11-16T18:22:46.378638+0000","last_active":"2023-11-16T18:23:39.510801+0000","last_peered":"2023-11-16T18:23:39.510801+0000","last_clean":"2023-11-16T18:23:39.510801+0000","last_became_active":"2023-11-16T18:22:46.378341+0000","last_became_peered":"2023-11-16T18:22:46.378341+0000","last_unstale":"2023-11-16T18:23:39.510801+0000","last_undegraded":"2023-11-16T18:23:39.510801+0000","last_fullsized":"2023-11-16T18:23:39.510801+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:36:28.625148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.857732+0000","last_change":"2023-11-16T18:22:46.843431+0000","last_active":"2023-11-16T18:24:37.857732+0000","last_peered":"2023-11-16T18:24:37.857732+0000","last_clean":"2023-11-16T18:24:37.857732+0000","last_became_active":"2023-11-16T18:22:46.842992+0000","last_became_peered":"2023-11-16T18:22:46.842992+0000","last_unstale":"2023-11-16T18:24:37.857732+0000","last_undegraded":"2023-11-16T18:24:37.857732+0000","last_fullsized":"2023-11-16T18:24:37.857732+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:23:28.821023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503912+0000","last_change":"2023-11-16T18:22:51.122640+0000","last_active":"2023-11-16T18:23:39.503912+0000","last_peered":"2023-11-16T18:23:39.503912+0000","last_clean":"2023-11-16T18:23:39.503912+0000","last_became_active":"2023-11-16T18:22:51.122006+0000","last_became_peered":"2023-11-16T18:22:51.122006+0000","last_unstale":"2023-11-16T18:23:39.503912+0000","last_undegraded":"2023-11-16T18:23:39.503912+0000","last_fullsized":"2023-11-16T18:23:39.503912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:27:30.406233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.504268+0000","last_change":"2023-11-16T18:22:46.384837+0000","last_active":"2023-11-16T18:23:39.504268+0000","last_peered":"2023-11-16T18:23:39.504268+0000","last_clean":"2023-11-16T18:23:39.504268+0000","last_became_active":"2023-11-16T18:22:46.384541+0000","last_became_peered":"2023-11-16T18:22:46.384541+0000","last_unstale":"2023-11-16T18:23:39.504268+0000","last_undegraded":"2023-11-16T18:23:39.504268+0000","last_fullsized":"2023-11-16T18:23:39.504268+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:06:22.863895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502460+0000","last_change":"2023-11-16T18:22:46.839967+0000","last_active":"2023-11-16T18:23:39.502460+0000","last_peered":"2023-11-16T18:23:39.502460+0000","last_clean":"2023-11-16T18:23:39.502460+0000","last_became_active":"2023-11-16T18:22:46.839645+0000","last_became_peered":"2023-11-16T18:22:46.839645+0000","last_unstale":"2023-11-16T18:23:39.502460+0000","last_undegraded":"2023-11-16T18:23:39.502460+0000","last_fullsized":"2023-11-16T18:23:39.502460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:04:34.616063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507240+0000","last_change":"2023-11-16T18:22:51.126548+0000","last_active":"2023-11-16T18:23:39.507240+0000","last_peered":"2023-11-16T18:23:39.507240+0000","last_clean":"2023-11-16T18:23:39.507240+0000","last_became_active":"2023-11-16T18:22:51.126039+0000","last_became_peered":"2023-11-16T18:22:51.126039+0000","last_unstale":"2023-11-16T18:23:39.507240+0000","last_undegraded":"2023-11-16T18:23:39.507240+0000","last_fullsized":"2023-11-16T18:23:39.507240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:22:07.650645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.511343+0000","last_change":"2023-11-16T18:22:46.383947+0000","last_active":"2023-11-16T18:23:39.511343+0000","last_peered":"2023-11-16T18:23:39.511343+0000","last_clean":"2023-11-16T18:23:39.511343+0000","last_became_active":"2023-11-16T18:22:46.383661+0000","last_became_peered":"2023-11-16T18:22:46.383661+0000","last_unstale":"2023-11-16T18:23:39.511343+0000","last_undegraded":"2023-11-16T18:23:39.511343+0000","last_fullsized":"2023-11-16T18:23:39.511343+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:34.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503182+0000","last_change":"2023-11-16T18:22:46.836877+0000","last_active":"2023-11-16T18:23:39.503182+0000","last_peered":"2023-11-16T18:23:39.503182+0000","last_clean":"2023-11-16T18:23:39.503182+0000","last_became_active":"2023-11-16T18:22:46.836477+0000","last_became_peered":"2023-11-16T18:22:46.836477+0000","last_unstale":"2023-11-16T18:23:39.503182+0000","last_undegraded":"2023-11-16T18:23:39.503182+0000","last_fullsized":"2023-11-16T18:23:39.503182+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:38:49.681616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.505171+0000","last_change":"2023-11-16T18:22:51.126930+0000","last_active":"2023-11-16T18:23:39.505171+0000","last_peered":"2023-11-16T18:23:39.505171+0000","last_clean":"2023-11-16T18:23:39.505171+0000","last_became_active":"2023-11-16T18:22:51.126409+0000","last_became_peered":"2023-11-16T18:22:51.126409+0000","last_unstale":"2023-11-16T18:23:39.505171+0000","last_undegraded":"2023-11-16T18:23:39.505171+0000","last_fullsized":"2023-11-16T18:23:39.505171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:31:50.012322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508533+0000","last_change":"2023-11-16T18:22:46.380820+0000","last_active":"2023-11-16T18:23:39.508533+0000","last_peered":"2023-11-16T18:23:39.508533+0000","last_clean":"2023-11-16T18:23:39.508533+0000","last_became_active":"2023-11-16T18:22:46.380446+0000","last_became_peered":"2023-11-16T18:22:46.380446+0000","last_unstale":"2023-11-16T18:23:39.508533+0000","last_undegraded":"2023-11-16T18:23:39.508533+0000","last_fullsized":"2023-11-16T18:23:39.508533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:20:07.923022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.502955+0000","last_change":"2023-11-16T18:22:46.838153+0000","last_active":"2023-11-16T18:23:39.502955+0000","last_peered":"2023-11-16T18:23:39.502955+0000","last_clean":"2023-11-16T18:23:39.502955+0000","last_became_active":"2023-11-16T18:22:46.837836+0000","last_became_peered":"2023-11-16T18:22:46.837836+0000","last_unstale":"2023-11-16T18:23:39.502955+0000","last_undegraded":"2023-11-16T18:23:39.502955+0000","last_fullsized":"2023-11-16T18:23:39.502955+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:24:42.477381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.507608+0000","last_change":"2023-11-16T18:22:51.119993+0000","last_active":"2023-11-16T18:23:39.507608+0000","last_peered":"2023-11-16T18:23:39.507608+0000","last_clean":"2023-11-16T18:23:39.507608+0000","last_became_active":"2023-11-16T18:22:51.119401+0000","last_became_peered":"2023-11-16T18:22:51.119401+0000","last_unstale":"2023-11-16T18:23:39.507608+0000","last_undegraded":"2023-11-16T18:23:39.507608+0000","last_fullsized":"2023-11-16T18:23:39.507608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:50.097321+0000","last_clean_scrub_stamp":"2023-11-16T18:22:50.097321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:19:31.451143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585944+0000","last_change":"2023-11-16T18:22:46.383399+0000","last_active":"2023-11-16T18:23:39.585944+0000","last_peered":"2023-11-16T18:23:39.585944+0000","last_clean":"2023-11-16T18:23:39.585944+0000","last_became_active":"2023-11-16T18:22:46.382938+0000","last_became_peered":"2023-11-16T18:22:46.382938+0000","last_unstale":"2023-11-16T18:23:39.585944+0000","last_undegraded":"2023-11-16T18:23:39.585944+0000","last_fullsized":"2023-11-16T18:23:39.585944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:45.355170+0000","last_clean_scrub_stamp":"2023-11-16T18:22:45.355170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:29.725086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.503688+0000","last_change":"2023-11-16T18:22:46.836035+0000","last_active":"2023-11-16T18:23:39.503688+0000","last_peered":"2023-11-16T18:23:39.503688+0000","last_clean":"2023-11-16T18:23:39.503688+0000","last_became_active":"2023-11-16T18:22:46.835709+0000","last_became_peered":"2023-11-16T18:22:46.835709+0000","last_unstale":"2023-11-16T18:23:39.503688+0000","last_undegraded":"2023-11-16T18:23:39.503688+0000","last_fullsized":"2023-11-16T18:23:39.503688+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:32:57.558106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.848398+0000","last_change":"2023-11-16T18:22:46.820535+0000","last_active":"2023-11-16T18:24:37.848398+0000","last_peered":"2023-11-16T18:24:37.848398+0000","last_clean":"2023-11-16T18:24:37.848398+0000","last_became_active":"2023-11-16T18:22:46.820009+0000","last_became_peered":"2023-11-16T18:22:46.820009+0000","last_unstale":"2023-11-16T18:24:37.848398+0000","last_undegraded":"2023-11-16T18:24:37.848398+0000","last_fullsized":"2023-11-16T18:24:37.848398+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:50:39.196560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510673+0000","last_change":"2023-11-16T18:22:46.831762+0000","last_active":"2023-11-16T18:23:39.510673+0000","last_peered":"2023-11-16T18:23:39.510673+0000","last_clean":"2023-11-16T18:23:39.510673+0000","last_became_active":"2023-11-16T18:22:46.831429+0000","last_became_peered":"2023-11-16T18:22:46.831429+0000","last_unstale":"2023-11-16T18:23:39.510673+0000","last_undegraded":"2023-11-16T18:23:39.510673+0000","last_fullsized":"2023-11-16T18:23:39.510673+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:15:44.338906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.510533+0000","last_change":"2023-11-16T18:22:46.838133+0000","last_active":"2023-11-16T18:23:39.510533+0000","last_peered":"2023-11-16T18:23:39.510533+0000","last_clean":"2023-11-16T18:23:39.510533+0000","last_became_active":"2023-11-16T18:22:46.837845+0000","last_became_peered":"2023-11-16T18:22:46.837845+0000","last_unstale":"2023-11-16T18:23:39.510533+0000","last_undegraded":"2023-11-16T18:23:39.510533+0000","last_fullsized":"2023-11-16T18:23:39.510533+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:13:06.102358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.585337+0000","last_change":"2023-11-16T18:23:39.585337+0000","last_active":"2023-11-16T18:23:39.585337+0000","last_peered":"2023-11-16T18:23:39.585337+0000","last_clean":"2023-11-16T18:23:39.585337+0000","last_became_active":"2023-11-16T18:23:39.584874+0000","last_became_peered":"2023-11-16T18:23:39.584874+0000","last_unstale":"2023-11-16T18:23:39.585337+0000","last_undegraded":"2023-11-16T18:23:39.585337+0000","last_fullsized":"2023-11-16T18:23:39.585337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:24:54.928438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.842092+0000","last_change":"2023-11-16T18:22:46.823192+0000","last_active":"2023-11-16T18:24:37.842092+0000","last_peered":"2023-11-16T18:24:37.842092+0000","last_clean":"2023-11-16T18:24:37.842092+0000","last_became_active":"2023-11-16T18:22:46.822930+0000","last_became_peered":"2023-11-16T18:22:46.822930+0000","last_unstale":"2023-11-16T18:24:37.842092+0000","last_undegraded":"2023-11-16T18:24:37.842092+0000","last_fullsized":"2023-11-16T18:24:37.842092+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:49:48.831997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.848926+0000","last_change":"2023-11-16T18:22:46.825815+0000","last_active":"2023-11-16T18:24:37.848926+0000","last_peered":"2023-11-16T18:24:37.848926+0000","last_clean":"2023-11-16T18:24:37.848926+0000","last_became_active":"2023-11-16T18:22:46.825517+0000","last_became_peered":"2023-11-16T18:22:46.825517+0000","last_unstale":"2023-11-16T18:24:37.848926+0000","last_undegraded":"2023-11-16T18:24:37.848926+0000","last_fullsized":"2023-11-16T18:24:37.848926+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:32:02.468288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.511101+0000","last_change":"2023-11-16T18:22:46.826825+0000","last_active":"2023-11-16T18:23:39.511101+0000","last_peered":"2023-11-16T18:23:39.511101+0000","last_clean":"2023-11-16T18:23:39.511101+0000","last_became_active":"2023-11-16T18:22:46.826495+0000","last_became_peered":"2023-11-16T18:22:46.826495+0000","last_unstale":"2023-11-16T18:23:39.511101+0000","last_undegraded":"2023-11-16T18:23:39.511101+0000","last_fullsized":"2023-11-16T18:23:39.511101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:07:42.411162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.841815+0000","last_change":"2023-11-16T18:22:46.844090+0000","last_active":"2023-11-16T18:24:37.841815+0000","last_peered":"2023-11-16T18:24:37.841815+0000","last_clean":"2023-11-16T18:24:37.841815+0000","last_became_active":"2023-11-16T18:22:46.843641+0000","last_became_peered":"2023-11-16T18:22:46.843641+0000","last_unstale":"2023-11-16T18:24:37.841815+0000","last_undegraded":"2023-11-16T18:24:37.841815+0000","last_fullsized":"2023-11-16T18:24:37.841815+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:00:12.291257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508321+0000","last_change":"2023-11-16T18:22:46.838340+0000","last_active":"2023-11-16T18:23:39.508321+0000","last_peered":"2023-11-16T18:23:39.508321+0000","last_clean":"2023-11-16T18:23:39.508321+0000","last_became_active":"2023-11-16T18:22:46.838039+0000","last_became_peered":"2023-11-16T18:22:46.838039+0000","last_unstale":"2023-11-16T18:23:39.508321+0000","last_undegraded":"2023-11-16T18:23:39.508321+0000","last_fullsized":"2023-11-16T18:23:39.508321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:09:28.607361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.853480+0000","last_change":"2023-11-16T18:22:46.841739+0000","last_active":"2023-11-16T18:24:37.853480+0000","last_peered":"2023-11-16T18:24:37.853480+0000","last_clean":"2023-11-16T18:24:37.853480+0000","last_became_active":"2023-11-16T18:22:46.841450+0000","last_became_peered":"2023-11-16T18:22:46.841450+0000","last_unstale":"2023-11-16T18:24:37.853480+0000","last_undegraded":"2023-11-16T18:24:37.853480+0000","last_fullsized":"2023-11-16T18:24:37.853480+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:42:53.482129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:23:39.508013+0000","last_change":"2023-11-16T18:22:46.839060+0000","last_active":"2023-11-16T18:23:39.508013+0000","last_peered":"2023-11-16T18:23:39.508013+0000","last_clean":"2023-11-16T18:23:39.508013+0000","last_became_active":"2023-11-16T18:22:46.838725+0000","last_became_peered":"2023-11-16T18:22:46.838725+0000","last_unstale":"2023-11-16T18:23:39.508013+0000","last_undegraded":"2023-11-16T18:23:39.508013+0000","last_fullsized":"2023-11-16T18:23:39.508013+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:33:35.374697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.853570+0000","last_change":"2023-11-16T18:22:46.833439+0000","last_active":"2023-11-16T18:24:37.853570+0000","last_peered":"2023-11-16T18:24:37.853570+0000","last_clean":"2023-11-16T18:24:37.853570+0000","last_became_active":"2023-11-16T18:22:46.833111+0000","last_became_peered":"2023-11-16T18:22:46.833111+0000","last_unstale":"2023-11-16T18:24:37.853570+0000","last_undegraded":"2023-11-16T18:24:37.853570+0000","last_fullsized":"2023-11-16T18:24:37.853570+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:10:36.066370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T18:24:37.848188+0000","last_change":"2023-11-16T18:22:46.825185+0000","last_active":"2023-11-16T18:24:37.848188+0000","last_peered":"2023-11-16T18:24:37.848188+0000","last_clean":"2023-11-16T18:24:37.848188+0000","last_became_active":"2023-11-16T18:22:46.824866+0000","last_became_peered":"2023-11-16T18:22:46.824866+0000","last_unstale":"2023-11-16T18:24:37.848188+0000","last_undegraded":"2023-11-16T18:24:37.848188+0000","last_fullsized":"2023-11-16T18:24:37.848188+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T18:22:46.356914+0000","last_clean_scrub_stamp":"2023-11-16T18:22:46.356914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:46:05.263494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":173,"ondisk_log_size":173,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":215374,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65767,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":549,"num_read_kb":345,"num_write":159,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":61913,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":197301,"data_compressed":9627,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481827,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6140,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739012,"statfs":{"total":95995035648,"available":95988748288,"internally_reserved":0,"allocated":970752,"data_stored":485806,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.56000000000000005,"15min":0.57499999999999996},"min":{"1min":0.33800000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.66100000000000003,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.66500000000000004,"15min":0.69299999999999995},"min":{"1min":0.379,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":3.0590000000000002,"15min":3.0590000000000002},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63500000000000001,"15min":0.64100000000000001},"min":{"1min":0.41099999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.503,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81599999999999995,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.53900000000000003}]},{"osd":2,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.49299999999999999,"15min":0.48499999999999999},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84999999999999998,"5min":0.874,"15min":0.874},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.52600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.64200000000000002,"15min":0.64700000000000002},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67600000000000005,"15min":0.68000000000000005},"min":{"1min":0.52700000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.86499999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.58299999999999996}]},{"osd":5,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63100000000000001,"15min":0.63500000000000001},"min":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.91100000000000003,"5min":1.339,"15min":1.339},"last":0.629},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59699999999999998,"15min":0.60499999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84499999999999997,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.56399999999999995}]},{"osd":6,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.73599999999999999,"15min":0.754},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.1140000000000001,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.80700000000000005,"15min":0.82899999999999996},"min":{"1min":0.54200000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.196,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.746,"15min":0.75},"min":{"1min":0.53800000000000003,"5min":0.443,"15min":0.443},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.75,"15min":0.752},"min":{"1min":0.56100000000000005,"5min":0.502,"15min":0.502},"max":{"1min":0.97499999999999998,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.72799999999999998}]},{"osd":8,"last update":"Thu Nov 16 18:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.80300000000000005}]},{"osd":9,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.86599999999999999,"15min":0.88300000000000001},"min":{"1min":0.53100000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.161,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.73799999999999999,"15min":0.74099999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.74199999999999999}]},{"osd":10,"last update":"Thu Nov 16 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80400000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0209999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.755},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.83099999999999996,"15min":0.83699999999999997},"min":{"1min":0.59599999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.032,"5min":1.476,"15min":1.476},"last":0.83599999999999997}]}]},{"osd":10,"up_from":69,"seq":296352743462,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6596,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738556,"statfs":{"total":95995035648,"available":95988281344,"internally_reserved":0,"allocated":1110016,"data_stored":1138555,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.35299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.51800000000000002},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.53700000000000003}]},{"osd":2,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.82899999999999996,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.85699999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.73699999999999999}]},{"osd":3,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.659,"5min":1.659,"15min":1.659},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.72499999999999998,"15min":0.73299999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.629,"15min":0.63},"min":{"1min":0.44,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76800000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.36499999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.85099999999999998,"5min":0.996,"15min":0.996},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.99399999999999999,"5min":1.206,"15min":1.206},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.745},"min":{"1min":0.55200000000000005,"5min":0.502,"15min":0.502},"max":{"1min":0.97799999999999998,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.77400000000000002,"15min":0.77700000000000002},"min":{"1min":0.499,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.57399999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.60699999999999998}]},{"osd":7,"last update":"Thu Nov 16 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55000000000000004}]},{"osd":8,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.84199999999999997,"15min":0.84999999999999998},"min":{"1min":0.63100000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.98899999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0189999999999999,"5min":1.155,"15min":1.155},"last":0.71499999999999997}]},{"osd":9,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.65600000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.628},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.85699999999999998,"15min":0.85999999999999999},"min":{"1min":0.59799999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.75600000000000001}]},{"osd":11,"last update":"Thu Nov 16 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.63200000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.80400000000000005,"15min":0.80800000000000005},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":0.92400000000000004,"5min":1.034,"15min":1.034},"last":0.79700000000000004}]}]},{"osd":9,"up_from":64,"seq":274877906987,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6340,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":978944,"data_stored":485930,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.433,"15min":0.45500000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54900000000000004,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.371,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56499999999999995,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.56299999999999994}]},{"osd":1,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.64000000000000001,"15min":0.66800000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73599999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.60599999999999998,"15min":0.63},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.75700000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63400000000000001}]},{"osd":2,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.64000000000000001,"15min":0.65800000000000003},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82499999999999996,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61899999999999999,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.80100000000000005,"5min":1.024,"15min":1.024},"last":0.67100000000000004}]},{"osd":3,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.51900000000000002,"15min":0.54900000000000004},"min":{"1min":0.29399999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63200000000000001,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.505,"15min":0.55000000000000004},"min":{"1min":0.29799999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.58599999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.70199999999999996}]},{"osd":4,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.67200000000000004,"15min":0.71299999999999997},"min":{"1min":0.44,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.70799999999999996,"15min":0.747},"min":{"1min":0.377,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85299999999999998,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.73999999999999999}]},{"osd":5,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.69299999999999995,"15min":0.72099999999999997},"min":{"1min":0.45900000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.89700000000000002,"5min":1.357,"15min":1.357},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.72899999999999998,"15min":0.76500000000000001},"min":{"1min":0.51600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.88300000000000001,"5min":1.196,"15min":1.196},"last":0.77300000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.379},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.42699999999999999}]},{"osd":7,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.77800000000000002,"15min":0.83099999999999996},"min":{"1min":0.504,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.81899999999999995,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.73399999999999999,"15min":0.78600000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78900000000000003,"5min":1.157,"15min":1.157},"last":0.85099999999999998}]},{"osd":8,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.749,"15min":0.79800000000000004},"min":{"1min":0.49399999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.85299999999999998,"5min":1.248,"15min":1.248},"last":1.004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.79700000000000004,"15min":0.83999999999999997},"min":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.90500000000000003,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.92200000000000004}]},{"osd":10,"last update":"Thu Nov 16 18:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.76700000000000002,"15min":0.78900000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.879,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.752,"15min":0.76600000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.873,"5min":1.022,"15min":1.022},"last":0.82999999999999996}]},{"osd":11,"last update":"Thu Nov 16 18:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.874},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.88600000000000001}]}]},{"osd":8,"up_from":57,"seq":244813135920,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6460,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738692,"statfs":{"total":95995035648,"available":95988420608,"internally_reserved":0,"allocated":970752,"data_stored":485738,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,11,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.42399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":6.899},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.40799999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":5.3760000000000003}]},{"osd":1,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.59099999999999997,"15min":0.56799999999999995},"min":{"1min":0.45400000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.876,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":8.3309999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.626,"15min":0.59599999999999997},"min":{"1min":0.56100000000000005,"5min":0.316,"15min":0.316},"max":{"1min":0.84299999999999997,"5min":0.872,"15min":0.872},"last":0.97399999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.46100000000000002,"15min":0.44700000000000001},"min":{"1min":0.38500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71099999999999997,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.504,"15min":0.48199999999999998},"min":{"1min":0.36799999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72799999999999998,"5min":1.05,"15min":1.05},"last":0.998}]},{"osd":3,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91900000000000004,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":6.6440000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.628,"15min":0.628},"min":{"1min":0.48999999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0760000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.82799999999999996}]},{"osd":4,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.621,"15min":0.60499999999999998},"min":{"1min":0.46500000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":20.532},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.63900000000000001,"15min":0.60999999999999999},"min":{"1min":0.47299999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":14.776999999999999}]},{"osd":5,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.56100000000000005,"15min":0.52500000000000002},"min":{"1min":0.44400000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.55300000000000005,"15min":0.51500000000000001},"min":{"1min":0.53700000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.44900000000000001}]},{"osd":6,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67500000000000004,"15min":0.66300000000000003},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.71299999999999997,"15min":0.69699999999999995},"min":{"1min":0.64600000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.71999999999999997,"15min":0.68000000000000005},"min":{"1min":0.64000000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.143,"5min":1.266,"15min":1.266},"last":2.2400000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.69899999999999995,"15min":0.67300000000000004},"min":{"1min":0.624,"5min":0.434,"15min":0.434},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":2.3479999999999999}]},{"osd":9,"last update":"Thu Nov 16 18:24:25 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.72999999999999998,"15min":0.71399999999999997},"min":{"1min":0.56000000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0269999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.751,"15min":0.72799999999999998},"min":{"1min":0.55200000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":10.176}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.72499999999999998,"15min":0.71099999999999997},"min":{"1min":0.60499999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.036,"5min":1.276,"15min":1.276},"last":3.9489999999999998},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.76400000000000001,"15min":0.74399999999999999},"min":{"1min":0.74299999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":3.5649999999999999}]},{"osd":11,"last update":"Thu Nov 16 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":8.0890000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":4.6399999999999997}]}]},{"osd":7,"up_from":49,"seq":210453397556,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6580,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738572,"statfs":{"total":95995035648,"available":95988297728,"internally_reserved":0,"allocated":1028096,"data_stored":1082370,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.60699999999999998,"15min":0.59499999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81999999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81699999999999995,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.44500000000000001,"15min":0.42799999999999999},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61499999999999999,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.48099999999999998,"15min":0.46600000000000003},"min":{"1min":0.36299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71899999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63100000000000001,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65000000000000002,"15min":0.622},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.876,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85699999999999998,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.439},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.42299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.83599999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47699999999999998}]},{"osd":4,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.57799999999999996,"15min":0.55200000000000005},"min":{"1min":0.43099999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.77900000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.372},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.372,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76300000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70099999999999996,"15min":0.67900000000000005},"min":{"1min":0.47099999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89500000000000002,"5min":1.075,"15min":1.075},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.75600000000000001,"15min":0.748},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.94999999999999996,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.55500000000000005}]},{"osd":6,"last update":"Thu Nov 16 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.71599999999999997,"15min":0.69999999999999996},"min":{"1min":0.52600000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.74399999999999999,"15min":0.73199999999999998},"min":{"1min":0.47799999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.45500000000000002}]},{"osd":8,"last update":"Thu Nov 16 18:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.76000000000000001,"15min":0.745},"min":{"1min":0.60599999999999998,"5min":0.433,"15min":0.433},"max":{"1min":0.98899999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.623},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.78800000000000003,"15min":0.78000000000000003},"min":{"1min":0.57399999999999995,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.018,"5min":1.107,"15min":1.107},"last":0.56299999999999994}]},{"osd":9,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79500000000000004,"15min":0.80400000000000005},"min":{"1min":0.57199999999999995,"5min":0.443,"15min":0.443},"max":{"1min":1.0549999999999999,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.83099999999999996,"15min":0.81699999999999995},"min":{"1min":0.59699999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0369999999999999,"5min":3.4119999999999999,"15min":3.4119999999999999},"last":0.504}]},{"osd":10,"last update":"Thu Nov 16 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.71399999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.99099999999999999,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81000000000000005,"15min":0.80900000000000005},"min":{"1min":0.61199999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.071,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.67400000000000004}]},{"osd":11,"last update":"Thu Nov 16 18:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.59699999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561081,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6384,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738768,"statfs":{"total":95995035648,"available":95988498432,"internally_reserved":0,"allocated":958464,"data_stored":481106,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.39800000000000002,"15min":0.36899999999999999},"min":{"1min":0.30199999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.50600000000000001,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.434},{"interface":"front","average":{"1min":0.374,"5min":0.40500000000000003,"15min":0.38100000000000001},"min":{"1min":0.315,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.44800000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63,"15min":0.59799999999999998},"min":{"1min":0.44,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77500000000000002,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.53300000000000003},"min":{"1min":0.40799999999999997,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.751,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.754}]},{"osd":2,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.41599999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71199999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.621,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.754,"5min":1.391,"15min":1.391},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52600000000000002,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61199999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.45500000000000002,"15min":0.437},"min":{"1min":0.33000000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79800000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.377}]},{"osd":4,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.61399999999999999,"15min":0.60699999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.76700000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.69299999999999995,"15min":0.66500000000000004},"min":{"1min":0.504,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.76100000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Nov 16 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60299999999999998,"15min":0.58099999999999996},"min":{"1min":0.40300000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.68400000000000005,"5min":1.004,"15min":1.004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.64500000000000002,"15min":0.625},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82799999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75800000000000001,"15min":0.75},"min":{"1min":0.505,"5min":0.497,"15min":0.497},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.79100000000000004,"15min":0.77600000000000002},"min":{"1min":0.54600000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.83099999999999996}]},{"osd":8,"last update":"Thu Nov 16 18:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.80300000000000005,"15min":0.81899999999999995},"min":{"1min":0.497,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.95199999999999996,"5min":1.429,"15min":1.429},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.441,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89500000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.52200000000000002}]},{"osd":9,"last update":"Thu Nov 16 18:24:25 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.68000000000000005,"15min":0.73699999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70099999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.625},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.79900000000000004,"15min":0.85699999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.84699999999999998,"5min":1.298,"15min":1.298},"last":0.61099999999999999}]},{"osd":10,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.85999999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.70899999999999996}]},{"osd":11,"last update":"Thu Nov 16 18:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.65200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7476,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737676,"statfs":{"total":95995035648,"available":95987380224,"internally_reserved":0,"allocated":1028096,"data_stored":543180,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:33 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.52800000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.375},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53300000000000003,"15min":0.53800000000000003},"min":{"1min":0.34200000000000003,"5min":0.31,"15min":0.31},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.35199999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.60299999999999998,"15min":0.54800000000000004},"min":{"1min":0.45000000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.91700000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.61599999999999999,"15min":0.56100000000000005},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.754}]},{"osd":3,"last update":"Thu Nov 16 18:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59199999999999997,"15min":0.56899999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.59799999999999998,"15min":0.57399999999999995},"min":{"1min":0.47899999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.45200000000000001,"15min":0.40799999999999997},"min":{"1min":0.372,"5min":0.252,"15min":0.252},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.47399999999999998,"15min":0.41999999999999998},"min":{"1min":0.39300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80000000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Nov 16 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.624,"15min":0.60699999999999998},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93799999999999994,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.66700000000000004,"15min":0.63500000000000001},"min":{"1min":0.52800000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92600000000000005,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.71199999999999997}]},{"osd":6,"last update":"Thu Nov 16 18:24:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.629,"15min":0.61199999999999999},"min":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.63400000000000001,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.41199999999999998}]},{"osd":7,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.56899999999999995,"15min":0.53900000000000003},"min":{"1min":0.42399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.88200000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.63},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.622,"15min":0.58199999999999996},"min":{"1min":0.46500000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.91200000000000003,"5min":0.998,"15min":0.998},"last":0.52400000000000002}]},{"osd":8,"last update":"Thu Nov 16 18:24:01 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.69899999999999995,"15min":0.66900000000000004},"min":{"1min":0.48399999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0449999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.69999999999999996,"15min":0.67000000000000004},"min":{"1min":0.61799999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73799999999999999}]},{"osd":9,"last update":"Thu Nov 16 18:24:26 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.73299999999999998,"15min":0.72199999999999998},"min":{"1min":0.52300000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.749,"5min":0.755,"15min":0.749},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.58099999999999996}]},{"osd":10,"last update":"Thu Nov 16 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.77700000000000002,"15min":0.76900000000000002},"min":{"1min":0.57899999999999996,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.80900000000000005,"15min":0.79500000000000004},"min":{"1min":0.66000000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.65700000000000003}]},{"osd":11,"last update":"Thu Nov 16 18:23:48 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.79900000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7300,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737852,"statfs":{"total":95995035648,"available":95987560448,"internally_reserved":0,"allocated":978944,"data_stored":482221,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 18:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.55300000000000005,"15min":0.5},"min":{"1min":0.36799999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.923,"5min":2.923,"15min":2.923},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.56699999999999995,"15min":0.503},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.2109999999999999,"5min":3.2109999999999999,"15min":3.2109999999999999},"last":0.72599999999999998}]},{"osd":2,"last update":"Thu Nov 16 18:23:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60999999999999999,"15min":0.71599999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.93799999999999994,"5min":5.3650000000000002,"15min":5.3650000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.627,"15min":0.67100000000000004},"min":{"1min":0.437,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.8929999999999998,"5min":3.4009999999999998,"15min":3.4009999999999998},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Nov 16 18:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.7810000000000001,"5min":3.7810000000000001,"15min":3.7810000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":3.294,"5min":3.294,"15min":3.294},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.60599999999999998,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.0249999999999999,"5min":3.0249999999999999,"15min":3.0249999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.61299999999999999,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":3.242,"5min":3.242,"15min":3.242},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Nov 16 18:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.48399999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.63200000000000001,"15min":0.59599999999999997},"min":{"1min":0.40699999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.851,"5min":2.851,"15min":2.851},"last":0.64500000000000002}]},{"osd":6,"last update":"Thu Nov 16 18:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55500000000000005,"15min":0.51900000000000002},"min":{"1min":0.41699999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65700000000000003,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.32800000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.64100000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Nov 16 18:23:54 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.0499999999999998,"5min":3.0499999999999998,"15min":3.0499999999999998},"last":0.77500000000000002}]},{"osd":8,"last update":"Thu Nov 16 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.70299999999999996,"15min":0.72499999999999998},"min":{"1min":0.54200000000000004,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.036,"5min":1.052,"15min":1.052},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.73399999999999999,"15min":0.74299999999999999},"min":{"1min":0.54000000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.9470000000000001,"5min":2.9470000000000001,"15min":2.9470000000000001},"last":0.81699999999999995}]},{"osd":9,"last update":"Thu Nov 16 18:24:28 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.46000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.002,"5min":3.3180000000000001,"15min":3.3180000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.70299999999999996,"15min":0.73099999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.84599999999999997,"5min":2.79,"15min":2.79},"last":0.60599999999999998}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.78600000000000003,"15min":0.78100000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.0009999999999999,"5min":3.0009999999999999,"15min":3.0009999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.77000000000000002,"15min":0.75800000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":3.2679999999999998,"5min":3.2679999999999998,"15min":3.2679999999999998},"last":0.83599999999999997}]},{"osd":11,"last update":"Thu Nov 16 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.70599999999999996},"min":{"1min":0.58699999999999997,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.871},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.751,"15min":0.73499999999999999},"min":{"1min":0.59299999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.9740000000000002,"5min":2.9740000000000002,"15min":2.9740000000000002},"last":0.89200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7484,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737668,"statfs":{"total":95995035648,"available":95987372032,"internally_reserved":0,"allocated":970752,"data_stored":481744,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.54200000000000004,"15min":0.48199999999999998},"min":{"1min":0.34899999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.51800000000000002,"15min":0.46899999999999997},"min":{"1min":0.39500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.44700000000000001}]},{"osd":1,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.57399999999999995,"15min":0.52400000000000002},"min":{"1min":0.42699999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.073,"5min":1.47,"15min":1.47},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.59199999999999997,"15min":0.54800000000000004},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96599999999999997,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.71399999999999997}]},{"osd":3,"last update":"Thu Nov 16 18:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.54400000000000004},"min":{"1min":0.36499999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82999999999999996,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.61499999999999999,"15min":0.55300000000000005},"min":{"1min":0.47399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.89800000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.65200000000000002}]},{"osd":4,"last update":"Thu Nov 16 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.52700000000000002}]},{"osd":5,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.47599999999999998,"15min":0.42499999999999999},"min":{"1min":0.32700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.52500000000000002,"15min":0.48099999999999998},"min":{"1min":0.40300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.02,"5min":1.339,"15min":1.339},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Nov 16 18:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.64800000000000002,"15min":0.61899999999999999},"min":{"1min":0.45500000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.433},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.68799999999999994,"15min":0.63800000000000001},"min":{"1min":0.51300000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.67400000000000004}]},{"osd":7,"last update":"Thu Nov 16 18:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.63200000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:24:10 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.61599999999999999,"15min":0.58199999999999996},"min":{"1min":0.46899999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.622},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.60599999999999998,"15min":0.57799999999999996},"min":{"1min":0.45700000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.97099999999999997,"5min":1.377,"15min":1.377},"last":0.41199999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.54400000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92500000000000004,"5min":1.409,"15min":1.409},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.755,"15min":0.746},"min":{"1min":0.60799999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.59199999999999997}]},{"osd":10,"last update":"Thu Nov 16 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.73899999999999999,"15min":0.71899999999999997},"min":{"1min":0.60399999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.77200000000000002,"15min":0.76200000000000001},"min":{"1min":0.58799999999999997,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.242,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.66600000000000004}]},{"osd":11,"last update":"Thu Nov 16 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.64300000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6776,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738376,"statfs":{"total":95995035648,"available":95988097024,"internally_reserved":0,"allocated":966656,"data_stored":481027,"data_compressed":17590,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.378,"15min":0.36399999999999999},"min":{"1min":0.21099999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.47899999999999998,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.39400000000000002,"15min":0.38600000000000001},"min":{"1min":0.26500000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.59499999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.68500000000000005}]},{"osd":1,"last update":"Thu Nov 16 18:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55000000000000004,"15min":0.52400000000000002},"min":{"1min":0.35599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.77400000000000002,"5min":1.129,"15min":1.129},"last":0.495},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56399999999999995,"15min":0.53800000000000003},"min":{"1min":0.434,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.54800000000000004}]},{"osd":2,"last update":"Thu Nov 16 18:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59899999999999998,"15min":0.56899999999999995},"min":{"1min":0.42699999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83399999999999996,"5min":1.038,"15min":1.038},"last":0.628},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.58199999999999996,"15min":0.56999999999999995},"min":{"1min":0.371,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73399999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.53000000000000003}]},{"osd":4,"last update":"Thu Nov 16 18:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63500000000000001,"15min":0.60299999999999998},"min":{"1min":0.498,"5min":0.378,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":0.628},"min":{"1min":0.439,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79800000000000004,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Thu Nov 16 18:23:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.60299999999999998,"15min":0.61299999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70799999999999996,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.63100000000000001,"15min":0.623},"min":{"1min":0.45400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.81399999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.58699999999999997}]},{"osd":6,"last update":"Thu Nov 16 18:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.54200000000000004,"15min":0.49399999999999999},"min":{"1min":0.38600000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.63500000000000001,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.30199999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.72199999999999998,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.46300000000000002}]},{"osd":7,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.44600000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81399999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69599999999999995,"15min":0.68600000000000005},"min":{"1min":0.46700000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81499999999999995,"5min":1.087,"15min":1.087},"last":0.56699999999999995}]},{"osd":8,"last update":"Thu Nov 16 18:22:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.60799999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.752,"15min":0.81000000000000005},"min":{"1min":0.39000000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79700000000000004,"5min":1.135,"15min":1.135},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.70099999999999996,"15min":0.747},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.748,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.63600000000000001}]},{"osd":10,"last update":"Thu Nov 16 18:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.77000000000000002,"15min":0.78800000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.77600000000000002,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.82699999999999996,"15min":0.85199999999999998},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.81799999999999995,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.67000000000000004}]},{"osd":11,"last update":"Thu Nov 16 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.69899999999999995}]}]},{"osd":4,"up_from":32,"seq":137438953537,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6608,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738544,"statfs":{"total":95995035648,"available":95988269056,"internally_reserved":0,"allocated":991232,"data_stored":546980,"data_compressed":27217,"data_compressed_allocated":36864,"data_compressed_original":106496,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:22:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.504,"15min":0.48699999999999999},"min":{"1min":0.39900000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.752,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.48899999999999999,"15min":0.46400000000000002},"min":{"1min":0.41799999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.502}]},{"osd":1,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40600000000000003,"15min":0.377},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.56299999999999994,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.501},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.439,"15min":0.41299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.60699999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.52500000000000002}]},{"osd":2,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59199999999999997,"15min":0.56599999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85399999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.61699999999999999,"15min":0.57499999999999996},"min":{"1min":0.51500000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83599999999999997,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.68200000000000005}]},{"osd":3,"last update":"Thu Nov 16 18:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57199999999999995,"15min":0.54000000000000004},"min":{"1min":0.40100000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.03,"5min":0.64800000000000002,"15min":0.56200000000000006},"min":{"1min":0.40300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":10.048,"5min":10.048,"15min":10.048},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Nov 16 18:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.66400000000000003,"15min":0.61799999999999999},"min":{"1min":0.47599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.88300000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66500000000000004,"15min":0.63900000000000001},"min":{"1min":0.48399999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.99299999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.65400000000000003}]},{"osd":6,"last update":"Thu Nov 16 18:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.61599999999999999,"15min":0.59399999999999997},"min":{"1min":0.47799999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84899999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.68000000000000005,"15min":0.64400000000000002},"min":{"1min":0.53900000000000003,"5min":0.432,"15min":0.432},"max":{"1min":0.93200000000000005,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.86599999999999999}]},{"osd":7,"last update":"Thu Nov 16 18:23:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.626,"15min":0.57599999999999996},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95199999999999996,"5min":1.077,"15min":1.077},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56899999999999995,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76900000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63200000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.72499999999999998,"15min":0.72099999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98299999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73599999999999999,"15min":0.73199999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95499999999999996,"5min":1.109,"15min":1.109},"last":0.72699999999999998}]},{"osd":9,"last update":"Thu Nov 16 18:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.751,"15min":0.76100000000000001},"min":{"1min":0.59099999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.92400000000000004,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80600000000000005,"15min":0.80900000000000005},"min":{"1min":0.65000000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.96999999999999997,"5min":1.117,"15min":1.117},"last":0.94499999999999995}]},{"osd":10,"last update":"Thu Nov 16 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.63400000000000001,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":0.997,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78200000000000003,"15min":0.78500000000000003},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96799999999999997,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.83899999999999997}]},{"osd":11,"last update":"Thu Nov 16 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.88100000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790012,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6716,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738436,"statfs":{"total":95995035648,"available":95988158464,"internally_reserved":0,"allocated":1036288,"data_stored":1077616,"data_compressed":53759,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51900000000000002,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.69899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.59399999999999997,"15min":0.54000000000000004},"min":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76100000000000001,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.432}]},{"osd":1,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.54600000000000004,"15min":0.53300000000000003},"min":{"1min":0.41799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.83499999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.55100000000000005,"15min":0.52200000000000002},"min":{"1min":0.34599999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79000000000000004,"5min":0.873,"15min":0.873},"last":0.70599999999999996}]},{"osd":2,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.39400000000000002},"min":{"1min":0.312,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58599999999999997,"5min":0.626,"15min":0.626},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.41599999999999998,"15min":0.38600000000000001},"min":{"1min":0.316,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70399999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.64500000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55200000000000005,"15min":0.54200000000000004},"min":{"1min":0.42399999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.72499999999999998,"5min":0.88,"15min":0.88},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Nov 16 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.58999999999999997,"15min":0.57199999999999995},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.85999999999999999,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.58499999999999996,"15min":0.56899999999999995},"min":{"1min":0.47099999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.872,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Nov 16 18:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.46899999999999997,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81100000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.46300000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77500000000000002,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.54600000000000004}]},{"osd":7,"last update":"Thu Nov 16 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.64000000000000001,"15min":0.629},"min":{"1min":0.48699999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.63800000000000001,"15min":0.625},"min":{"1min":0.44800000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.89600000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.75900000000000001}]},{"osd":8,"last update":"Thu Nov 16 18:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.41399999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.45200000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.77600000000000002,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66400000000000003}]},{"osd":9,"last update":"Thu Nov 16 18:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66500000000000004,"15min":0.66100000000000003},"min":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.79400000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70299999999999996,"15min":0.70199999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.86899999999999999,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.77300000000000002}]},{"osd":10,"last update":"Thu Nov 16 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.72699999999999998,"15min":0.71599999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.70499999999999996,"15min":0.69299999999999995},"min":{"1min":0.53800000000000003,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93799999999999994,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.79100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34529,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36169,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":62141,"data_compressed":9627,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T18:24:39.512 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:24:39.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.667+0000 7f6d04c45700 1 -- 172.21.15.2:0/90707411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d000ff430 msgr2=0x7f6d000ff8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:39.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.667+0000 7f6d04c45700 1 --2- 172.21.15.2:0/90707411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d000ff430 0x7f6d000ff8a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009990 tx=0x7f6ce8009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.667+0000 7f6d04c45700 1 -- 172.21.15.2:0/90707411 shutdown_connections 2023-11-16T18:24:39.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.667+0000 7f6d04c45700 1 --2- 172.21.15.2:0/90707411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d000ffde0 0x7f6d00107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.667+0000 7f6d04c45700 1 --2- 172.21.15.2:0/90707411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d001007e0 0x7f6d00100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.667+0000 7f6d04c45700 1 --2- 172.21.15.2:0/90707411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d000ff430 0x7f6d000ff8a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.667+0000 7f6d04c45700 1 -- 172.21.15.2:0/90707411 >> 172.21.15.2:0/90707411 conn(0x7f6d000f9f50 msgr2=0x7f6d000fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:39.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.667+0000 7f6d04c45700 1 -- 172.21.15.2:0/90707411 shutdown_connections 2023-11-16T18:24:39.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.667+0000 7f6d04c45700 1 -- 172.21.15.2:0/90707411 wait complete. 2023-11-16T18:24:39.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.668+0000 7f6d04c45700 1 Processor -- start 2023-11-16T18:24:39.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.668+0000 7f6d04c45700 1 -- start start 2023-11-16T18:24:39.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.668+0000 7f6d04c45700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d000ff430 0x7f6d00196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:39.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.668+0000 7f6d04c45700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d000ffde0 0x7f6d001965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:39.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.668+0000 7f6d04c45700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d001007e0 0x7f6d0019d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:39.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.668+0000 7f6d04c45700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0019dbe0 con 0x7f6d000ffde0 2023-11-16T18:24:39.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.668+0000 7f6d04c45700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0019dd20 con 0x7f6d000ff430 2023-11-16T18:24:39.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.668+0000 7f6d04c45700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0019deb0 con 0x7f6d001007e0 2023-11-16T18:24:39.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.668+0000 7f6cfdd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d000ffde0 0x7f6d001965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:39.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.668+0000 7f6cfdd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d000ffde0 0x7f6d001965a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49754/0 (socket says 172.21.15.2:49754) 2023-11-16T18:24:39.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.668+0000 7f6cfdd9b700 1 -- 172.21.15.2:0/1580774945 learned_addr learned my addr 172.21.15.2:0/1580774945 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:39.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.668+0000 7f6cfed9d700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d001007e0 0x7f6d0019d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:39.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.668+0000 7f6cfe59c700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d000ff430 0x7f6d00196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:39.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.669+0000 7f6cfdd9b700 1 -- 172.21.15.2:0/1580774945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d001007e0 msgr2=0x7f6d0019d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:39.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.669+0000 7f6cfdd9b700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d001007e0 0x7f6d0019d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.669+0000 7f6cfdd9b700 1 -- 172.21.15.2:0/1580774945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d000ff430 msgr2=0x7f6d00196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:39.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.669+0000 7f6cfdd9b700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d000ff430 0x7f6d00196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.669+0000 7f6cfdd9b700 1 -- 172.21.15.2:0/1580774945 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce80095f0 con 0x7f6d000ffde0 2023-11-16T18:24:39.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.669+0000 7f6cfdd9b700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d000ffde0 0x7f6d001965a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8004730 tx=0x7f6ce8004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:39.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.669+0000 7f6cf77fe700 1 -- 172.21.15.2:0/1580774945 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce801d070 con 0x7f6d000ffde0 2023-11-16T18:24:39.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.669+0000 7f6cf77fe700 1 -- 172.21.15.2:0/1580774945 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ce8004c90 con 0x7f6d000ffde0 2023-11-16T18:24:39.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.669+0000 7f6d04c45700 1 -- 172.21.15.2:0/1580774945 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d0010abc0 con 0x7f6d000ffde0 2023-11-16T18:24:39.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.669+0000 7f6cf77fe700 1 -- 172.21.15.2:0/1580774945 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce80054b0 con 0x7f6d000ffde0 2023-11-16T18:24:39.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.669+0000 7f6d04c45700 1 -- 172.21.15.2:0/1580774945 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d0010aa30 con 0x7f6d000ffde0 2023-11-16T18:24:39.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.670+0000 7f6d04c45700 1 -- 172.21.15.2:0/1580774945 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d001072e0 con 0x7f6d000ffde0 2023-11-16T18:24:39.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.671+0000 7f6cf77fe700 1 -- 172.21.15.2:0/1580774945 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f6ce800fbc0 con 0x7f6d000ffde0 2023-11-16T18:24:39.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.672+0000 7f6cf77fe700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6cec0a1150 0x7f6cec0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:39.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.672+0000 7f6cfe59c700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6cec0a1150 0x7f6cec0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:39.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.672+0000 7f6cf77fe700 1 -- 172.21.15.2:0/1580774945 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f6ce80cdf40 con 0x7f6d000ffde0 2023-11-16T18:24:39.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.672+0000 7f6cfe59c700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6cec0a1150 0x7f6cec0a3600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6d00067ba0 tx=0x7f6cf000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:39.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.676+0000 7f6cf77fe700 1 -- 172.21.15.2:0/1580774945 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6ce805a730 con 0x7f6d000ffde0 2023-11-16T18:24:39.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:39 smithi002 ceph-mon[98807]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 340 B/s wr, 5 op/s 2023-11-16T18:24:39.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:39 smithi002 ceph-mon[98807]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:24:39.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2023-11-16T18:24:39.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 8]}]': finished 2023-11-16T18:24:39.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2023-11-16T18:24:39.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:39 smithi002 ceph-mon[98807]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2023-11-16T18:24:39.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:39 smithi002 ceph-mon[98807]: osdmap e89: 12 total, 12 up, 12 in 2023-11-16T18:24:39.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:39 smithi002 ceph-mon[98807]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-11-16T18:24:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.882+0000 7f6d04c45700 1 -- 172.21.15.2:0/1580774945 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d00061950 con 0x7f6d000ffde0 2023-11-16T18:24:39.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.884+0000 7f6cf77fe700 1 -- 172.21.15.2:0/1580774945 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24570 (secure 0 0 0) 0x7f6ce8097360 con 0x7f6d000ffde0 2023-11-16T18:24:39.886 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:39.887 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":90,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:24:39.813936+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:24:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.886+0000 7f6d04c45700 1 -- 172.21.15.2:0/1580774945 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6cec0a1150 msgr2=0x7f6cec0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.886+0000 7f6d04c45700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6cec0a1150 0x7f6cec0a3600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6d00067ba0 tx=0x7f6cf000a380 comp rx=0 tx=0).stop 2023-11-16T18:24:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.886+0000 7f6d04c45700 1 -- 172.21.15.2:0/1580774945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d000ffde0 msgr2=0x7f6d001965a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.886+0000 7f6d04c45700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d000ffde0 0x7f6d001965a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8004730 tx=0x7f6ce8004760 comp rx=0 tx=0).stop 2023-11-16T18:24:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.886+0000 7f6d04c45700 1 -- 172.21.15.2:0/1580774945 shutdown_connections 2023-11-16T18:24:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.886+0000 7f6d04c45700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f6cec0a1150 0x7f6cec0a3600 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.886+0000 7f6d04c45700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d001007e0 0x7f6d0019d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.886+0000 7f6d04c45700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d000ff430 0x7f6d00196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.886+0000 7f6d04c45700 1 --2- 172.21.15.2:0/1580774945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d000ffde0 0x7f6d001965a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.886+0000 7f6d04c45700 1 -- 172.21.15.2:0/1580774945 >> 172.21.15.2:0/1580774945 conn(0x7f6d000f9f50 msgr2=0x7f6d000fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.886+0000 7f6d04c45700 1 -- 172.21.15.2:0/1580774945 shutdown_connections 2023-11-16T18:24:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:39.886+0000 7f6d04c45700 1 -- 172.21.15.2:0/1580774945 wait complete. 2023-11-16T18:24:39.901 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T18:22:44.902889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T18:22:46.187275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T18:22:49.539148+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T18:24:39.902 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T18:24:40.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.029+0000 7f2468786700 1 -- 172.21.15.2:0/3822151479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24601033a0 msgr2=0x7f24601057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.029+0000 7f2468786700 1 --2- 172.21.15.2:0/3822151479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24601033a0 0x7f24601057c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2450009990 tx=0x7f2450009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.029+0000 7f2468786700 1 -- 172.21.15.2:0/3822151479 shutdown_connections 2023-11-16T18:24:40.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.029+0000 7f2468786700 1 --2- 172.21.15.2:0/3822151479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2460105e50 0x7f2460108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.029+0000 7f2468786700 1 --2- 172.21.15.2:0/3822151479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2460068aa0 0x7f2460102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.029+0000 7f2468786700 1 --2- 172.21.15.2:0/3822151479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24601033a0 0x7f24601057c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.029+0000 7f2468786700 1 -- 172.21.15.2:0/3822151479 >> 172.21.15.2:0/3822151479 conn(0x7f24600fa750 msgr2=0x7f24600fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.029+0000 7f2468786700 1 -- 172.21.15.2:0/3822151479 shutdown_connections 2023-11-16T18:24:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.029+0000 7f2468786700 1 -- 172.21.15.2:0/3822151479 wait complete. 2023-11-16T18:24:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.029+0000 7f2468786700 1 Processor -- start 2023-11-16T18:24:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2468786700 1 -- start start 2023-11-16T18:24:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2468786700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2460068aa0 0x7f24601007c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2468786700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24601033a0 0x7f24600fee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2468786700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2460105e50 0x7f24600ff340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2468786700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24600ffae0 con 0x7f24601033a0 2023-11-16T18:24:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2468786700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24600ffc50 con 0x7f2460068aa0 2023-11-16T18:24:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2468786700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246019ffa0 con 0x7f2460105e50 2023-11-16T18:24:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2465d21700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24601033a0 0x7f24600fee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2465d21700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24601033a0 0x7f24600fee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49768/0 (socket says 172.21.15.2:49768) 2023-11-16T18:24:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2465d21700 1 -- 172.21.15.2:0/3079358289 learned_addr learned my addr 172.21.15.2:0/3079358289 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2466522700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2460068aa0 0x7f24601007c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2466d23700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2460105e50 0x7f24600ff340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2465d21700 1 -- 172.21.15.2:0/3079358289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2460105e50 msgr2=0x7f24600ff340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2465d21700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2460105e50 0x7f24600ff340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2465d21700 1 -- 172.21.15.2:0/3079358289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2460068aa0 msgr2=0x7f24601007c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2465d21700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2460068aa0 0x7f24601007c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.038 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2465d21700 1 -- 172.21.15.2:0/3079358289 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24500095f0 con 0x7f24601033a0 2023-11-16T18:24:40.038 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2466522700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2460068aa0 0x7f24601007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:40.038 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2465d21700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24601033a0 0x7f24600fee00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f24500059d0 tx=0x7f245000f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:40.038 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f24577fe700 1 -- 172.21.15.2:0/3079358289 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245001d070 con 0x7f24601033a0 2023-11-16T18:24:40.038 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f2468786700 1 -- 172.21.15.2:0/3079358289 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246010bbf0 con 0x7f24601033a0 2023-11-16T18:24:40.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f24577fe700 1 -- 172.21.15.2:0/3079358289 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f245000fd60 con 0x7f24601033a0 2023-11-16T18:24:40.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.030+0000 7f24577fe700 1 -- 172.21.15.2:0/3079358289 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2450017650 con 0x7f24601033a0 2023-11-16T18:24:40.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.031+0000 7f2468786700 1 -- 172.21.15.2:0/3079358289 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246010ba60 con 0x7f24601033a0 2023-11-16T18:24:40.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.032+0000 7f2468786700 1 -- 172.21.15.2:0/3079358289 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24601a0540 con 0x7f24601033a0 2023-11-16T18:24:40.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.033+0000 7f24577fe700 1 -- 172.21.15.2:0/3079358289 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f2450004830 con 0x7f24601033a0 2023-11-16T18:24:40.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.034+0000 7f24577fe700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f244c0a1150 0x7f244c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:40.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.034+0000 7f2466522700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f244c0a1150 0x7f244c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:40.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.034+0000 7f24577fe700 1 -- 172.21.15.2:0/3079358289 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f24500cea50 con 0x7f24601033a0 2023-11-16T18:24:40.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.034+0000 7f2466522700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f244c0a1150 0x7f244c0a3600 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f245c005fd0 tx=0x7f245c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:40.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.037+0000 7f24577fe700 1 -- 172.21.15.2:0/3079358289 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2450097eb0 con 0x7f24601033a0 2023-11-16T18:24:40.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:39 smithi040 ceph-mon[100251]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 340 B/s wr, 5 op/s 2023-11-16T18:24:40.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:39 smithi040 ceph-mon[100251]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:24:40.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2023-11-16T18:24:40.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 8]}]': finished 2023-11-16T18:24:40.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2023-11-16T18:24:40.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:39 smithi040 ceph-mon[100251]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2023-11-16T18:24:40.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:39 smithi040 ceph-mon[100251]: osdmap e89: 12 total, 12 up, 12 in 2023-11-16T18:24:40.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:39 smithi040 ceph-mon[100251]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-11-16T18:24:40.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.238+0000 7f2468786700 1 -- 172.21.15.2:0/3079358289 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f24601a0c20 con 0x7f24601033a0 2023-11-16T18:24:40.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.238+0000 7f24577fe700 1 -- 172.21.15.2:0/3079358289 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f245009b4d0 con 0x7f24601033a0 2023-11-16T18:24:40.243 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:24:40.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.240+0000 7f2468786700 1 -- 172.21.15.2:0/3079358289 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f244c0a1150 msgr2=0x7f244c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.240+0000 7f2468786700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f244c0a1150 0x7f244c0a3600 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f245c005fd0 tx=0x7f245c005e30 comp rx=0 tx=0).stop 2023-11-16T18:24:40.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.240+0000 7f2468786700 1 -- 172.21.15.2:0/3079358289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24601033a0 msgr2=0x7f24600fee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.240+0000 7f2468786700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24601033a0 0x7f24600fee00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f24500059d0 tx=0x7f245000f510 comp rx=0 tx=0).stop 2023-11-16T18:24:40.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.240+0000 7f2468786700 1 -- 172.21.15.2:0/3079358289 shutdown_connections 2023-11-16T18:24:40.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.240+0000 7f2468786700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f244c0a1150 0x7f244c0a3600 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.240+0000 7f2468786700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2460105e50 0x7f24600ff340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.240+0000 7f2468786700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2460068aa0 0x7f24601007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.240+0000 7f2468786700 1 --2- 172.21.15.2:0/3079358289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24601033a0 0x7f24600fee00 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.240+0000 7f2468786700 1 -- 172.21.15.2:0/3079358289 >> 172.21.15.2:0/3079358289 conn(0x7f24600fa750 msgr2=0x7f2460101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:40.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.240+0000 7f2468786700 1 -- 172.21.15.2:0/3079358289 shutdown_connections 2023-11-16T18:24:40.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.241+0000 7f2468786700 1 -- 172.21.15.2:0/3079358289 wait complete. 2023-11-16T18:24:40.255 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T18:24:40.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:39 smithi188 ceph-mon[100639]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 340 B/s wr, 5 op/s 2023-11-16T18:24:40.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:39 smithi188 ceph-mon[100639]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:24:40.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2023-11-16T18:24:40.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 8]}]': finished 2023-11-16T18:24:40.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2023-11-16T18:24:40.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:39 smithi188 ceph-mon[100639]: from='mgr.14152 172.21.15.2:0/3695340816' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2023-11-16T18:24:40.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:39 smithi188 ceph-mon[100639]: osdmap e89: 12 total, 12 up, 12 in 2023-11-16T18:24:40.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:39 smithi188 ceph-mon[100639]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-11-16T18:24:40.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.382+0000 7fed1afb0700 1 -- 172.21.15.2:0/1022683198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed140ffe70 msgr2=0x7fed141002e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.382+0000 7fed1afb0700 1 --2- 172.21.15.2:0/1022683198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed140ffe70 0x7fed141002e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00b0a0 tx=0x7fed0c00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 -- 172.21.15.2:0/1022683198 shutdown_connections 2023-11-16T18:24:40.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 --2- 172.21.15.2:0/1022683198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed140ffe70 0x7fed141002e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 --2- 172.21.15.2:0/1022683198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed140feb70 0x7fed140fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 --2- 172.21.15.2:0/1022683198 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed140ff4c0 0x7fed140ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 -- 172.21.15.2:0/1022683198 >> 172.21.15.2:0/1022683198 conn(0x7fed140fa7a0 msgr2=0x7fed140fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:40.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 -- 172.21.15.2:0/1022683198 shutdown_connections 2023-11-16T18:24:40.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 -- 172.21.15.2:0/1022683198 wait complete. 2023-11-16T18:24:40.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 Processor -- start 2023-11-16T18:24:40.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 -- start start 2023-11-16T18:24:40.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed140feb70 0x7fed141982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:40.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed140ff4c0 0x7fed141987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:40.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed140ffe70 0x7fed1419f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:40.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1419fe30 con 0x7fed140ff4c0 2023-11-16T18:24:40.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1419ff70 con 0x7fed140feb70 2023-11-16T18:24:40.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.383+0000 7fed1afb0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed141a0100 con 0x7fed140ffe70 2023-11-16T18:24:40.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed13fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed140ff4c0 0x7fed141987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:40.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed13fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed140ff4c0 0x7fed141987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49782/0 (socket says 172.21.15.2:49782) 2023-11-16T18:24:40.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed13fff700 1 -- 172.21.15.2:0/1133418390 learned_addr learned my addr 172.21.15.2:0/1133418390 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:40.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed18d4c700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed140feb70 0x7fed141982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:40.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed1954d700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed140ffe70 0x7fed1419f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:40.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed13fff700 1 -- 172.21.15.2:0/1133418390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed140ffe70 msgr2=0x7fed1419f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed13fff700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed140ffe70 0x7fed1419f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed13fff700 1 -- 172.21.15.2:0/1133418390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed140feb70 msgr2=0x7fed141982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed13fff700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed140feb70 0x7fed141982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed13fff700 1 -- 172.21.15.2:0/1133418390 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c009d00 con 0x7fed140ff4c0 2023-11-16T18:24:40.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed13fff700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed140ff4c0 0x7fed141987f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fed0000d950 tx=0x7fed0000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:40.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed11ffb700 1 -- 172.21.15.2:0/1133418390 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0000f890 con 0x7fed140ff4c0 2023-11-16T18:24:40.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed11ffb700 1 -- 172.21.15.2:0/1133418390 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed00010460 con 0x7fed140ff4c0 2023-11-16T18:24:40.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed11ffb700 1 -- 172.21.15.2:0/1133418390 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0000e610 con 0x7fed140ff4c0 2023-11-16T18:24:40.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.384+0000 7fed1afb0700 1 -- 172.21.15.2:0/1133418390 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1410de40 con 0x7fed140ff4c0 2023-11-16T18:24:40.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.385+0000 7fed1afb0700 1 -- 172.21.15.2:0/1133418390 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1410dcb0 con 0x7fed140ff4c0 2023-11-16T18:24:40.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.386+0000 7fed1afb0700 1 -- 172.21.15.2:0/1133418390 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed14109950 con 0x7fed140ff4c0 2023-11-16T18:24:40.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.387+0000 7fed11ffb700 1 -- 172.21.15.2:0/1133418390 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fed000108d0 con 0x7fed140ff4c0 2023-11-16T18:24:40.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.387+0000 7fed11ffb700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fed040a1150 0x7fed040a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:40.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.387+0000 7fed11ffb700 1 -- 172.21.15.2:0/1133418390 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fed000cc9d0 con 0x7fed140ff4c0 2023-11-16T18:24:40.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.387+0000 7fed18d4c700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fed040a1150 0x7fed040a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:40.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.388+0000 7fed18d4c700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fed040a1150 0x7fed040a3600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fed08007c60 tx=0x7fed080073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:40.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.392+0000 7fed11ffb700 1 -- 172.21.15.2:0/1133418390 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fed00095da0 con 0x7fed140ff4c0 2023-11-16T18:24:40.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.592+0000 7fed1afb0700 1 -- 172.21.15.2:0/1133418390 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed14061700 con 0x7fed140ff4c0 2023-11-16T18:24:40.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.592+0000 7fed11ffb700 1 -- 172.21.15.2:0/1133418390 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7fed00016020 con 0x7fed140ff4c0 2023-11-16T18:24:40.594 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:24:40.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.594+0000 7fed1afb0700 1 -- 172.21.15.2:0/1133418390 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fed040a1150 msgr2=0x7fed040a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.594+0000 7fed1afb0700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fed040a1150 0x7fed040a3600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fed08007c60 tx=0x7fed080073d0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.594+0000 7fed1afb0700 1 -- 172.21.15.2:0/1133418390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed140ff4c0 msgr2=0x7fed141987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.594+0000 7fed1afb0700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed140ff4c0 0x7fed141987f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fed0000d950 tx=0x7fed0000dc60 comp rx=0 tx=0).stop 2023-11-16T18:24:40.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.594+0000 7fed1afb0700 1 -- 172.21.15.2:0/1133418390 shutdown_connections 2023-11-16T18:24:40.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.594+0000 7fed1afb0700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fed040a1150 0x7fed040a3600 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.594+0000 7fed1afb0700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed140ffe70 0x7fed1419f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.594+0000 7fed1afb0700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed140feb70 0x7fed141982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.594+0000 7fed1afb0700 1 --2- 172.21.15.2:0/1133418390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed140ff4c0 0x7fed141987f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.594+0000 7fed1afb0700 1 -- 172.21.15.2:0/1133418390 >> 172.21.15.2:0/1133418390 conn(0x7fed140fa7a0 msgr2=0x7fed140fc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:40.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.594+0000 7fed1afb0700 1 -- 172.21.15.2:0/1133418390 shutdown_connections 2023-11-16T18:24:40.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.594+0000 7fed1afb0700 1 -- 172.21.15.2:0/1133418390 wait complete. 2023-11-16T18:24:40.608 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T18:24:40.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.735+0000 7fec4ed03700 1 -- 172.21.15.2:0/1946863921 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec48100d20 msgr2=0x7fec481075e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.735+0000 7fec4ed03700 1 --2- 172.21.15.2:0/1946863921 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec48100d20 0x7fec481075e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fec4000b0a0 tx=0x7fec4000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.735+0000 7fec4ed03700 1 -- 172.21.15.2:0/1946863921 shutdown_connections 2023-11-16T18:24:40.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.735+0000 7fec4ed03700 1 --2- 172.21.15.2:0/1946863921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec480ff1a0 0x7fec480ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.735+0000 7fec4ed03700 1 --2- 172.21.15.2:0/1946863921 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec48100d20 0x7fec481075e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.735+0000 7fec4ed03700 1 --2- 172.21.15.2:0/1946863921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec48100390 0x7fec481007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.735+0000 7fec4ed03700 1 -- 172.21.15.2:0/1946863921 >> 172.21.15.2:0/1946863921 conn(0x7fec480fa7a0 msgr2=0x7fec480fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:40.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.735+0000 7fec4ed03700 1 -- 172.21.15.2:0/1946863921 shutdown_connections 2023-11-16T18:24:40.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.735+0000 7fec4ed03700 1 -- 172.21.15.2:0/1946863921 wait complete. 2023-11-16T18:24:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ed03700 1 Processor -- start 2023-11-16T18:24:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ed03700 1 -- start start 2023-11-16T18:24:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ed03700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec480ff1a0 0x7fec48194fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ed03700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec48100390 0x7fec481954f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ed03700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec48100d20 0x7fec4819c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ed03700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4819cb30 con 0x7fec480ff1a0 2023-11-16T18:24:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ed03700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4819cc70 con 0x7fec48100d20 2023-11-16T18:24:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ed03700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4819ce00 con 0x7fec48100390 2023-11-16T18:24:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ca9f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec480ff1a0 0x7fec48194fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ca9f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec480ff1a0 0x7fec48194fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49794/0 (socket says 172.21.15.2:49794) 2023-11-16T18:24:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ca9f700 1 -- 172.21.15.2:0/1543216555 learned_addr learned my addr 172.21.15.2:0/1543216555 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec47fff700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec48100390 0x7fec481954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4d2a0700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec48100d20 0x7fec4819c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ca9f700 1 -- 172.21.15.2:0/1543216555 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec48100390 msgr2=0x7fec481954f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ca9f700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec48100390 0x7fec481954f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ca9f700 1 -- 172.21.15.2:0/1543216555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec48100d20 msgr2=0x7fec4819c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ca9f700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec48100d20 0x7fec4819c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ca9f700 1 -- 172.21.15.2:0/1543216555 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec40009d00 con 0x7fec480ff1a0 2023-11-16T18:24:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4d2a0700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec48100d20 0x7fec4819c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.736+0000 7fec4ca9f700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec480ff1a0 0x7fec48194fb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00e970 tx=0x7fec3c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.737+0000 7fec45ffb700 1 -- 172.21.15.2:0/1543216555 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c009790 con 0x7fec480ff1a0 2023-11-16T18:24:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.737+0000 7fec45ffb700 1 -- 172.21.15.2:0/1543216555 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec3c00eec0 con 0x7fec480ff1a0 2023-11-16T18:24:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.737+0000 7fec4ed03700 1 -- 172.21.15.2:0/1543216555 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec4810ab60 con 0x7fec480ff1a0 2023-11-16T18:24:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.737+0000 7fec45ffb700 1 -- 172.21.15.2:0/1543216555 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c003cf0 con 0x7fec480ff1a0 2023-11-16T18:24:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.737+0000 7fec4ed03700 1 -- 172.21.15.2:0/1543216555 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4810a9d0 con 0x7fec480ff1a0 2023-11-16T18:24:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.738+0000 7fec4ed03700 1 -- 172.21.15.2:0/1543216555 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec48106710 con 0x7fec480ff1a0 2023-11-16T18:24:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.739+0000 7fec45ffb700 1 -- 172.21.15.2:0/1543216555 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fec3c00ccd0 con 0x7fec480ff1a0 2023-11-16T18:24:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.739+0000 7fec45ffb700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fec380a11d0 0x7fec380a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.739+0000 7fec45ffb700 1 -- 172.21.15.2:0/1543216555 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fec3c014070 con 0x7fec480ff1a0 2023-11-16T18:24:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.739+0000 7fec47fff700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fec380a11d0 0x7fec380a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.740+0000 7fec47fff700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fec380a11d0 0x7fec380a3680 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fec34009c60 tx=0x7fec34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.744+0000 7fec45ffb700 1 -- 172.21.15.2:0/1543216555 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fec3c0d3080 con 0x7fec480ff1a0 2023-11-16T18:24:40.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.942+0000 7fec4ed03700 1 -- 172.21.15.2:0/1543216555 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fec48061b00 con 0x7fec480ff1a0 2023-11-16T18:24:40.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.942+0000 7fec45ffb700 1 -- 172.21.15.2:0/1543216555 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fec3c096210 con 0x7fec480ff1a0 2023-11-16T18:24:40.946 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T18:24:40.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.944+0000 7fec4ed03700 1 -- 172.21.15.2:0/1543216555 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fec380a11d0 msgr2=0x7fec380a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.944+0000 7fec4ed03700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fec380a11d0 0x7fec380a3680 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fec34009c60 tx=0x7fec34009260 comp rx=0 tx=0).stop 2023-11-16T18:24:40.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.944+0000 7fec4ed03700 1 -- 172.21.15.2:0/1543216555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec480ff1a0 msgr2=0x7fec48194fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:40.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.944+0000 7fec4ed03700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec480ff1a0 0x7fec48194fb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00e970 tx=0x7fec3c00ec80 comp rx=0 tx=0).stop 2023-11-16T18:24:40.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.944+0000 7fec4ed03700 1 -- 172.21.15.2:0/1543216555 shutdown_connections 2023-11-16T18:24:40.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.944+0000 7fec4ed03700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fec380a11d0 0x7fec380a3680 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.944+0000 7fec4ed03700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec48100d20 0x7fec4819c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.944+0000 7fec4ed03700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec480ff1a0 0x7fec48194fb0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.944+0000 7fec4ed03700 1 --2- 172.21.15.2:0/1543216555 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec48100390 0x7fec481954f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:40.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.944+0000 7fec4ed03700 1 -- 172.21.15.2:0/1543216555 >> 172.21.15.2:0/1543216555 conn(0x7fec480fa7a0 msgr2=0x7fec48103850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:40.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.945+0000 7fec4ed03700 1 -- 172.21.15.2:0/1543216555 shutdown_connections 2023-11-16T18:24:40.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:40.945+0000 7fec4ed03700 1 -- 172.21.15.2:0/1543216555 wait complete. 2023-11-16T18:24:40.958 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:40 smithi002 ceph-mon[98807]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:24:40.958 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:40 smithi002 ceph-mon[98807]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:24:40.958 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:40 smithi002 ceph-mon[98807]: osdmap e90: 12 total, 12 up, 12 in 2023-11-16T18:24:40.958 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:40 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1580774945' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:40.958 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:40 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3079358289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:40.958 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:40 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1133418390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:40.959 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T18:24:41.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.086+0000 7f27728c9700 1 -- 172.21.15.2:0/1542697074 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f276c100db0 msgr2=0x7f276c107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:41.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.086+0000 7f27728c9700 1 --2- 172.21.15.2:0/1542697074 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f276c100db0 0x7f276c107650 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f276400b0a0 tx=0x7f276400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.086+0000 7f27728c9700 1 -- 172.21.15.2:0/1542697074 shutdown_connections 2023-11-16T18:24:41.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.086+0000 7f27728c9700 1 --2- 172.21.15.2:0/1542697074 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f276c100db0 0x7f276c107650 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.086+0000 7f27728c9700 1 --2- 172.21.15.2:0/1542697074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f276c0ff170 0x7f276c0ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.086+0000 7f27728c9700 1 --2- 172.21.15.2:0/1542697074 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f276c100400 0x7f276c100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.086+0000 7f27728c9700 1 -- 172.21.15.2:0/1542697074 >> 172.21.15.2:0/1542697074 conn(0x7f276c0fa770 msgr2=0x7f276c0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:41.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.086+0000 7f27728c9700 1 -- 172.21.15.2:0/1542697074 shutdown_connections 2023-11-16T18:24:41.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.086+0000 7f27728c9700 1 -- 172.21.15.2:0/1542697074 wait complete. 2023-11-16T18:24:41.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.086+0000 7f27728c9700 1 Processor -- start 2023-11-16T18:24:41.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.086+0000 7f27728c9700 1 -- start start 2023-11-16T18:24:41.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f27728c9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f276c0ff170 0x7f276c069c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:41.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f27728c9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f276c100400 0x7f276c19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:41.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f27728c9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f276c100db0 0x7f276c19d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:41.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f27728c9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276c19db10 con 0x7f276c0ff170 2023-11-16T18:24:41.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f27728c9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276c19dc80 con 0x7f276c100db0 2023-11-16T18:24:41.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f27728c9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276c19de40 con 0x7f276c100400 2023-11-16T18:24:41.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f276bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f276c0ff170 0x7f276c069c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:41.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f276bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f276c0ff170 0x7f276c069c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49808/0 (socket says 172.21.15.2:49808) 2023-11-16T18:24:41.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f276bfff700 1 -- 172.21.15.2:0/820959497 learned_addr learned my addr 172.21.15.2:0/820959497 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:41.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f276b7fe700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f276c100400 0x7f276c19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:41.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f2770e66700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f276c100db0 0x7f276c19d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:41.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f276bfff700 1 -- 172.21.15.2:0/820959497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f276c100400 msgr2=0x7f276c19ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:41.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f276bfff700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f276c100400 0x7f276c19ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f276bfff700 1 -- 172.21.15.2:0/820959497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f276c100db0 msgr2=0x7f276c19d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:41.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f276bfff700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f276c100db0 0x7f276c19d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f276bfff700 1 -- 172.21.15.2:0/820959497 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2764009d00 con 0x7f276c0ff170 2023-11-16T18:24:41.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f276bfff700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f276c0ff170 0x7f276c069c40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f275c00c9b0 tx=0x7f275c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:41.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.087+0000 7f27697fa700 1 -- 172.21.15.2:0/820959497 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275c007b00 con 0x7f276c0ff170 2023-11-16T18:24:41.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.088+0000 7f27697fa700 1 -- 172.21.15.2:0/820959497 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f275c007c60 con 0x7f276c0ff170 2023-11-16T18:24:41.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.088+0000 7f27728c9700 1 -- 172.21.15.2:0/820959497 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276c10ab10 con 0x7f276c0ff170 2023-11-16T18:24:41.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.088+0000 7f27697fa700 1 -- 172.21.15.2:0/820959497 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275c018660 con 0x7f276c0ff170 2023-11-16T18:24:41.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.088+0000 7f27728c9700 1 -- 172.21.15.2:0/820959497 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f276c10a980 con 0x7f276c0ff170 2023-11-16T18:24:41.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.088+0000 7f27728c9700 1 -- 172.21.15.2:0/820959497 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f276c1060e0 con 0x7f276c0ff170 2023-11-16T18:24:41.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.090+0000 7f27697fa700 1 -- 172.21.15.2:0/820959497 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f275c00f450 con 0x7f276c0ff170 2023-11-16T18:24:41.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.090+0000 7f27697fa700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f27580a1150 0x7f27580a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:41.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.090+0000 7f276b7fe700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f27580a1150 0x7f27580a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:41.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.090+0000 7f27697fa700 1 -- 172.21.15.2:0/820959497 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f275c0ccf20 con 0x7f276c0ff170 2023-11-16T18:24:41.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.091+0000 7f276b7fe700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f27580a1150 0x7f27580a3600 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2754005950 tx=0x7f275400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:41.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.094+0000 7f27697fa700 1 -- 172.21.15.2:0/820959497 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f275c096380 con 0x7f276c0ff170 2023-11-16T18:24:41.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:40 smithi040 ceph-mon[100251]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:24:41.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:40 smithi040 ceph-mon[100251]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:24:41.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:40 smithi040 ceph-mon[100251]: osdmap e90: 12 total, 12 up, 12 in 2023-11-16T18:24:41.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:40 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1580774945' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:41.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:40 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3079358289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:41.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:40 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1133418390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:41.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:40 smithi188 ceph-mon[100639]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:24:41.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:40 smithi188 ceph-mon[100639]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T18:24:41.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:40 smithi188 ceph-mon[100639]: osdmap e90: 12 total, 12 up, 12 in 2023-11-16T18:24:41.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:40 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1580774945' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:41.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:40 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3079358289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:24:41.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:40 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1133418390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:24:41.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.294+0000 7f27728c9700 1 -- 172.21.15.2:0/820959497 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f276c061900 con 0x7f276c0ff170 2023-11-16T18:24:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.295+0000 7f27697fa700 1 -- 172.21.15.2:0/820959497 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f275c0999a0 con 0x7f276c0ff170 2023-11-16T18:24:41.298 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:24:41.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.297+0000 7f27728c9700 1 -- 172.21.15.2:0/820959497 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f27580a1150 msgr2=0x7f27580a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:41.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.297+0000 7f27728c9700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f27580a1150 0x7f27580a3600 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2754005950 tx=0x7f275400b410 comp rx=0 tx=0).stop 2023-11-16T18:24:41.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.297+0000 7f27728c9700 1 -- 172.21.15.2:0/820959497 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f276c0ff170 msgr2=0x7f276c069c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:41.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.297+0000 7f27728c9700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f276c0ff170 0x7f276c069c40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f275c00c9b0 tx=0x7f275c00ccc0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.297+0000 7f27728c9700 1 -- 172.21.15.2:0/820959497 shutdown_connections 2023-11-16T18:24:41.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.297+0000 7f27728c9700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f27580a1150 0x7f27580a3600 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.297+0000 7f27728c9700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f276c100db0 0x7f276c19d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.297+0000 7f27728c9700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f276c0ff170 0x7f276c069c40 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.297+0000 7f27728c9700 1 --2- 172.21.15.2:0/820959497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f276c100400 0x7f276c19ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.297+0000 7f27728c9700 1 -- 172.21.15.2:0/820959497 >> 172.21.15.2:0/820959497 conn(0x7f276c0fa770 msgr2=0x7f276c0fca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:41.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.297+0000 7f27728c9700 1 -- 172.21.15.2:0/820959497 shutdown_connections 2023-11-16T18:24:41.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.297+0000 7f27728c9700 1 -- 172.21.15.2:0/820959497 wait complete. 2023-11-16T18:24:41.311 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:24:41.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.435+0000 7f5c8b094700 1 -- 172.21.15.2:0/2898808193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c84102060 msgr2=0x7f5c84108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:41.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.435+0000 7f5c8b094700 1 --2- 172.21.15.2:0/2898808193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c84102060 0x7f5c84108920 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00b0a0 tx=0x7f5c7c00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.435+0000 7f5c8b094700 1 -- 172.21.15.2:0/2898808193 shutdown_connections 2023-11-16T18:24:41.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.435+0000 7f5c8b094700 1 --2- 172.21.15.2:0/2898808193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c841004e0 0x7f5c841008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.435+0000 7f5c8b094700 1 --2- 172.21.15.2:0/2898808193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c84102060 0x7f5c84108920 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.435+0000 7f5c8b094700 1 --2- 172.21.15.2:0/2898808193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c841016d0 0x7f5c84101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.435+0000 7f5c8b094700 1 -- 172.21.15.2:0/2898808193 >> 172.21.15.2:0/2898808193 conn(0x7f5c840fbac0 msgr2=0x7f5c840fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:41.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.435+0000 7f5c8b094700 1 -- 172.21.15.2:0/2898808193 shutdown_connections 2023-11-16T18:24:41.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.435+0000 7f5c8b094700 1 -- 172.21.15.2:0/2898808193 wait complete. 2023-11-16T18:24:41.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.435+0000 7f5c8b094700 1 Processor -- start 2023-11-16T18:24:41.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.435+0000 7f5c8b094700 1 -- start start 2023-11-16T18:24:41.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c8b094700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c841004e0 0x7f5c84196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:41.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c8b094700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c841016d0 0x7f5c84196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:41.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c8b094700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c84102060 0x7f5c8419d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:41.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c8b094700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8419de80 con 0x7f5c841016d0 2023-11-16T18:24:41.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c8b094700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8419dfc0 con 0x7f5c841004e0 2023-11-16T18:24:41.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c8b094700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8419e150 con 0x7f5c84102060 2023-11-16T18:24:41.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c83fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c841016d0 0x7f5c84196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:41.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c83fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c841016d0 0x7f5c84196840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49824/0 (socket says 172.21.15.2:49824) 2023-11-16T18:24:41.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c83fff700 1 -- 172.21.15.2:0/814251523 learned_addr learned my addr 172.21.15.2:0/814251523 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:41.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c88e30700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c841004e0 0x7f5c84196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:41.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c89631700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c84102060 0x7f5c8419d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:41.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c83fff700 1 -- 172.21.15.2:0/814251523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c84102060 msgr2=0x7f5c8419d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:41.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c83fff700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c84102060 0x7f5c8419d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c83fff700 1 -- 172.21.15.2:0/814251523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c841004e0 msgr2=0x7f5c84196300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:41.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c83fff700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c841004e0 0x7f5c84196300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c83fff700 1 -- 172.21.15.2:0/814251523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c009d00 con 0x7f5c841016d0 2023-11-16T18:24:41.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c83fff700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c841016d0 0x7f5c84196840 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5c7000e9a0 tx=0x7f5c7000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:41.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c81ffb700 1 -- 172.21.15.2:0/814251523 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70009790 con 0x7f5c841016d0 2023-11-16T18:24:41.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c81ffb700 1 -- 172.21.15.2:0/814251523 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c7000cbd0 con 0x7f5c841016d0 2023-11-16T18:24:41.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c8b094700 1 -- 172.21.15.2:0/814251523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8410be50 con 0x7f5c841016d0 2023-11-16T18:24:41.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.436+0000 7f5c81ffb700 1 -- 172.21.15.2:0/814251523 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70018680 con 0x7f5c841016d0 2023-11-16T18:24:41.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.437+0000 7f5c8b094700 1 -- 172.21.15.2:0/814251523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8410bcc0 con 0x7f5c841016d0 2023-11-16T18:24:41.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.438+0000 7f5c8b094700 1 -- 172.21.15.2:0/814251523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8406a2f0 con 0x7f5c841016d0 2023-11-16T18:24:41.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.439+0000 7f5c81ffb700 1 -- 172.21.15.2:0/814251523 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f5c7000cd40 con 0x7f5c841016d0 2023-11-16T18:24:41.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.439+0000 7f5c81ffb700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5c740a1150 0x7f5c740a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:41.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.440+0000 7f5c88e30700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5c740a1150 0x7f5c740a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:41.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.440+0000 7f5c81ffb700 1 -- 172.21.15.2:0/814251523 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f5c70014070 con 0x7f5c841016d0 2023-11-16T18:24:41.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.440+0000 7f5c88e30700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5c740a1150 0x7f5c740a3600 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5c78005950 tx=0x7f5c780058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:41.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.443+0000 7f5c81ffb700 1 -- 172.21.15.2:0/814251523 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5c70096850 con 0x7f5c841016d0 2023-11-16T18:24:41.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.688+0000 7f5c8b094700 1 -- 172.21.15.2:0/814251523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c841072f0 con 0x7f5c841016d0 2023-11-16T18:24:41.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.688+0000 7f5c81ffb700 1 -- 172.21.15.2:0/814251523 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10774 (secure 0 0 0) 0x7f5c70099e70 con 0x7f5c841016d0 2023-11-16T18:24:41.690 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:41.690 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:23:07.909835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:24:41.693 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.690+0000 7f5c8b094700 1 -- 172.21.15.2:0/814251523 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5c740a1150 msgr2=0x7f5c740a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:41.693 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.691+0000 7f5c8b094700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5c740a1150 0x7f5c740a3600 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5c78005950 tx=0x7f5c780058e0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.693 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.691+0000 7f5c8b094700 1 -- 172.21.15.2:0/814251523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c841016d0 msgr2=0x7f5c84196840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:41.693 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.691+0000 7f5c8b094700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c841016d0 0x7f5c84196840 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5c7000e9a0 tx=0x7f5c7000ed60 comp rx=0 tx=0).stop 2023-11-16T18:24:41.693 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.691+0000 7f5c8b094700 1 -- 172.21.15.2:0/814251523 shutdown_connections 2023-11-16T18:24:41.693 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.691+0000 7f5c8b094700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f5c740a1150 0x7f5c740a3600 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.693 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.691+0000 7f5c8b094700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c84102060 0x7f5c8419d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.693 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.691+0000 7f5c8b094700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c841004e0 0x7f5c84196300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.693 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.691+0000 7f5c8b094700 1 --2- 172.21.15.2:0/814251523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c841016d0 0x7f5c84196840 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.691+0000 7f5c8b094700 1 -- 172.21.15.2:0/814251523 >> 172.21.15.2:0/814251523 conn(0x7f5c840fbac0 msgr2=0x7f5c840fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:41.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.691+0000 7f5c8b094700 1 -- 172.21.15.2:0/814251523 shutdown_connections 2023-11-16T18:24:41.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.691+0000 7f5c8b094700 1 -- 172.21.15.2:0/814251523 wait complete. 2023-11-16T18:24:41.694 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T18:24:41.705 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:24:41.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.831+0000 7f2100cdb700 1 -- 172.21.15.2:0/544154956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20fc1014c0 msgr2=0x7f20fc101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.831+0000 7f2100cdb700 1 --2- 172.21.15.2:0/544154956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20fc1014c0 0x7f20fc101910 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f20f0009990 tx=0x7f20f0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.831+0000 7f2100cdb700 1 -- 172.21.15.2:0/544154956 shutdown_connections 2023-11-16T18:24:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.831+0000 7f2100cdb700 1 --2- 172.21.15.2:0/544154956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc1002d0 0x7f20fc1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.831+0000 7f2100cdb700 1 --2- 172.21.15.2:0/544154956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20fc101e50 0x7f20fc108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.831+0000 7f2100cdb700 1 --2- 172.21.15.2:0/544154956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20fc1014c0 0x7f20fc101910 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.831+0000 7f2100cdb700 1 -- 172.21.15.2:0/544154956 >> 172.21.15.2:0/544154956 conn(0x7f20fc0fb8b0 msgr2=0x7f20fc0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.831+0000 7f2100cdb700 1 -- 172.21.15.2:0/544154956 shutdown_connections 2023-11-16T18:24:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.831+0000 7f2100cdb700 1 -- 172.21.15.2:0/544154956 wait complete. 2023-11-16T18:24:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.832+0000 7f2100cdb700 1 Processor -- start 2023-11-16T18:24:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.832+0000 7f2100cdb700 1 -- start start 2023-11-16T18:24:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f2100cdb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20fc1002d0 0x7f20fc1960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f2100cdb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc1014c0 0x7f20fc196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f2100cdb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20fc101e50 0x7f20fc19d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f2100cdb700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc19dc50 con 0x7f20fc101e50 2023-11-16T18:24:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f2100cdb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc19dd90 con 0x7f20fc1002d0 2023-11-16T18:24:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f2100cdb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc19df20 con 0x7f20fc1014c0 2023-11-16T18:24:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20fad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20fc101e50 0x7f20fc19d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20fad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20fc101e50 0x7f20fc19d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49840/0 (socket says 172.21.15.2:49840) 2023-11-16T18:24:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20fad9d700 1 -- 172.21.15.2:0/636408729 learned_addr learned my addr 172.21.15.2:0/636408729 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20fa59c700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20fc1002d0 0x7f20fc1960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:41.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20f9d9b700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc1014c0 0x7f20fc196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:41.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20fad9d700 1 -- 172.21.15.2:0/636408729 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc1014c0 msgr2=0x7f20fc196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:41.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20fad9d700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc1014c0 0x7f20fc196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20fad9d700 1 -- 172.21.15.2:0/636408729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20fc1002d0 msgr2=0x7f20fc1960d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:41.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20fad9d700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20fc1002d0 0x7f20fc1960d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:41.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20fad9d700 1 -- 172.21.15.2:0/636408729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f00095f0 con 0x7f20fc101e50 2023-11-16T18:24:41.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20fad9d700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20fc101e50 0x7f20fc19d680 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f20f400ef30 tx=0x7f20f400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:41.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20eb7fe700 1 -- 172.21.15.2:0/636408729 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f4018070 con 0x7f20fc101e50 2023-11-16T18:24:41.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20eb7fe700 1 -- 172.21.15.2:0/636408729 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20f400f040 con 0x7f20fc101e50 2023-11-16T18:24:41.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f2100cdb700 1 -- 172.21.15.2:0/636408729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20fc10bc80 con 0x7f20fc101e50 2023-11-16T18:24:41.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f20eb7fe700 1 -- 172.21.15.2:0/636408729 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f400cd00 con 0x7f20fc101e50 2023-11-16T18:24:41.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.833+0000 7f2100cdb700 1 -- 172.21.15.2:0/636408729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20fc10baf0 con 0x7f20fc101e50 2023-11-16T18:24:41.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.835+0000 7f2100cdb700 1 -- 172.21.15.2:0/636408729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20fc107860 con 0x7f20fc101e50 2023-11-16T18:24:41.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.836+0000 7f20eb7fe700 1 -- 172.21.15.2:0/636408729 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f20f4003a00 con 0x7f20fc101e50 2023-11-16T18:24:41.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.836+0000 7f20eb7fe700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f20e40a11d0 0x7f20e40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:41.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.836+0000 7f20fa59c700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f20e40a11d0 0x7f20e40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:41.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.836+0000 7f20eb7fe700 1 -- 172.21.15.2:0/636408729 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f20f40cc6e0 con 0x7f20fc101e50 2023-11-16T18:24:41.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.837+0000 7f20fa59c700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f20e40a11d0 0x7f20e40a3680 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f20fc101320 tx=0x7f20ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:41.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:41.840+0000 7f20eb7fe700 1 -- 172.21.15.2:0/636408729 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f20f4095b40 con 0x7f20fc101e50 2023-11-16T18:24:42.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.041+0000 7f2100cdb700 1 -- 172.21.15.2:0/636408729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20fc02cfe0 con 0x7f20fc101e50 2023-11-16T18:24:42.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.043+0000 7f20eb7fe700 1 -- 172.21.15.2:0/636408729 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24570 (secure 0 0 0) 0x7f20f4099160 con 0x7f20fc101e50 2023-11-16T18:24:42.045 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:24:42.045 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":90,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:24:39.813936+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:24:42.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.045+0000 7f2100cdb700 1 -- 172.21.15.2:0/636408729 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f20e40a11d0 msgr2=0x7f20e40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.045+0000 7f2100cdb700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f20e40a11d0 0x7f20e40a3680 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f20fc101320 tx=0x7f20ec009260 comp rx=0 tx=0).stop 2023-11-16T18:24:42.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.045+0000 7f2100cdb700 1 -- 172.21.15.2:0/636408729 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20fc101e50 msgr2=0x7f20fc19d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.045+0000 7f2100cdb700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20fc101e50 0x7f20fc19d680 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f20f400ef30 tx=0x7f20f400c3a0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.045+0000 7f2100cdb700 1 -- 172.21.15.2:0/636408729 shutdown_connections 2023-11-16T18:24:42.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.045+0000 7f2100cdb700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f20e40a11d0 0x7f20e40a3680 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.045+0000 7f2100cdb700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20fc101e50 0x7f20fc19d680 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.045+0000 7f2100cdb700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20fc1002d0 0x7f20fc1960d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.045+0000 7f2100cdb700 1 --2- 172.21.15.2:0/636408729 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc1014c0 0x7f20fc196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.045+0000 7f2100cdb700 1 -- 172.21.15.2:0/636408729 >> 172.21.15.2:0/636408729 conn(0x7f20fc0fb8b0 msgr2=0x7f20fc1049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.045+0000 7f2100cdb700 1 -- 172.21.15.2:0/636408729 shutdown_connections 2023-11-16T18:24:42.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.045+0000 7f2100cdb700 1 -- 172.21.15.2:0/636408729 wait complete. 2023-11-16T18:24:42.058 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:41 smithi002 ceph-mon[98807]: pgmap v321: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 638 B/s wr, 5 op/s 2023-11-16T18:24:42.058 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:41 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1543216555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:42.058 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:41 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/820959497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:42.058 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:41 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/814251523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:42.060 DEBUG:tasks.fwd_scrub:Yielding 2023-11-16T18:24:42.060 INFO:teuthology.run_tasks:Running task exec... 2023-11-16T18:24:42.070 INFO:teuthology.task.exec:Executing custom commands... 2023-11-16T18:24:42.070 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T18:24:42.071 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2023-11-16T18:24:42.071 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2023-11-16T18:24:42.071 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T18:24:42.071 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T18:24:42.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.198+0000 7f868d19a700 1 -- 172.21.15.2:0/2594485227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8688101e00 msgr2=0x7f86881086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.198+0000 7f868d19a700 1 --2- 172.21.15.2:0/2594485227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8688101e00 0x7f86881086c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f868000b0a0 tx=0x7f868000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.198+0000 7f868d19a700 1 -- 172.21.15.2:0/2594485227 shutdown_connections 2023-11-16T18:24:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.198+0000 7f868d19a700 1 --2- 172.21.15.2:0/2594485227 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688100280 0x7f8688100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.198+0000 7f868d19a700 1 --2- 172.21.15.2:0/2594485227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8688101e00 0x7f86881086c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.198+0000 7f868d19a700 1 --2- 172.21.15.2:0/2594485227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8688101470 0x7f86881018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.198+0000 7f868d19a700 1 -- 172.21.15.2:0/2594485227 >> 172.21.15.2:0/2594485227 conn(0x7f86880fb8a0 msgr2=0x7f86880fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.198+0000 7f868d19a700 1 -- 172.21.15.2:0/2594485227 shutdown_connections 2023-11-16T18:24:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.198+0000 7f868d19a700 1 -- 172.21.15.2:0/2594485227 wait complete. 2023-11-16T18:24:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.198+0000 7f868d19a700 1 Processor -- start 2023-11-16T18:24:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868d19a700 1 -- start start 2023-11-16T18:24:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868d19a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8688100280 0x7f86880748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868d19a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8688101470 0x7f8688072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868d19a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688101e00 0x7f8688073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868d19a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8688073bc0 con 0x7f8688101470 2023-11-16T18:24:42.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868d19a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8688073d30 con 0x7f8688100280 2023-11-16T18:24:42.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868d19a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86881a0180 con 0x7f8688101e00 2023-11-16T18:24:42.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8688101470 0x7f8688072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8688101470 0x7f8688072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49852/0 (socket says 172.21.15.2:49852) 2023-11-16T18:24:42.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868659c700 1 -- 172.21.15.2:0/2837976314 learned_addr learned my addr 172.21.15.2:0/2837976314 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:42.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f8686d9d700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8688100280 0x7f86880748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868759e700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688101e00 0x7f8688073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868659c700 1 -- 172.21.15.2:0/2837976314 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688101e00 msgr2=0x7f8688073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868659c700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688101e00 0x7f8688073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868659c700 1 -- 172.21.15.2:0/2837976314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8688100280 msgr2=0x7f86880748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868659c700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8688100280 0x7f86880748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.199+0000 7f868659c700 1 -- 172.21.15.2:0/2837976314 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8680009d00 con 0x7f8688101470 2023-11-16T18:24:42.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.200+0000 7f868659c700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8688101470 0x7f8688072ee0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f867c00d950 tx=0x7f867c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.200+0000 7f866ffff700 1 -- 172.21.15.2:0/2837976314 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c009940 con 0x7f8688101470 2023-11-16T18:24:42.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.200+0000 7f866ffff700 1 -- 172.21.15.2:0/2837976314 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f867c010460 con 0x7f8688101470 2023-11-16T18:24:42.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.200+0000 7f868d19a700 1 -- 172.21.15.2:0/2837976314 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868810bc00 con 0x7f8688101470 2023-11-16T18:24:42.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.200+0000 7f866ffff700 1 -- 172.21.15.2:0/2837976314 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c00f620 con 0x7f8688101470 2023-11-16T18:24:42.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.200+0000 7f868d19a700 1 -- 172.21.15.2:0/2837976314 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868810ba70 con 0x7f8688101470 2023-11-16T18:24:42.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.201+0000 7f868d19a700 1 -- 172.21.15.2:0/2837976314 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8688107770 con 0x7f8688101470 2023-11-16T18:24:42.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.201+0000 7f2830279700 1 -- 172.21.15.2:0/3575243965 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2828101e10 msgr2=0x7f28281086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.201+0000 7f2830279700 1 --2- 172.21.15.2:0/3575243965 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2828101e10 0x7f28281086d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f282000b0a0 tx=0x7f282000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.202+0000 7f2830279700 1 -- 172.21.15.2:0/3575243965 shutdown_connections 2023-11-16T18:24:42.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.202+0000 7f2830279700 1 --2- 172.21.15.2:0/3575243965 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2828101e10 0x7f28281086d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.202+0000 7f2830279700 1 --2- 172.21.15.2:0/3575243965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2828100290 0x7f28281006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.202+0000 7f2830279700 1 --2- 172.21.15.2:0/3575243965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2828101480 0x7f28281018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.202+0000 7f2830279700 1 -- 172.21.15.2:0/3575243965 >> 172.21.15.2:0/3575243965 conn(0x7f28280fb8b0 msgr2=0x7f28280fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.202+0000 7f2830279700 1 -- 172.21.15.2:0/3575243965 shutdown_connections 2023-11-16T18:24:42.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.202+0000 7f2830279700 1 -- 172.21.15.2:0/3575243965 wait complete. 2023-11-16T18:24:42.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.202+0000 7f2830279700 1 Processor -- start 2023-11-16T18:24:42.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.202+0000 7f866ffff700 1 -- 172.21.15.2:0/2837976314 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f867c0108d0 con 0x7f8688101470 2023-11-16T18:24:42.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f2830279700 1 -- start start 2023-11-16T18:24:42.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f2830279700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2828100290 0x7f2828195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f2830279700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2828101e10 0x7f28281964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f2830279700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282819b510 0x7f282819d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f2830279700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282819dec0 con 0x7f282819b510 2023-11-16T18:24:42.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f2830279700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282819e000 con 0x7f2828100290 2023-11-16T18:24:42.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f2830279700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282819e190 con 0x7f2828101e10 2023-11-16T18:24:42.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f282e816700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282819b510 0x7f282819d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f282e816700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282819b510 0x7f282819d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49866/0 (socket says 172.21.15.2:49866) 2023-11-16T18:24:42.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f282e816700 1 -- 172.21.15.2:0/1782896208 learned_addr learned my addr 172.21.15.2:0/1782896208 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:42.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f282e015700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2828100290 0x7f2828195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f866ffff700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f86700a1150 0x7f86700a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f282d814700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2828101e10 0x7f28281964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f8686d9d700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f86700a1150 0x7f86700a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f866ffff700 1 -- 172.21.15.2:0/2837976314 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f867c0ccac0 con 0x7f8688101470 2023-11-16T18:24:42.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f282e816700 1 -- 172.21.15.2:0/1782896208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2828101e10 msgr2=0x7f28281964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f282e816700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2828101e10 0x7f28281964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.203+0000 7f282e816700 1 -- 172.21.15.2:0/1782896208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2828100290 msgr2=0x7f2828195f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.204+0000 7f282e816700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2828100290 0x7f2828195f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.204+0000 7f282e816700 1 -- 172.21.15.2:0/1782896208 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2820009d00 con 0x7f282819b510 2023-11-16T18:24:42.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.204+0000 7f282e015700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2828100290 0x7f2828195f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:24:42.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.204+0000 7f8686d9d700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f86700a1150 0x7f86700a3600 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f86881012d0 tx=0x7f8678009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.204+0000 7f282e816700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282819b510 0x7f282819d8f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f28200062a0 tx=0x7f282000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.207+0000 7f281f7fe700 1 -- 172.21.15.2:0/1782896208 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282000e040 con 0x7f282819b510 2023-11-16T18:24:42.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.207+0000 7f281f7fe700 1 -- 172.21.15.2:0/1782896208 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28200094c0 con 0x7f282819b510 2023-11-16T18:24:42.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.207+0000 7f2830279700 1 -- 172.21.15.2:0/1782896208 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282810bc70 con 0x7f282819b510 2023-11-16T18:24:42.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.207+0000 7f281f7fe700 1 -- 172.21.15.2:0/1782896208 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28200052a0 con 0x7f282819b510 2023-11-16T18:24:42.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.207+0000 7f2830279700 1 -- 172.21.15.2:0/1782896208 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282810bae0 con 0x7f282819b510 2023-11-16T18:24:42.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.207+0000 7f866ffff700 1 -- 172.21.15.2:0/2837976314 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f867c095e50 con 0x7f8688101470 2023-11-16T18:24:42.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:41 smithi040 ceph-mon[100251]: pgmap v321: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 638 B/s wr, 5 op/s 2023-11-16T18:24:42.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:41 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1543216555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:42.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:41 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/820959497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:42.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:41 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/814251523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:42.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.210+0000 7f281f7fe700 1 -- 172.21.15.2:0/1782896208 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f282001d470 con 0x7f282819b510 2023-11-16T18:24:42.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.210+0000 7f283c405b80 1 Processor -- start 2023-11-16T18:24:42.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f281f7fe700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f28140a11d0 0x7f28140a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f283c405b80 1 -- start start 2023-11-16T18:24:42.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f282d814700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f28140a11d0 0x7f28140a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f281f7fe700 1 -- 172.21.15.2:0/1782896208 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f28200ce300 con 0x7f282819b510 2023-11-16T18:24:42.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f283c405b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fbc70 0x55f64d7fc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f283c405b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fc5c0 0x55f64d7fcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f283c405b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fd150 0x55f64d7fd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f283c405b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64d7010e0 con 0x55f64d7fd150 2023-11-16T18:24:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f283c405b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64d753920 con 0x55f64d7fc5c0 2023-11-16T18:24:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f283c405b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64d74cdb0 con 0x55f64d7fbc70 2023-11-16T18:24:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f282e816700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fd150 0x55f64d7fd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f282e816700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fd150 0x55f64d7fd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49874/0 (socket says 172.21.15.2:49874) 2023-11-16T18:24:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f282e816700 1 -- 172.21.15.2:0/3331590425 learned_addr learned my addr 172.21.15.2:0/3331590425 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f282e015700 1 --2- 172.21.15.2:0/3331590425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fbc70 0x55f64d7fc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f282d814700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f28140a11d0 0x7f28140a3680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2828100fc0 tx=0x7f2818009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f282d814700 1 --2- 172.21.15.2:0/3331590425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fc5c0 0x55f64d7fcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f282e816700 1 -- 172.21.15.2:0/3331590425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fbc70 msgr2=0x55f64d7fc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f282e816700 1 --2- 172.21.15.2:0/3331590425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fbc70 0x55f64d7fc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.211+0000 7f282e816700 1 -- 172.21.15.2:0/3331590425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fc5c0 msgr2=0x55f64d7fcc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.212+0000 7f282e816700 1 --2- 172.21.15.2:0/3331590425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fc5c0 0x55f64d7fcc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.212+0000 7f282e816700 1 -- 172.21.15.2:0/3331590425 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f64d5ce6b0 con 0x55f64d7fd150 2023-11-16T18:24:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.212+0000 7f282d814700 1 --2- 172.21.15.2:0/3331590425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fc5c0 0x55f64d7fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.212+0000 7f282e816700 1 --2- 172.21.15.2:0/3331590425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fd150 0x55f64d7fd5f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f28200a3270 tx=0x7f28200097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567d2ea90f11572a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.212+0000 7f281cdac700 1 -- 172.21.15.2:0/3331590425 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28200ce170 con 0x55f64d7fd150 2023-11-16T18:24:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.212+0000 7f281cdac700 1 -- 172.21.15.2:0/3331590425 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f282002d2f0 con 0x55f64d7fd150 2023-11-16T18:24:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.212+0000 7f281cdac700 1 -- 172.21.15.2:0/3331590425 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2820099810 con 0x55f64d7fd150 2023-11-16T18:24:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 -- 172.21.15.2:0/3331590425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fd150 msgr2=0x55f64d7fd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 --2- 172.21.15.2:0/3331590425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fd150 0x55f64d7fd5f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f28200a3270 tx=0x7f28200097e0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 -- 172.21.15.2:0/3331590425 shutdown_connections 2023-11-16T18:24:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 --2- 172.21.15.2:0/3331590425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fbc70 0x55f64d7fc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 --2- 172.21.15.2:0/3331590425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fd150 0x55f64d7fd5f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 --2- 172.21.15.2:0/3331590425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fc5c0 0x55f64d7fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 -- 172.21.15.2:0/3331590425 >> 172.21.15.2:0/3331590425 conn(0x55f64d7f13e0 msgr2=0x55f64d7f66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 -- 172.21.15.2:0/3331590425 shutdown_connections 2023-11-16T18:24:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 -- 172.21.15.2:0/3331590425 wait complete. 2023-11-16T18:24:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 -- start start 2023-11-16T18:24:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 Processor -- start 2023-11-16T18:24:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d80dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fc5c0 0x55f64d80e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fd150 0x55f64d815180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64d6ee960 con 0x55f64d7fbc70 2023-11-16T18:24:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64d6ee600 con 0x55f64d7fd150 2023-11-16T18:24:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64d5d09a0 con 0x55f64d7fc5c0 2023-11-16T18:24:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.213+0000 7f283c405b80 10 client.0 _ll_register_callbacks cb 0x55f64d70c800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:24:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f282e015700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d80dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f282e015700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d80dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49878/0 (socket says 172.21.15.2:49878) 2023-11-16T18:24:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f282e015700 1 -- 172.21.15.2:0/1034209648 learned_addr learned my addr 172.21.15.2:0/1034209648 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:42.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f282e816700 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fd150 0x55f64d815180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f282d814700 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fc5c0 0x55f64d80e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f282e015700 1 -- 172.21.15.2:0/1034209648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fc5c0 msgr2=0x55f64d80e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f282e015700 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fc5c0 0x55f64d80e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f282e015700 1 -- 172.21.15.2:0/1034209648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fd150 msgr2=0x55f64d815180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f282e015700 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fd150 0x55f64d815180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f282e015700 1 -- 172.21.15.2:0/1034209648 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282810bae0 con 0x55f64d7fbc70 2023-11-16T18:24:42.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f282d814700 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fc5c0 0x55f64d80e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:42.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f282e015700 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d80dc00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2824012d20 tx=0x7f282400f6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3966f049c4b81192 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f2811ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:24:42.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f2811ffb700 1 -- 172.21.15.2:0/1034209648 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2824017070 con 0x55f64d7fbc70 2023-11-16T18:24:42.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f2811ffb700 1 -- 172.21.15.2:0/1034209648 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f282400fc30 con 0x55f64d7fbc70 2023-11-16T18:24:42.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f283c405b80 1 -- 172.21.15.2:0/1034209648 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f64d7eeba0 con 0x55f64d7fbc70 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f2811ffb700 1 -- 172.21.15.2:0/1034209648 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f282400fdf0 con 0x55f64d7fbc70 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f283c405b80 10 client.15555 fetch_fsmap learned FSMap version 24 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.214+0000 7f283c405b80 1 -- 172.21.15.2:0/1034209648 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55f64d713210 con 0x55f64d7fbc70 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.215+0000 7f2811ffb700 1 -- 172.21.15.2:0/1034209648 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f64d7eeba0 con 0x55f64d7fbc70 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.215+0000 7f2811ffb700 1 -- 172.21.15.2:0/1034209648 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2824003b70 con 0x55f64d7fbc70 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.215+0000 7f283c405b80 10 client.15555 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.215+0000 7f283c405b80 10 client.15555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.215+0000 7f283c405b80 20 client.15555 populate_metadata read hostname 'smithi002' 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.215+0000 7f283c405b80 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55f64d819990 0x55f64d81bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.215+0000 7f283c405b80 4 client.15555 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.215+0000 7f283c405b80 1 -- 172.21.15.2:0/1034209648 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f64d70f120 con 0x55f64d819990 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.215+0000 7f282d814700 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55f64d819990 0x55f64d81bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.216+0000 7f282d814700 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55f64d819990 0x55f64d81bda0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc028a3a657b5693 server_cookie=247c0d7d86c3c780 in_seq=0 out_seq=0 2023-11-16T18:24:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.216+0000 7f2811ffb700 10 client.15555 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:24:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.216+0000 7f2811ffb700 1 -- 172.21.15.2:0/1034209648 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f64d70f120 con 0x55f64d819990 2023-11-16T18:24:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.216+0000 7f2811ffb700 10 client.15555 handle_command_reply: tid=0 2023-11-16T18:24:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.217+0000 7f283c405b80 1 client.15555 shutdown 2023-11-16T18:24:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.217+0000 7f283c405b80 1 -- 172.21.15.2:0/1034209648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 msgr2=0x55f64d80dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.217+0000 7f283c405b80 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d80dc00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2824012d20 tx=0x7f282400f6a0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.217+0000 7f283c405b80 1 -- 172.21.15.2:0/1034209648 shutdown_connections 2023-11-16T18:24:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.217+0000 7f283c405b80 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55f64d819990 0x55f64d81bda0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.217+0000 7f283c405b80 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fd150 0x55f64d815180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.217+0000 7f2811ffb700 0 client.15555 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:24:42.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.217+0000 7f283c405b80 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d80dc00 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.217+0000 7f283c405b80 1 --2- 172.21.15.2:0/1034209648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fc5c0 0x55f64d80e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.217+0000 7f283c405b80 1 -- 172.21.15.2:0/1034209648 >> 172.21.15.2:0/1034209648 conn(0x55f64d7f13e0 msgr2=0x55f64d7f5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.217+0000 7f283c405b80 1 -- 172.21.15.2:0/1034209648 shutdown_connections 2023-11-16T18:24:42.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.217+0000 7f283c405b80 1 -- 172.21.15.2:0/1034209648 wait complete. 2023-11-16T18:24:42.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.217+0000 7f283c405b80 20 client.15555 trim_cache size 0 max 16384 2023-11-16T18:24:42.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.234+0000 7f283c405b80 1 Processor -- start 2023-11-16T18:24:42.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.234+0000 7f283c405b80 1 -- start start 2023-11-16T18:24:42.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.234+0000 7f283c405b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d80e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.234+0000 7f283c405b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fc5c0 0x55f64d815140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.234+0000 7f283c405b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fd150 0x55f64d80bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.234+0000 7f283c405b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64d7f8220 con 0x55f64d7fbc70 2023-11-16T18:24:42.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.234+0000 7f283c405b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64d7eea30 con 0x55f64d7fc5c0 2023-11-16T18:24:42.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.234+0000 7f283c405b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64d808610 con 0x55f64d7fd150 2023-11-16T18:24:42.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.234+0000 7f282e015700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d80e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f282e015700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d80e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49886/0 (socket says 172.21.15.2:49886) 2023-11-16T18:24:42.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f282e015700 1 -- 172.21.15.2:0/1633418958 learned_addr learned my addr 172.21.15.2:0/1633418958 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:42.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f282d814700 1 --2- 172.21.15.2:0/1633418958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fc5c0 0x55f64d815140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f282e816700 1 --2- 172.21.15.2:0/1633418958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fd150 0x55f64d80bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f282e015700 1 -- 172.21.15.2:0/1633418958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fd150 msgr2=0x55f64d80bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f282e015700 1 --2- 172.21.15.2:0/1633418958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fd150 0x55f64d80bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f282e015700 1 -- 172.21.15.2:0/1633418958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fc5c0 msgr2=0x55f64d815140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f282e015700 1 --2- 172.21.15.2:0/1633418958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fc5c0 0x55f64d815140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f282e015700 1 -- 172.21.15.2:0/1633418958 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f64d7f2b80 con 0x55f64d7fbc70 2023-11-16T18:24:42.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f282e816700 1 --2- 172.21.15.2:0/1633418958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fd150 0x55f64d80bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:24:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f282e015700 1 --2- 172.21.15.2:0/1633418958 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d80e0c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2824012cf0 tx=0x7f2824010550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235f3017ef2094df server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f281d5ad700 1 -- 172.21.15.2:0/1633418958 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2824017070 con 0x55f64d7fbc70 2023-11-16T18:24:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f281d5ad700 1 -- 172.21.15.2:0/1633418958 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2824018040 con 0x55f64d7fbc70 2023-11-16T18:24:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.235+0000 7f281d5ad700 1 -- 172.21.15.2:0/1633418958 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2824010be0 con 0x55f64d7fbc70 2023-11-16T18:24:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.236+0000 7f283c405b80 1 -- 172.21.15.2:0/1633418958 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 msgr2=0x55f64d80e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.236+0000 7f283c405b80 1 --2- 172.21.15.2:0/1633418958 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d80e0c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2824012cf0 tx=0x7f2824010550 comp rx=0 tx=0).stop 2023-11-16T18:24:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.236+0000 7f283c405b80 1 -- 172.21.15.2:0/1633418958 shutdown_connections 2023-11-16T18:24:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.236+0000 7f283c405b80 1 --2- 172.21.15.2:0/1633418958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fd150 0x55f64d80bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.236+0000 7f283c405b80 1 --2- 172.21.15.2:0/1633418958 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d80e0c0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.236+0000 7f283c405b80 1 --2- 172.21.15.2:0/1633418958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fc5c0 0x55f64d815140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.236+0000 7f283c405b80 1 -- 172.21.15.2:0/1633418958 >> 172.21.15.2:0/1633418958 conn(0x55f64d7f13e0 msgr2=0x55f64d80d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.236+0000 7f283c405b80 1 -- 172.21.15.2:0/1633418958 shutdown_connections 2023-11-16T18:24:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.236+0000 7f283c405b80 1 -- 172.21.15.2:0/1633418958 wait complete. 2023-11-16T18:24:42.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f283c405b80 1 -- start start 2023-11-16T18:24:42.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f283c405b80 1 Processor -- start 2023-11-16T18:24:42.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f283c405b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d815050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f283c405b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fc5c0 0x55f64d811af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f283c405b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fd150 0x55f64d812030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f283c405b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64d812720 con 0x55f64d7fbc70 2023-11-16T18:24:42.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f283c405b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64d86d840 con 0x55f64d7fd150 2023-11-16T18:24:42.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f283c405b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64d86da00 con 0x55f64d7fc5c0 2023-11-16T18:24:42.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f283c405b80 10 client.0 _ll_register_callbacks cb 0x55f64d70c800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:24:42.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f282e015700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d815050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f282e015700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d815050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49892/0 (socket says 172.21.15.2:49892) 2023-11-16T18:24:42.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f282e015700 1 -- 172.21.15.2:0/1513945083 learned_addr learned my addr 172.21.15.2:0/1513945083 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:42.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f282d814700 1 --2- 172.21.15.2:0/1513945083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fc5c0 0x55f64d811af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.237+0000 7f282e816700 1 --2- 172.21.15.2:0/1513945083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fd150 0x55f64d812030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f282e015700 1 -- 172.21.15.2:0/1513945083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fc5c0 msgr2=0x55f64d811af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f282e015700 1 --2- 172.21.15.2:0/1513945083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fc5c0 0x55f64d811af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f282e015700 1 -- 172.21.15.2:0/1513945083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fd150 msgr2=0x55f64d812030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f282e015700 1 --2- 172.21.15.2:0/1513945083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fd150 0x55f64d812030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f282e015700 1 -- 172.21.15.2:0/1513945083 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f64d713210 con 0x55f64d7fbc70 2023-11-16T18:24:42.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f282e015700 1 --2- 172.21.15.2:0/1513945083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d815050 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f2824003f00 tx=0x7f28240128a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a42794b5b4b7ea server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f2811ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:24:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f2811ffb700 1 -- 172.21.15.2:0/1513945083 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2824017070 con 0x55f64d7fbc70 2023-11-16T18:24:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f2811ffb700 1 -- 172.21.15.2:0/1513945083 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f2824009260 con 0x55f64d7fbc70 2023-11-16T18:24:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f283c405b80 1 -- 172.21.15.2:0/1513945083 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f282002d2f0 con 0x55f64d7fbc70 2023-11-16T18:24:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f2811ffb700 1 -- 172.21.15.2:0/1513945083 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2824005320 con 0x55f64d7fbc70 2023-11-16T18:24:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f283c405b80 10 client.15561 fetch_fsmap learned FSMap version 24 2023-11-16T18:24:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f283c405b80 1 -- 172.21.15.2:0/1513945083 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55f64d712f60 con 0x55f64d7fbc70 2023-11-16T18:24:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.238+0000 7f2811ffb700 1 -- 172.21.15.2:0/1513945083 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f282002d2f0 con 0x55f64d7fbc70 2023-11-16T18:24:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.239+0000 7f2811ffb700 1 -- 172.21.15.2:0/1513945083 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f28240038c0 con 0x55f64d7fbc70 2023-11-16T18:24:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.239+0000 7f283c405b80 10 client.15561 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:24:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.239+0000 7f283c405b80 10 client.15561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:24:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.239+0000 7f283c405b80 20 client.15561 populate_metadata read hostname 'smithi002' 2023-11-16T18:24:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.239+0000 7f283c405b80 1 --2- 172.21.15.2:0/1513945083 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55f64d86ef70 0x55f64d871350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.239+0000 7f283c405b80 4 client.15561 mds_command: new command op to 24473 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T18:24:42.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.239+0000 7f283c405b80 1 -- 172.21.15.2:0/1513945083 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f64d71b6e0 con 0x55f64d86ef70 2023-11-16T18:24:42.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.239+0000 7f282d814700 1 --2- 172.21.15.2:0/1513945083 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55f64d86ef70 0x55f64d871350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.239+0000 7f2811ffb700 10 client.15561 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:24:42.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.239+0000 7f282d814700 1 --2- 172.21.15.2:0/1513945083 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55f64d86ef70 0x55f64d871350 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80ec9954d6dcdfb6 server_cookie=f73752f1c5739fac in_seq=0 out_seq=0 2023-11-16T18:24:42.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f2811ffb700 1 -- 172.21.15.2:0/1513945083 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f64d71b6e0 con 0x55f64d86ef70 2023-11-16T18:24:42.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f2811ffb700 10 client.15561 handle_command_reply: tid=0 2023-11-16T18:24:42.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f283c405b80 1 client.15561 shutdown 2023-11-16T18:24:42.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f283c405b80 1 -- 172.21.15.2:0/1513945083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 msgr2=0x55f64d815050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f283c405b80 1 --2- 172.21.15.2:0/1513945083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d815050 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f2824003f00 tx=0x7f28240128a0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f283c405b80 1 -- 172.21.15.2:0/1513945083 shutdown_connections 2023-11-16T18:24:42.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f283c405b80 1 --2- 172.21.15.2:0/1513945083 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55f64d86ef70 0x55f64d871350 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f283c405b80 1 --2- 172.21.15.2:0/1513945083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64d7fd150 0x55f64d812030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f283c405b80 1 --2- 172.21.15.2:0/1513945083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f64d7fbc70 0x55f64d815050 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f2811ffb700 0 client.15561 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:24:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f283c405b80 1 --2- 172.21.15.2:0/1513945083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f64d7fc5c0 0x55f64d811af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f283c405b80 1 -- 172.21.15.2:0/1513945083 >> 172.21.15.2:0/1513945083 conn(0x55f64d7f13e0 msgr2=0x55f64d7fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f283c405b80 1 -- 172.21.15.2:0/1513945083 shutdown_connections 2023-11-16T18:24:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f283c405b80 1 -- 172.21.15.2:0/1513945083 wait complete. 2023-11-16T18:24:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.241+0000 7f283c405b80 20 client.15561 trim_cache size 0 max 16384 2023-11-16T18:24:42.256 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:24:42.256 INFO:teuthology.orchestra.run.smithi002.stdout: "return_code": 0, 2023-11-16T18:24:42.257 INFO:teuthology.orchestra.run.smithi002.stdout: "scrub_tag": "a7a9ed8c-d014-4a6d-a850-f2e632e800ce", 2023-11-16T18:24:42.257 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": "asynchronous" 2023-11-16T18:24:42.257 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:24:42.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.242+0000 7f2830279700 1 -- 172.21.15.2:0/1782896208 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f28140a11d0 msgr2=0x7f28140a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.242+0000 7f2830279700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f28140a11d0 0x7f28140a3680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2828100fc0 tx=0x7f2818009260 comp rx=0 tx=0).stop 2023-11-16T18:24:42.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.242+0000 7f2830279700 1 -- 172.21.15.2:0/1782896208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282819b510 msgr2=0x7f282819d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.242+0000 7f2830279700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282819b510 0x7f282819d8f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f28200062a0 tx=0x7f282000bd60 comp rx=0 tx=0).stop 2023-11-16T18:24:42.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.242+0000 7f2830279700 1 -- 172.21.15.2:0/1782896208 shutdown_connections 2023-11-16T18:24:42.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.242+0000 7f2830279700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f28140a11d0 0x7f28140a3680 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.242+0000 7f2830279700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282819b510 0x7f282819d8f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.242+0000 7f2830279700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2828100290 0x7f2828195f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.242+0000 7f2830279700 1 --2- 172.21.15.2:0/1782896208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2828101e10 0x7f28281964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.242+0000 7f2830279700 1 -- 172.21.15.2:0/1782896208 >> 172.21.15.2:0/1782896208 conn(0x7f28280fb8b0 msgr2=0x7f28280fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.242+0000 7f2830279700 1 -- 172.21.15.2:0/1782896208 shutdown_connections 2023-11-16T18:24:42.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.242+0000 7f2830279700 1 -- 172.21.15.2:0/1782896208 wait complete. 2023-11-16T18:24:42.259 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:24:42.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:41 smithi188 ceph-mon[100639]: pgmap v321: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 638 B/s wr, 5 op/s 2023-11-16T18:24:42.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:41 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1543216555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:24:42.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:41 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/820959497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:24:42.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:41 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/814251523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:24:42.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.386+0000 7fa353d44700 1 -- 172.21.15.2:0/2633694751 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa34c068b00 msgr2=0x7fa34c102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.386+0000 7fa353d44700 1 --2- 172.21.15.2:0/2633694751 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa34c068b00 0x7fa34c102e00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa348009990 tx=0x7fa348009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.386+0000 7fa353d44700 1 -- 172.21.15.2:0/2633694751 shutdown_connections 2023-11-16T18:24:42.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.386+0000 7fa353d44700 1 --2- 172.21.15.2:0/2633694751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa34c105e50 0x7fa34c108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.386+0000 7fa353d44700 1 --2- 172.21.15.2:0/2633694751 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa34c068b00 0x7fa34c102e00 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.386+0000 7fa353d44700 1 --2- 172.21.15.2:0/2633694751 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa34c103340 0x7fa34c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.386+0000 7fa353d44700 1 -- 172.21.15.2:0/2633694751 >> 172.21.15.2:0/2633694751 conn(0x7fa34c0fa750 msgr2=0x7fa34c0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.386+0000 7fa353d44700 1 -- 172.21.15.2:0/2633694751 shutdown_connections 2023-11-16T18:24:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.386+0000 7fa353d44700 1 -- 172.21.15.2:0/2633694751 wait complete. 2023-11-16T18:24:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.386+0000 7fa353d44700 1 Processor -- start 2023-11-16T18:24:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.386+0000 7fa353d44700 1 -- start start 2023-11-16T18:24:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa353d44700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa34c068b00 0x7fa34c195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa353d44700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa34c103340 0x7fa34c1964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa353d44700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa34c105e50 0x7fa34c19d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa353d44700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c19db10 con 0x7fa34c103340 2023-11-16T18:24:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa353d44700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c19dc50 con 0x7fa34c105e50 2023-11-16T18:24:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa353d44700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c19dde0 con 0x7fa34c068b00 2023-11-16T18:24:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa3512df700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa34c103340 0x7fa34c1964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa3512df700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa34c103340 0x7fa34c1964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49908/0 (socket says 172.21.15.2:49908) 2023-11-16T18:24:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa3512df700 1 -- 172.21.15.2:0/4016441940 learned_addr learned my addr 172.21.15.2:0/4016441940 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa351ae0700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa34c068b00 0x7fa34c195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa3522e1700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa34c105e50 0x7fa34c19d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa3512df700 1 -- 172.21.15.2:0/4016441940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa34c068b00 msgr2=0x7fa34c195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa3512df700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa34c068b00 0x7fa34c195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa3512df700 1 -- 172.21.15.2:0/4016441940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa34c105e50 msgr2=0x7fa34c19d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa3512df700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa34c105e50 0x7fa34c19d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa3512df700 1 -- 172.21.15.2:0/4016441940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3480095f0 con 0x7fa34c103340 2023-11-16T18:24:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa351ae0700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa34c068b00 0x7fa34c195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa3512df700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa34c103340 0x7fa34c1964d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00d950 tx=0x7fa33c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa342ffd700 1 -- 172.21.15.2:0/4016441940 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c009b40 con 0x7fa34c103340 2023-11-16T18:24:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa353d44700 1 -- 172.21.15.2:0/4016441940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34c10bbd0 con 0x7fa34c103340 2023-11-16T18:24:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa342ffd700 1 -- 172.21.15.2:0/4016441940 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa33c009ca0 con 0x7fa34c103340 2023-11-16T18:24:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa342ffd700 1 -- 172.21.15.2:0/4016441940 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c00f620 con 0x7fa34c103340 2023-11-16T18:24:42.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.387+0000 7fa353d44700 1 -- 172.21.15.2:0/4016441940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c10ba40 con 0x7fa34c103340 2023-11-16T18:24:42.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.390+0000 7fa342ffd700 1 -- 172.21.15.2:0/4016441940 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fa33c010460 con 0x7fa34c103340 2023-11-16T18:24:42.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.390+0000 7fa342ffd700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa3380a1150 0x7fa3380a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.390+0000 7fa342ffd700 1 -- 172.21.15.2:0/4016441940 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fa33c0cd040 con 0x7fa34c103340 2023-11-16T18:24:42.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.390+0000 7fa351ae0700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa3380a1150 0x7fa3380a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.391+0000 7fa351ae0700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa3380a1150 0x7fa3380a3600 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa348006010 tx=0x7fa34800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.393+0000 7fa35fed0b80 1 Processor -- start 2023-11-16T18:24:42.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.393+0000 7fa35fed0b80 1 -- start start 2023-11-16T18:24:42.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa35fed0b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b22f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa35fed0b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22f550 0x55630b22fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa35fed0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b230580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa35fed0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630b134060 con 0x55630b230110 2023-11-16T18:24:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa35fed0b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630b1868a0 con 0x55630b22f550 2023-11-16T18:24:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa35fed0b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630b17fd30 con 0x55630b22ec00 2023-11-16T18:24:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa3522e1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b230580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa3522e1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b230580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49918/0 (socket says 172.21.15.2:49918) 2023-11-16T18:24:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa3522e1700 1 -- 172.21.15.2:0/4206479910 learned_addr learned my addr 172.21.15.2:0/4206479910 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa3512df700 1 --2- 172.21.15.2:0/4206479910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22f550 0x55630b22fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa351ae0700 1 --2- 172.21.15.2:0/4206479910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b22f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa3522e1700 1 -- 172.21.15.2:0/4206479910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 msgr2=0x55630b22f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa3522e1700 1 --2- 172.21.15.2:0/4206479910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b22f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa3522e1700 1 -- 172.21.15.2:0/4206479910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22f550 msgr2=0x55630b22fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa3522e1700 1 --2- 172.21.15.2:0/4206479910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22f550 0x55630b22fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa3522e1700 1 -- 172.21.15.2:0/4206479910 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55630b001630 con 0x55630b230110 2023-11-16T18:24:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa351ae0700 1 --2- 172.21.15.2:0/4206479910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b22f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa3522e1700 1 --2- 172.21.15.2:0/4206479910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b230580 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa3440060b0 tx=0x7fa344007500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef804782dfb07cac server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.394+0000 7fa337fff700 1 -- 172.21.15.2:0/4206479910 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344003a00 con 0x55630b230110 2023-11-16T18:24:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.395+0000 7fa337fff700 1 -- 172.21.15.2:0/4206479910 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa344003b60 con 0x55630b230110 2023-11-16T18:24:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.395+0000 7fa337fff700 1 -- 172.21.15.2:0/4206479910 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440146f0 con 0x55630b230110 2023-11-16T18:24:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.395+0000 7fa35fed0b80 1 -- 172.21.15.2:0/4206479910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 msgr2=0x55630b230580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.395+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/4206479910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b230580 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa3440060b0 tx=0x7fa344007500 comp rx=0 tx=0).stop 2023-11-16T18:24:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.395+0000 7fa35fed0b80 1 -- 172.21.15.2:0/4206479910 shutdown_connections 2023-11-16T18:24:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.395+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/4206479910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b22f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.395+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/4206479910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b230580 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.395+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/4206479910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22f550 0x55630b22fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.395+0000 7fa35fed0b80 1 -- 172.21.15.2:0/4206479910 >> 172.21.15.2:0/4206479910 conn(0x55630b224360 msgr2=0x55630b229670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.395+0000 7fa35fed0b80 1 -- 172.21.15.2:0/4206479910 shutdown_connections 2023-11-16T18:24:42.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.395+0000 7fa35fed0b80 1 -- 172.21.15.2:0/4206479910 wait complete. 2023-11-16T18:24:42.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.396+0000 7fa35fed0b80 1 -- start start 2023-11-16T18:24:42.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.396+0000 7fa35fed0b80 1 Processor -- start 2023-11-16T18:24:42.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.396+0000 7fa35fed0b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b2385b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.396+0000 7fa35fed0b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22f550 0x55630b238af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.396+0000 7fa35fed0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b23fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.396+0000 7fa35fed0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630b1218e0 con 0x55630b230110 2023-11-16T18:24:42.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.396+0000 7fa35fed0b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630b121580 con 0x55630b22f550 2023-11-16T18:24:42.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.396+0000 7fa35fed0b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630b003920 con 0x55630b22ec00 2023-11-16T18:24:42.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.396+0000 7fa35fed0b80 10 client.0 _ll_register_callbacks cb 0x55630b13f780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:24:42.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.396+0000 7fa3522e1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b23fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa351ae0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b2385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3522e1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b23fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49924/0 (socket says 172.21.15.2:49924) 2023-11-16T18:24:42.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa351ae0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b2385b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:41450/0 (socket says 172.21.15.2:41450) 2023-11-16T18:24:42.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3522e1700 1 -- 172.21.15.2:0/3559758654 learned_addr learned my addr 172.21.15.2:0/3559758654 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:42.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3512df700 1 --2- 172.21.15.2:0/3559758654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22f550 0x55630b238af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3522e1700 1 -- 172.21.15.2:0/3559758654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 msgr2=0x55630b2385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3522e1700 1 --2- 172.21.15.2:0/3559758654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b2385b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3522e1700 1 -- 172.21.15.2:0/3559758654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22f550 msgr2=0x55630b238af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3522e1700 1 --2- 172.21.15.2:0/3559758654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22f550 0x55630b238af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3522e1700 1 -- 172.21.15.2:0/3559758654 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa344009920 con 0x55630b230110 2023-11-16T18:24:42.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3512df700 1 --2- 172.21.15.2:0/3559758654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22f550 0x55630b238af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:24:42.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3522e1700 1 --2- 172.21.15.2:0/3559758654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b23fb30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa344004440 tx=0x7fa344004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b181ccf33b689c28 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3357fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:24:42.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3357fa700 1 -- 172.21.15.2:0/3559758654 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344003a00 con 0x55630b230110 2023-11-16T18:24:42.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3357fa700 1 -- 172.21.15.2:0/3559758654 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fa344014670 con 0x55630b230110 2023-11-16T18:24:42.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3559758654 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55630b221b20 con 0x55630b230110 2023-11-16T18:24:42.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.397+0000 7fa3357fa700 1 -- 172.21.15.2:0/3559758654 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa344004690 con 0x55630b230110 2023-11-16T18:24:42.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.398+0000 7fa35fed0b80 10 client.15573 fetch_fsmap learned FSMap version 24 2023-11-16T18:24:42.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.398+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3559758654 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55630b146190 con 0x55630b230110 2023-11-16T18:24:42.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.398+0000 7fa3357fa700 1 -- 172.21.15.2:0/3559758654 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55630b221b20 con 0x55630b230110 2023-11-16T18:24:42.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.398+0000 7fa3357fa700 1 -- 172.21.15.2:0/3559758654 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa34401c040 con 0x55630b230110 2023-11-16T18:24:42.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.398+0000 7fa35fed0b80 10 client.15573 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:24:42.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.398+0000 7fa35fed0b80 10 client.15573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:24:42.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.398+0000 7fa35fed0b80 20 client.15573 populate_metadata read hostname 'smithi002' 2023-11-16T18:24:42.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.398+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3559758654 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55630b2442c0 0x55630b2466d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.398+0000 7fa35fed0b80 4 client.15573 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:24:42.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.398+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3559758654 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55630b1420a0 con 0x55630b2442c0 2023-11-16T18:24:42.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.398+0000 7fa351ae0700 1 --2- 172.21.15.2:0/3559758654 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55630b2442c0 0x55630b2466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.399+0000 7fa3357fa700 10 client.15573 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:24:42.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.399+0000 7fa351ae0700 1 --2- 172.21.15.2:0/3559758654 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55630b2442c0 0x55630b2466d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22e2a7b491567ec2 server_cookie=797bf7da0dabf7df in_seq=0 out_seq=0 2023-11-16T18:24:42.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa3357fa700 1 -- 172.21.15.2:0/3559758654 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55630b1420a0 con 0x55630b2442c0 2023-11-16T18:24:42.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa3357fa700 10 client.15573 handle_command_reply: tid=0 2023-11-16T18:24:42.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa35fed0b80 1 client.15573 shutdown 2023-11-16T18:24:42.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3559758654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 msgr2=0x55630b23fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3559758654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b23fb30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa344004440 tx=0x7fa344004470 comp rx=0 tx=0).stop 2023-11-16T18:24:42.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3559758654 shutdown_connections 2023-11-16T18:24:42.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3559758654 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55630b2442c0 0x55630b2466d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3559758654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b2385b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3559758654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b23fb30 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3559758654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22f550 0x55630b238af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa3357fa700 0 client.15573 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:24:42.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3559758654 >> 172.21.15.2:0/3559758654 conn(0x55630b224360 msgr2=0x55630b228de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3559758654 shutdown_connections 2023-11-16T18:24:42.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3559758654 wait complete. 2023-11-16T18:24:42.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.400+0000 7fa35fed0b80 20 client.15573 trim_cache size 0 max 16384 2023-11-16T18:24:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.417+0000 7fa35fed0b80 1 Processor -- start 2023-11-16T18:24:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.417+0000 7fa35fed0b80 1 -- start start 2023-11-16T18:24:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.417+0000 7fa35fed0b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b2385b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.417+0000 7fa35fed0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b22f550 0x55630b23faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.417+0000 7fa35fed0b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b230110 0x55630b238af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.417+0000 7fa35fed0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630b237a60 con 0x55630b22f550 2023-11-16T18:24:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.417+0000 7fa35fed0b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630b238120 con 0x55630b230110 2023-11-16T18:24:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.417+0000 7fa35fed0b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630b22aae0 con 0x55630b22ec00 2023-11-16T18:24:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.417+0000 7fa3512df700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b22f550 0x55630b23faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.417+0000 7fa3512df700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b22f550 0x55630b23faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49930/0 (socket says 172.21.15.2:49930) 2023-11-16T18:24:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.417+0000 7fa3512df700 1 -- 172.21.15.2:0/3537732853 learned_addr learned my addr 172.21.15.2:0/3537732853 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.418+0000 7fa351ae0700 1 --2- 172.21.15.2:0/3537732853 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b2385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.418+0000 7fa3522e1700 1 --2- 172.21.15.2:0/3537732853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b230110 0x55630b238af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.418+0000 7fa3512df700 1 -- 172.21.15.2:0/3537732853 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 msgr2=0x55630b2385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.418+0000 7fa3512df700 1 --2- 172.21.15.2:0/3537732853 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b2385b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.418+0000 7fa3512df700 1 -- 172.21.15.2:0/3537732853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b230110 msgr2=0x55630b238af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.418+0000 7fa3512df700 1 --2- 172.21.15.2:0/3537732853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b230110 0x55630b238af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.418+0000 7fa3512df700 1 -- 172.21.15.2:0/3537732853 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55630b145ee0 con 0x55630b22f550 2023-11-16T18:24:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.418+0000 7fa3522e1700 1 --2- 172.21.15.2:0/3537732853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b230110 0x55630b238af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.418+0000 7fa3512df700 1 --2- 172.21.15.2:0/3537732853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b22f550 0x55630b23faf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa33c096b40 tx=0x7fa33c010c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab10f0091dc5725 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.418+0000 7fa340ff9700 1 -- 172.21.15.2:0/3537732853 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0a2520 con 0x55630b22f550 2023-11-16T18:24:42.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.418+0000 7fa340ff9700 1 -- 172.21.15.2:0/3537732853 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa33c098470 con 0x55630b22f550 2023-11-16T18:24:42.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.418+0000 7fa340ff9700 1 -- 172.21.15.2:0/3537732853 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0995c0 con 0x55630b22f550 2023-11-16T18:24:42.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.419+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3537732853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b22f550 msgr2=0x55630b23faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.419+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3537732853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b22f550 0x55630b23faf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa33c096b40 tx=0x7fa33c010c40 comp rx=0 tx=0).stop 2023-11-16T18:24:42.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.419+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3537732853 shutdown_connections 2023-11-16T18:24:42.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.419+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3537732853 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22ec00 0x55630b2385b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.419+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3537732853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b230110 0x55630b238af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.419+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3537732853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b22f550 0x55630b23faf0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.419+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3537732853 >> 172.21.15.2:0/3537732853 conn(0x55630b224360 msgr2=0x55630b232f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.419+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3537732853 shutdown_connections 2023-11-16T18:24:42.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.419+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3537732853 wait complete. 2023-11-16T18:24:42.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.419+0000 7fa35fed0b80 1 -- start start 2023-11-16T18:24:42.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.419+0000 7fa35fed0b80 1 Processor -- start 2023-11-16T18:24:42.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa35fed0b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22ec00 0x55630b23bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa35fed0b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22f550 0x55630b23c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa35fed0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b23f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa35fed0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630b23f6a0 con 0x55630b230110 2023-11-16T18:24:42.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa35fed0b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630b23f810 con 0x55630b22ec00 2023-11-16T18:24:42.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa35fed0b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630b23f9d0 con 0x55630b22f550 2023-11-16T18:24:42.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa35fed0b80 10 client.0 _ll_register_callbacks cb 0x7fa33c00df70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:24:42.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3522e1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b23f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3522e1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b23f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49938/0 (socket says 172.21.15.2:49938) 2023-11-16T18:24:42.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3522e1700 1 -- 172.21.15.2:0/3548812533 learned_addr learned my addr 172.21.15.2:0/3548812533 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:42.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa351ae0700 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22ec00 0x55630b23bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3512df700 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22f550 0x55630b23c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3522e1700 1 -- 172.21.15.2:0/3548812533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22f550 msgr2=0x55630b23c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3522e1700 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22f550 0x55630b23c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3522e1700 1 -- 172.21.15.2:0/3548812533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22ec00 msgr2=0x55630b23bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:24:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3522e1700 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22ec00 0x55630b23bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3522e1700 1 -- 172.21.15.2:0/3548812533 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c10ba40 con 0x55630b230110 2023-11-16T18:24:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3512df700 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22f550 0x55630b23c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:24:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3522e1700 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b23f160 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa344004020 tx=0x7fa344009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c9dcaacc453de7c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3357fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:24:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3357fa700 1 -- 172.21.15.2:0/3548812533 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440073e0 con 0x55630b230110 2023-11-16T18:24:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa3357fa700 1 -- 172.21.15.2:0/3548812533 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fa34401eb60 con 0x55630b230110 2023-11-16T18:24:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.420+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3548812533 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa344003b60 con 0x55630b230110 2023-11-16T18:24:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.421+0000 7fa3357fa700 1 -- 172.21.15.2:0/3548812533 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa344007850 con 0x55630b230110 2023-11-16T18:24:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.421+0000 7fa35fed0b80 10 client.15579 fetch_fsmap learned FSMap version 24 2023-11-16T18:24:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.421+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3548812533 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55630af96c60 con 0x55630b230110 2023-11-16T18:24:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.421+0000 7fa3357fa700 1 -- 172.21.15.2:0/3548812533 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa344003b60 con 0x55630b230110 2023-11-16T18:24:42.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.421+0000 7fa3357fa700 1 -- 172.21.15.2:0/3548812533 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa34401ed20 con 0x55630b230110 2023-11-16T18:24:42.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.421+0000 7fa35fed0b80 10 client.15579 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:24:42.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.421+0000 7fa35fed0b80 10 client.15579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:24:42.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.421+0000 7fa35fed0b80 20 client.15579 populate_metadata read hostname 'smithi002' 2023-11-16T18:24:42.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.421+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55630b299010 0x55630b29b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:42.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.421+0000 7fa35fed0b80 4 client.15579 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:24:42.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.421+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3548812533 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55630b14e660 con 0x55630b299010 2023-11-16T18:24:42.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.421+0000 7fa351ae0700 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55630b299010 0x55630b29b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:42.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa351ae0700 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55630b299010 0x55630b29b3f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74ab0adcc001facd server_cookie=d31008bf371feed8 in_seq=0 out_seq=0 2023-11-16T18:24:42.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa3357fa700 10 client.15579 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:24:42.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa3357fa700 1 -- 172.21.15.2:0/3548812533 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55630b14e660 con 0x55630b299010 2023-11-16T18:24:42.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa3357fa700 10 client.15579 handle_command_reply: tid=0 2023-11-16T18:24:42.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa35fed0b80 1 client.15579 shutdown 2023-11-16T18:24:42.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3548812533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 msgr2=0x55630b23f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b23f160 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa344004020 tx=0x7fa344009650 comp rx=0 tx=0).stop 2023-11-16T18:24:42.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3548812533 shutdown_connections 2023-11-16T18:24:42.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55630b299010 0x55630b29b3f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55630b230110 0x55630b23f160 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55630b22ec00 0x55630b23bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa35fed0b80 1 --2- 172.21.15.2:0/3548812533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55630b22f550 0x55630b23c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3548812533 >> 172.21.15.2:0/3548812533 conn(0x55630b224360 msgr2=0x55630b232f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa3357fa700 0 client.15579 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:24:42.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3548812533 shutdown_connections 2023-11-16T18:24:42.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.422+0000 7fa35fed0b80 1 -- 172.21.15.2:0/3548812533 wait complete. 2023-11-16T18:24:42.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.423+0000 7fa35fed0b80 20 client.15579 trim_cache size 0 max 16384 2023-11-16T18:24:42.432 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:24:42.432 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-16T18:24:42.432 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": { 2023-11-16T18:24:42.432 INFO:teuthology.orchestra.run.smithi002.stdout: "a7a9ed8c-d014-4a6d-a850-f2e632e800ce": { 2023-11-16T18:24:42.432 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/", 2023-11-16T18:24:42.432 INFO:teuthology.orchestra.run.smithi002.stdout: "tag": "a7a9ed8c-d014-4a6d-a850-f2e632e800ce", 2023-11-16T18:24:42.433 INFO:teuthology.orchestra.run.smithi002.stdout: "options": "recursive,force" 2023-11-16T18:24:42.433 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:24:42.433 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:24:42.433 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:24:42.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.423+0000 7fa353d44700 1 -- 172.21.15.2:0/4016441940 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa3380a1150 msgr2=0x7fa3380a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.423+0000 7fa353d44700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa3380a1150 0x7fa3380a3600 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa348006010 tx=0x7fa34800b540 comp rx=0 tx=0).stop 2023-11-16T18:24:42.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.423+0000 7fa353d44700 1 -- 172.21.15.2:0/4016441940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa34c103340 msgr2=0x7fa34c1964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.423+0000 7fa353d44700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa34c103340 0x7fa34c1964d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00d950 tx=0x7fa33c00dd10 comp rx=0 tx=0).stop 2023-11-16T18:24:42.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.423+0000 7fa353d44700 1 -- 172.21.15.2:0/4016441940 shutdown_connections 2023-11-16T18:24:42.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.423+0000 7fa353d44700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7fa3380a1150 0x7fa3380a3600 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.423+0000 7fa353d44700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa34c068b00 0x7fa34c195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.423+0000 7fa353d44700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa34c105e50 0x7fa34c19d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.423+0000 7fa353d44700 1 --2- 172.21.15.2:0/4016441940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa34c103340 0x7fa34c1964d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.423+0000 7fa353d44700 1 -- 172.21.15.2:0/4016441940 >> 172.21.15.2:0/4016441940 conn(0x7fa34c0fa750 msgr2=0x7fa34c101680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.423+0000 7fa353d44700 1 -- 172.21.15.2:0/4016441940 shutdown_connections 2023-11-16T18:24:42.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.423+0000 7fa353d44700 1 -- 172.21.15.2:0/4016441940 wait complete. 2023-11-16T18:24:42.440 INFO:tasks.cephfs.filesystem:scrub status for tag:a7a9ed8c-d014-4a6d-a850-f2e632e800ce - {'path': '/', 'tag': 'a7a9ed8c-d014-4a6d-a850-f2e632e800ce', 'options': 'recursive,force'} 2023-11-16T18:24:42.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.482+0000 7f868d19a700 1 -- 172.21.15.2:0/2837976314 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f86880619b0 con 0x7f8688101470 2023-11-16T18:24:42.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.843+0000 7f866ffff700 1 -- 172.21.15.2:0/2837976314 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 19) v1 ==== 135731+0+0 (secure 0 0 0) 0x7f867c0105d0 con 0x7f8688101470 2023-11-16T18:24:42.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.844+0000 7f866ffff700 1 -- 172.21.15.2:0/2837976314 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f867c090e20 con 0x7f8688101470 2023-11-16T18:24:42.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.847+0000 7f868d19a700 1 -- 172.21.15.2:0/2837976314 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f86700a1150 msgr2=0x7f86700a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.847+0000 7f868d19a700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f86700a1150 0x7f86700a3600 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f86881012d0 tx=0x7f8678009260 comp rx=0 tx=0).stop 2023-11-16T18:24:42.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.847+0000 7f868d19a700 1 -- 172.21.15.2:0/2837976314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8688101470 msgr2=0x7f8688072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:42.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.847+0000 7f868d19a700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8688101470 0x7f8688072ee0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f867c00d950 tx=0x7f867c00dc60 comp rx=0 tx=0).stop 2023-11-16T18:24:42.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.847+0000 7f868d19a700 1 -- 172.21.15.2:0/2837976314 shutdown_connections 2023-11-16T18:24:42.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.847+0000 7f868d19a700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f86700a1150 0x7f86700a3600 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.847+0000 7f868d19a700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688101e00 0x7f8688073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.847+0000 7f868d19a700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8688100280 0x7f86880748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.847+0000 7f868d19a700 1 --2- 172.21.15.2:0/2837976314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8688101470 0x7f8688072ee0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.847+0000 7f868d19a700 1 -- 172.21.15.2:0/2837976314 >> 172.21.15.2:0/2837976314 conn(0x7f86880fb8a0 msgr2=0x7f86881048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.847+0000 7f868d19a700 1 -- 172.21.15.2:0/2837976314 shutdown_connections 2023-11-16T18:24:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:42.847+0000 7f868d19a700 1 -- 172.21.15.2:0/2837976314 wait complete. 2023-11-16T18:24:42.864 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2023-11-16T18:24:43.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.007+0000 7f173c950700 1 -- 172.21.15.2:0/2420996472 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1738105e20 msgr2=0x7f17381082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:43.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.007+0000 7f173c950700 1 --2- 172.21.15.2:0/2420996472 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1738105e20 0x7f17381082d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f173000b0a0 tx=0x7f173000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.007+0000 7f173c950700 1 -- 172.21.15.2:0/2420996472 shutdown_connections 2023-11-16T18:24:43.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.007+0000 7f173c950700 1 --2- 172.21.15.2:0/2420996472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1738069e90 0x7f1738102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.007+0000 7f173c950700 1 --2- 172.21.15.2:0/2420996472 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1738105e20 0x7f17381082d0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.007+0000 7f173c950700 1 --2- 172.21.15.2:0/2420996472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1738103310 0x7f1738105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.007+0000 7f173c950700 1 -- 172.21.15.2:0/2420996472 >> 172.21.15.2:0/2420996472 conn(0x7f17380fa780 msgr2=0x7f17380fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:43.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.007+0000 7f173c950700 1 -- 172.21.15.2:0/2420996472 shutdown_connections 2023-11-16T18:24:43.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.007+0000 7f173c950700 1 -- 172.21.15.2:0/2420996472 wait complete. 2023-11-16T18:24:43.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.008+0000 7f173c950700 1 Processor -- start 2023-11-16T18:24:43.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.008+0000 7f173c950700 1 -- start start 2023-11-16T18:24:43.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.008+0000 7f173c950700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1738069e90 0x7f1738100730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:43.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.008+0000 7f173c950700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1738103310 0x7f17380fed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:43.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.008+0000 7f173c950700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1738105e20 0x7f17380ff2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:43.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.008+0000 7f173c950700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17380ffa50 con 0x7f1738069e90 2023-11-16T18:24:43.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.008+0000 7f173c950700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17380ffbc0 con 0x7f1738105e20 2023-11-16T18:24:43.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.008+0000 7f173c950700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17381a0170 con 0x7f1738103310 2023-11-16T18:24:43.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f173659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1738069e90 0x7f1738100730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f1736d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1738105e20 0x7f17380ff2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f1735d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1738103310 0x7f17380fed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f173659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1738069e90 0x7f1738100730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49978/0 (socket says 172.21.15.2:49978) 2023-11-16T18:24:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f173659c700 1 -- 172.21.15.2:0/4132872887 learned_addr learned my addr 172.21.15.2:0/4132872887 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f173659c700 1 -- 172.21.15.2:0/4132872887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1738103310 msgr2=0x7f17380fed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f173659c700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1738103310 0x7f17380fed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f173659c700 1 -- 172.21.15.2:0/4132872887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1738105e20 msgr2=0x7f17380ff2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f173659c700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1738105e20 0x7f17380ff2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f173659c700 1 -- 172.21.15.2:0/4132872887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1730009d00 con 0x7f1738069e90 2023-11-16T18:24:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f1735d9b700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1738103310 0x7f17380fed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:24:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f1736d9d700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1738105e20 0x7f17380ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:24:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f173659c700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1738069e90 0x7f1738100730 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f172800e970 tx=0x7f172800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f17277fe700 1 -- 172.21.15.2:0/4132872887 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1728009790 con 0x7f1738069e90 2023-11-16T18:24:43.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f173c950700 1 -- 172.21.15.2:0/4132872887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173810bc10 con 0x7f1738069e90 2023-11-16T18:24:43.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f17277fe700 1 -- 172.21.15.2:0/4132872887 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f172800c7e0 con 0x7f1738069e90 2023-11-16T18:24:43.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f17277fe700 1 -- 172.21.15.2:0/4132872887 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17280054e0 con 0x7f1738069e90 2023-11-16T18:24:43.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.009+0000 7f173c950700 1 -- 172.21.15.2:0/4132872887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173810ba80 con 0x7f1738069e90 2023-11-16T18:24:43.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.010+0000 7f173c950700 1 -- 172.21.15.2:0/4132872887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17380fc2f0 con 0x7f1738069e90 2023-11-16T18:24:43.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.011+0000 7f17277fe700 1 -- 172.21.15.2:0/4132872887 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 135731+0+0 (secure 0 0 0) 0x7f172800c950 con 0x7f1738069e90 2023-11-16T18:24:43.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.012+0000 7f17277fe700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f17200a11a0 0x7f17200a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:43.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.012+0000 7f1735d9b700 1 -- 172.21.15.2:0/4132872887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f17200a11a0 msgr2=0x7f17200a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3369431713 2023-11-16T18:24:43.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.012+0000 7f17277fe700 1 -- 172.21.15.2:0/4132872887 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f1728014070 con 0x7f1738069e90 2023-11-16T18:24:43.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.012+0000 7f1735d9b700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f17200a11a0 0x7f17200a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T18:24:43.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.016+0000 7f17277fe700 1 -- 172.21.15.2:0/4132872887 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f17280962d0 con 0x7f1738069e90 2023-11-16T18:24:43.097 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:42 smithi188 ceph-mon[100639]: pgmap v322: 129 pgs: 129 active+clean; 646 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-11-16T18:24:43.097 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:42 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/636408729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:43.097 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:42 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2837976314' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-11-16T18:24:43.097 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: ignoring --setuser ceph since I am not root 2023-11-16T18:24:43.097 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: ignoring --setgroup ceph since I am not root 2023-11-16T18:24:43.097 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:42 smithi188 ceph-mgr[102802]: -- 172.21.15.188:0/2034075545 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ee24665520 con 0x55ee2463ac00 2023-11-16T18:24:43.098 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:43.015+0000 7fa8986b0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T18:24:43.098 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:43.095+0000 7fa8986b0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T18:24:43.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:42 smithi002 ceph-mon[98807]: pgmap v322: 129 pgs: 129 active+clean; 646 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-11-16T18:24:43.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:42 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/636408729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:43.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:42 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2837976314' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-11-16T18:24:43.106 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:42 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: ignoring --setuser ceph since I am not root 2023-11-16T18:24:43.106 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:42 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: ignoring --setgroup ceph since I am not root 2023-11-16T18:24:43.106 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:42 smithi002 ceph-mgr[99075]: -- 172.21.15.2:0/3172364447 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557d92f11520 con 0x557d92ee7000 2023-11-16T18:24:43.106 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:43 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:43.048+0000 7fb6ea0ec200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T18:24:43.141 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:42 smithi040 ceph-mon[100251]: pgmap v322: 129 pgs: 129 active+clean; 646 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-11-16T18:24:43.141 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:42 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/636408729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:24:43.141 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:42 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2837976314' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-11-16T18:24:43.141 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:42 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: ignoring --setuser ceph since I am not root 2023-11-16T18:24:43.141 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:42 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: ignoring --setgroup ceph since I am not root 2023-11-16T18:24:43.141 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:42 smithi040 ceph-mgr[101731]: -- 172.21.15.40:0/2861831901 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5573de057520 con 0x5573de02d000 2023-11-16T18:24:43.141 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:43.041+0000 7fc84567f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T18:24:43.141 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:43.138+0000 7fc84567f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T18:24:43.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.213+0000 7f1735d9b700 1 -- 172.21.15.2:0/4132872887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f17200a11a0 msgr2=0x7f17200a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3369431713 2023-11-16T18:24:43.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.213+0000 7f1735d9b700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f17200a11a0 0x7f17200a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T18:24:43.227 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.225+0000 7f173c950700 1 -- 172.21.15.2:0/4132872887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f173802cfe0 con 0x7f1738069e90 2023-11-16T18:24:43.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.230+0000 7f17277fe700 1 -- 172.21.15.2:0/4132872887 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f17280998f0 con 0x7f1738069e90 2023-11-16T18:24:43.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.232+0000 7f173c950700 1 -- 172.21.15.2:0/4132872887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f17200a11a0 msgr2=0x7f17200a3650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T18:24:43.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.232+0000 7f173c950700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f17200a11a0 0x7f17200a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.232+0000 7f173c950700 1 -- 172.21.15.2:0/4132872887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1738069e90 msgr2=0x7f1738100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:43.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.232+0000 7f173c950700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1738069e90 0x7f1738100730 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f172800e970 tx=0x7f172800ec80 comp rx=0 tx=0).stop 2023-11-16T18:24:43.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.232+0000 7f173c950700 1 -- 172.21.15.2:0/4132872887 shutdown_connections 2023-11-16T18:24:43.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.232+0000 7f173c950700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f17200a11a0 0x7f17200a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.232+0000 7f173c950700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1738105e20 0x7f17380ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.232+0000 7f173c950700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1738069e90 0x7f1738100730 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.232+0000 7f173c950700 1 --2- 172.21.15.2:0/4132872887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1738103310 0x7f17380fed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.232+0000 7f173c950700 1 -- 172.21.15.2:0/4132872887 >> 172.21.15.2:0/4132872887 conn(0x7f17380fa780 msgr2=0x7f1738101650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:43.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.232+0000 7f173c950700 1 -- 172.21.15.2:0/4132872887 shutdown_connections 2023-11-16T18:24:43.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.232+0000 7f173c950700 1 -- 172.21.15.2:0/4132872887 wait complete. 2023-11-16T18:24:43.247 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2023-11-16T18:24:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.374+0000 7feec24b2700 1 -- 172.21.15.2:0/3787631249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc0ff140 msgr2=0x7feebc0ff550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.374+0000 7feec24b2700 1 --2- 172.21.15.2:0/3787631249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc0ff140 0x7feebc0ff550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7feeac0098f0 tx=0x7feeac009c00 comp rx=0 tx=0).stop 2023-11-16T18:24:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.374+0000 7feec24b2700 1 -- 172.21.15.2:0/3787631249 shutdown_connections 2023-11-16T18:24:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.374+0000 7feec24b2700 1 --2- 172.21.15.2:0/3787631249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feebc100cc0 0x7feebc107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.374+0000 7feec24b2700 1 --2- 172.21.15.2:0/3787631249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc0ff140 0x7feebc0ff550 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.374+0000 7feec24b2700 1 --2- 172.21.15.2:0/3787631249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feebc100330 0x7feebc100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.374+0000 7feec24b2700 1 -- 172.21.15.2:0/3787631249 >> 172.21.15.2:0/3787631249 conn(0x7feebc0fa720 msgr2=0x7feebc0fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.374+0000 7feec24b2700 1 -- 172.21.15.2:0/3787631249 shutdown_connections 2023-11-16T18:24:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.374+0000 7feec24b2700 1 -- 172.21.15.2:0/3787631249 wait complete. 2023-11-16T18:24:43.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.375+0000 7feec24b2700 1 Processor -- start 2023-11-16T18:24:43.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.375+0000 7feec24b2700 1 -- start start 2023-11-16T18:24:43.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feec24b2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc0ff140 0x7feebc068bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:43.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feec24b2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feebc100330 0x7feebc19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:43.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feec24b2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feebc100cc0 0x7feebc19d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:43.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feec24b2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc19dbe0 con 0x7feebc100330 2023-11-16T18:24:43.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feec24b2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc19dd50 con 0x7feebc0ff140 2023-11-16T18:24:43.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feec24b2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc19df10 con 0x7feebc100cc0 2023-11-16T18:24:43.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feebb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feebc100330 0x7feebc19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:43.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feebb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feebc100330 0x7feebc19ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50020/0 (socket says 172.21.15.2:50020) 2023-11-16T18:24:43.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feebb7fe700 1 -- 172.21.15.2:0/2077441041 learned_addr learned my addr 172.21.15.2:0/2077441041 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:43.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feebbfff700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc0ff140 0x7feebc068bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:43.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feec0a4f700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feebc100cc0 0x7feebc19d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:43.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feebb7fe700 1 -- 172.21.15.2:0/2077441041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feebc100cc0 msgr2=0x7feebc19d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:43.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feebb7fe700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feebc100cc0 0x7feebc19d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feebb7fe700 1 -- 172.21.15.2:0/2077441041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc0ff140 msgr2=0x7feebc068bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:43.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feebb7fe700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc0ff140 0x7feebc068bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feebb7fe700 1 -- 172.21.15.2:0/2077441041 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb4009d00 con 0x7feebc100330 2023-11-16T18:24:43.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feebbfff700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc0ff140 0x7feebc068bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:24:43.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.376+0000 7feebb7fe700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feebc100330 0x7feebc19ae00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7feeb000e9a0 tx=0x7feeb000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:43.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.377+0000 7feeb97fa700 1 -- 172.21.15.2:0/2077441041 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb0009790 con 0x7feebc100330 2023-11-16T18:24:43.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.377+0000 7feec24b2700 1 -- 172.21.15.2:0/2077441041 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac009520 con 0x7feebc100330 2023-11-16T18:24:43.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.377+0000 7feeb97fa700 1 -- 172.21.15.2:0/2077441041 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feeb0010430 con 0x7feebc100330 2023-11-16T18:24:43.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.377+0000 7feeb97fa700 1 -- 172.21.15.2:0/2077441041 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb0018680 con 0x7feebc100330 2023-11-16T18:24:43.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.377+0000 7feec24b2700 1 -- 172.21.15.2:0/2077441041 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feebc10aaf0 con 0x7feebc100330 2023-11-16T18:24:43.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.378+0000 7feec24b2700 1 -- 172.21.15.2:0/2077441041 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feebc10ac80 con 0x7feebc100330 2023-11-16T18:24:43.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.379+0000 7feeb97fa700 1 -- 172.21.15.2:0/2077441041 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 135731+0+0 (secure 0 0 0) 0x7feeb00105a0 con 0x7feebc100330 2023-11-16T18:24:43.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.379+0000 7feeb97fa700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feea40a11a0 0x7feea40a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:43.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.379+0000 7feebbfff700 1 -- 172.21.15.2:0/2077441041 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feea40a11a0 msgr2=0x7feea40a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3369431713 2023-11-16T18:24:43.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.379+0000 7feeb97fa700 1 -- 172.21.15.2:0/2077441041 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7feeb0014070 con 0x7feebc100330 2023-11-16T18:24:43.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.379+0000 7feebbfff700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feea40a11a0 0x7feea40a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T18:24:43.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.388+0000 7feeb97fa700 1 -- 172.21.15.2:0/2077441041 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7feeb0051730 con 0x7feebc100330 2023-11-16T18:24:43.448 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:43.185+0000 7fa8986b0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T18:24:43.449 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:43.278+0000 7fa8986b0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T18:24:43.449 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:43.446+0000 7fa8986b0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T18:24:43.483 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:43.248+0000 7fc84567f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T18:24:43.483 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:43.359+0000 7fc84567f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T18:24:43.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.580+0000 7feebbfff700 1 -- 172.21.15.2:0/2077441041 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feea40a11a0 msgr2=0x7feea40a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3369431713 2023-11-16T18:24:43.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.580+0000 7feebbfff700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feea40a11a0 0x7feea40a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T18:24:43.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.591+0000 7feec24b2700 1 -- 172.21.15.2:0/2077441041 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7feebc061770 con 0x7feebc100330 2023-11-16T18:24:43.595 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:43 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:43.148+0000 7fb6ea0ec200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T18:24:43.595 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:43 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:43.259+0000 7fb6ea0ec200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T18:24:43.595 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:43 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:43.378+0000 7fb6ea0ec200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T18:24:43.595 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:43 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:43.591+0000 7fb6ea0ec200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T18:24:43.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.596+0000 7feeb97fa700 1 -- 172.21.15.2:0/2077441041 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7feeb0099d10 con 0x7feebc100330 2023-11-16T18:24:43.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.598+0000 7feec24b2700 1 -- 172.21.15.2:0/2077441041 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feea40a11a0 msgr2=0x7feea40a3650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T18:24:43.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.598+0000 7feec24b2700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feea40a11a0 0x7feea40a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.598+0000 7feec24b2700 1 -- 172.21.15.2:0/2077441041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feebc100330 msgr2=0x7feebc19ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:43.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.598+0000 7feec24b2700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feebc100330 0x7feebc19ae00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7feeb000e9a0 tx=0x7feeb000ecb0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.598+0000 7feec24b2700 1 -- 172.21.15.2:0/2077441041 shutdown_connections 2023-11-16T18:24:43.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.598+0000 7feec24b2700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7feea40a11a0 0x7feea40a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.598+0000 7feec24b2700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feebc100cc0 0x7feebc19d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.598+0000 7feec24b2700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc0ff140 0x7feebc068bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.598+0000 7feec24b2700 1 --2- 172.21.15.2:0/2077441041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feebc100330 0x7feebc19ae00 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.598+0000 7feec24b2700 1 -- 172.21.15.2:0/2077441041 >> 172.21.15.2:0/2077441041 conn(0x7feebc0fa720 msgr2=0x7feebc103910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:43.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.598+0000 7feec24b2700 1 -- 172.21.15.2:0/2077441041 shutdown_connections 2023-11-16T18:24:43.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.598+0000 7feec24b2700 1 -- 172.21.15.2:0/2077441041 wait complete. 2023-11-16T18:24:43.614 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M' 2023-11-16T18:24:43.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.744+0000 7f411d3a0700 1 -- 172.21.15.2:0/3618096431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41180feb30 msgr2=0x7f41180fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:43.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.744+0000 7f411d3a0700 1 --2- 172.21.15.2:0/3618096431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41180feb30 0x7f41180fef40 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4108009990 tx=0x7f4108009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.744+0000 7f411d3a0700 1 -- 172.21.15.2:0/3618096431 shutdown_connections 2023-11-16T18:24:43.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.744+0000 7f411d3a0700 1 --2- 172.21.15.2:0/3618096431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41180ffe30 0x7f41181002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.744+0000 7f411d3a0700 1 --2- 172.21.15.2:0/3618096431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41180feb30 0x7f41180fef40 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.744+0000 7f411d3a0700 1 --2- 172.21.15.2:0/3618096431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41180ff480 0x7f41180ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.744+0000 7f411d3a0700 1 -- 172.21.15.2:0/3618096431 >> 172.21.15.2:0/3618096431 conn(0x7f41180fa720 msgr2=0x7f41180fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:43.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.744+0000 7f411d3a0700 1 -- 172.21.15.2:0/3618096431 shutdown_connections 2023-11-16T18:24:43.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.744+0000 7f411d3a0700 1 -- 172.21.15.2:0/3618096431 wait complete. 2023-11-16T18:24:43.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f411d3a0700 1 Processor -- start 2023-11-16T18:24:43.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f411d3a0700 1 -- start start 2023-11-16T18:24:43.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f411d3a0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41180feb30 0x7f41181981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:43.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f411d3a0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41180ff480 0x7f41181986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:43.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f411d3a0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41180ffe30 0x7f411819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:43.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f411d3a0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411819fd30 con 0x7f41180ff480 2023-11-16T18:24:43.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f411d3a0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411819fe70 con 0x7f41180feb30 2023-11-16T18:24:43.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f411d3a0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41181a0000 con 0x7f41180ffe30 2023-11-16T18:24:43.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f41167fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41180ff480 0x7f41181986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:43.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f41167fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41180ff480 0x7f41181986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50066/0 (socket says 172.21.15.2:50066) 2023-11-16T18:24:43.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f4116ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41180feb30 0x7f41181981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:43.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f41167fc700 1 -- 172.21.15.2:0/310042898 learned_addr learned my addr 172.21.15.2:0/310042898 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:43.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f41177fe700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41180ffe30 0x7f411819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:43.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f41167fc700 1 -- 172.21.15.2:0/310042898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41180ffe30 msgr2=0x7f411819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f41167fc700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41180ffe30 0x7f411819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f41167fc700 1 -- 172.21.15.2:0/310042898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41180feb30 msgr2=0x7f41181981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f41167fc700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41180feb30 0x7f41181981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f41167fc700 1 -- 172.21.15.2:0/310042898 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41080095f0 con 0x7f41180ff480 2023-11-16T18:24:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.745+0000 7f4116ffd700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41180feb30 0x7f41181981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:24:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.746+0000 7f41167fc700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41180ff480 0x7f41181986f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f410c00e9a0 tx=0x7f410c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.746+0000 7f40fffff700 1 -- 172.21.15.2:0/310042898 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c009790 con 0x7f41180ff480 2023-11-16T18:24:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.746+0000 7f411d3a0700 1 -- 172.21.15.2:0/310042898 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411810dd90 con 0x7f41180ff480 2023-11-16T18:24:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.746+0000 7f40fffff700 1 -- 172.21.15.2:0/310042898 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f410c010430 con 0x7f41180ff480 2023-11-16T18:24:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.746+0000 7f40fffff700 1 -- 172.21.15.2:0/310042898 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c018680 con 0x7f41180ff480 2023-11-16T18:24:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.746+0000 7f411d3a0700 1 -- 172.21.15.2:0/310042898 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f411810dc00 con 0x7f41180ff480 2023-11-16T18:24:43.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.747+0000 7f411d3a0700 1 -- 172.21.15.2:0/310042898 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4118109910 con 0x7f41180ff480 2023-11-16T18:24:43.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.748+0000 7f40fffff700 1 -- 172.21.15.2:0/310042898 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 135731+0+0 (secure 0 0 0) 0x7f410c0105a0 con 0x7f41180ff480 2023-11-16T18:24:43.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.749+0000 7f40fffff700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 0x7f41000a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:43.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.749+0000 7f4116ffd700 1 -- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 msgr2=0x7f41000a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3369431713 2023-11-16T18:24:43.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.749+0000 7f40fffff700 1 -- 172.21.15.2:0/310042898 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f410c014070 con 0x7f41180ff480 2023-11-16T18:24:43.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.749+0000 7f4116ffd700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 0x7f41000a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T18:24:43.759 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.757+0000 7f40fffff700 1 -- 172.21.15.2:0/310042898 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f410c096470 con 0x7f41180ff480 2023-11-16T18:24:43.839 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:43.566+0000 7fc84567f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T18:24:43.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:43 smithi002 ceph-mon[98807]: scrub queued for path: / 2023-11-16T18:24:43.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:43 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:24:43.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:43 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:24:43.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:43 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:24:43.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:43 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2837976314' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-11-16T18:24:43.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:43 smithi002 ceph-mon[98807]: mgrmap e19: x(active, since 8m), standbys: z, y 2023-11-16T18:24:43.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:43 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/4132872887' entity='client.admin' 2023-11-16T18:24:43.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:43 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2077441041' entity='client.admin' 2023-11-16T18:24:43.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.950+0000 7f4116ffd700 1 -- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 msgr2=0x7f41000a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3369431713 2023-11-16T18:24:43.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.950+0000 7f4116ffd700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 0x7f41000a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T18:24:43.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:43.986+0000 7f411d3a0700 1 -- 172.21.15.2:0/310042898 --> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f4118061a30 con 0x7f41000a11a0 2023-11-16T18:24:44.046 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:43.755+0000 7fa8986b0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T18:24:44.047 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:43.845+0000 7fa8986b0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T18:24:44.047 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:43.974+0000 7fa8986b0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T18:24:44.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-mon[100639]: scrub queued for path: / 2023-11-16T18:24:44.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:24:44.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:24:44.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:24:44.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2837976314' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-11-16T18:24:44.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-mon[100639]: mgrmap e19: x(active, since 8m), standbys: z, y 2023-11-16T18:24:44.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/4132872887' entity='client.admin' 2023-11-16T18:24:44.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:43 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2077441041' entity='client.admin' 2023-11-16T18:24:44.223 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:43.961+0000 7fc84567f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T18:24:44.223 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:44 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:44.069+0000 7fc84567f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T18:24:44.224 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-mon[100251]: scrub queued for path: / 2023-11-16T18:24:44.224 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:24:44.224 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:24:44.224 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:24:44.224 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2837976314' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-11-16T18:24:44.224 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-mon[100251]: mgrmap e19: x(active, since 8m), standbys: z, y 2023-11-16T18:24:44.224 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/4132872887' entity='client.admin' 2023-11-16T18:24:44.224 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:43 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2077441041' entity='client.admin' 2023-11-16T18:24:44.263 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:44 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:43.998+0000 7fb6ea0ec200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T18:24:44.263 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:44 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:44.107+0000 7fb6ea0ec200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T18:24:44.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:44.351+0000 7f4116ffd700 1 -- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 msgr2=0x7f41000a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3369431713 2023-11-16T18:24:44.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:44.351+0000 7f4116ffd700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 0x7f41000a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-16T18:24:44.483 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:44 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:44.221+0000 7fc84567f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T18:24:44.483 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:44 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:44.337+0000 7fc84567f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T18:24:44.486 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:44.068+0000 7fa8986b0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T18:24:44.605 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:44 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:44.260+0000 7fb6ea0ec200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T18:24:44.606 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:44 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:44.376+0000 7fb6ea0ec200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T18:24:45.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:45.153+0000 7f4116ffd700 1 -- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 msgr2=0x7f41000a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3369431713 2023-11-16T18:24:45.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:45.153+0000 7f4116ffd700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 0x7f41000a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-16T18:24:45.233 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:44 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:44.847+0000 7fc84567f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T18:24:45.355 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:44 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:44.890+0000 7fb6ea0ec200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T18:24:45.546 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:44.484+0000 7fa8986b0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T18:24:45.546 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:45 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:45.045+0000 7fa8986b0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T18:24:45.547 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:45 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:45.215+0000 7fa8986b0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T18:24:45.855 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:45 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:45.579+0000 7fb6ea0ec200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T18:24:45.855 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:45 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:45.773+0000 7fb6ea0ec200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T18:24:45.983 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:45 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:45.531+0000 7fc84567f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T18:24:45.983 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:45 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:45.723+0000 7fc84567f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T18:24:46.236 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:45 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:45.974+0000 7fa8986b0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:24:46.236 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:46 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:46.065+0000 7fa8986b0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T18:24:46.511 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:46 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:46.234+0000 7fa8986b0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:24:46.755 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:46.754+0000 7f4116ffd700 1 -- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 msgr2=0x7f41000a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3369431713 2023-11-16T18:24:46.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:46.754+0000 7f4116ffd700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 0x7f41000a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-16T18:24:46.796 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:46 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:46.510+0000 7fa8986b0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T18:24:46.796 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:46 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:46.736+0000 7fa8986b0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T18:24:46.983 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:46 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:46.684+0000 7fc84567f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:24:46.983 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:46 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:46.794+0000 7fc84567f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T18:24:47.083 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:46 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:46.745+0000 7fb6ea0ec200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:24:47.084 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:46 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:46.859+0000 7fb6ea0ec200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T18:24:47.146 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:46 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:46.885+0000 7fa8986b0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T18:24:47.364 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:47 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:47.016+0000 7fc84567f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:24:47.546 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:47 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:47.144+0000 7fa8986b0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T18:24:47.615 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:47 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:47.362+0000 7fc84567f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T18:24:47.691 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:47.082+0000 7fb6ea0ec200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T18:24:47.691 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:47.435+0000 7fb6ea0ec200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T18:24:47.983 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:47 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:47.614+0000 7fc84567f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T18:24:47.983 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:47 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:47.795+0000 7fc84567f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T18:24:48.105 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:47.690+0000 7fb6ea0ec200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T18:24:48.105 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:47.874+0000 7fb6ea0ec200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T18:24:48.162 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:47 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:47.842+0000 7fa8986b0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T18:24:48.162 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:47 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:47.972+0000 7fa8986b0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T18:24:48.483 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:48 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:48.122+0000 7fc84567f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T18:24:48.487 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:48 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:48.160+0000 7fa8986b0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T18:24:48.487 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:48 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:48.272+0000 7fa8986b0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T18:24:48.605 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:48 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:48.209+0000 7fb6ea0ec200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T18:24:48.796 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:48 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:48.485+0000 7fa8986b0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T18:24:49.233 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:48 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:48.945+0000 7fc84567f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T18:24:49.233 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:49 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:49.103+0000 7fc84567f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T18:24:49.355 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:49 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:49.047+0000 7fb6ea0ec200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T18:24:49.355 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:49 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:49.208+0000 7fb6ea0ec200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T18:24:49.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:49.418+0000 7f40fffff700 1 -- 172.21.15.2:0/310042898 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 20) v1 ==== 135731+0+0 (secure 0 0 0) 0x7f410c010880 con 0x7f41180ff480 2023-11-16T18:24:49.733 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:49 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:49.344+0000 7fc84567f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T18:24:49.734 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:49 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:49.482+0000 7fc84567f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T18:24:49.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:49 smithi040 ceph-mon[100251]: Standby manager daemon z restarted 2023-11-16T18:24:49.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:49 smithi040 ceph-mon[100251]: Standby manager daemon z started 2023-11-16T18:24:49.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:49 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.188:0/32493312' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T18:24:49.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:49 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.188:0/32493312' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T18:24:49.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:49 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.188:0/32493312' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T18:24:49.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:49 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.188:0/32493312' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T18:24:49.797 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:24:49 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z[102778]: 2023-11-16T18:24:49.342+0000 7fa8986b0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T18:24:49.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:49 smithi188 ceph-mon[100639]: Standby manager daemon z restarted 2023-11-16T18:24:49.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:49 smithi188 ceph-mon[100639]: Standby manager daemon z started 2023-11-16T18:24:49.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:49 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.188:0/32493312' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T18:24:49.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:49 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.188:0/32493312' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T18:24:49.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:49 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.188:0/32493312' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T18:24:49.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:49 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.188:0/32493312' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T18:24:49.855 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:49 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:49.462+0000 7fb6ea0ec200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T18:24:49.855 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:49 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:49.602+0000 7fb6ea0ec200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T18:24:49.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:49 smithi002 ceph-mon[98807]: Standby manager daemon z restarted 2023-11-16T18:24:49.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:49 smithi002 ceph-mon[98807]: Standby manager daemon z started 2023-11-16T18:24:49.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:49 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.188:0/32493312' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T18:24:49.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:49 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.188:0/32493312' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T18:24:49.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:49 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.188:0/32493312' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T18:24:49.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:49 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.188:0/32493312' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T18:24:49.959 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:49.958+0000 7f4116ffd700 1 -- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 msgr2=0x7f41000a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3369431713 2023-11-16T18:24:49.959 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:49.958+0000 7f4116ffd700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 0x7f41000a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-16T18:24:50.233 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:49 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:49.753+0000 7fc84567f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T18:24:50.355 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:49 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:49.883+0000 7fb6ea0ec200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T18:24:50.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:50 smithi040 ceph-mon[100251]: mgrmap e20: x(active, since 8m), standbys: y, z 2023-11-16T18:24:50.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:50 smithi188 ceph-mon[100639]: mgrmap e20: x(active, since 8m), standbys: y, z 2023-11-16T18:24:50.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:50 smithi002 ceph-mon[98807]: mgrmap e20: x(active, since 8m), standbys: y, z 2023-11-16T18:24:50.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:50.961+0000 7f40fffff700 1 -- 172.21.15.2:0/310042898 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mgrmap(e 21) v1 ==== 135400+0+0 (secure 0 0 0) 0x7f410c0ca8a0 con 0x7f41180ff480 2023-11-16T18:24:50.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:50.962+0000 7f40fffff700 1 -- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 msgr2=0x7f41000a3650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T18:24:50.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:50.962+0000 7f40fffff700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/3369431713,v1:172.21.15.2:6801/3369431713] conn(0x7f41000a11a0 0x7f41000a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:51.112 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:24:50 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y[101707]: 2023-11-16T18:24:50.789+0000 7fc84567f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T18:24:51.205 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:50 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:50.935+0000 7fb6ea0ec200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T18:24:51.205 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:51.179+0000 7fb64964a700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:24:51.205 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:51.179+0000 7fb64964a700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:24:51.205 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:51.179+0000 7fb64964a700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:24:51.205 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:51.179+0000 7fb64964a700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:24:51.206 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:51.179+0000 7fb64964a700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:24:51.206 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:51.182+0000 7fb646544700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:24:51.206 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:51.182+0000 7fb646544700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:24:51.206 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:51.182+0000 7fb646544700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:24:51.206 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:51.182+0000 7fb646544700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:24:51.207 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:24:51.182+0000 7fb646544700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:24:51.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: Standby manager daemon y restarted 2023-11-16T18:24:51.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: Standby manager daemon y started 2023-11-16T18:24:51.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.40:0/1509074593' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T18:24:51.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.40:0/1509074593' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T18:24:51.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.40:0/1509074593' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T18:24:51.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.? 172.21.15.40:0/1509074593' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T18:24:51.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: Active manager daemon x restarted 2023-11-16T18:24:51.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: Activating manager daemon x 2023-11-16T18:24:51.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: osdmap e91: 12 total, 12 up, 12 in 2023-11-16T18:24:51.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: mgrmap e21: x(active, starting, since 0.012926s), standbys: z, y 2023-11-16T18:24:51.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:24:51.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:24:51.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:24:51.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T18:24:51.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T18:24:51.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T18:24:51.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T18:24:51.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T18:24:51.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T18:24:51.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T18:24:51.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T18:24:51.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T18:24:51.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T18:24:51.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T18:24:51.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T18:24:51.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T18:24:51.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T18:24:51.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T18:24:51.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:24:51.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:24:51.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:24:51.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:24:51.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:24:51.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:24:51.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:24:51.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:24:51.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:24:51.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:24:51.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:24:51.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:24:51.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T18:24:51.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T18:24:51.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T18:24:51.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: Manager daemon x is now available 2023-11-16T18:24:51.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:24:51.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:24:51.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T18:24:51.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:51 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T18:24:51.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: Standby manager daemon y restarted 2023-11-16T18:24:51.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: Standby manager daemon y started 2023-11-16T18:24:51.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.40:0/1509074593' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T18:24:51.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.40:0/1509074593' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T18:24:51.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.40:0/1509074593' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T18:24:51.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.? 172.21.15.40:0/1509074593' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T18:24:51.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: Active manager daemon x restarted 2023-11-16T18:24:51.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: Activating manager daemon x 2023-11-16T18:24:51.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: osdmap e91: 12 total, 12 up, 12 in 2023-11-16T18:24:51.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: mgrmap e21: x(active, starting, since 0.012926s), standbys: z, y 2023-11-16T18:24:51.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:24:51.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:24:51.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:24:51.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T18:24:51.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T18:24:51.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T18:24:51.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T18:24:51.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T18:24:51.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T18:24:51.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T18:24:51.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T18:24:51.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T18:24:51.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T18:24:51.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T18:24:51.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T18:24:51.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T18:24:51.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T18:24:51.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T18:24:51.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:24:51.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:24:51.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:24:51.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:24:51.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:24:51.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:24:51.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:24:51.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:24:51.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:24:51.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:24:51.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:24:51.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:24:51.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T18:24:51.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T18:24:51.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T18:24:51.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: Manager daemon x is now available 2023-11-16T18:24:51.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:24:51.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:24:51.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T18:24:51.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:51 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T18:24:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: Standby manager daemon y restarted 2023-11-16T18:24:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: Standby manager daemon y started 2023-11-16T18:24:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.40:0/1509074593' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T18:24:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.40:0/1509074593' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T18:24:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.40:0/1509074593' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T18:24:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.? 172.21.15.40:0/1509074593' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T18:24:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: Active manager daemon x restarted 2023-11-16T18:24:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: Activating manager daemon x 2023-11-16T18:24:51.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: osdmap e91: 12 total, 12 up, 12 in 2023-11-16T18:24:51.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: mgrmap e21: x(active, starting, since 0.012926s), standbys: z, y 2023-11-16T18:24:51.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T18:24:51.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T18:24:51.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T18:24:51.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T18:24:51.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T18:24:51.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T18:24:51.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T18:24:51.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T18:24:51.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T18:24:51.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T18:24:51.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T18:24:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T18:24:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T18:24:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T18:24:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T18:24:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T18:24:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T18:24:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T18:24:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T18:24:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T18:24:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T18:24:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T18:24:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T18:24:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T18:24:51.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T18:24:51.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T18:24:51.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T18:24:51.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T18:24:51.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T18:24:51.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T18:24:51.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T18:24:51.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T18:24:51.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T18:24:51.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: Manager daemon x is now available 2023-11-16T18:24:51.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:24:51.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:24:51.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T18:24:51.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:51 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T18:24:51.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:51.968+0000 7f40fffff700 1 -- 172.21.15.2:0/310042898 <== mon.0 v2:172.21.15.2:3300/0 9 ==== mgrmap(e 22) v1 ==== 135527+0+0 (secure 0 0 0) 0x7f410c096840 con 0x7f41180ff480 2023-11-16T18:24:51.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:51.969+0000 7f40fffff700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f410001b470 0x7f410001b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:51.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:51.969+0000 7f40fffff700 1 -- 172.21.15.2:0/310042898 --> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f4118061a30 con 0x7f410001b470 2023-11-16T18:24:51.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:51.972+0000 7f4116ffd700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f410001b470 0x7f410001b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:51.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:51.974+0000 7f4116ffd700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f410001b470 0x7f410001b840 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4108005b40 tx=0x7f4108005a90 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:52.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.037+0000 7f40fffff700 1 -- 172.21.15.2:0/310042898 <== mgr.15594 v2:172.21.15.2:6800/2202170290 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4118061a30 con 0x7f410001b470 2023-11-16T18:24:52.038 INFO:teuthology.orchestra.run.smithi002.stdout:Schedule set for path / 2023-11-16T18:24:52.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.039+0000 7f411d3a0700 1 -- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f410001b470 msgr2=0x7f410001b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:52.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.039+0000 7f411d3a0700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f410001b470 0x7f410001b840 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4108005b40 tx=0x7f4108005a90 comp rx=0 tx=0).stop 2023-11-16T18:24:52.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.039+0000 7f411d3a0700 1 -- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41180ff480 msgr2=0x7f41181986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:52.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.039+0000 7f411d3a0700 1 --2- 172.21.15.2:0/310042898 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41180ff480 0x7f41181986f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f410c00e9a0 tx=0x7f410c00ecb0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.039+0000 7f41177fe700 1 -- 172.21.15.2:0/310042898 reap_dead start 2023-11-16T18:24:52.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.039+0000 7f411d3a0700 1 -- 172.21.15.2:0/310042898 shutdown_connections 2023-11-16T18:24:52.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.039+0000 7f411d3a0700 1 -- 172.21.15.2:0/310042898 >> 172.21.15.2:0/310042898 conn(0x7f41180fa720 msgr2=0x7f41180fc450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:52.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.039+0000 7f411d3a0700 1 -- 172.21.15.2:0/310042898 shutdown_connections 2023-11-16T18:24:52.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.039+0000 7f411d3a0700 1 -- 172.21.15.2:0/310042898 wait complete. 2023-11-16T18:24:52.053 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h' 2023-11-16T18:24:52.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.201+0000 7f6b60b8d700 1 -- 172.21.15.2:0/2205070879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b5c0ff140 msgr2=0x7f6b5c0ff550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:52.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.201+0000 7f6b60b8d700 1 --2- 172.21.15.2:0/2205070879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b5c0ff140 0x7f6b5c0ff550 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00b0a0 tx=0x7f6b4c00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.201+0000 7f6b60b8d700 1 -- 172.21.15.2:0/2205070879 shutdown_connections 2023-11-16T18:24:52.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.201+0000 7f6b60b8d700 1 --2- 172.21.15.2:0/2205070879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b5c100cc0 0x7f6b5c107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.201+0000 7f6b60b8d700 1 --2- 172.21.15.2:0/2205070879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b5c0ff140 0x7f6b5c0ff550 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.201+0000 7f6b60b8d700 1 --2- 172.21.15.2:0/2205070879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b5c100330 0x7f6b5c100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.201+0000 7f6b60b8d700 1 -- 172.21.15.2:0/2205070879 >> 172.21.15.2:0/2205070879 conn(0x7f6b5c0fa720 msgr2=0x7f6b5c0fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:52.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.201+0000 7f6b60b8d700 1 -- 172.21.15.2:0/2205070879 shutdown_connections 2023-11-16T18:24:52.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.201+0000 7f6b60b8d700 1 -- 172.21.15.2:0/2205070879 wait complete. 2023-11-16T18:24:52.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.202+0000 7f6b60b8d700 1 Processor -- start 2023-11-16T18:24:52.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.202+0000 7f6b60b8d700 1 -- start start 2023-11-16T18:24:52.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.202+0000 7f6b60b8d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b5c0ff140 0x7f6b5c068bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:52.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.202+0000 7f6b60b8d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b5c100330 0x7f6b5c19af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:52.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.202+0000 7f6b60b8d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b5c100cc0 0x7f6b5c19d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:52.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.202+0000 7f6b60b8d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c19dd00 con 0x7f6b5c100cc0 2023-11-16T18:24:52.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.202+0000 7f6b60b8d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c19de70 con 0x7f6b5c100330 2023-11-16T18:24:52.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.202+0000 7f6b60b8d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c19e030 con 0x7f6b5c0ff140 2023-11-16T18:24:52.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.202+0000 7f6b5ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b5c100cc0 0x7f6b5c19d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:52.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.202+0000 7f6b5ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b5c100cc0 0x7f6b5c19d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50322/0 (socket says 172.21.15.2:50322) 2023-11-16T18:24:52.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.202+0000 7f6b5ad9d700 1 -- 172.21.15.2:0/3030540988 learned_addr learned my addr 172.21.15.2:0/3030540988 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:52.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.202+0000 7f6b59d9b700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b5c100330 0x7f6b5c19af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:52.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.202+0000 7f6b5a59c700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b5c0ff140 0x7f6b5c068bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:52.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.203+0000 7f6b5ad9d700 1 -- 172.21.15.2:0/3030540988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b5c0ff140 msgr2=0x7f6b5c068bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:52.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.203+0000 7f6b5ad9d700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b5c0ff140 0x7f6b5c068bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.203+0000 7f6b5ad9d700 1 -- 172.21.15.2:0/3030540988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b5c100330 msgr2=0x7f6b5c19af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:52.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.203+0000 7f6b5ad9d700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b5c100330 0x7f6b5c19af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.203+0000 7f6b5ad9d700 1 -- 172.21.15.2:0/3030540988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c009d00 con 0x7f6b5c100cc0 2023-11-16T18:24:52.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.203+0000 7f6b5ad9d700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b5c100cc0 0x7f6b5c19d560 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6b54005fd0 tx=0x7f6b5400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:52.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.203+0000 7f6b4b7fe700 1 -- 172.21.15.2:0/3030540988 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5400f890 con 0x7f6b5c100cc0 2023-11-16T18:24:52.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.203+0000 7f6b4b7fe700 1 -- 172.21.15.2:0/3030540988 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b54010460 con 0x7f6b5c100cc0 2023-11-16T18:24:52.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.203+0000 7f6b60b8d700 1 -- 172.21.15.2:0/3030540988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5c10aaf0 con 0x7f6b5c100cc0 2023-11-16T18:24:52.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.203+0000 7f6b4b7fe700 1 -- 172.21.15.2:0/3030540988 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5400e610 con 0x7f6b5c100cc0 2023-11-16T18:24:52.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.203+0000 7f6b60b8d700 1 -- 172.21.15.2:0/3030540988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5c10a960 con 0x7f6b5c100cc0 2023-11-16T18:24:52.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.204+0000 7f6b60b8d700 1 -- 172.21.15.2:0/3030540988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5c106710 con 0x7f6b5c100cc0 2023-11-16T18:24:52.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.205+0000 7f6b4b7fe700 1 -- 172.21.15.2:0/3030540988 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 22) v1 ==== 135527+0+0 (secure 0 0 0) 0x7f6b540105d0 con 0x7f6b5c100cc0 2023-11-16T18:24:52.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.206+0000 7f6b4b7fe700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b440a0f50 0x7f6b440a3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:52.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.206+0000 7f6b5a59c700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b440a0f50 0x7f6b440a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:52.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.206+0000 7f6b4b7fe700 1 -- 172.21.15.2:0/3030540988 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6b540cc6e0 con 0x7f6b5c100cc0 2023-11-16T18:24:52.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.206+0000 7f6b5a59c700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b440a0f50 0x7f6b440a3400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c007a60 tx=0x7f6b4c00b7f0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:52.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.211+0000 7f6b4b7fe700 1 -- 172.21.15.2:0/3030540988 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6b54095c90 con 0x7f6b5c100cc0 2023-11-16T18:24:52.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.445+0000 7f6b60b8d700 1 -- 172.21.15.2:0/3030540988 --> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b5c061c40 con 0x7f6b440a0f50 2023-11-16T18:24:52.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.501+0000 7f6b4b7fe700 1 -- 172.21.15.2:0/3030540988 <== mgr.15594 v2:172.21.15.2:6800/2202170290 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f6b5c061c40 con 0x7f6b440a0f50 2023-11-16T18:24:52.502 INFO:teuthology.orchestra.run.smithi002.stdout:Retention added to path / 2023-11-16T18:24:52.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.503+0000 7f6b60b8d700 1 -- 172.21.15.2:0/3030540988 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b440a0f50 msgr2=0x7f6b440a3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:52.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.504+0000 7f6b60b8d700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b440a0f50 0x7f6b440a3400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c007a60 tx=0x7f6b4c00b7f0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.504+0000 7f6b60b8d700 1 -- 172.21.15.2:0/3030540988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b5c100cc0 msgr2=0x7f6b5c19d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:52.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.504+0000 7f6b60b8d700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b5c100cc0 0x7f6b5c19d560 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6b54005fd0 tx=0x7f6b5400dc30 comp rx=0 tx=0).stop 2023-11-16T18:24:52.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.504+0000 7f6b60b8d700 1 -- 172.21.15.2:0/3030540988 shutdown_connections 2023-11-16T18:24:52.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.504+0000 7f6b60b8d700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b440a0f50 0x7f6b440a3400 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.504+0000 7f6b60b8d700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b5c0ff140 0x7f6b5c068bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.504+0000 7f6b60b8d700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b5c100cc0 0x7f6b5c19d560 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.504+0000 7f6b60b8d700 1 --2- 172.21.15.2:0/3030540988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b5c100330 0x7f6b5c19af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.504+0000 7f6b60b8d700 1 -- 172.21.15.2:0/3030540988 >> 172.21.15.2:0/3030540988 conn(0x7f6b5c0fa720 msgr2=0x7f6b5c103850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:52.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.504+0000 7f6b60b8d700 1 -- 172.21.15.2:0/3030540988 shutdown_connections 2023-11-16T18:24:52.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.504+0000 7f6b60b8d700 1 -- 172.21.15.2:0/3030540988 wait complete. 2023-11-16T18:24:52.520 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-11-16T18:24:52.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.656+0000 7ff23071c700 1 -- 172.21.15.2:0/2777718943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff228100cc0 msgr2=0x7ff2281030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:52.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.656+0000 7ff23071c700 1 --2- 172.21.15.2:0/2777718943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff228100cc0 0x7ff2281030a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff224009990 tx=0x7ff224009ca0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.657+0000 7ff23071c700 1 -- 172.21.15.2:0/2777718943 shutdown_connections 2023-11-16T18:24:52.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.657+0000 7ff23071c700 1 --2- 172.21.15.2:0/2777718943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff228105f00 0x7ff228108360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.657+0000 7ff23071c700 1 --2- 172.21.15.2:0/2777718943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff228100cc0 0x7ff2281030a0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.657+0000 7ff23071c700 1 --2- 172.21.15.2:0/2777718943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2281035e0 0x7ff2281059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.657+0000 7ff23071c700 1 -- 172.21.15.2:0/2777718943 >> 172.21.15.2:0/2777718943 conn(0x7ff2280fa720 msgr2=0x7ff2280fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:52.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.657+0000 7ff23071c700 1 -- 172.21.15.2:0/2777718943 shutdown_connections 2023-11-16T18:24:52.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.657+0000 7ff23071c700 1 -- 172.21.15.2:0/2777718943 wait complete. 2023-11-16T18:24:52.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.657+0000 7ff23071c700 1 Processor -- start 2023-11-16T18:24:52.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.657+0000 7ff23071c700 1 -- start start 2023-11-16T18:24:52.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.657+0000 7ff23071c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff228100cc0 0x7ff228195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:52.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.657+0000 7ff23071c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2281035e0 0x7ff2281964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:52.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff23071c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff228105f00 0x7ff22819d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:52.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff23071c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22819db10 con 0x7ff2281035e0 2023-11-16T18:24:52.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff23071c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22819dc50 con 0x7ff228105f00 2023-11-16T18:24:52.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff23071c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22819dde0 con 0x7ff228100cc0 2023-11-16T18:24:52.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff22dcb7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2281035e0 0x7ff2281964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:52.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff22dcb7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2281035e0 0x7ff2281964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50332/0 (socket says 172.21.15.2:50332) 2023-11-16T18:24:52.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff22dcb7700 1 -- 172.21.15.2:0/2666934966 learned_addr learned my addr 172.21.15.2:0/2666934966 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:52.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff22e4b8700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff228100cc0 0x7ff228195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:52.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff22ecb9700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff228105f00 0x7ff22819d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:52.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff22dcb7700 1 -- 172.21.15.2:0/2666934966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff228100cc0 msgr2=0x7ff228195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:52.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff22dcb7700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff228100cc0 0x7ff228195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff22dcb7700 1 -- 172.21.15.2:0/2666934966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff228105f00 msgr2=0x7ff22819d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:52.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff22dcb7700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff228105f00 0x7ff22819d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff22dcb7700 1 -- 172.21.15.2:0/2666934966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2240095f0 con 0x7ff2281035e0 2023-11-16T18:24:52.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff22dcb7700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2281035e0 0x7ff2281964d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff21800e9a0 tx=0x7ff21800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:52.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff21f7fe700 1 -- 172.21.15.2:0/2666934966 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218009790 con 0x7ff2281035e0 2023-11-16T18:24:52.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff23071c700 1 -- 172.21.15.2:0/2666934966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22810bbc0 con 0x7ff2281035e0 2023-11-16T18:24:52.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff21f7fe700 1 -- 172.21.15.2:0/2666934966 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff218010430 con 0x7ff2281035e0 2023-11-16T18:24:52.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.658+0000 7ff21f7fe700 1 -- 172.21.15.2:0/2666934966 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218018680 con 0x7ff2281035e0 2023-11-16T18:24:52.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.659+0000 7ff23071c700 1 -- 172.21.15.2:0/2666934966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22810ba30 con 0x7ff2281035e0 2023-11-16T18:24:52.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.660+0000 7ff23071c700 1 -- 172.21.15.2:0/2666934966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2280fc290 con 0x7ff2281035e0 2023-11-16T18:24:52.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.661+0000 7ff21f7fe700 1 -- 172.21.15.2:0/2666934966 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 22) v1 ==== 135527+0+0 (secure 0 0 0) 0x7ff2180105a0 con 0x7ff2281035e0 2023-11-16T18:24:52.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.661+0000 7ff21f7fe700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff2140a0f50 0x7ff2140a3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:52.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.661+0000 7ff21f7fe700 1 -- 172.21.15.2:0/2666934966 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff218014070 con 0x7ff2281035e0 2023-11-16T18:24:52.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.661+0000 7ff22e4b8700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff2140a0f50 0x7ff2140a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:52.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.662+0000 7ff22e4b8700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff2140a0f50 0x7ff2140a3400 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff22400b2e0 tx=0x7ff224005fb0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:52.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.666+0000 7ff21f7fe700 1 -- 172.21.15.2:0/2666934966 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff218051730 con 0x7ff2281035e0 2023-11-16T18:24:52.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:52 smithi040 ceph-mon[100251]: [16/Nov/2023:18:24:51] ENGINE Bus STARTING 2023-11-16T18:24:52.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:52 smithi040 ceph-mon[100251]: mgrmap e22: x(active, since 1.01898s), standbys: z, y 2023-11-16T18:24:52.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:52 smithi188 ceph-mon[100639]: [16/Nov/2023:18:24:51] ENGINE Bus STARTING 2023-11-16T18:24:52.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:52 smithi188 ceph-mon[100639]: mgrmap e22: x(active, since 1.01898s), standbys: z, y 2023-11-16T18:24:52.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:52 smithi002 ceph-mon[98807]: [16/Nov/2023:18:24:51] ENGINE Bus STARTING 2023-11-16T18:24:52.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:52 smithi002 ceph-mon[98807]: mgrmap e22: x(active, since 1.01898s), standbys: z, y 2023-11-16T18:24:52.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.904+0000 7ff23071c700 1 -- 172.21.15.2:0/2666934966 --> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7ff228061b70 con 0x7ff2140a0f50 2023-11-16T18:24:52.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.913+0000 7ff21f7fe700 1 -- 172.21.15.2:0/2666934966 <== mgr.15594 v2:172.21.15.2:6800/2202170290 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+283 (secure 0 0 0) 0x7ff228061b70 con 0x7ff2140a0f50 2023-11-16T18:24:52.914 INFO:teuthology.orchestra.run.smithi002.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-11-16T00:00:00", "created": "2023-11-16T18:24:51", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2023-11-16T18:24:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.914+0000 7ff23071c700 1 -- 172.21.15.2:0/2666934966 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff2140a0f50 msgr2=0x7ff2140a3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.915+0000 7ff23071c700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff2140a0f50 0x7ff2140a3400 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff22400b2e0 tx=0x7ff224005fb0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.915+0000 7ff23071c700 1 -- 172.21.15.2:0/2666934966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2281035e0 msgr2=0x7ff2281964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.915+0000 7ff23071c700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2281035e0 0x7ff2281964d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff21800e9a0 tx=0x7ff21800ecb0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.915+0000 7ff23071c700 1 -- 172.21.15.2:0/2666934966 shutdown_connections 2023-11-16T18:24:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.915+0000 7ff23071c700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff2140a0f50 0x7ff2140a3400 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.915+0000 7ff23071c700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff228100cc0 0x7ff228195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.915+0000 7ff23071c700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff228105f00 0x7ff22819d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.915+0000 7ff23071c700 1 --2- 172.21.15.2:0/2666934966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2281035e0 0x7ff2281964d0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.915+0000 7ff23071c700 1 -- 172.21.15.2:0/2666934966 >> 172.21.15.2:0/2666934966 conn(0x7ff2280fa720 msgr2=0x7ff228101920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.915+0000 7ff23071c700 1 -- 172.21.15.2:0/2666934966 shutdown_connections 2023-11-16T18:24:52.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:52.915+0000 7ff23071c700 1 -- 172.21.15.2:0/2666934966 wait complete. 2023-11-16T18:24:52.928 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-11-16T18:24:53.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.060+0000 7fd53d5d9700 1 -- 172.21.15.2:0/3544700711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd538100cd0 msgr2=0x7fd538107590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:53.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.060+0000 7fd53d5d9700 1 --2- 172.21.15.2:0/3544700711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd538100cd0 0x7fd538107590 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd53000b0a0 tx=0x7fd53000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:24:53.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.060+0000 7fd53d5d9700 1 -- 172.21.15.2:0/3544700711 shutdown_connections 2023-11-16T18:24:53.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.060+0000 7fd53d5d9700 1 --2- 172.21.15.2:0/3544700711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd538100cd0 0x7fd538107590 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:53.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.060+0000 7fd53d5d9700 1 --2- 172.21.15.2:0/3544700711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5380ff150 0x7fd5380ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:53.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.060+0000 7fd53d5d9700 1 --2- 172.21.15.2:0/3544700711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd538100340 0x7fd538100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:53.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.060+0000 7fd53d5d9700 1 -- 172.21.15.2:0/3544700711 >> 172.21.15.2:0/3544700711 conn(0x7fd5380fa750 msgr2=0x7fd5380fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:53.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.060+0000 7fd53d5d9700 1 -- 172.21.15.2:0/3544700711 shutdown_connections 2023-11-16T18:24:53.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.060+0000 7fd53d5d9700 1 -- 172.21.15.2:0/3544700711 wait complete. 2023-11-16T18:24:53.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.061+0000 7fd53d5d9700 1 Processor -- start 2023-11-16T18:24:53.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.061+0000 7fd53d5d9700 1 -- start start 2023-11-16T18:24:53.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.061+0000 7fd53d5d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5380ff150 0x7fd538196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:53.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.061+0000 7fd53d5d9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd538100340 0x7fd538196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:53.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.061+0000 7fd53d5d9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd538100cd0 0x7fd53819d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:53.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.061+0000 7fd53d5d9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53819db90 con 0x7fd5380ff150 2023-11-16T18:24:53.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.061+0000 7fd53d5d9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53819dcd0 con 0x7fd538100cd0 2023-11-16T18:24:53.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.061+0000 7fd53d5d9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53819de60 con 0x7fd538100340 2023-11-16T18:24:53.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.061+0000 7fd536ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5380ff150 0x7fd538196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:53.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.061+0000 7fd536ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5380ff150 0x7fd538196010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50344/0 (socket says 172.21.15.2:50344) 2023-11-16T18:24:53.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.061+0000 7fd536ffd700 1 -- 172.21.15.2:0/615224933 learned_addr learned my addr 172.21.15.2:0/615224933 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:24:53.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.061+0000 7fd5377fe700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd538100cd0 0x7fd53819d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:53.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.061+0000 7fd5367fc700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd538100340 0x7fd538196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:53.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.062+0000 7fd536ffd700 1 -- 172.21.15.2:0/615224933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd538100340 msgr2=0x7fd538196550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:53.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.062+0000 7fd536ffd700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd538100340 0x7fd538196550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:53.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.062+0000 7fd536ffd700 1 -- 172.21.15.2:0/615224933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd538100cd0 msgr2=0x7fd53819d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:53.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.062+0000 7fd536ffd700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd538100cd0 0x7fd53819d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:53.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.062+0000 7fd536ffd700 1 -- 172.21.15.2:0/615224933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd530009d00 con 0x7fd5380ff150 2023-11-16T18:24:53.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.062+0000 7fd5377fe700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd538100cd0 0x7fd53819d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:24:53.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.062+0000 7fd536ffd700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5380ff150 0x7fd538196010 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00e970 tx=0x7fd52c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:53.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.062+0000 7fd51ffff700 1 -- 172.21.15.2:0/615224933 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c009790 con 0x7fd5380ff150 2023-11-16T18:24:53.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.062+0000 7fd53d5d9700 1 -- 172.21.15.2:0/615224933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53810aac0 con 0x7fd5380ff150 2023-11-16T18:24:53.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.062+0000 7fd51ffff700 1 -- 172.21.15.2:0/615224933 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd52c00c7e0 con 0x7fd5380ff150 2023-11-16T18:24:53.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.062+0000 7fd51ffff700 1 -- 172.21.15.2:0/615224933 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c003c70 con 0x7fd5380ff150 2023-11-16T18:24:53.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.062+0000 7fd53d5d9700 1 -- 172.21.15.2:0/615224933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53810a930 con 0x7fd5380ff150 2023-11-16T18:24:53.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.063+0000 7fd53d5d9700 1 -- 172.21.15.2:0/615224933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd538067370 con 0x7fd5380ff150 2023-11-16T18:24:53.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.064+0000 7fd51ffff700 1 -- 172.21.15.2:0/615224933 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 22) v1 ==== 135527+0+0 (secure 0 0 0) 0x7fd52c00cc50 con 0x7fd5380ff150 2023-11-16T18:24:53.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.065+0000 7fd51ffff700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd5200a0f50 0x7fd5200a3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:24:53.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.065+0000 7fd5367fc700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd5200a0f50 0x7fd5200a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:24:53.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.065+0000 7fd51ffff700 1 -- 172.21.15.2:0/615224933 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd52c014070 con 0x7fd5380ff150 2023-11-16T18:24:53.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.065+0000 7fd5367fc700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd5200a0f50 0x7fd5200a3400 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd52800ac50 tx=0x7fd52800a380 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:24:53.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.068+0000 7fd51ffff700 1 -- 172.21.15.2:0/615224933 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd52c096240 con 0x7fd5380ff150 2023-11-16T18:24:53.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.298+0000 7fd53d5d9700 1 -- 172.21.15.2:0/615224933 --> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fd53819ea20 con 0x7fd5200a0f50 2023-11-16T18:24:53.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.305+0000 7fd51ffff700 1 -- 172.21.15.2:0/615224933 <== mgr.15594 v2:172.21.15.2:6800/2202170290 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fd53819ea20 con 0x7fd5200a0f50 2023-11-16T18:24:53.307 INFO:teuthology.orchestra.run.smithi002.stdout:/ 1M 6M3h 2023-11-16T18:24:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.307+0000 7fd53d5d9700 1 -- 172.21.15.2:0/615224933 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd5200a0f50 msgr2=0x7fd5200a3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.308+0000 7fd53d5d9700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd5200a0f50 0x7fd5200a3400 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd52800ac50 tx=0x7fd52800a380 comp rx=0 tx=0).stop 2023-11-16T18:24:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.308+0000 7fd53d5d9700 1 -- 172.21.15.2:0/615224933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5380ff150 msgr2=0x7fd538196010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:24:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.308+0000 7fd53d5d9700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5380ff150 0x7fd538196010 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00e970 tx=0x7fd52c00ec80 comp rx=0 tx=0).stop 2023-11-16T18:24:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.308+0000 7fd53d5d9700 1 -- 172.21.15.2:0/615224933 shutdown_connections 2023-11-16T18:24:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.308+0000 7fd53d5d9700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd5200a0f50 0x7fd5200a3400 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.308+0000 7fd53d5d9700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd538100cd0 0x7fd53819d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.308+0000 7fd53d5d9700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5380ff150 0x7fd538196010 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.308+0000 7fd53d5d9700 1 --2- 172.21.15.2:0/615224933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd538100340 0x7fd538196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:24:53.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.308+0000 7fd53d5d9700 1 -- 172.21.15.2:0/615224933 >> 172.21.15.2:0/615224933 conn(0x7fd5380fa750 msgr2=0x7fd538068bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:24:53.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.308+0000 7fd53d5d9700 1 -- 172.21.15.2:0/615224933 shutdown_connections 2023-11-16T18:24:53.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:24:53.308+0000 7fd53d5d9700 1 -- 172.21.15.2:0/615224933 wait complete. 2023-11-16T18:24:53.322 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2023-11-16T18:24:53.352 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2023-11-16T18:24:53.363 INFO:teuthology.run_tasks:Running task background_exec... 2023-11-16T18:24:53.441 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T18:24:53.442 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2023-11-16T18:24:53.444 INFO:teuthology.run_tasks:Running task workunit... 2023-11-16T18:24:53.457 INFO:tasks.workunit:Pulling workunits from ref 1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-16T18:24:53.458 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-11-16T18:24:53.458 INFO:tasks.workunit:timeout=3h 2023-11-16T18:24:53.458 INFO:tasks.workunit:cleanup=True 2023-11-16T18:24:53.458 DEBUG:teuthology.orchestra.run.smithi002:> stat -- /home/ubuntu/cephtest/mnt.0 2023-11-16T18:24:53.479 INFO:teuthology.orchestra.run.smithi002.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-11-16T18:24:53.479 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-11-16T18:24:53.479 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 7ch/124d Inode: 1099511628283 Links: 2 2023-11-16T18:24:53.479 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T18:24:53.479 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:cephfs_t:s0 2023-11-16T18:24:53.479 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-16 18:23:09.144562373 +0000 2023-11-16T18:24:53.479 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-16 18:23:09.144562373 +0000 2023-11-16T18:24:53.480 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-16 18:24:33.948836345 +0000 2023-11-16T18:24:53.480 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2023-11-16 18:23:09.144562373 +0000 2023-11-16T18:24:53.480 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-11-16T18:24:53.480 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-11-16T18:24:53.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:53 smithi002 ceph-mon[98807]: [16/Nov/2023:18:24:51] ENGINE Serving on http://172.21.15.2:8765 2023-11-16T18:24:53.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:53 smithi002 ceph-mon[98807]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:53.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:53 smithi002 ceph-mon[98807]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:24:53.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:53 smithi002 ceph-mon[98807]: [16/Nov/2023:18:24:52] ENGINE Serving on https://172.21.15.2:7150 2023-11-16T18:24:53.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:53 smithi002 ceph-mon[98807]: [16/Nov/2023:18:24:52] ENGINE Bus STARTED 2023-11-16T18:24:53.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:53 smithi002 ceph-mon[98807]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:53.577 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/chrisphoffman/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-16T18:24:53.595 INFO:tasks.workunit.client.0.smithi002.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-11-16T18:24:53.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:53 smithi040 ceph-mon[100251]: [16/Nov/2023:18:24:51] ENGINE Serving on http://172.21.15.2:8765 2023-11-16T18:24:53.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:53 smithi040 ceph-mon[100251]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:53.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:53 smithi040 ceph-mon[100251]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:24:53.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:53 smithi040 ceph-mon[100251]: [16/Nov/2023:18:24:52] ENGINE Serving on https://172.21.15.2:7150 2023-11-16T18:24:53.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:53 smithi040 ceph-mon[100251]: [16/Nov/2023:18:24:52] ENGINE Bus STARTED 2023-11-16T18:24:53.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:53 smithi040 ceph-mon[100251]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:53.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:53 smithi188 ceph-mon[100639]: [16/Nov/2023:18:24:51] ENGINE Serving on http://172.21.15.2:8765 2023-11-16T18:24:53.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:53 smithi188 ceph-mon[100639]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:53.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:53 smithi188 ceph-mon[100639]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:24:53.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:53 smithi188 ceph-mon[100639]: [16/Nov/2023:18:24:52] ENGINE Serving on https://172.21.15.2:7150 2023-11-16T18:24:53.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:53 smithi188 ceph-mon[100639]: [16/Nov/2023:18:24:52] ENGINE Bus STARTED 2023-11-16T18:24:53.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:53 smithi188 ceph-mon[100639]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:54.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:54 smithi040 ceph-mon[100251]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:54.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:54 smithi040 ceph-mon[100251]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:24:54.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:54 smithi040 ceph-mon[100251]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:54.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:54 smithi040 ceph-mon[100251]: mgrmap e23: x(active, since 3s), standbys: z, y 2023-11-16T18:24:54.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:54 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:54 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:54 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:54 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:54 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:54 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:54 smithi188 ceph-mon[100639]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:54.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:54 smithi188 ceph-mon[100639]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:24:54.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:54 smithi188 ceph-mon[100639]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:54.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:54 smithi188 ceph-mon[100639]: mgrmap e23: x(active, since 3s), standbys: z, y 2023-11-16T18:24:54.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:54 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:54 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:54 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:54 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:54 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:54 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:54 smithi002 ceph-mon[98807]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:54.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:54 smithi002 ceph-mon[98807]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:24:54.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:54 smithi002 ceph-mon[98807]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:24:54.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:54 smithi002 ceph-mon[98807]: mgrmap e23: x(active, since 3s), standbys: z, y 2023-11-16T18:24:54.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:54 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:54 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:54 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:54 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:54 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:54.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:54 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:24:56.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi002 to 197.2M 2023-11-16T18:24:56.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: Unable to set osd_memory_target on smithi002 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-11-16T18:24:56.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi040 to 197.2M 2023-11-16T18:24:56.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: Unable to set osd_memory_target on smithi040 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-11-16T18:24:56.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi188 to 196.2M 2023-11-16T18:24:56.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: Unable to set osd_memory_target on smithi188 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T18:24:56.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:24:56.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:24:56.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:24:56.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T18:24:56.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: Updating smithi040:/etc/ceph/ceph.conf 2023-11-16T18:24:56.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: Updating smithi188:/etc/ceph/ceph.conf 2023-11-16T18:24:56.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: Updating smithi002:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:24:56.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: Updating smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:24:56.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:24:56.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:56 smithi040 ceph-mon[100251]: scrub summary: idle 2023-11-16T18:24:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:24:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi002 to 197.2M 2023-11-16T18:24:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: Unable to set osd_memory_target on smithi002 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-11-16T18:24:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi040 to 197.2M 2023-11-16T18:24:56.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: Unable to set osd_memory_target on smithi040 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-11-16T18:24:56.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi188 to 196.2M 2023-11-16T18:24:56.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: Unable to set osd_memory_target on smithi188 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T18:24:56.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:24:56.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:24:56.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:24:56.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T18:24:56.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: Updating smithi040:/etc/ceph/ceph.conf 2023-11-16T18:24:56.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: Updating smithi188:/etc/ceph/ceph.conf 2023-11-16T18:24:56.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: Updating smithi002:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:24:56.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: Updating smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:24:56.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:24:56.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:56 smithi188 ceph-mon[100639]: scrub summary: idle 2023-11-16T18:24:56.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:24:56.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:56.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi002 to 197.2M 2023-11-16T18:24:56.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: Unable to set osd_memory_target on smithi002 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-11-16T18:24:56.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi040 to 197.2M 2023-11-16T18:24:56.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:24:56.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: Unable to set osd_memory_target on smithi040 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-11-16T18:24:56.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi188 to 196.2M 2023-11-16T18:24:56.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: Unable to set osd_memory_target on smithi188 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T18:24:56.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:24:56.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:24:56.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:24:56.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T18:24:56.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: Updating smithi040:/etc/ceph/ceph.conf 2023-11-16T18:24:56.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: Updating smithi188:/etc/ceph/ceph.conf 2023-11-16T18:24:56.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: Updating smithi002:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:24:56.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: Updating smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:24:56.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T18:24:56.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:56 smithi002 ceph-mon[98807]: scrub summary: idle 2023-11-16T18:24:57.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:24:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:24:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:24:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: Updating smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:24:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: Updating smithi002:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:24:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:24:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:24:57.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:24:57.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:24:57.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:24:57.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:24:57.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:24:57.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:24:57.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:24:57.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:24:57.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:57 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:24:57.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:24:57.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:24:57.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:24:57.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: Updating smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:24:57.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: Updating smithi002:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:24:57.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:24:57.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:24:57.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:24:57.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:24:57.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:24:57.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:24:57.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:24:57.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:24:57.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:57.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:24:57.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:24:57.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:57 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:24:58.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:24:58.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:24:58.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2023-11-16T18:24:58.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: Updating smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:24:58.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: Updating smithi002:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:24:58.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: Updating smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T18:24:58.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:58.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:58.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:58.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:58.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:58.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:58.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:58.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:24:58.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:24:58.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:24:58.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:24:58.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:24:58.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:24:58.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:24:58.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:24:58.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:24:58.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:24:58.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:57 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:24:59.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:24:58 smithi040 ceph-mon[100251]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 90 KiB/s wr, 56 op/s 2023-11-16T18:24:59.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:24:58 smithi188 ceph-mon[100639]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 90 KiB/s wr, 56 op/s 2023-11-16T18:24:59.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:24:58 smithi002 ceph-mon[98807]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 90 KiB/s wr, 56 op/s 2023-11-16T18:25:00.856 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:25:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:25:00.512+0000 7fb6271c6700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:25:00.856 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:25:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:25:00.512+0000 7fb6271c6700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:25:00.857 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:25:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:25:00.512+0000 7fb6271c6700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:25:00.857 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:25:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:25:00.512+0000 7fb6271c6700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:25:00.857 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:25:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:25:00.512+0000 7fb6271c6700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T18:25:01.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:00 smithi040 ceph-mon[100251]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 132 KiB/s wr, 51 op/s 2023-11-16T18:25:01.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:00 smithi188 ceph-mon[100639]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 132 KiB/s wr, 51 op/s 2023-11-16T18:25:01.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:00 smithi002 ceph-mon[98807]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 132 KiB/s wr, 51 op/s 2023-11-16T18:25:03.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:02 smithi040 ceph-mon[100251]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 103 KiB/s wr, 39 op/s 2023-11-16T18:25:03.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:02 smithi040 ceph-mon[100251]: mgrmap e24: x(active, since 10s), standbys: z, y 2023-11-16T18:25:03.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:02 smithi188 ceph-mon[100639]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 103 KiB/s wr, 39 op/s 2023-11-16T18:25:03.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:02 smithi188 ceph-mon[100639]: mgrmap e24: x(active, since 10s), standbys: z, y 2023-11-16T18:25:03.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:02 smithi002 ceph-mon[98807]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 103 KiB/s wr, 39 op/s 2023-11-16T18:25:03.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:02 smithi002 ceph-mon[98807]: mgrmap e24: x(active, since 10s), standbys: z, y 2023-11-16T18:25:05.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:04 smithi040 ceph-mon[100251]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 109 KiB/s wr, 49 op/s 2023-11-16T18:25:05.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:04 smithi188 ceph-mon[100639]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 109 KiB/s wr, 49 op/s 2023-11-16T18:25:05.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:04 smithi002 ceph-mon[98807]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 109 KiB/s wr, 49 op/s 2023-11-16T18:25:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.614+0000 7fd6d60db700 1 -- 172.21.15.2:0/983731718 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d0104400 msgr2=0x7fd6d010bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.614+0000 7fd6d60db700 1 --2- 172.21.15.2:0/983731718 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d0104400 0x7fd6d010bfd0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800b0d0 tx=0x7fd6c800b3e0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.614+0000 7fd6d60db700 1 -- 172.21.15.2:0/983731718 shutdown_connections 2023-11-16T18:25:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.614+0000 7fd6d60db700 1 --2- 172.21.15.2:0/983731718 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d0104400 0x7fd6d010bfd0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.614+0000 7fd6d60db700 1 --2- 172.21.15.2:0/983731718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d00ff3c0 0x7fd6d00ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.614+0000 7fd6d60db700 1 --2- 172.21.15.2:0/983731718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6d00ffd60 0x7fd6d0103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.614+0000 7fd6d60db700 1 -- 172.21.15.2:0/983731718 >> 172.21.15.2:0/983731718 conn(0x7fd6d00763a0 msgr2=0x7fd6d00767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.614+0000 7fd6d60db700 1 -- 172.21.15.2:0/983731718 shutdown_connections 2023-11-16T18:25:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.614+0000 7fd6d60db700 1 -- 172.21.15.2:0/983731718 wait complete. 2023-11-16T18:25:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.614+0000 7fd6d60db700 1 Processor -- start 2023-11-16T18:25:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6d60db700 1 -- start start 2023-11-16T18:25:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6d60db700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d00ff3c0 0x7fd6d0197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6d60db700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6d00ffd60 0x7fd6d0197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6d60db700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d0104400 0x7fd6d01a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6d60db700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d0198190 con 0x7fd6d00ff3c0 2023-11-16T18:25:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6d60db700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d01a28d0 con 0x7fd6d0104400 2023-11-16T18:25:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6d60db700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d01a2a90 con 0x7fd6d00ffd60 2023-11-16T18:25:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6cf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d00ff3c0 0x7fd6d0197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6cf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d00ff3c0 0x7fd6d0197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50380/0 (socket says 172.21.15.2:50380) 2023-11-16T18:25:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6cf7fe700 1 -- 172.21.15.2:0/3003865206 learned_addr learned my addr 172.21.15.2:0/3003865206 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6ceffd700 1 --2- 172.21.15.2:0/3003865206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6d00ffd60 0x7fd6d0197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6cf7fe700 1 -- 172.21.15.2:0/3003865206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6d00ffd60 msgr2=0x7fd6d0197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6cf7fe700 1 --2- 172.21.15.2:0/3003865206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6d00ffd60 0x7fd6d0197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6cf7fe700 1 -- 172.21.15.2:0/3003865206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d0104400 msgr2=0x7fd6d01a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6cf7fe700 1 --2- 172.21.15.2:0/3003865206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d0104400 0x7fd6d01a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6cf7fe700 1 -- 172.21.15.2:0/3003865206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c8009d00 con 0x7fd6d00ff3c0 2023-11-16T18:25:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.615+0000 7fd6cf7fe700 1 --2- 172.21.15.2:0/3003865206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d00ff3c0 0x7fd6d0197310 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000d980 tx=0x7fd6c000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.616+0000 7fd6ccff9700 1 -- 172.21.15.2:0/3003865206 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c000bb60 con 0x7fd6d00ff3c0 2023-11-16T18:25:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.616+0000 7fd6ccff9700 1 -- 172.21.15.2:0/3003865206 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6c0007640 con 0x7fd6d00ff3c0 2023-11-16T18:25:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.616+0000 7fd6ccff9700 1 -- 172.21.15.2:0/3003865206 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0004e90 con 0x7fd6d00ff3c0 2023-11-16T18:25:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.616+0000 7fd6d60db700 1 -- 172.21.15.2:0/3003865206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d010f220 con 0x7fd6d00ff3c0 2023-11-16T18:25:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.616+0000 7fd6d60db700 1 -- 172.21.15.2:0/3003865206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d010f090 con 0x7fd6d00ff3c0 2023-11-16T18:25:06.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.618+0000 7fd6ccff9700 1 -- 172.21.15.2:0/3003865206 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7fd6c000bcc0 con 0x7fd6d00ff3c0 2023-11-16T18:25:06.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.619+0000 7fd6ccff9700 1 --2- 172.21.15.2:0/3003865206 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd6b80a1460 0x7fd6b80a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.619+0000 7fd6ccff9700 1 -- 172.21.15.2:0/3003865206 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd6c00cd170 con 0x7fd6d00ff3c0 2023-11-16T18:25:06.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.619+0000 7fd6ceffd700 1 --2- 172.21.15.2:0/3003865206 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd6b80a1460 0x7fd6b80a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.620+0000 7fd6ceffd700 1 --2- 172.21.15.2:0/3003865206 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd6b80a1460 0x7fd6b80a3910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0068920 tx=0x7fd6c4009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.620+0000 7fd6e2267b80 1 Processor -- start 2023-11-16T18:25:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.620+0000 7fd6e2267b80 1 -- start start 2023-11-16T18:25:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.620+0000 7fd6e2267b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 0x560b329dc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.620+0000 7fd6e2267b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b329dc720 0x560b329d6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.620+0000 7fd6e2267b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329dcb90 0x560b329d6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.620+0000 7fd6e2267b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b328dc000 con 0x560b328326b0 2023-11-16T18:25:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.620+0000 7fd6e2267b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b3292e840 con 0x560b329dc720 2023-11-16T18:25:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.620+0000 7fd6e2267b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b32927cd0 con 0x560b329dcb90 2023-11-16T18:25:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.620+0000 7fd6cf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 0x560b329dc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.621+0000 7fd6cf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 0x560b329dc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50388/0 (socket says 172.21.15.2:50388) 2023-11-16T18:25:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.621+0000 7fd6cf7fe700 1 -- 172.21.15.2:0/2567755501 learned_addr learned my addr 172.21.15.2:0/2567755501 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.621+0000 7fd6cffff700 1 --2- 172.21.15.2:0/2567755501 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329dcb90 0x560b329d6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.621+0000 7fd6ceffd700 1 --2- 172.21.15.2:0/2567755501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b329dc720 0x560b329d6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.621+0000 7fd6cf7fe700 1 -- 172.21.15.2:0/2567755501 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329dcb90 msgr2=0x560b329d6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.621+0000 7fd6cf7fe700 1 --2- 172.21.15.2:0/2567755501 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329dcb90 0x560b329d6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.621+0000 7fd6cf7fe700 1 -- 172.21.15.2:0/2567755501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b329dc720 msgr2=0x560b329d6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.621+0000 7fd6cf7fe700 1 --2- 172.21.15.2:0/2567755501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b329dc720 0x560b329d6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.621+0000 7fd6cf7fe700 1 -- 172.21.15.2:0/2567755501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b327a95d0 con 0x560b328326b0 2023-11-16T18:25:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.621+0000 7fd6cf7fe700 1 --2- 172.21.15.2:0/2567755501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 0x560b329dc1e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0099980 tx=0x7fd6c000dee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a7dc5fd44acb313 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.621+0000 7fd6b5ffb700 1 -- 172.21.15.2:0/2567755501 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0099c80 con 0x560b328326b0 2023-11-16T18:25:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.621+0000 7fd6b5ffb700 1 -- 172.21.15.2:0/2567755501 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6c009d070 con 0x560b328326b0 2023-11-16T18:25:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.621+0000 7fd6b5ffb700 1 -- 172.21.15.2:0/2567755501 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00a1ea0 con 0x560b328326b0 2023-11-16T18:25:06.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.622+0000 7fd6e2267b80 1 -- 172.21.15.2:0/2567755501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 msgr2=0x560b329dc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:06.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.622+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/2567755501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 0x560b329dc1e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0099980 tx=0x7fd6c000dee0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.622+0000 7fd6e2267b80 1 -- 172.21.15.2:0/2567755501 shutdown_connections 2023-11-16T18:25:06.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.622+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/2567755501 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329dcb90 0x560b329d6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.622+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/2567755501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 0x560b329dc1e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.622+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/2567755501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b329dc720 0x560b329d6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.622+0000 7fd6e2267b80 1 -- 172.21.15.2:0/2567755501 >> 172.21.15.2:0/2567755501 conn(0x560b329cbf50 msgr2=0x560b329d1270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:06.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.622+0000 7fd6e2267b80 1 -- 172.21.15.2:0/2567755501 shutdown_connections 2023-11-16T18:25:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.622+0000 7fd6e2267b80 1 -- 172.21.15.2:0/2567755501 wait complete. 2023-11-16T18:25:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.622+0000 7fd6e2267b80 1 -- start start 2023-11-16T18:25:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6e2267b80 1 Processor -- start 2023-11-16T18:25:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6e2267b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b328326b0 0x560b329e2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6e2267b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329dc720 0x560b329dfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6e2267b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dcb90 0x560b329e0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6e2267b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b328c9880 con 0x560b329dcb90 2023-11-16T18:25:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6e2267b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b328c9520 con 0x560b328326b0 2023-11-16T18:25:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6e2267b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b327ab8c0 con 0x560b329dc720 2023-11-16T18:25:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6e2267b80 10 client.0 _ll_register_callbacks cb 0x560b328e7720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6cffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dcb90 0x560b329e0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6cffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dcb90 0x560b329e0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50398/0 (socket says 172.21.15.2:50398) 2023-11-16T18:25:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6cffff700 1 -- 172.21.15.2:0/729103815 learned_addr learned my addr 172.21.15.2:0/729103815 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6cf7fe700 1 --2- 172.21.15.2:0/729103815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b328326b0 0x560b329e2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6ceffd700 1 --2- 172.21.15.2:0/729103815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329dc720 0x560b329dfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6cffff700 1 -- 172.21.15.2:0/729103815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329dc720 msgr2=0x560b329dfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6cffff700 1 --2- 172.21.15.2:0/729103815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329dc720 0x560b329dfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6cffff700 1 -- 172.21.15.2:0/729103815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b328326b0 msgr2=0x560b329e2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6cffff700 1 --2- 172.21.15.2:0/729103815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b328326b0 0x560b329e2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.623+0000 7fd6cffff700 1 -- 172.21.15.2:0/729103815 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d010f090 con 0x560b329dcb90 2023-11-16T18:25:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6cffff700 1 --2- 172.21.15.2:0/729103815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dcb90 0x560b329e0330 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8008fb0 tx=0x7fd6c8008fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e9422046441913 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6aeffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6aeffd700 1 -- 172.21.15.2:0/729103815 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c800e030 con 0x560b329dcb90 2023-11-16T18:25:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6aeffd700 1 -- 172.21.15.2:0/729103815 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd6c80074e0 con 0x560b329dcb90 2023-11-16T18:25:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6e2267b80 1 -- 172.21.15.2:0/729103815 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560b329c9710 con 0x560b329dcb90 2023-11-16T18:25:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6aeffd700 1 -- 172.21.15.2:0/729103815 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6c80076a0 con 0x560b329dcb90 2023-11-16T18:25:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6e2267b80 10 client.15690 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6e2267b80 1 -- 172.21.15.2:0/729103815 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560b328ee130 con 0x560b329dcb90 2023-11-16T18:25:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6aeffd700 1 -- 172.21.15.2:0/729103815 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560b329c9710 con 0x560b329dcb90 2023-11-16T18:25:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6aeffd700 1 -- 172.21.15.2:0/729103815 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd6c8019070 con 0x560b329dcb90 2023-11-16T18:25:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6e2267b80 10 client.15690 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6e2267b80 10 client.15690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6e2267b80 20 client.15690 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/729103815 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560b329f0010 0x560b329f2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6e2267b80 4 client.15690 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:25:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.624+0000 7fd6e2267b80 1 -- 172.21.15.2:0/729103815 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560b328ea040 con 0x560b329f0010 2023-11-16T18:25:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.625+0000 7fd6cf7fe700 1 --2- 172.21.15.2:0/729103815 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560b329f0010 0x560b329f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.625+0000 7fd6aeffd700 10 client.15690 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.625+0000 7fd6cf7fe700 1 --2- 172.21.15.2:0/729103815 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560b329f0010 0x560b329f2420 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ecd2ef2a657b58e server_cookie=24468c84964d144f in_seq=0 out_seq=0 2023-11-16T18:25:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6aeffd700 1 -- 172.21.15.2:0/729103815 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560b328ea040 con 0x560b329f0010 2023-11-16T18:25:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6aeffd700 10 client.15690 handle_command_reply: tid=0 2023-11-16T18:25:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6e2267b80 1 client.15690 shutdown 2023-11-16T18:25:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6e2267b80 1 -- 172.21.15.2:0/729103815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dcb90 msgr2=0x560b329e0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/729103815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dcb90 0x560b329e0330 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8008fb0 tx=0x7fd6c8008fe0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6e2267b80 1 -- 172.21.15.2:0/729103815 shutdown_connections 2023-11-16T18:25:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/729103815 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560b329f0010 0x560b329f2420 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/729103815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dcb90 0x560b329e0330 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/729103815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b328326b0 0x560b329e2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/729103815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329dc720 0x560b329dfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6e2267b80 1 -- 172.21.15.2:0/729103815 >> 172.21.15.2:0/729103815 conn(0x560b329cbf50 msgr2=0x560b329d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6aeffd700 0 client.15690 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:06.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6e2267b80 1 -- 172.21.15.2:0/729103815 shutdown_connections 2023-11-16T18:25:06.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6e2267b80 1 -- 172.21.15.2:0/729103815 wait complete. 2023-11-16T18:25:06.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.626+0000 7fd6e2267b80 20 client.15690 trim_cache size 0 max 16384 2023-11-16T18:25:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.644+0000 7fd6e2267b80 1 Processor -- start 2023-11-16T18:25:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.644+0000 7fd6e2267b80 1 -- start start 2023-11-16T18:25:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.644+0000 7fd6e2267b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329cbf50 0x560b329e25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.644+0000 7fd6e2267b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 0x560b329d9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.644+0000 7fd6e2267b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b329dc720 0x560b329da3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.644+0000 7fd6e2267b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b329cbba0 con 0x560b328326b0 2023-11-16T18:25:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.644+0000 7fd6e2267b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b329da9d0 con 0x560b329dc720 2023-11-16T18:25:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.644+0000 7fd6e2267b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b329ddf70 con 0x560b329cbf50 2023-11-16T18:25:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.644+0000 7fd6ceffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 0x560b329d9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.644+0000 7fd6ceffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 0x560b329d9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50404/0 (socket says 172.21.15.2:50404) 2023-11-16T18:25:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.644+0000 7fd6ceffd700 1 -- 172.21.15.2:0/3822278097 learned_addr learned my addr 172.21.15.2:0/3822278097 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.645+0000 7fd6cf7fe700 1 --2- 172.21.15.2:0/3822278097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329cbf50 0x560b329e25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.645+0000 7fd6cffff700 1 --2- 172.21.15.2:0/3822278097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b329dc720 0x560b329da3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.645+0000 7fd6ceffd700 1 -- 172.21.15.2:0/3822278097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329cbf50 msgr2=0x560b329e25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.645+0000 7fd6ceffd700 1 --2- 172.21.15.2:0/3822278097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329cbf50 0x560b329e25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.645+0000 7fd6ceffd700 1 -- 172.21.15.2:0/3822278097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b329dc720 msgr2=0x560b329da3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.645+0000 7fd6ceffd700 1 --2- 172.21.15.2:0/3822278097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b329dc720 0x560b329da3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.645+0000 7fd6ceffd700 1 -- 172.21.15.2:0/3822278097 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b328ede80 con 0x560b328326b0 2023-11-16T18:25:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.645+0000 7fd6cffff700 1 --2- 172.21.15.2:0/3822278097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b329dc720 0x560b329da3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:25:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.645+0000 7fd6ceffd700 1 --2- 172.21.15.2:0/3822278097 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 0x560b329d9e60 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd6d006a3c0 tx=0x7fd6c4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e739c63f674cefb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.645+0000 7fd6b67fc700 1 -- 172.21.15.2:0/3822278097 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4029640 con 0x560b328326b0 2023-11-16T18:25:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.645+0000 7fd6b67fc700 1 -- 172.21.15.2:0/3822278097 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6c4015070 con 0x560b328326b0 2023-11-16T18:25:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.645+0000 7fd6b67fc700 1 -- 172.21.15.2:0/3822278097 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c401e830 con 0x560b328326b0 2023-11-16T18:25:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.646+0000 7fd6e2267b80 1 -- 172.21.15.2:0/3822278097 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 msgr2=0x560b329d9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.646+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/3822278097 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 0x560b329d9e60 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd6d006a3c0 tx=0x7fd6c4013000 comp rx=0 tx=0).stop 2023-11-16T18:25:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.646+0000 7fd6e2267b80 1 -- 172.21.15.2:0/3822278097 shutdown_connections 2023-11-16T18:25:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.646+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/3822278097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329cbf50 0x560b329e25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.646+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/3822278097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b329dc720 0x560b329da3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.646+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/3822278097 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b328326b0 0x560b329d9e60 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.646+0000 7fd6e2267b80 1 -- 172.21.15.2:0/3822278097 >> 172.21.15.2:0/3822278097 conn(0x560b329cd710 msgr2=0x560b329ec290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.646+0000 7fd6e2267b80 1 -- 172.21.15.2:0/3822278097 shutdown_connections 2023-11-16T18:25:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.646+0000 7fd6e2267b80 1 -- 172.21.15.2:0/3822278097 wait complete. 2023-11-16T18:25:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.646+0000 7fd6e2267b80 1 -- start start 2023-11-16T18:25:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.647+0000 7fd6e2267b80 1 Processor -- start 2023-11-16T18:25:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.647+0000 7fd6e2267b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b328326b0 0x560b329e5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.647+0000 7fd6e2267b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329cbf50 0x560b329e5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.647+0000 7fd6e2267b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dc720 0x560b329e5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.647+0000 7fd6e2267b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b329e7660 con 0x560b329dc720 2023-11-16T18:25:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.647+0000 7fd6e2267b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b32a47310 con 0x560b328326b0 2023-11-16T18:25:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.647+0000 7fd6e2267b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b32a474d0 con 0x560b329cbf50 2023-11-16T18:25:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.647+0000 7fd6e2267b80 10 client.0 _ll_register_callbacks cb 0x7fd6c4005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.647+0000 7fd6cffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dc720 0x560b329e5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.647+0000 7fd6cffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dc720 0x560b329e5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50412/0 (socket says 172.21.15.2:50412) 2023-11-16T18:25:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.647+0000 7fd6cffff700 1 -- 172.21.15.2:0/911024748 learned_addr learned my addr 172.21.15.2:0/911024748 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.647+0000 7fd6ceffd700 1 --2- 172.21.15.2:0/911024748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329cbf50 0x560b329e5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.647+0000 7fd6cf7fe700 1 --2- 172.21.15.2:0/911024748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b328326b0 0x560b329e5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6cffff700 1 -- 172.21.15.2:0/911024748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329cbf50 msgr2=0x560b329e5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6cffff700 1 --2- 172.21.15.2:0/911024748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329cbf50 0x560b329e5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6cffff700 1 -- 172.21.15.2:0/911024748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b328326b0 msgr2=0x560b329e5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6cffff700 1 --2- 172.21.15.2:0/911024748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b328326b0 0x560b329e5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6cffff700 1 -- 172.21.15.2:0/911024748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c4010c20 con 0x560b329dc720 2023-11-16T18:25:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6cffff700 1 --2- 172.21.15.2:0/911024748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dc720 0x560b329e5a80 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd6c801e040 tx=0x7fd6c800bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba8347549775ea14 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6aeffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6aeffd700 1 -- 172.21.15.2:0/911024748 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c800e040 con 0x560b329dc720 2023-11-16T18:25:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6aeffd700 1 -- 172.21.15.2:0/911024748 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd6c801cbd0 con 0x560b329dc720 2023-11-16T18:25:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6e2267b80 1 -- 172.21.15.2:0/911024748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6c009d070 con 0x560b329dc720 2023-11-16T18:25:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6aeffd700 1 -- 172.21.15.2:0/911024748 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6c800b730 con 0x560b329dc720 2023-11-16T18:25:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6e2267b80 10 client.15696 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6e2267b80 1 -- 172.21.15.2:0/911024748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560b3273ec00 con 0x560b329dc720 2023-11-16T18:25:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.648+0000 7fd6aeffd700 1 -- 172.21.15.2:0/911024748 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6c009d070 con 0x560b329dc720 2023-11-16T18:25:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.649+0000 7fd6aeffd700 1 -- 172.21.15.2:0/911024748 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd6c8038070 con 0x560b329dc720 2023-11-16T18:25:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.649+0000 7fd6e2267b80 10 client.15696 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.649+0000 7fd6e2267b80 10 client.15696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.649+0000 7fd6e2267b80 20 client.15696 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.649+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/911024748 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560b329dcb90 0x560b32a4a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.649+0000 7fd6e2267b80 4 client.15696 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:25:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.649+0000 7fd6e2267b80 1 -- 172.21.15.2:0/911024748 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560b328f6600 con 0x560b329dcb90 2023-11-16T18:25:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.649+0000 7fd6cf7fe700 1 --2- 172.21.15.2:0/911024748 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560b329dcb90 0x560b32a4a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.649+0000 7fd6aeffd700 10 client.15696 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.649+0000 7fd6cf7fe700 1 --2- 172.21.15.2:0/911024748 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560b329dcb90 0x560b32a4a7e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18c817e58e11b3e9 server_cookie=c927dbef0b143ee9 in_seq=0 out_seq=0 2023-11-16T18:25:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.663+0000 7fd6aeffd700 1 -- 172.21.15.2:0/911024748 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560b328f6600 con 0x560b329dcb90 2023-11-16T18:25:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.663+0000 7fd6aeffd700 10 client.15696 handle_command_reply: tid=0 2023-11-16T18:25:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.663+0000 7fd6e2267b80 1 client.15696 shutdown 2023-11-16T18:25:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.663+0000 7fd6e2267b80 1 -- 172.21.15.2:0/911024748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dc720 msgr2=0x560b329e5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/911024748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dc720 0x560b329e5a80 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd6c801e040 tx=0x7fd6c800bb30 comp rx=0 tx=0).stop 2023-11-16T18:25:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6e2267b80 1 -- 172.21.15.2:0/911024748 shutdown_connections 2023-11-16T18:25:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/911024748 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560b329dcb90 0x560b32a4a7e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/911024748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560b329dc720 0x560b329e5a80 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/911024748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560b328326b0 0x560b329e5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6aeffd700 0 client.15696 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6e2267b80 1 --2- 172.21.15.2:0/911024748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b329cbf50 0x560b329e5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6e2267b80 1 -- 172.21.15.2:0/911024748 >> 172.21.15.2:0/911024748 conn(0x560b329cd710 msgr2=0x560b329d9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6e2267b80 1 -- 172.21.15.2:0/911024748 shutdown_connections 2023-11-16T18:25:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6e2267b80 1 -- 172.21.15.2:0/911024748 wait complete. 2023-11-16T18:25:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6e2267b80 20 client.15696 trim_cache size 0 max 16384 2023-11-16T18:25:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:25:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:25:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:25:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:25:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6d60db700 1 -- 172.21.15.2:0/3003865206 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd6b80a1460 msgr2=0x7fd6b80a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6d60db700 1 --2- 172.21.15.2:0/3003865206 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd6b80a1460 0x7fd6b80a3910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0068920 tx=0x7fd6c4009260 comp rx=0 tx=0).stop 2023-11-16T18:25:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6d60db700 1 -- 172.21.15.2:0/3003865206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d00ff3c0 msgr2=0x7fd6d0197310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6d60db700 1 --2- 172.21.15.2:0/3003865206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d00ff3c0 0x7fd6d0197310 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000d980 tx=0x7fd6c000dd40 comp rx=0 tx=0).stop 2023-11-16T18:25:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6d60db700 1 -- 172.21.15.2:0/3003865206 shutdown_connections 2023-11-16T18:25:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6d60db700 1 --2- 172.21.15.2:0/3003865206 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd6b80a1460 0x7fd6b80a3910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6d60db700 1 --2- 172.21.15.2:0/3003865206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d0104400 0x7fd6d01a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6d60db700 1 --2- 172.21.15.2:0/3003865206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d00ff3c0 0x7fd6d0197310 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6d60db700 1 --2- 172.21.15.2:0/3003865206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6d00ffd60 0x7fd6d0197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.664+0000 7fd6d60db700 1 -- 172.21.15.2:0/3003865206 >> 172.21.15.2:0/3003865206 conn(0x7fd6d00763a0 msgr2=0x7fd6d00fc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.665+0000 7fd6d60db700 1 -- 172.21.15.2:0/3003865206 shutdown_connections 2023-11-16T18:25:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:06.665+0000 7fd6d60db700 1 -- 172.21.15.2:0/3003865206 wait complete. 2023-11-16T18:25:07.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:06 smithi040 ceph-mon[100251]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 100 KiB/s wr, 45 op/s 2023-11-16T18:25:07.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:06 smithi188 ceph-mon[100639]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 100 KiB/s wr, 45 op/s 2023-11-16T18:25:07.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:06 smithi002 ceph-mon[98807]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 100 KiB/s wr, 45 op/s 2023-11-16T18:25:09.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:08 smithi040 ceph-mon[100251]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 100 KiB/s wr, 47 op/s 2023-11-16T18:25:09.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:08 smithi188 ceph-mon[100639]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 100 KiB/s wr, 47 op/s 2023-11-16T18:25:09.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:08 smithi002 ceph-mon[98807]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 100 KiB/s wr, 47 op/s 2023-11-16T18:25:11.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:10 smithi040 ceph-mon[100251]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 64 KiB/s wr, 29 op/s 2023-11-16T18:25:11.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:10 smithi188 ceph-mon[100639]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 64 KiB/s wr, 29 op/s 2023-11-16T18:25:11.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:10 smithi002 ceph-mon[98807]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 64 KiB/s wr, 29 op/s 2023-11-16T18:25:12.441 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:25:12.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.575+0000 7f22dd339700 1 -- 172.21.15.2:0/2808918959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d8101d90 msgr2=0x7f22d810a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:12.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.575+0000 7f22dd339700 1 --2- 172.21.15.2:0/2808918959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d8101d90 0x7f22d810a280 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009990 tx=0x7f22cc009ca0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.575+0000 7f22dd339700 1 -- 172.21.15.2:0/2808918959 shutdown_connections 2023-11-16T18:25:12.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.575+0000 7f22dd339700 1 --2- 172.21.15.2:0/2808918959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22d810a850 0x7f22d810cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.575+0000 7f22dd339700 1 --2- 172.21.15.2:0/2808918959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22d8101480 0x7f22d8101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.575+0000 7f22dd339700 1 --2- 172.21.15.2:0/2808918959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d8101d90 0x7f22d810a280 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.575+0000 7f22dd339700 1 -- 172.21.15.2:0/2808918959 >> 172.21.15.2:0/2808918959 conn(0x7f22d8076230 msgr2=0x7f22d8076630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:12.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.575+0000 7f22dd339700 1 -- 172.21.15.2:0/2808918959 shutdown_connections 2023-11-16T18:25:12.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.575+0000 7f22dd339700 1 -- 172.21.15.2:0/2808918959 wait complete. 2023-11-16T18:25:12.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.577+0000 7f22dd339700 1 Processor -- start 2023-11-16T18:25:12.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.577+0000 7f22dd339700 1 -- start start 2023-11-16T18:25:12.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.577+0000 7f22dd339700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22d8101480 0x7f22d80ff0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.577+0000 7f22dd339700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22d8101d90 0x7f22d80ff610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.577+0000 7f22dd339700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d810a850 0x7f22d81a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.577+0000 7f22dd339700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d80ffe40 con 0x7f22d8101d90 2023-11-16T18:25:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.577+0000 7f22dd339700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d81a4b00 con 0x7f22d8101480 2023-11-16T18:25:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.577+0000 7f22dd339700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d81a4c40 con 0x7f22d810a850 2023-11-16T18:25:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22d6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22d8101480 0x7f22d80ff0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22d67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22d8101d90 0x7f22d80ff610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22d6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22d8101480 0x7f22d80ff0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:40858/0 (socket says 172.21.15.2:40858) 2023-11-16T18:25:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22d67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22d8101d90 0x7f22d80ff610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50424/0 (socket says 172.21.15.2:50424) 2023-11-16T18:25:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22d6ffd700 1 -- 172.21.15.2:0/2234793053 learned_addr learned my addr 172.21.15.2:0/2234793053 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22d77fe700 1 --2- 172.21.15.2:0/2234793053 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d810a850 0x7f22d81a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22d6ffd700 1 -- 172.21.15.2:0/2234793053 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d810a850 msgr2=0x7f22d81a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22d6ffd700 1 --2- 172.21.15.2:0/2234793053 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d810a850 0x7f22d81a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22d6ffd700 1 -- 172.21.15.2:0/2234793053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22d8101d90 msgr2=0x7f22d80ff610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22d6ffd700 1 --2- 172.21.15.2:0/2234793053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22d8101d90 0x7f22d80ff610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22d6ffd700 1 -- 172.21.15.2:0/2234793053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22cc0095f0 con 0x7f22d8101480 2023-11-16T18:25:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22d6ffd700 1 --2- 172.21.15.2:0/2234793053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22d8101480 0x7f22d80ff0d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f22c800b530 tx=0x7f22c800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22bffff700 1 -- 172.21.15.2:0/2234793053 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c8011830 con 0x7f22d8101480 2023-11-16T18:25:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22dd339700 1 -- 172.21.15.2:0/2234793053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d8110350 con 0x7f22d8101480 2023-11-16T18:25:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22bffff700 1 -- 172.21.15.2:0/2234793053 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22c8011e50 con 0x7f22d8101480 2023-11-16T18:25:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.578+0000 7f22bffff700 1 -- 172.21.15.2:0/2234793053 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c800f2e0 con 0x7f22d8101480 2023-11-16T18:25:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.579+0000 7f22dd339700 1 -- 172.21.15.2:0/2234793053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d81101c0 con 0x7f22d8101480 2023-11-16T18:25:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.582+0000 7f22bffff700 1 -- 172.21.15.2:0/2234793053 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f22c80105a0 con 0x7f22d8101480 2023-11-16T18:25:12.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.582+0000 7f22e94c5b80 1 Processor -- start 2023-11-16T18:25:12.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.582+0000 7f22e94c5b80 1 -- start start 2023-11-16T18:25:12.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22bffff700 1 --2- 172.21.15.2:0/2234793053 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f22c00a13e0 0x7f22c00a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22e94c5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b10bb60 0x56165b10bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22e94c5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 0x56165b110370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22e94c5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b1108b0 0x56165b110d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22e94c5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56165b011060 con 0x56165b1108b0 2023-11-16T18:25:12.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22e94c5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56165b0638a0 con 0x56165b10bb60 2023-11-16T18:25:12.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22e94c5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56165b05cd30 con 0x56165b10c470 2023-11-16T18:25:12.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22d6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b10bb60 0x56165b10bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22d77fe700 1 --2- 172.21.15.2:0/2234793053 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f22c00a13e0 0x7f22c00a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22d6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b10bb60 0x56165b10bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:40864/0 (socket says 172.21.15.2:40864) 2023-11-16T18:25:12.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22d6ffd700 1 -- 172.21.15.2:0/864612434 learned_addr learned my addr 172.21.15.2:0/864612434 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:12.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22d67fc700 1 --2- 172.21.15.2:0/864612434 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 0x56165b110370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22d67fc700 1 -- 172.21.15.2:0/864612434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b10bb60 msgr2=0x56165b10bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:12.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22d67fc700 1 --2- 172.21.15.2:0/864612434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b10bb60 0x56165b10bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22d67fc700 1 -- 172.21.15.2:0/864612434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b1108b0 msgr2=0x56165b110d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:12.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.583+0000 7f22d67fc700 1 --2- 172.21.15.2:0/864612434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b1108b0 0x56165b110d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.584+0000 7f22d67fc700 1 -- 172.21.15.2:0/864612434 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56165aede630 con 0x56165b10c470 2023-11-16T18:25:12.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.584+0000 7f22d6ffd700 1 --2- 172.21.15.2:0/864612434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b10bb60 0x56165b10bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:12.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.584+0000 7f22bffff700 1 -- 172.21.15.2:0/2234793053 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f22c80ccbd0 con 0x7f22d8101480 2023-11-16T18:25:12.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.584+0000 7f22d77fe700 1 --2- 172.21.15.2:0/2234793053 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f22c00a13e0 0x7f22c00a3890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f22d0009ec0 tx=0x7f22d0009920 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:12.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.584+0000 7f22d67fc700 1 --2- 172.21.15.2:0/864612434 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 0x56165b110370 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00b2e0 tx=0x7f22cc005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=529b78e3efcef93d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:12.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.584+0000 7f22bd7fa700 1 -- 172.21.15.2:0/864612434 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22cc01d070 con 0x56165b10c470 2023-11-16T18:25:12.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.584+0000 7f22bd7fa700 1 -- 172.21.15.2:0/864612434 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22cc01e040 con 0x56165b10c470 2023-11-16T18:25:12.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.584+0000 7f22bd7fa700 1 -- 172.21.15.2:0/864612434 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22cc0049d0 con 0x56165b10c470 2023-11-16T18:25:12.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.585+0000 7f22e94c5b80 1 -- 172.21.15.2:0/864612434 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 msgr2=0x56165b110370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:12.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.585+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/864612434 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 0x56165b110370 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00b2e0 tx=0x7f22cc005bc0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.585+0000 7f22e94c5b80 1 -- 172.21.15.2:0/864612434 shutdown_connections 2023-11-16T18:25:12.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.585+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/864612434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b1108b0 0x56165b110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.585+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/864612434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b10bb60 0x56165b10bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.585+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/864612434 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 0x56165b110370 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.585+0000 7f22e94c5b80 1 -- 172.21.15.2:0/864612434 >> 172.21.15.2:0/864612434 conn(0x56165b101360 msgr2=0x56165b106b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:12.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.585+0000 7f22e94c5b80 1 -- 172.21.15.2:0/864612434 shutdown_connections 2023-11-16T18:25:12.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.585+0000 7f22e94c5b80 1 -- 172.21.15.2:0/864612434 wait complete. 2023-11-16T18:25:12.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.585+0000 7f22e94c5b80 1 -- start start 2023-11-16T18:25:12.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.585+0000 7f22e94c5b80 1 Processor -- start 2023-11-16T18:25:12.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22e94c5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b1153c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22e94c5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 0x56165b115900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22e94c5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b1108b0 0x56165b115e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22e94c5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56165affe8e0 con 0x56165b10bb60 2023-11-16T18:25:12.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22e94c5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56165affe580 con 0x56165b1108b0 2023-11-16T18:25:12.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22e94c5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56165aee0920 con 0x56165b10c470 2023-11-16T18:25:12.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22e94c5b80 10 client.0 _ll_register_callbacks cb 0x56165b01c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:12.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22d6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b1153c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22d6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b1153c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50434/0 (socket says 172.21.15.2:50434) 2023-11-16T18:25:12.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22d6ffd700 1 -- 172.21.15.2:0/2026221073 learned_addr learned my addr 172.21.15.2:0/2026221073 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:12.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22d67fc700 1 --2- 172.21.15.2:0/2026221073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 0x56165b115900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22d77fe700 1 --2- 172.21.15.2:0/2026221073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b1108b0 0x56165b115e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22d6ffd700 1 -- 172.21.15.2:0/2026221073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 msgr2=0x56165b115900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:12.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22d6ffd700 1 --2- 172.21.15.2:0/2026221073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 0x56165b115900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22d6ffd700 1 -- 172.21.15.2:0/2026221073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b1108b0 msgr2=0x56165b115e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:12.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22d6ffd700 1 --2- 172.21.15.2:0/2026221073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b1108b0 0x56165b115e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.586+0000 7f22d6ffd700 1 -- 172.21.15.2:0/2026221073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d810d950 con 0x56165b10bb60 2023-11-16T18:25:12.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.587+0000 7f22d6ffd700 1 --2- 172.21.15.2:0/2026221073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b1153c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f22c80ca050 tx=0x7f22c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b7544b39225bc5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:12.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.587+0000 7f22baff5700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:12.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22baff5700 1 -- 172.21.15.2:0/2026221073 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c8094200 con 0x56165b10bb60 2023-11-16T18:25:12.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22baff5700 1 -- 172.21.15.2:0/2026221073 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f22c80b92c0 con 0x56165b10bb60 2023-11-16T18:25:12.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22e94c5b80 1 -- 172.21.15.2:0/2026221073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56165b0feb20 con 0x56165b10bb60 2023-11-16T18:25:12.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22baff5700 1 -- 172.21.15.2:0/2026221073 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22c8021b30 con 0x56165b10bb60 2023-11-16T18:25:12.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22e94c5b80 10 client.15702 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:12.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22e94c5b80 1 -- 172.21.15.2:0/2026221073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56165b023190 con 0x56165b10bb60 2023-11-16T18:25:12.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22baff5700 1 -- 172.21.15.2:0/2026221073 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56165b0feb20 con 0x56165b10bb60 2023-11-16T18:25:12.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22baff5700 1 -- 172.21.15.2:0/2026221073 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f22c8098ce0 con 0x56165b10bb60 2023-11-16T18:25:12.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22e94c5b80 10 client.15702 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:12.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22e94c5b80 10 client.15702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:12.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22e94c5b80 20 client.15702 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:12.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/2026221073 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56165b11c840 0x56165b11ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22e94c5b80 4 client.15702 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:25:12.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.591+0000 7f22e94c5b80 1 -- 172.21.15.2:0/2026221073 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56165b01f0a0 con 0x56165b11c840 2023-11-16T18:25:12.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.592+0000 7f22d67fc700 1 --2- 172.21.15.2:0/2026221073 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56165b11c840 0x56165b11ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.592+0000 7f22d67fc700 1 --2- 172.21.15.2:0/2026221073 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56165b11c840 0x56165b11ec50 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1355986863f7def server_cookie=9dfbed66155080ba in_seq=0 out_seq=0 2023-11-16T18:25:12.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.592+0000 7f22baff5700 10 client.15702 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:12.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22baff5700 1 -- 172.21.15.2:0/2026221073 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56165b01f0a0 con 0x56165b11c840 2023-11-16T18:25:12.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22baff5700 10 client.15702 handle_command_reply: tid=0 2023-11-16T18:25:12.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22e94c5b80 1 client.15702 shutdown 2023-11-16T18:25:12.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22e94c5b80 1 -- 172.21.15.2:0/2026221073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 msgr2=0x56165b1153c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:12.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/2026221073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b1153c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f22c80ca050 tx=0x7f22c8004750 comp rx=0 tx=0).stop 2023-11-16T18:25:12.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22e94c5b80 1 -- 172.21.15.2:0/2026221073 shutdown_connections 2023-11-16T18:25:12.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/2026221073 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56165b11c840 0x56165b11ec50 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/2026221073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b1108b0 0x56165b115e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/2026221073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b1153c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22baff5700 0 client.15702 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:12.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/2026221073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 0x56165b115900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22e94c5b80 1 -- 172.21.15.2:0/2026221073 >> 172.21.15.2:0/2026221073 conn(0x56165b101360 msgr2=0x56165b111d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:12.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22e94c5b80 1 -- 172.21.15.2:0/2026221073 shutdown_connections 2023-11-16T18:25:12.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.593+0000 7f22e94c5b80 1 -- 172.21.15.2:0/2026221073 wait complete. 2023-11-16T18:25:12.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.594+0000 7f22e94c5b80 20 client.15702 trim_cache size 0 max 16384 2023-11-16T18:25:12.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22e94c5b80 1 Processor -- start 2023-11-16T18:25:12.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22e94c5b80 1 -- start start 2023-11-16T18:25:12.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22e94c5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b123580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22e94c5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 0x56165b105550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22e94c5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b1108b0 0x56165b105a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22e94c5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56165b113480 con 0x56165b10bb60 2023-11-16T18:25:12.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22e94c5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56165b10c840 con 0x56165b1108b0 2023-11-16T18:25:12.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22e94c5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56165b113940 con 0x56165b10c470 2023-11-16T18:25:12.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22d6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b123580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22d6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b123580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50442/0 (socket says 172.21.15.2:50442) 2023-11-16T18:25:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22d6ffd700 1 -- 172.21.15.2:0/2558340869 learned_addr learned my addr 172.21.15.2:0/2558340869 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22d67fc700 1 --2- 172.21.15.2:0/2558340869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 0x56165b105550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22d6ffd700 1 -- 172.21.15.2:0/2558340869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 msgr2=0x56165b105550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22d6ffd700 1 --2- 172.21.15.2:0/2558340869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 0x56165b105550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22d6ffd700 1 -- 172.21.15.2:0/2558340869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b1108b0 msgr2=0x56165b105a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.611+0000 7f22d77fe700 1 --2- 172.21.15.2:0/2558340869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b1108b0 0x56165b105a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.612+0000 7f22d6ffd700 1 --2- 172.21.15.2:0/2558340869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b1108b0 0x56165b105a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.612+0000 7f22d6ffd700 1 -- 172.21.15.2:0/2558340869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56165b106100 con 0x56165b10bb60 2023-11-16T18:25:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.612+0000 7f22d77fe700 1 --2- 172.21.15.2:0/2558340869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b1108b0 0x56165b105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:25:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.612+0000 7f22d6ffd700 1 --2- 172.21.15.2:0/2558340869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b123580 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f22c800bb90 tx=0x7f22c80c9fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ebff5905e3b768 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.612+0000 7f22bdffb700 1 -- 172.21.15.2:0/2558340869 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c8010f00 con 0x56165b10bb60 2023-11-16T18:25:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.612+0000 7f22bdffb700 1 -- 172.21.15.2:0/2558340869 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22c8011990 con 0x56165b10bb60 2023-11-16T18:25:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.612+0000 7f22bdffb700 1 -- 172.21.15.2:0/2558340869 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c80a17b0 con 0x56165b10bb60 2023-11-16T18:25:12.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.613+0000 7f22e94c5b80 1 -- 172.21.15.2:0/2558340869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 msgr2=0x56165b123580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:12.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.613+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/2558340869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b123580 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f22c800bb90 tx=0x7f22c80c9fc0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.613+0000 7f22e94c5b80 1 -- 172.21.15.2:0/2558340869 shutdown_connections 2023-11-16T18:25:12.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.613+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/2558340869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b1108b0 0x56165b105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.613+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/2558340869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b123580 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.613+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/2558340869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b10c470 0x56165b105550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.613+0000 7f22e94c5b80 1 -- 172.21.15.2:0/2558340869 >> 172.21.15.2:0/2558340869 conn(0x56165b101360 msgr2=0x56165b118c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:12.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.613+0000 7f22e94c5b80 1 -- 172.21.15.2:0/2558340869 shutdown_connections 2023-11-16T18:25:12.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.613+0000 7f22e94c5b80 1 -- 172.21.15.2:0/2558340869 wait complete. 2023-11-16T18:25:12.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22e94c5b80 1 -- start start 2023-11-16T18:25:12.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22e94c5b80 1 Processor -- start 2023-11-16T18:25:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22e94c5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b184dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22e94c5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b10c470 0x56165b17e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22e94c5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b1108b0 0x56165b17ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22e94c5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56165b17f110 con 0x56165b10bb60 2023-11-16T18:25:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22e94c5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56165b182860 con 0x56165b10c470 2023-11-16T18:25:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22e94c5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56165b182a20 con 0x56165b1108b0 2023-11-16T18:25:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22e94c5b80 10 client.0 _ll_register_callbacks cb 0x7f22c80c9f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22d6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b184dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22d6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b184dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50448/0 (socket says 172.21.15.2:50448) 2023-11-16T18:25:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22d6ffd700 1 -- 172.21.15.2:0/57964419 learned_addr learned my addr 172.21.15.2:0/57964419 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22d77fe700 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b1108b0 0x56165b17ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.614+0000 7f22d67fc700 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b10c470 0x56165b17e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22d6ffd700 1 -- 172.21.15.2:0/57964419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b1108b0 msgr2=0x56165b17ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:12.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22d6ffd700 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b1108b0 0x56165b17ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22d6ffd700 1 -- 172.21.15.2:0/57964419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b10c470 msgr2=0x56165b17e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:12.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22d6ffd700 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b10c470 0x56165b17e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22d6ffd700 1 -- 172.21.15.2:0/57964419 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56165b023190 con 0x56165b10bb60 2023-11-16T18:25:12.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22d67fc700 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b10c470 0x56165b17e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:25:12.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22d6ffd700 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b184dc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f22c8097b40 tx=0x7f22c8011d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a551b7489a6d96d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:12.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22baff5700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:12.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22baff5700 1 -- 172.21.15.2:0/57964419 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c80a1c40 con 0x56165b10bb60 2023-11-16T18:25:12.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22baff5700 1 -- 172.21.15.2:0/57964419 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f22c80c8020 con 0x56165b10bb60 2023-11-16T18:25:12.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22e94c5b80 1 -- 172.21.15.2:0/57964419 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22cc01e040 con 0x56165b10bb60 2023-11-16T18:25:12.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22baff5700 1 -- 172.21.15.2:0/57964419 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22c809d820 con 0x56165b10bb60 2023-11-16T18:25:12.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22e94c5b80 10 client.15708 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:12.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22e94c5b80 1 -- 172.21.15.2:0/57964419 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56165b022ee0 con 0x56165b10bb60 2023-11-16T18:25:12.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.615+0000 7f22baff5700 1 -- 172.21.15.2:0/57964419 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22cc01e040 con 0x56165b10bb60 2023-11-16T18:25:12.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.616+0000 7f22baff5700 1 -- 172.21.15.2:0/57964419 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f22c808c070 con 0x56165b10bb60 2023-11-16T18:25:12.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.616+0000 7f22e94c5b80 10 client.15708 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:12.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.616+0000 7f22e94c5b80 10 client.15708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:12.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.616+0000 7f22e94c5b80 20 client.15708 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:12.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.616+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56165b19c5a0 0x56165b19e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.616+0000 7f22e94c5b80 4 client.15708 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:25:12.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.616+0000 7f22e94c5b80 1 -- 172.21.15.2:0/57964419 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56165b02b660 con 0x56165b19c5a0 2023-11-16T18:25:12.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.616+0000 7f22d67fc700 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56165b19c5a0 0x56165b19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22d67fc700 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56165b19c5a0 0x56165b19e980 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f91039524c099af6 server_cookie=be1a88de87b87da7 in_seq=0 out_seq=0 2023-11-16T18:25:12.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22baff5700 10 client.15708 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:12.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22baff5700 1 -- 172.21.15.2:0/57964419 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56165b02b660 con 0x56165b19c5a0 2023-11-16T18:25:12.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22baff5700 10 client.15708 handle_command_reply: tid=0 2023-11-16T18:25:12.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22e94c5b80 1 client.15708 shutdown 2023-11-16T18:25:12.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22e94c5b80 1 -- 172.21.15.2:0/57964419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 msgr2=0x56165b184dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:12.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b184dc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f22c8097b40 tx=0x7f22c8011d00 comp rx=0 tx=0).stop 2023-11-16T18:25:12.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22e94c5b80 1 -- 172.21.15.2:0/57964419 shutdown_connections 2023-11-16T18:25:12.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56165b19c5a0 0x56165b19e980 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56165b1108b0 0x56165b17ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56165b10bb60 0x56165b184dc0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22e94c5b80 1 --2- 172.21.15.2:0/57964419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56165b10c470 0x56165b17e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22e94c5b80 1 -- 172.21.15.2:0/57964419 >> 172.21.15.2:0/57964419 conn(0x56165b101360 msgr2=0x56165b105550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:12.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22baff5700 0 client.15708 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:12.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22e94c5b80 1 -- 172.21.15.2:0/57964419 shutdown_connections 2023-11-16T18:25:12.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22e94c5b80 1 -- 172.21.15.2:0/57964419 wait complete. 2023-11-16T18:25:12.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.617+0000 7f22e94c5b80 20 client.15708 trim_cache size 0 max 16384 2023-11-16T18:25:12.625 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:25:12.625 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "no active scrubs running", 2023-11-16T18:25:12.625 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": {} 2023-11-16T18:25:12.625 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:25:12.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.618+0000 7f22dd339700 1 -- 172.21.15.2:0/2234793053 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f22c00a13e0 msgr2=0x7f22c00a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:12.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.618+0000 7f22dd339700 1 --2- 172.21.15.2:0/2234793053 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f22c00a13e0 0x7f22c00a3890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f22d0009ec0 tx=0x7f22d0009920 comp rx=0 tx=0).stop 2023-11-16T18:25:12.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.618+0000 7f22dd339700 1 -- 172.21.15.2:0/2234793053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22d8101480 msgr2=0x7f22d80ff0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:12.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.618+0000 7f22dd339700 1 --2- 172.21.15.2:0/2234793053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22d8101480 0x7f22d80ff0d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f22c800b530 tx=0x7f22c800b840 comp rx=0 tx=0).stop 2023-11-16T18:25:12.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.618+0000 7f22dd339700 1 -- 172.21.15.2:0/2234793053 shutdown_connections 2023-11-16T18:25:12.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.618+0000 7f22dd339700 1 --2- 172.21.15.2:0/2234793053 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f22c00a13e0 0x7f22c00a3890 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.618+0000 7f22dd339700 1 --2- 172.21.15.2:0/2234793053 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d810a850 0x7f22d81a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.618+0000 7f22dd339700 1 --2- 172.21.15.2:0/2234793053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22d8101480 0x7f22d80ff0d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.618+0000 7f22dd339700 1 --2- 172.21.15.2:0/2234793053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22d8101d90 0x7f22d80ff610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.618+0000 7f22dd339700 1 -- 172.21.15.2:0/2234793053 >> 172.21.15.2:0/2234793053 conn(0x7f22d8076230 msgr2=0x7f22d80fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:12.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.618+0000 7f22dd339700 1 -- 172.21.15.2:0/2234793053 shutdown_connections 2023-11-16T18:25:12.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.618+0000 7f22dd339700 1 -- 172.21.15.2:0/2234793053 wait complete. 2023-11-16T18:25:12.632 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T18:25:12.632 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:25:12.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.759+0000 7fed55d8b700 1 -- 172.21.15.2:0/2448369127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed500ff410 msgr2=0x7fed500ff7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:12.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.759+0000 7fed55d8b700 1 --2- 172.21.15.2:0/2448369127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed500ff410 0x7fed500ff7e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fed400099c0 tx=0x7fed40009cd0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.759+0000 7fed55d8b700 1 -- 172.21.15.2:0/2448369127 shutdown_connections 2023-11-16T18:25:12.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.759+0000 7fed55d8b700 1 --2- 172.21.15.2:0/2448369127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed50104450 0x7fed5010c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.759+0000 7fed55d8b700 1 --2- 172.21.15.2:0/2448369127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed500ff410 0x7fed500ff7e0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.759+0000 7fed55d8b700 1 --2- 172.21.15.2:0/2448369127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed500ffdb0 0x7fed50103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.759+0000 7fed55d8b700 1 -- 172.21.15.2:0/2448369127 >> 172.21.15.2:0/2448369127 conn(0x7fed500fac20 msgr2=0x7fed500fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:12.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.759+0000 7fed55d8b700 1 -- 172.21.15.2:0/2448369127 shutdown_connections 2023-11-16T18:25:12.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.759+0000 7fed55d8b700 1 -- 172.21.15.2:0/2448369127 wait complete. 2023-11-16T18:25:12.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.760+0000 7fed55d8b700 1 Processor -- start 2023-11-16T18:25:12.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.760+0000 7fed55d8b700 1 -- start start 2023-11-16T18:25:12.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.760+0000 7fed55d8b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed500ff410 0x7fed50196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.760+0000 7fed55d8b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed500ffdb0 0x7fed50196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.760+0000 7fed55d8b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed50104450 0x7fed5019adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.760+0000 7fed55d8b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5019b430 con 0x7fed500ff410 2023-11-16T18:25:12.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.760+0000 7fed55d8b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5019b5a0 con 0x7fed500ffdb0 2023-11-16T18:25:12.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.760+0000 7fed55d8b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5019b760 con 0x7fed50104450 2023-11-16T18:25:12.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.760+0000 7fed4f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed500ff410 0x7fed50196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.760+0000 7fed4f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed500ff410 0x7fed50196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50462/0 (socket says 172.21.15.2:50462) 2023-11-16T18:25:12.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.760+0000 7fed4f7fe700 1 -- 172.21.15.2:0/3752832232 learned_addr learned my addr 172.21.15.2:0/3752832232 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:12.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.760+0000 7fed4effd700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed500ffdb0 0x7fed50196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.760+0000 7fed4ffff700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed50104450 0x7fed5019adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.761+0000 7fed4effd700 1 -- 172.21.15.2:0/3752832232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed50104450 msgr2=0x7fed5019adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:12.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.761+0000 7fed4effd700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed50104450 0x7fed5019adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.761+0000 7fed4effd700 1 -- 172.21.15.2:0/3752832232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed500ff410 msgr2=0x7fed50196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:12.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.761+0000 7fed4effd700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed500ff410 0x7fed50196450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:12.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.761+0000 7fed4effd700 1 -- 172.21.15.2:0/3752832232 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed400095f0 con 0x7fed500ffdb0 2023-11-16T18:25:12.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.761+0000 7fed4effd700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed500ffdb0 0x7fed50196990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fed50100180 tx=0x7fed4400eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:12.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.761+0000 7fed4cff9700 1 -- 172.21.15.2:0/3752832232 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44010900 con 0x7fed500ffdb0 2023-11-16T18:25:12.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.761+0000 7fed55d8b700 1 -- 172.21.15.2:0/3752832232 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed501103c0 con 0x7fed500ffdb0 2023-11-16T18:25:12.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.761+0000 7fed4cff9700 1 -- 172.21.15.2:0/3752832232 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed4400c430 con 0x7fed500ffdb0 2023-11-16T18:25:12.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.761+0000 7fed4cff9700 1 -- 172.21.15.2:0/3752832232 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44018680 con 0x7fed500ffdb0 2023-11-16T18:25:12.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.761+0000 7fed55d8b700 1 -- 172.21.15.2:0/3752832232 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed50110230 con 0x7fed500ffdb0 2023-11-16T18:25:12.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.763+0000 7fed55d8b700 1 -- 172.21.15.2:0/3752832232 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed50110550 con 0x7fed500ffdb0 2023-11-16T18:25:12.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.764+0000 7fed4cff9700 1 -- 172.21.15.2:0/3752832232 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7fed4400c5a0 con 0x7fed500ffdb0 2023-11-16T18:25:12.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.765+0000 7fed4cff9700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fed380a13e0 0x7fed380a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:12.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.765+0000 7fed4f7fe700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fed380a13e0 0x7fed380a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:12.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.765+0000 7fed4cff9700 1 -- 172.21.15.2:0/3752832232 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fed44014070 con 0x7fed500ffdb0 2023-11-16T18:25:12.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.765+0000 7fed4f7fe700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fed380a13e0 0x7fed380a3890 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fed400036c0 tx=0x7fed40005fb0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:12.770 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:12.768+0000 7fed4cff9700 1 -- 172.21.15.2:0/3752832232 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fed440961e0 con 0x7fed500ffdb0 2023-11-16T18:25:13.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:12 smithi040 ceph-mon[100251]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 22 op/s 2023-11-16T18:25:13.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.253+0000 7fed55d8b700 1 -- 172.21.15.2:0/3752832232 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed50067f40 con 0x7fed500ffdb0 2023-11-16T18:25:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.254+0000 7fed4cff9700 1 -- 172.21.15.2:0/3752832232 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10774 (secure 0 0 0) 0x7fed44099800 con 0x7fed500ffdb0 2023-11-16T18:25:13.256 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:25:13.257 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:23:07.909835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:25:13.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.257+0000 7fed55d8b700 1 -- 172.21.15.2:0/3752832232 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fed380a13e0 msgr2=0x7fed380a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.257+0000 7fed55d8b700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fed380a13e0 0x7fed380a3890 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fed400036c0 tx=0x7fed40005fb0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.257+0000 7fed55d8b700 1 -- 172.21.15.2:0/3752832232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed500ffdb0 msgr2=0x7fed50196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.257+0000 7fed55d8b700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed500ffdb0 0x7fed50196990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fed50100180 tx=0x7fed4400eca0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.257+0000 7fed55d8b700 1 -- 172.21.15.2:0/3752832232 shutdown_connections 2023-11-16T18:25:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.257+0000 7fed55d8b700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fed380a13e0 0x7fed380a3890 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.257+0000 7fed55d8b700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed50104450 0x7fed5019adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.257+0000 7fed55d8b700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed500ff410 0x7fed50196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.257+0000 7fed55d8b700 1 --2- 172.21.15.2:0/3752832232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed500ffdb0 0x7fed50196990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.257+0000 7fed55d8b700 1 -- 172.21.15.2:0/3752832232 >> 172.21.15.2:0/3752832232 conn(0x7fed500fac20 msgr2=0x7fed500fb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.257+0000 7fed55d8b700 1 -- 172.21.15.2:0/3752832232 shutdown_connections 2023-11-16T18:25:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.257+0000 7fed55d8b700 1 -- 172.21.15.2:0/3752832232 wait complete. 2023-11-16T18:25:13.261 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T18:25:13.274 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:12 smithi002 ceph-mon[98807]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 22 op/s 2023-11-16T18:25:13.276 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-11-16T18:25:13.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:12 smithi188 ceph-mon[100639]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 22 op/s 2023-11-16T18:25:13.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.469+0000 7f6ba27d2700 1 -- 172.21.15.2:0/3976599797 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b9c0ff430 msgr2=0x7f6b9c0ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.469+0000 7f6ba27d2700 1 --2- 172.21.15.2:0/3976599797 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b9c0ff430 0x7f6b9c0ff800 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c0099c0 tx=0x7f6b8c009cd0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.469+0000 7f6ba27d2700 1 -- 172.21.15.2:0/3976599797 shutdown_connections 2023-11-16T18:25:13.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.469+0000 7f6ba27d2700 1 --2- 172.21.15.2:0/3976599797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c104470 0x7f6b9c10c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.469+0000 7f6ba27d2700 1 --2- 172.21.15.2:0/3976599797 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b9c0ff430 0x7f6b9c0ff800 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.469+0000 7f6ba27d2700 1 --2- 172.21.15.2:0/3976599797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b9c0ffdd0 0x7f6b9c103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.469+0000 7f6ba27d2700 1 -- 172.21.15.2:0/3976599797 >> 172.21.15.2:0/3976599797 conn(0x7f6b9c0763a0 msgr2=0x7f6b9c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:13.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.469+0000 7f6ba27d2700 1 -- 172.21.15.2:0/3976599797 shutdown_connections 2023-11-16T18:25:13.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.469+0000 7f6ba27d2700 1 -- 172.21.15.2:0/3976599797 wait complete. 2023-11-16T18:25:13.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.470+0000 7f6ba27d2700 1 Processor -- start 2023-11-16T18:25:13.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.470+0000 7f6ba27d2700 1 -- start start 2023-11-16T18:25:13.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.470+0000 7f6ba27d2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c0ff430 0x7f6b9c10afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.470+0000 7f6ba27d2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b9c0ffdd0 0x7f6b9c10b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.470+0000 7f6ba27d2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b9c104470 0x7f6b9c1b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.470+0000 7f6ba27d2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c1b1660 con 0x7f6b9c104470 2023-11-16T18:25:13.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.470+0000 7f6ba27d2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c1b17d0 con 0x7f6b9c0ff430 2023-11-16T18:25:13.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.470+0000 7f6ba27d2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c1b1990 con 0x7f6b9c0ffdd0 2023-11-16T18:25:13.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.470+0000 7f6ba0d6f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b9c104470 0x7f6b9c1b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.470+0000 7f6ba0d6f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b9c104470 0x7f6b9c1b1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50478/0 (socket says 172.21.15.2:50478) 2023-11-16T18:25:13.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.470+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/3261617272 learned_addr learned my addr 172.21.15.2:0/3261617272 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:13.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.470+0000 7f6b9b7fe700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b9c0ffdd0 0x7f6b9c10b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.471+0000 7f6b9bfff700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c0ff430 0x7f6b9c10afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.471+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/3261617272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b9c0ffdd0 msgr2=0x7f6b9c10b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.471+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b9c0ffdd0 0x7f6b9c10b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.471+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/3261617272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c0ff430 msgr2=0x7f6b9c10afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.471+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c0ff430 0x7f6b9c10afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.471+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/3261617272 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c0095f0 con 0x7f6b9c104470 2023-11-16T18:25:13.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.471+0000 7f6b9b7fe700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b9c0ffdd0 0x7f6b9c10b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:13.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.471+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b9c104470 0x7f6b9c1b1060 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400ef30 tx=0x7f6b9400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:13.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.471+0000 7f6b997fa700 1 -- 172.21.15.2:0/3261617272 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94018070 con 0x7f6b9c104470 2023-11-16T18:25:13.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.471+0000 7f6ba27d2700 1 -- 172.21.15.2:0/3261617272 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9c1103a0 con 0x7f6b9c104470 2023-11-16T18:25:13.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.471+0000 7f6ba27d2700 1 -- 172.21.15.2:0/3261617272 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9c110210 con 0x7f6b9c104470 2023-11-16T18:25:13.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.471+0000 7f6b997fa700 1 -- 172.21.15.2:0/3261617272 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b9400f040 con 0x7f6b9c104470 2023-11-16T18:25:13.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.471+0000 7f6b997fa700 1 -- 172.21.15.2:0/3261617272 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400cd00 con 0x7f6b9c104470 2023-11-16T18:25:13.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.475+0000 7f6b997fa700 1 -- 172.21.15.2:0/3261617272 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f6b94004ad0 con 0x7f6b9c104470 2023-11-16T18:25:13.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.476+0000 7f6b997fa700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b840a1460 0x7f6b840a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.476+0000 7f6b997fa700 1 -- 172.21.15.2:0/3261617272 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6b940cccb0 con 0x7f6b9c104470 2023-11-16T18:25:13.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.476+0000 7f6b9bfff700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b840a1460 0x7f6b840a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.476+0000 7f6bae95eb80 1 Processor -- start 2023-11-16T18:25:13.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.477+0000 7f6b9bfff700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b840a1460 0x7f6b840a3910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009990 tx=0x7f6b8c005fb0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:13.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.477+0000 7f6bae95eb80 1 -- start start 2023-11-16T18:25:13.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.477+0000 7f6bae95eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f3b60 0x556ada3f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.477+0000 7f6bae95eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3f8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.477+0000 7f6bae95eb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f88b0 0x556ada3f8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.477+0000 7f6bae95eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ada2f9060 con 0x556ada3f4470 2023-11-16T18:25:13.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.477+0000 7f6bae95eb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ada34b8a0 con 0x556ada3f88b0 2023-11-16T18:25:13.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.477+0000 7f6bae95eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ada344d30 con 0x556ada3f3b60 2023-11-16T18:25:13.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.477+0000 7f6b9b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3f8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.477+0000 7f6b9bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f3b60 0x556ada3f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.477+0000 7f6b9bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f3b60 0x556ada3f3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:42014/0 (socket says 172.21.15.2:42014) 2023-11-16T18:25:13.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.477+0000 7f6b9bfff700 1 -- 172.21.15.2:0/542047987 learned_addr learned my addr 172.21.15.2:0/542047987 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:13.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.478+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/542047987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f88b0 0x556ada3f8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.478+0000 7f6b9bfff700 1 -- 172.21.15.2:0/542047987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f88b0 msgr2=0x556ada3f8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.478+0000 7f6b9bfff700 1 --2- 172.21.15.2:0/542047987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f88b0 0x556ada3f8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.478+0000 7f6b9bfff700 1 -- 172.21.15.2:0/542047987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 msgr2=0x556ada3f8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.478+0000 7f6b9bfff700 1 --2- 172.21.15.2:0/542047987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3f8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.478+0000 7f6b9bfff700 1 -- 172.21.15.2:0/542047987 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ada1c6630 con 0x556ada3f3b60 2023-11-16T18:25:13.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.478+0000 7f6b9bfff700 1 --2- 172.21.15.2:0/542047987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f3b60 0x556ada3f3f30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c0099c0 tx=0x7f6b8c01f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c26d16b00627f6cc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:13.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.478+0000 7f6b827fc700 1 -- 172.21.15.2:0/542047987 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c00b990 con 0x556ada3f3b60 2023-11-16T18:25:13.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.478+0000 7f6b827fc700 1 -- 172.21.15.2:0/542047987 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b8c01e040 con 0x556ada3f3b60 2023-11-16T18:25:13.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.478+0000 7f6b827fc700 1 -- 172.21.15.2:0/542047987 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c0306e0 con 0x556ada3f3b60 2023-11-16T18:25:13.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.479+0000 7f6bae95eb80 1 -- 172.21.15.2:0/542047987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f3b60 msgr2=0x556ada3f3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.479+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/542047987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f3b60 0x556ada3f3f30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c0099c0 tx=0x7f6b8c01f000 comp rx=0 tx=0).stop 2023-11-16T18:25:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.479+0000 7f6bae95eb80 1 -- 172.21.15.2:0/542047987 shutdown_connections 2023-11-16T18:25:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.479+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/542047987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f3b60 0x556ada3f3f30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.479+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/542047987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f88b0 0x556ada3f8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.479+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/542047987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3f8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.479+0000 7f6bae95eb80 1 -- 172.21.15.2:0/542047987 >> 172.21.15.2:0/542047987 conn(0x556ada3e9360 msgr2=0x556ada3eeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.479+0000 7f6bae95eb80 1 -- 172.21.15.2:0/542047987 shutdown_connections 2023-11-16T18:25:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.479+0000 7f6bae95eb80 1 -- 172.21.15.2:0/542047987 wait complete. 2023-11-16T18:25:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.480+0000 7f6bae95eb80 1 -- start start 2023-11-16T18:25:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.480+0000 7f6bae95eb80 1 Processor -- start 2023-11-16T18:25:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.480+0000 7f6bae95eb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f3b60 0x556ada4000e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.480+0000 7f6bae95eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3fd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6bae95eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f88b0 0x556ada3fda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6bae95eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ada2e68e0 con 0x556ada3f4470 2023-11-16T18:25:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6bae95eb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ada2e6580 con 0x556ada3f3b60 2023-11-16T18:25:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6bae95eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ada1c8920 con 0x556ada3f88b0 2023-11-16T18:25:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6bae95eb80 10 client.0 _ll_register_callbacks cb 0x556ada304780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6b9b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3fd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6b9b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3fd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50494/0 (socket says 172.21.15.2:50494) 2023-11-16T18:25:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6b9b7fe700 1 -- 172.21.15.2:0/2351168661 learned_addr learned my addr 172.21.15.2:0/2351168661 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/2351168661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f88b0 0x556ada3fda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6b9bfff700 1 --2- 172.21.15.2:0/2351168661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f3b60 0x556ada4000e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/2351168661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f3b60 msgr2=0x556ada4000e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/2351168661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f3b60 0x556ada4000e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/2351168661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 msgr2=0x556ada3fd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/2351168661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3fd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/2351168661 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9c068e00 con 0x556ada3f88b0 2023-11-16T18:25:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/2351168661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f88b0 0x556ada3fda30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b940c9f10 tx=0x7f6b94099d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4289648d8a2cc440 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.481+0000 7f6b7b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:25:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6b7b7fe700 1 -- 172.21.15.2:0/2351168661 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b940b92d0 con 0x556ada3f88b0 2023-11-16T18:25:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6b7b7fe700 1 -- 172.21.15.2:0/2351168661 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6b9400ce60 con 0x556ada3f88b0 2023-11-16T18:25:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2351168661 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556ada3e6b20 con 0x556ada3f88b0 2023-11-16T18:25:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6b7b7fe700 1 -- 172.21.15.2:0/2351168661 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b94099850 con 0x556ada3f88b0 2023-11-16T18:25:13.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6bae95eb80 10 client.25294 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:13.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2351168661 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556ada30b190 con 0x556ada3f88b0 2023-11-16T18:25:13.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6b7b7fe700 1 -- 172.21.15.2:0/2351168661 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556ada3e6b20 con 0x556ada3f88b0 2023-11-16T18:25:13.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6b7b7fe700 1 -- 172.21.15.2:0/2351168661 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6b940ca1f0 con 0x556ada3f88b0 2023-11-16T18:25:13.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6bae95eb80 10 client.25294 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:13.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6bae95eb80 10 client.25294 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2023-11-16T18:25:13.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6bae95eb80 20 client.25294 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:13.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2351168661 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x556ada40d9a0 0x556ada40fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6bae95eb80 4 client.25294 mds_command: new command op to 24446 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:25:13.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.482+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2351168661 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556ada3070a0 con 0x556ada40d9a0 2023-11-16T18:25:13.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.483+0000 7f6b9bfff700 1 --2- 172.21.15.2:0/2351168661 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x556ada40d9a0 0x556ada40fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.483+0000 7f6b7b7fe700 10 client.25294 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:25:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.483+0000 7f6b9bfff700 1 --2- 172.21.15.2:0/2351168661 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x556ada40d9a0 0x556ada40fdb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=96025cde923a2d1 server_cookie=8987da3bc5f3e050 in_seq=0 out_seq=0 2023-11-16T18:25:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.484+0000 7f6b7b7fe700 1 -- 172.21.15.2:0/2351168661 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x556ada3070a0 con 0x556ada40d9a0 2023-11-16T18:25:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.484+0000 7f6b7b7fe700 10 client.25294 handle_command_reply: tid=0 2023-11-16T18:25:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.484+0000 7f6bae95eb80 1 client.25294 shutdown 2023-11-16T18:25:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.484+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2351168661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f88b0 msgr2=0x556ada3fda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.484+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2351168661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f88b0 0x556ada3fda30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b940c9f10 tx=0x7f6b94099d90 comp rx=0 tx=0).stop 2023-11-16T18:25:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.485+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2351168661 shutdown_connections 2023-11-16T18:25:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.485+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2351168661 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x556ada40d9a0 0x556ada40fdb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.485+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2351168661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f88b0 0x556ada3fda30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.485+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2351168661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f3b60 0x556ada4000e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.485+0000 7f6b7b7fe700 0 client.25294 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:25:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.485+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2351168661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3fd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.485+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2351168661 >> 172.21.15.2:0/2351168661 conn(0x556ada3e9360 msgr2=0x556ada3f9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:13.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.485+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2351168661 shutdown_connections 2023-11-16T18:25:13.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.485+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2351168661 wait complete. 2023-11-16T18:25:13.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.485+0000 7f6bae95eb80 20 client.25294 trim_cache size 0 max 16384 2023-11-16T18:25:13.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.510+0000 7f6bae95eb80 1 Processor -- start 2023-11-16T18:25:13.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.510+0000 7f6bae95eb80 1 -- start start 2023-11-16T18:25:13.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.510+0000 7f6bae95eb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f3b60 0x556ada3ffc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.510+0000 7f6bae95eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3ee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.510+0000 7f6bae95eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f88b0 0x556ada3eeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.510+0000 7f6bae95eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ada3e9170 con 0x556ada3f4470 2023-11-16T18:25:13.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.510+0000 7f6bae95eb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ada3fb850 con 0x556ada3f3b60 2023-11-16T18:25:13.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.510+0000 7f6bae95eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ada3f74c0 con 0x556ada3f88b0 2023-11-16T18:25:13.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.510+0000 7f6b9b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.511+0000 7f6b9b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3ee560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50498/0 (socket says 172.21.15.2:50498) 2023-11-16T18:25:13.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.511+0000 7f6b9bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f3b60 0x556ada3ffc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.511+0000 7f6b9b7fe700 1 -- 172.21.15.2:0/2782473141 learned_addr learned my addr 172.21.15.2:0/2782473141 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:13.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.511+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/2782473141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f88b0 0x556ada3eeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.511+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/2782473141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f3b60 msgr2=0x556ada3ffc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.511+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/2782473141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f3b60 0x556ada3ffc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.511+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/2782473141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 msgr2=0x556ada3ee560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.511+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/2782473141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3ee560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.511+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/2782473141 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ada3f8f50 con 0x556ada3f88b0 2023-11-16T18:25:13.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.511+0000 7f6b9b7fe700 1 --2- 172.21.15.2:0/2782473141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3ee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:13.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.511+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/2782473141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f88b0 0x556ada3eeaa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6b940ca690 tx=0x7f6b94009ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16e7614585a907bc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.511+0000 7f6b82ffd700 1 -- 172.21.15.2:0/2782473141 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b940c5e90 con 0x556ada3f88b0 2023-11-16T18:25:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.512+0000 7f6b82ffd700 1 -- 172.21.15.2:0/2782473141 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b9400ce60 con 0x556ada3f88b0 2023-11-16T18:25:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.512+0000 7f6b82ffd700 1 -- 172.21.15.2:0/2782473141 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b940a1c10 con 0x556ada3f88b0 2023-11-16T18:25:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.512+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2782473141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f88b0 msgr2=0x556ada3eeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.512+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2782473141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f88b0 0x556ada3eeaa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6b940ca690 tx=0x7f6b94009ed0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.513+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2782473141 shutdown_connections 2023-11-16T18:25:13.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.513+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2782473141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f88b0 0x556ada3eeaa0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.513+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2782473141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f3b60 0x556ada3ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.513+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2782473141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f4470 0x556ada3ee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.513+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2782473141 >> 172.21.15.2:0/2782473141 conn(0x556ada3e9360 msgr2=0x556ada3f6930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:13.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.513+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2782473141 shutdown_connections 2023-11-16T18:25:13.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.513+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2782473141 wait complete. 2023-11-16T18:25:13.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.513+0000 7f6bae95eb80 1 -- start start 2023-11-16T18:25:13.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.513+0000 7f6bae95eb80 1 Processor -- start 2023-11-16T18:25:13.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6bae95eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f3b60 0x556ada4029c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6bae95eb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f4470 0x556ada402f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6bae95eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f88b0 0x556ada403440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6bae95eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ada460900 con 0x556ada3f88b0 2023-11-16T18:25:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6bae95eb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ada460a40 con 0x556ada3f4470 2023-11-16T18:25:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6bae95eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ada460bb0 con 0x556ada3f3b60 2023-11-16T18:25:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6bae95eb80 10 client.0 _ll_register_callbacks cb 0x7f6b940991b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6ba0d6f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f88b0 0x556ada403440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6ba0d6f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f88b0 0x556ada403440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50506/0 (socket says 172.21.15.2:50506) 2023-11-16T18:25:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/2526426107 learned_addr learned my addr 172.21.15.2:0/2526426107 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6b9bfff700 1 --2- 172.21.15.2:0/2526426107 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f3b60 0x556ada4029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6b9b7fe700 1 --2- 172.21.15.2:0/2526426107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f4470 0x556ada402f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/2526426107 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f3b60 msgr2=0x556ada4029c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/2526426107 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f3b60 0x556ada4029c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/2526426107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f4470 msgr2=0x556ada402f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/2526426107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f4470 0x556ada402f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.514+0000 7f6ba0d6f700 1 -- 172.21.15.2:0/2526426107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b90012450 con 0x556ada3f88b0 2023-11-16T18:25:13.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.515+0000 7f6ba0d6f700 1 --2- 172.21.15.2:0/2526426107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f88b0 0x556ada403440 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400c6f0 tx=0x7f6b94004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224171f4e43bb49c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:13.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.515+0000 7f6b7b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:13.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.515+0000 7f6b7b7fe700 1 -- 172.21.15.2:0/2526426107 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9409dce0 con 0x556ada3f88b0 2023-11-16T18:25:13.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.515+0000 7f6b7b7fe700 1 -- 172.21.15.2:0/2526426107 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6b9409de40 con 0x556ada3f88b0 2023-11-16T18:25:13.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.515+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2526426107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b8c01e040 con 0x556ada3f88b0 2023-11-16T18:25:13.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.515+0000 7f6b7b7fe700 1 -- 172.21.15.2:0/2526426107 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b940b92e0 con 0x556ada3f88b0 2023-11-16T18:25:13.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.515+0000 7f6bae95eb80 10 client.15732 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:13.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.515+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2526426107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556ada30b190 con 0x556ada3f88b0 2023-11-16T18:25:13.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.515+0000 7f6b7b7fe700 1 -- 172.21.15.2:0/2526426107 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b8c01e040 con 0x556ada3f88b0 2023-11-16T18:25:13.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.516+0000 7f6b7b7fe700 1 -- 172.21.15.2:0/2526426107 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6b940987b0 con 0x556ada3f88b0 2023-11-16T18:25:13.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.516+0000 7f6bae95eb80 10 client.15732 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:13.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.516+0000 7f6bae95eb80 10 client.15732 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2023-11-16T18:25:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.516+0000 7f6bae95eb80 20 client.15732 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.516+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2526426107 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x556ada461de0 0x556ada4641c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.516+0000 7f6bae95eb80 4 client.15732 mds_command: new command op to 24446 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:25:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.516+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2526426107 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556ada313660 con 0x556ada461de0 2023-11-16T18:25:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.516+0000 7f6b9bfff700 1 --2- 172.21.15.2:0/2526426107 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x556ada461de0 0x556ada4641c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6b7b7fe700 10 client.15732 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:25:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6b9bfff700 1 --2- 172.21.15.2:0/2526426107 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x556ada461de0 0x556ada4641c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a37cb7eb142d1fd1 server_cookie=e5be91016d4d6cbb in_seq=0 out_seq=0 2023-11-16T18:25:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6b7b7fe700 1 -- 172.21.15.2:0/2526426107 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556ada313660 con 0x556ada461de0 2023-11-16T18:25:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6b7b7fe700 10 client.15732 handle_command_reply: tid=0 2023-11-16T18:25:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6bae95eb80 1 client.15732 shutdown 2023-11-16T18:25:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2526426107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f88b0 msgr2=0x556ada403440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2526426107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f88b0 0x556ada403440 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400c6f0 tx=0x7f6b94004df0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2526426107 shutdown_connections 2023-11-16T18:25:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2526426107 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x556ada461de0 0x556ada4641c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2526426107 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556ada3f3b60 0x556ada4029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2526426107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556ada3f88b0 0x556ada403440 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6bae95eb80 1 --2- 172.21.15.2:0/2526426107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ada3f4470 0x556ada402f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2526426107 >> 172.21.15.2:0/2526426107 conn(0x556ada3e9360 msgr2=0x556ada3f78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6b7b7fe700 0 client.15732 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:25:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2526426107 shutdown_connections 2023-11-16T18:25:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6bae95eb80 1 -- 172.21.15.2:0/2526426107 wait complete. 2023-11-16T18:25:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.517+0000 7f6bae95eb80 20 client.15732 trim_cache size 0 max 16384 2023-11-16T18:25:13.525 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:25:13.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.518+0000 7f6ba27d2700 1 -- 172.21.15.2:0/3261617272 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b840a1460 msgr2=0x7f6b840a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.518+0000 7f6ba27d2700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b840a1460 0x7f6b840a3910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009990 tx=0x7f6b8c005fb0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.518+0000 7f6ba27d2700 1 -- 172.21.15.2:0/3261617272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b9c104470 msgr2=0x7f6b9c1b1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.518+0000 7f6ba27d2700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b9c104470 0x7f6b9c1b1060 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400ef30 tx=0x7f6b9400c3a0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.518+0000 7f6ba27d2700 1 -- 172.21.15.2:0/3261617272 shutdown_connections 2023-11-16T18:25:13.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.518+0000 7f6ba27d2700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b840a1460 0x7f6b840a3910 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.518+0000 7f6ba27d2700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b9c104470 0x7f6b9c1b1060 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.518+0000 7f6ba27d2700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c0ff430 0x7f6b9c10afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.526 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.518+0000 7f6ba27d2700 1 --2- 172.21.15.2:0/3261617272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b9c0ffdd0 0x7f6b9c10b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.518+0000 7f6ba27d2700 1 -- 172.21.15.2:0/3261617272 >> 172.21.15.2:0/3261617272 conn(0x7f6b9c0763a0 msgr2=0x7f6b9c100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:13.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.518+0000 7f6ba27d2700 1 -- 172.21.15.2:0/3261617272 shutdown_connections 2023-11-16T18:25:13.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.518+0000 7f6ba27d2700 1 -- 172.21.15.2:0/3261617272 wait complete. 2023-11-16T18:25:13.537 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-11-16T18:25:13.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.732+0000 7fd11b6b0700 1 -- 172.21.15.2:0/2276259369 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1140fe910 msgr2=0x7fd1140fed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.732+0000 7fd11b6b0700 1 --2- 172.21.15.2:0/2276259369 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1140fe910 0x7fd1140fed80 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd104009990 tx=0x7fd104009ca0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.732+0000 7fd11b6b0700 1 -- 172.21.15.2:0/2276259369 shutdown_connections 2023-11-16T18:25:13.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.732+0000 7fd11b6b0700 1 --2- 172.21.15.2:0/2276259369 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd114104920 0x7fd114104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.732+0000 7fd11b6b0700 1 --2- 172.21.15.2:0/2276259369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1140ff2c0 0x7fd11410b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.732+0000 7fd11b6b0700 1 --2- 172.21.15.2:0/2276259369 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1140fe910 0x7fd1140fed80 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.732+0000 7fd11b6b0700 1 -- 172.21.15.2:0/2276259369 >> 172.21.15.2:0/2276259369 conn(0x7fd114075880 msgr2=0x7fd114075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:13.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.732+0000 7fd11b6b0700 1 -- 172.21.15.2:0/2276259369 shutdown_connections 2023-11-16T18:25:13.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.732+0000 7fd11b6b0700 1 -- 172.21.15.2:0/2276259369 wait complete. 2023-11-16T18:25:13.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.733+0000 7fd11b6b0700 1 Processor -- start 2023-11-16T18:25:13.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.733+0000 7fd11b6b0700 1 -- start start 2023-11-16T18:25:13.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.733+0000 7fd11b6b0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1140fe910 0x7fd11410ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.733+0000 7fd11b6b0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1140ff2c0 0x7fd11410d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.733+0000 7fd11b6b0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd114104920 0x7fd1141a4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.733+0000 7fd11b6b0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11410dbe0 con 0x7fd1140ff2c0 2023-11-16T18:25:13.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.733+0000 7fd11b6b0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1141a4a70 con 0x7fd114104920 2023-11-16T18:25:13.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.733+0000 7fd11b6b0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1141a4bb0 con 0x7fd1140fe910 2023-11-16T18:25:13.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.733+0000 7fd118c4b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1140ff2c0 0x7fd11410d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.733+0000 7fd118c4b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1140ff2c0 0x7fd11410d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50520/0 (socket says 172.21.15.2:50520) 2023-11-16T18:25:13.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.733+0000 7fd118c4b700 1 -- 172.21.15.2:0/717761024 learned_addr learned my addr 172.21.15.2:0/717761024 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:13.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.733+0000 7fd119c4d700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd114104920 0x7fd1141a4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.733+0000 7fd11944c700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1140fe910 0x7fd11410ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.734+0000 7fd11944c700 1 -- 172.21.15.2:0/717761024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd114104920 msgr2=0x7fd1141a4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.734+0000 7fd11944c700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd114104920 0x7fd1141a4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.734+0000 7fd11944c700 1 -- 172.21.15.2:0/717761024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1140ff2c0 msgr2=0x7fd11410d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.734+0000 7fd11944c700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1140ff2c0 0x7fd11410d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.734+0000 7fd11944c700 1 -- 172.21.15.2:0/717761024 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1040095f0 con 0x7fd1140fe910 2023-11-16T18:25:13.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.734+0000 7fd118c4b700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1140ff2c0 0x7fd11410d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:13.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.734+0000 7fd119c4d700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd114104920 0x7fd1141a4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:13.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.734+0000 7fd11944c700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1140fe910 0x7fd11410ce70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd11000e9c0 tx=0x7fd11000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:13.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.734+0000 7fd10a7fc700 1 -- 172.21.15.2:0/717761024 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd110004220 con 0x7fd1140fe910 2023-11-16T18:25:13.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.734+0000 7fd11b6b0700 1 -- 172.21.15.2:0/717761024 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd114110360 con 0x7fd1140fe910 2023-11-16T18:25:13.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.734+0000 7fd10a7fc700 1 -- 172.21.15.2:0/717761024 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd110004380 con 0x7fd1140fe910 2023-11-16T18:25:13.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.734+0000 7fd10a7fc700 1 -- 172.21.15.2:0/717761024 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd110003be0 con 0x7fd1140fe910 2023-11-16T18:25:13.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.734+0000 7fd11b6b0700 1 -- 172.21.15.2:0/717761024 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1141101d0 con 0x7fd1140fe910 2023-11-16T18:25:13.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.738+0000 7fd10a7fc700 1 -- 172.21.15.2:0/717761024 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7fd110003d40 con 0x7fd1140fe910 2023-11-16T18:25:13.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.739+0000 7fd10a7fc700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd1000a1460 0x7fd1000a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.739+0000 7fd10a7fc700 1 -- 172.21.15.2:0/717761024 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd110014070 con 0x7fd1140fe910 2023-11-16T18:25:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.739+0000 7fd118c4b700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd1000a1460 0x7fd1000a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.739+0000 7fd12783cb80 1 Processor -- start 2023-11-16T18:25:13.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd12783cb80 1 -- start start 2023-11-16T18:25:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd118c4b700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd1000a1460 0x7fd1000a3910 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd11410e660 tx=0x7fd104005960 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd12783cb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab05536740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd12783cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536c80 0x55ab0553ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd12783cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab0553b0c0 0x55ab0553b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd12783cb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab05433060 con 0x55ab05536370 2023-11-16T18:25:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd12783cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab054858a0 con 0x55ab0553b0c0 2023-11-16T18:25:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd12783cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0547ed30 con 0x55ab05536c80 2023-11-16T18:25:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd11944c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab05536740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd11944c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab05536740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50524/0 (socket says 172.21.15.2:50524) 2023-11-16T18:25:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd11944c700 1 -- 172.21.15.2:0/1881971121 learned_addr learned my addr 172.21.15.2:0/1881971121 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd118c4b700 1 --2- 172.21.15.2:0/1881971121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536c80 0x55ab0553ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd11944c700 1 -- 172.21.15.2:0/1881971121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536c80 msgr2=0x55ab0553ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd119c4d700 1 --2- 172.21.15.2:0/1881971121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab0553b0c0 0x55ab0553b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd11944c700 1 --2- 172.21.15.2:0/1881971121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536c80 0x55ab0553ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd11944c700 1 -- 172.21.15.2:0/1881971121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab0553b0c0 msgr2=0x55ab0553b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd11944c700 1 --2- 172.21.15.2:0/1881971121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab0553b0c0 0x55ab0553b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.740+0000 7fd11944c700 1 -- 172.21.15.2:0/1881971121 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab05300630 con 0x55ab05536370 2023-11-16T18:25:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.741+0000 7fd11944c700 1 --2- 172.21.15.2:0/1881971121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab05536740 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd110003ff0 tx=0x7fd1100999c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2713e446f990fce8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.745+0000 7fd0ff7fe700 1 -- 172.21.15.2:0/1881971121 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd110099fe0 con 0x55ab05536370 2023-11-16T18:25:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.745+0000 7fd0ff7fe700 1 -- 172.21.15.2:0/1881971121 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd11009e010 con 0x55ab05536370 2023-11-16T18:25:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.745+0000 7fd0ff7fe700 1 -- 172.21.15.2:0/1881971121 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd110099730 con 0x55ab05536370 2023-11-16T18:25:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.746+0000 7fd12783cb80 1 -- 172.21.15.2:0/1881971121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 msgr2=0x55ab05536740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.746+0000 7fd12783cb80 1 --2- 172.21.15.2:0/1881971121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab05536740 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd110003ff0 tx=0x7fd1100999c0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.746+0000 7fd12783cb80 1 -- 172.21.15.2:0/1881971121 shutdown_connections 2023-11-16T18:25:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.746+0000 7fd12783cb80 1 --2- 172.21.15.2:0/1881971121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab0553b0c0 0x55ab0553b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.746+0000 7fd12783cb80 1 --2- 172.21.15.2:0/1881971121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab05536740 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.746+0000 7fd12783cb80 1 --2- 172.21.15.2:0/1881971121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536c80 0x55ab0553ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.746+0000 7fd12783cb80 1 -- 172.21.15.2:0/1881971121 >> 172.21.15.2:0/1881971121 conn(0x55ab05523360 msgr2=0x55ab05528b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.746+0000 7fd12783cb80 1 -- 172.21.15.2:0/1881971121 shutdown_connections 2023-11-16T18:25:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.746+0000 7fd12783cb80 1 -- 172.21.15.2:0/1881971121 wait complete. 2023-11-16T18:25:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.746+0000 7fd12783cb80 1 -- start start 2023-11-16T18:25:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.746+0000 7fd12783cb80 1 Processor -- start 2023-11-16T18:25:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.747+0000 7fd12783cb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab055428d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.747+0000 7fd12783cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536c80 0x55ab0553fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.747+0000 7fd12783cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab0553b0c0 0x55ab05540290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.747+0000 7fd12783cb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab054208e0 con 0x55ab05536370 2023-11-16T18:25:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.747+0000 7fd12783cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab05420580 con 0x55ab0553b0c0 2023-11-16T18:25:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.747+0000 7fd12783cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab05302920 con 0x55ab05536c80 2023-11-16T18:25:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.747+0000 7fd12783cb80 10 client.0 _ll_register_callbacks cb 0x55ab0543e780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.747+0000 7fd11944c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab055428d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.747+0000 7fd11944c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab055428d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50530/0 (socket says 172.21.15.2:50530) 2023-11-16T18:25:13.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.747+0000 7fd11944c700 1 -- 172.21.15.2:0/4207175534 learned_addr learned my addr 172.21.15.2:0/4207175534 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:13.755 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.747+0000 7fd11944c700 1 -- 172.21.15.2:0/4207175534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536c80 msgr2=0x55ab0553fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.755 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.747+0000 7fd119c4d700 1 --2- 172.21.15.2:0/4207175534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab0553b0c0 0x55ab05540290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.755 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.747+0000 7fd11944c700 1 --2- 172.21.15.2:0/4207175534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536c80 0x55ab0553fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.755 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.748+0000 7fd11944c700 1 -- 172.21.15.2:0/4207175534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab0553b0c0 msgr2=0x55ab05540290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.748+0000 7fd11944c700 1 --2- 172.21.15.2:0/4207175534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab0553b0c0 0x55ab05540290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.748+0000 7fd11944c700 1 -- 172.21.15.2:0/4207175534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1141101d0 con 0x55ab05536370 2023-11-16T18:25:13.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.748+0000 7fd11944c700 1 --2- 172.21.15.2:0/4207175534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab055428d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd11000f010 tx=0x7fd1100997b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b2516c5939df13 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:13.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.748+0000 7fd0fcff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:13.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.748+0000 7fd0fcff9700 1 -- 172.21.15.2:0/4207175534 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11009a450 con 0x55ab05536370 2023-11-16T18:25:13.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.748+0000 7fd0fcff9700 1 -- 172.21.15.2:0/4207175534 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd1100c6ce0 con 0x55ab05536370 2023-11-16T18:25:13.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.748+0000 7fd12783cb80 1 -- 172.21.15.2:0/4207175534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ab05520b20 con 0x55ab05536370 2023-11-16T18:25:13.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.748+0000 7fd0fcff9700 1 -- 172.21.15.2:0/4207175534 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1100c6ea0 con 0x55ab05536370 2023-11-16T18:25:13.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.748+0000 7fd12783cb80 10 client.15744 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:13.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.748+0000 7fd12783cb80 1 -- 172.21.15.2:0/4207175534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55ab05445190 con 0x55ab05536370 2023-11-16T18:25:13.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.748+0000 7fd0fcff9700 1 -- 172.21.15.2:0/4207175534 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ab05520b20 con 0x55ab05536370 2023-11-16T18:25:13.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.750+0000 7fd0fcff9700 1 -- 172.21.15.2:0/4207175534 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd1100caa10 con 0x55ab05536370 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.750+0000 7fd12783cb80 10 client.15744 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.750+0000 7fd12783cb80 10 client.15744 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.750+0000 7fd12783cb80 20 client.15744 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.750+0000 7fd12783cb80 1 --2- 172.21.15.2:0/4207175534 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55ab05560d60 0x55ab05563140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.750+0000 7fd12783cb80 4 client.15744 mds_command: new command op to 24460 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.750+0000 7fd12783cb80 1 -- 172.21.15.2:0/4207175534 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ab054410a0 con 0x55ab05560d60 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.750+0000 7fd118c4b700 1 --2- 172.21.15.2:0/4207175534 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55ab05560d60 0x55ab05563140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.751+0000 7fd0fcff9700 10 client.15744 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.751+0000 7fd118c4b700 1 --2- 172.21.15.2:0/4207175534 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55ab05560d60 0x55ab05563140 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc86d2241c84097 server_cookie=b7ad055621c1981a in_seq=0 out_seq=0 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.751+0000 7fd0fcff9700 1 -- 172.21.15.2:0/4207175534 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ab054410a0 con 0x55ab05560d60 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.751+0000 7fd0fcff9700 10 client.15744 handle_command_reply: tid=0 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.751+0000 7fd12783cb80 1 client.15744 shutdown 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.752+0000 7fd12783cb80 1 -- 172.21.15.2:0/4207175534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 msgr2=0x55ab055428d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.752+0000 7fd12783cb80 1 --2- 172.21.15.2:0/4207175534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab055428d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd11000f010 tx=0x7fd1100997b0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.752+0000 7fd12783cb80 1 -- 172.21.15.2:0/4207175534 shutdown_connections 2023-11-16T18:25:13.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.752+0000 7fd12783cb80 1 --2- 172.21.15.2:0/4207175534 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55ab05560d60 0x55ab05563140 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.752+0000 7fd12783cb80 1 --2- 172.21.15.2:0/4207175534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab0553b0c0 0x55ab05540290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.752+0000 7fd12783cb80 1 --2- 172.21.15.2:0/4207175534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab055428d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.752+0000 7fd12783cb80 1 --2- 172.21.15.2:0/4207175534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536c80 0x55ab0553fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.752+0000 7fd12783cb80 1 -- 172.21.15.2:0/4207175534 >> 172.21.15.2:0/4207175534 conn(0x55ab05523360 msgr2=0x55ab0553c570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:13.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.752+0000 7fd0fcff9700 0 client.15744 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:25:13.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.752+0000 7fd12783cb80 1 -- 172.21.15.2:0/4207175534 shutdown_connections 2023-11-16T18:25:13.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.752+0000 7fd12783cb80 1 -- 172.21.15.2:0/4207175534 wait complete. 2023-11-16T18:25:13.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.752+0000 7fd12783cb80 20 client.15744 trim_cache size 0 max 16384 2023-11-16T18:25:13.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.777+0000 7fd12783cb80 1 Processor -- start 2023-11-16T18:25:13.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.777+0000 7fd12783cb80 1 -- start start 2023-11-16T18:25:13.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd12783cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536370 0x55ab055581c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd12783cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab05536c80 0x55ab05527550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd12783cb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab0553b0c0 0x55ab05562d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd12783cb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0552a4b0 con 0x55ab0553b0c0 2023-11-16T18:25:13.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd12783cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab05522f30 con 0x55ab05536c80 2023-11-16T18:25:13.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd12783cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab05541180 con 0x55ab05536370 2023-11-16T18:25:13.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd118c4b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab05536c80 0x55ab05527550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd119c4d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab0553b0c0 0x55ab05562d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd119c4d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab0553b0c0 0x55ab05562d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50542/0 (socket says 172.21.15.2:50542) 2023-11-16T18:25:13.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd118c4b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab05536c80 0x55ab05527550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:40978/0 (socket says 172.21.15.2:40978) 2023-11-16T18:25:13.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd119c4d700 1 -- 172.21.15.2:0/633743732 learned_addr learned my addr 172.21.15.2:0/633743732 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:13.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd11944c700 1 --2- 172.21.15.2:0/633743732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536370 0x55ab055581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd118c4b700 1 -- 172.21.15.2:0/633743732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536370 msgr2=0x55ab055581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd118c4b700 1 --2- 172.21.15.2:0/633743732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536370 0x55ab055581c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd118c4b700 1 -- 172.21.15.2:0/633743732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab0553b0c0 msgr2=0x55ab05562d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd118c4b700 1 --2- 172.21.15.2:0/633743732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab0553b0c0 0x55ab05562d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd118c4b700 1 -- 172.21.15.2:0/633743732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab05524b00 con 0x55ab05536c80 2023-11-16T18:25:13.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.778+0000 7fd119c4d700 1 --2- 172.21.15.2:0/633743732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab0553b0c0 0x55ab05562d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:13.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.779+0000 7fd118c4b700 1 --2- 172.21.15.2:0/633743732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab05536c80 0x55ab05527550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd10400b6f0 tx=0x7fd104020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41202b0fc956aa61 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:13.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.779+0000 7fd0fffff700 1 -- 172.21.15.2:0/633743732 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1040054a0 con 0x55ab05536c80 2023-11-16T18:25:13.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.779+0000 7fd0fffff700 1 -- 172.21.15.2:0/633743732 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd10402c040 con 0x55ab05536c80 2023-11-16T18:25:13.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.779+0000 7fd0fffff700 1 -- 172.21.15.2:0/633743732 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd104004c80 con 0x55ab05536c80 2023-11-16T18:25:13.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.780+0000 7fd12783cb80 1 -- 172.21.15.2:0/633743732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab05536c80 msgr2=0x55ab05527550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.780+0000 7fd12783cb80 1 --2- 172.21.15.2:0/633743732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab05536c80 0x55ab05527550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd10400b6f0 tx=0x7fd104020000 comp rx=0 tx=0).stop 2023-11-16T18:25:13.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.780+0000 7fd12783cb80 1 -- 172.21.15.2:0/633743732 shutdown_connections 2023-11-16T18:25:13.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.780+0000 7fd12783cb80 1 --2- 172.21.15.2:0/633743732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab05536370 0x55ab055581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.780+0000 7fd12783cb80 1 --2- 172.21.15.2:0/633743732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab0553b0c0 0x55ab05562d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.780+0000 7fd12783cb80 1 --2- 172.21.15.2:0/633743732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab05536c80 0x55ab05527550 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.780+0000 7fd12783cb80 1 -- 172.21.15.2:0/633743732 >> 172.21.15.2:0/633743732 conn(0x55ab05523360 msgr2=0x55ab0555d500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:13.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.780+0000 7fd12783cb80 1 -- 172.21.15.2:0/633743732 shutdown_connections 2023-11-16T18:25:13.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.780+0000 7fd12783cb80 1 -- 172.21.15.2:0/633743732 wait complete. 2023-11-16T18:25:13.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.780+0000 7fd12783cb80 1 -- start start 2023-11-16T18:25:13.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.781+0000 7fd12783cb80 1 Processor -- start 2023-11-16T18:25:13.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.781+0000 7fd12783cb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab05542340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.781+0000 7fd12783cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab05536c80 0x55ab05542880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.781+0000 7fd12783cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab0553b0c0 0x55ab055afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.781+0000 7fd12783cb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab055b02d0 con 0x55ab05536370 2023-11-16T18:25:13.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.781+0000 7fd12783cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab055b0440 con 0x55ab05536c80 2023-11-16T18:25:13.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.781+0000 7fd12783cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab055b0600 con 0x55ab0553b0c0 2023-11-16T18:25:13.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.781+0000 7fd12783cb80 10 client.0 _ll_register_callbacks cb 0x7fd110099b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:13.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.781+0000 7fd11944c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab05542340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.781+0000 7fd11944c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab05542340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50544/0 (socket says 172.21.15.2:50544) 2023-11-16T18:25:13.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.781+0000 7fd11944c700 1 -- 172.21.15.2:0/586930603 learned_addr learned my addr 172.21.15.2:0/586930603 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:13.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.781+0000 7fd118c4b700 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab05536c80 0x55ab05542880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd119c4d700 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab0553b0c0 0x55ab055afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd11944c700 1 -- 172.21.15.2:0/586930603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab0553b0c0 msgr2=0x55ab055afbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd11944c700 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab0553b0c0 0x55ab055afbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd11944c700 1 -- 172.21.15.2:0/586930603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab05536c80 msgr2=0x55ab05542880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:13.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd11944c700 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab05536c80 0x55ab05542880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd11944c700 1 -- 172.21.15.2:0/586930603 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10c011040 con 0x55ab05536370 2023-11-16T18:25:13.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd119c4d700 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab0553b0c0 0x55ab055afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:25:13.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd11944c700 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab05542340 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd1100964e0 tx=0x7fd1100045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5417a588e88c5175 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:13.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd0fcff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:13.789 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd0fcff9700 1 -- 172.21.15.2:0/586930603 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1100b83d0 con 0x55ab05536370 2023-11-16T18:25:13.789 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd0fcff9700 1 -- 172.21.15.2:0/586930603 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd1100b2720 con 0x55ab05536370 2023-11-16T18:25:13.789 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd12783cb80 1 -- 172.21.15.2:0/586930603 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd11009e010 con 0x55ab05536370 2023-11-16T18:25:13.789 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd0fcff9700 1 -- 172.21.15.2:0/586930603 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1100a1fa0 con 0x55ab05536370 2023-11-16T18:25:13.789 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd12783cb80 10 client.15750 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:13.789 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.782+0000 7fd12783cb80 1 -- 172.21.15.2:0/586930603 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7fd1140679c0 con 0x55ab05536370 2023-11-16T18:25:13.789 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.783+0000 7fd0fcff9700 1 -- 172.21.15.2:0/586930603 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd11009e010 con 0x55ab05536370 2023-11-16T18:25:13.789 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.783+0000 7fd0fcff9700 1 -- 172.21.15.2:0/586930603 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd1100832f0 con 0x55ab05536370 2023-11-16T18:25:13.789 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.783+0000 7fd12783cb80 10 client.15750 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:13.789 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.784+0000 7fd12783cb80 10 client.15750 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-11-16T18:25:13.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.784+0000 7fd12783cb80 20 client.15750 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:13.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.784+0000 7fd12783cb80 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55ab055571d0 0x55ab05532990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:13.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.784+0000 7fd12783cb80 4 client.15750 mds_command: new command op to 24460 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:25:13.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.784+0000 7fd12783cb80 1 -- 172.21.15.2:0/586930603 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ab0544d660 con 0x55ab055571d0 2023-11-16T18:25:13.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.784+0000 7fd118c4b700 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55ab055571d0 0x55ab05532990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:13.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.784+0000 7fd0fcff9700 10 client.15750 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:25:13.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.784+0000 7fd118c4b700 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55ab055571d0 0x55ab05532990 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e6fe1b76cd87601 server_cookie=7fbd2d1d3ac6ee4d in_seq=0 out_seq=0 2023-11-16T18:25:13.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd0fcff9700 1 -- 172.21.15.2:0/586930603 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ab0544d660 con 0x55ab055571d0 2023-11-16T18:25:13.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd0fcff9700 10 client.15750 handle_command_reply: tid=0 2023-11-16T18:25:13.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd12783cb80 1 client.15750 shutdown 2023-11-16T18:25:13.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd12783cb80 1 -- 172.21.15.2:0/586930603 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 msgr2=0x55ab05542340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd12783cb80 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab05542340 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd1100964e0 tx=0x7fd1100045b0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd12783cb80 1 -- 172.21.15.2:0/586930603 shutdown_connections 2023-11-16T18:25:13.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd12783cb80 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55ab055571d0 0x55ab05532990 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd12783cb80 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ab0553b0c0 0x55ab055afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd12783cb80 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ab05536370 0x55ab05542340 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd12783cb80 1 --2- 172.21.15.2:0/586930603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab05536c80 0x55ab05542880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd12783cb80 1 -- 172.21.15.2:0/586930603 >> 172.21.15.2:0/586930603 conn(0x55ab05523360 msgr2=0x55ab05527550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:13.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd0fcff9700 0 client.15750 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:25:13.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd12783cb80 1 -- 172.21.15.2:0/586930603 shutdown_connections 2023-11-16T18:25:13.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd12783cb80 1 -- 172.21.15.2:0/586930603 wait complete. 2023-11-16T18:25:13.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.785+0000 7fd12783cb80 20 client.15750 trim_cache size 0 max 16384 2023-11-16T18:25:13.791 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:25:13.792 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.786+0000 7fd11b6b0700 1 -- 172.21.15.2:0/717761024 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd1000a1460 msgr2=0x7fd1000a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.792 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.786+0000 7fd11b6b0700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd1000a1460 0x7fd1000a3910 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd11410e660 tx=0x7fd104005960 comp rx=0 tx=0).stop 2023-11-16T18:25:13.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.786+0000 7fd11b6b0700 1 -- 172.21.15.2:0/717761024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1140fe910 msgr2=0x7fd11410ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:13.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.786+0000 7fd11b6b0700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1140fe910 0x7fd11410ce70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd11000e9c0 tx=0x7fd11000ecd0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.786+0000 7fd11b6b0700 1 -- 172.21.15.2:0/717761024 shutdown_connections 2023-11-16T18:25:13.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.786+0000 7fd11b6b0700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd1000a1460 0x7fd1000a3910 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.786+0000 7fd11b6b0700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1140fe910 0x7fd11410ce70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.786+0000 7fd11b6b0700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd114104920 0x7fd1141a4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.786+0000 7fd11b6b0700 1 --2- 172.21.15.2:0/717761024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1140ff2c0 0x7fd11410d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:13.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.786+0000 7fd11b6b0700 1 -- 172.21.15.2:0/717761024 >> 172.21.15.2:0/717761024 conn(0x7fd114075880 msgr2=0x7fd1140fcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:13.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.786+0000 7fd11b6b0700 1 -- 172.21.15.2:0/717761024 shutdown_connections 2023-11-16T18:25:13.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.786+0000 7fd11b6b0700 1 -- 172.21.15.2:0/717761024 wait complete. 2023-11-16T18:25:13.804 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T18:25:14.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.998+0000 7f79399c1700 1 -- 172.21.15.2:0/62043533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7934104920 msgr2=0x7f7934104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:14.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.998+0000 7f79399c1700 1 --2- 172.21.15.2:0/62043533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7934104920 0x7f7934104cf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7924009990 tx=0x7f7924009ca0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.998+0000 7f79399c1700 1 -- 172.21.15.2:0/62043533 shutdown_connections 2023-11-16T18:25:14.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.998+0000 7f79399c1700 1 --2- 172.21.15.2:0/62043533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79340ff2c0 0x7f793410b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.998+0000 7f79399c1700 1 --2- 172.21.15.2:0/62043533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7934104920 0x7f7934104cf0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.998+0000 7f79399c1700 1 --2- 172.21.15.2:0/62043533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79340fe910 0x7f79340fed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.998+0000 7f79399c1700 1 -- 172.21.15.2:0/62043533 >> 172.21.15.2:0/62043533 conn(0x7f7934075880 msgr2=0x7f7934075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:14.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.998+0000 7f79399c1700 1 -- 172.21.15.2:0/62043533 shutdown_connections 2023-11-16T18:25:14.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.998+0000 7f79399c1700 1 -- 172.21.15.2:0/62043533 wait complete. 2023-11-16T18:25:14.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.999+0000 7f79399c1700 1 Processor -- start 2023-11-16T18:25:14.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:13.999+0000 7f79399c1700 1 -- start start 2023-11-16T18:25:14.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79399c1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79340fe910 0x7f7934194150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79399c1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79340ff2c0 0x7f7934194690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79399c1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7934104920 0x7f7934198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79399c1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7934199130 con 0x7f79340ff2c0 2023-11-16T18:25:14.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79399c1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79341992a0 con 0x7f79340fe910 2023-11-16T18:25:14.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79399c1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7934199460 con 0x7f7934104920 2023-11-16T18:25:14.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f7932ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79340fe910 0x7f7934194150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f7932ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79340fe910 0x7f7934194150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:40996/0 (socket says 172.21.15.2:40996) 2023-11-16T18:25:14.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79327fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79340ff2c0 0x7f7934194690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f7932ffd700 1 -- 172.21.15.2:0/4038977846 learned_addr learned my addr 172.21.15.2:0/4038977846 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:14.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79337fe700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7934104920 0x7f7934198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79327fc700 1 -- 172.21.15.2:0/4038977846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7934104920 msgr2=0x7f7934198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:14.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79327fc700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7934104920 0x7f7934198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79327fc700 1 -- 172.21.15.2:0/4038977846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79340fe910 msgr2=0x7f7934194150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:14.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79327fc700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79340fe910 0x7f7934194150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79327fc700 1 -- 172.21.15.2:0/4038977846 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79240095f0 con 0x7f79340ff2c0 2023-11-16T18:25:14.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f7932ffd700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79340fe910 0x7f7934194150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:14.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.000+0000 7f79337fe700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7934104920 0x7f7934198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:25:14.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.001+0000 7f79327fc700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79340ff2c0 0x7f7934194690 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f792800ea90 tx=0x7f792800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:14.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.001+0000 7f79389bf700 1 -- 172.21.15.2:0/4038977846 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792800ccd0 con 0x7f79340ff2c0 2023-11-16T18:25:14.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.001+0000 7f79399c1700 1 -- 172.21.15.2:0/4038977846 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f793410e0e0 con 0x7f79340ff2c0 2023-11-16T18:25:14.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.001+0000 7f79389bf700 1 -- 172.21.15.2:0/4038977846 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f792800ce30 con 0x7f79340ff2c0 2023-11-16T18:25:14.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.001+0000 7f79389bf700 1 -- 172.21.15.2:0/4038977846 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7928005280 con 0x7f79340ff2c0 2023-11-16T18:25:14.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.001+0000 7f79399c1700 1 -- 172.21.15.2:0/4038977846 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f793410df50 con 0x7f79340ff2c0 2023-11-16T18:25:14.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.005+0000 7f79389bf700 1 -- 172.21.15.2:0/4038977846 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f7928004750 con 0x7f79340ff2c0 2023-11-16T18:25:14.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.006+0000 7f79389bf700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f791c0a13e0 0x7f791c0a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.006+0000 7f7945b4db80 1 Processor -- start 2023-11-16T18:25:14.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.006+0000 7f79389bf700 1 -- 172.21.15.2:0/4038977846 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f79280cce70 con 0x7f79340ff2c0 2023-11-16T18:25:14.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.006+0000 7f7945b4db80 1 -- start start 2023-11-16T18:25:14.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.006+0000 7f7932ffd700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f791c0a13e0 0x7f791c0a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.006+0000 7f7945b4db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacbdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.006+0000 7f7945b4db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbe470 0x5629eacc2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.006+0000 7f7945b4db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 0x5629eacc2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.006+0000 7f7945b4db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629eabc3060 con 0x5629eacbe470 2023-11-16T18:25:14.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.006+0000 7f7945b4db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629eac158a0 con 0x5629eacc28b0 2023-11-16T18:25:14.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.006+0000 7f7945b4db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629eac0ed30 con 0x5629eacbdb60 2023-11-16T18:25:14.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.007+0000 7f79327fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbe470 0x5629eacc2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.007+0000 7f79327fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbe470 0x5629eacc2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50566/0 (socket says 172.21.15.2:50566) 2023-11-16T18:25:14.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.007+0000 7f79327fc700 1 -- 172.21.15.2:0/2559252517 learned_addr learned my addr 172.21.15.2:0/2559252517 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:14.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.007+0000 7f79337fe700 1 --2- 172.21.15.2:0/2559252517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 0x5629eacc2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.007+0000 7f7932ffd700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f791c0a13e0 0x7f791c0a3890 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f792400b2e0 tx=0x7f7924005fd0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:14.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.007+0000 7f7932ffd700 1 --2- 172.21.15.2:0/2559252517 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacbdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.007+0000 7f79327fc700 1 -- 172.21.15.2:0/2559252517 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 msgr2=0x5629eacbdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:14.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.007+0000 7f79327fc700 1 --2- 172.21.15.2:0/2559252517 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacbdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.007+0000 7f79327fc700 1 -- 172.21.15.2:0/2559252517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 msgr2=0x5629eacc2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:14.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.007+0000 7f79327fc700 1 --2- 172.21.15.2:0/2559252517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 0x5629eacc2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.007+0000 7f79327fc700 1 -- 172.21.15.2:0/2559252517 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629eaa90630 con 0x5629eacbe470 2023-11-16T18:25:14.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.007+0000 7f79327fc700 1 --2- 172.21.15.2:0/2559252517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbe470 0x5629eacc2370 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f792800c6b0 tx=0x7f79280992e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b29d68b6320a8a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:14.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.007+0000 7f7919ffb700 1 -- 172.21.15.2:0/2559252517 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792809d9b0 con 0x5629eacbe470 2023-11-16T18:25:14.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.008+0000 7f7919ffb700 1 -- 172.21.15.2:0/2559252517 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79280a1dd0 con 0x5629eacbe470 2023-11-16T18:25:14.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.008+0000 7f7919ffb700 1 -- 172.21.15.2:0/2559252517 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79280ca3b0 con 0x5629eacbe470 2023-11-16T18:25:14.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.008+0000 7f7945b4db80 1 -- 172.21.15.2:0/2559252517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbe470 msgr2=0x5629eacc2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:14.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.008+0000 7f7945b4db80 1 --2- 172.21.15.2:0/2559252517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbe470 0x5629eacc2370 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f792800c6b0 tx=0x7f79280992e0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.009+0000 7f7945b4db80 1 -- 172.21.15.2:0/2559252517 shutdown_connections 2023-11-16T18:25:14.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.009+0000 7f7945b4db80 1 --2- 172.21.15.2:0/2559252517 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacbdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.009+0000 7f7945b4db80 1 --2- 172.21.15.2:0/2559252517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 0x5629eacc2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.009+0000 7f7945b4db80 1 --2- 172.21.15.2:0/2559252517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbe470 0x5629eacc2370 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.009+0000 7f7945b4db80 1 -- 172.21.15.2:0/2559252517 >> 172.21.15.2:0/2559252517 conn(0x5629eacb3360 msgr2=0x5629eacb8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:14.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.009+0000 7f7945b4db80 1 -- 172.21.15.2:0/2559252517 shutdown_connections 2023-11-16T18:25:14.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.009+0000 7f7945b4db80 1 -- 172.21.15.2:0/2559252517 wait complete. 2023-11-16T18:25:14.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.009+0000 7f7945b4db80 1 -- start start 2023-11-16T18:25:14.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.009+0000 7f7945b4db80 1 Processor -- start 2023-11-16T18:25:14.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.010+0000 7f7945b4db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.010+0000 7f7945b4db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacbe470 0x5629eacc72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.010+0000 7f7945b4db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacc28b0 0x5629eacc7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.010+0000 7f7945b4db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629eabb08e0 con 0x5629eacc28b0 2023-11-16T18:25:14.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.010+0000 7f7945b4db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629eabb0580 con 0x5629eacbe470 2023-11-16T18:25:14.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.010+0000 7f7945b4db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629eaa92920 con 0x5629eacbdb60 2023-11-16T18:25:14.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.010+0000 7f7945b4db80 10 client.0 _ll_register_callbacks cb 0x5629eabce780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:14.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.010+0000 7f79337fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacc28b0 0x5629eacc7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.010+0000 7f79337fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacc28b0 0x5629eacc7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50576/0 (socket says 172.21.15.2:50576) 2023-11-16T18:25:14.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.010+0000 7f79337fe700 1 -- 172.21.15.2:0/1281080518 learned_addr learned my addr 172.21.15.2:0/1281080518 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:14.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.010+0000 7f7932ffd700 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.010+0000 7f79327fc700 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacbe470 0x5629eacc72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.011+0000 7f7932ffd700 1 -- 172.21.15.2:0/1281080518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacbe470 msgr2=0x5629eacc72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:14.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.011+0000 7f7932ffd700 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacbe470 0x5629eacc72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.011+0000 7f7932ffd700 1 -- 172.21.15.2:0/1281080518 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacc28b0 msgr2=0x5629eacc7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:14.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.011+0000 7f7932ffd700 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacc28b0 0x5629eacc7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.011+0000 7f7932ffd700 1 -- 172.21.15.2:0/1281080518 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f793410df50 con 0x5629eacbdb60 2023-11-16T18:25:14.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.011+0000 7f79337fe700 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacc28b0 0x5629eacc7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.011+0000 7f7932ffd700 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacca0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7924000c00 tx=0x7f792401a050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d4ea22ab5b74125 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.011+0000 7f7912ffd700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:25:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.011+0000 7f7912ffd700 1 -- 172.21.15.2:0/1281080518 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7924024c30 con 0x5629eacbdb60 2023-11-16T18:25:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.011+0000 7f7912ffd700 1 -- 172.21.15.2:0/1281080518 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f792400fe70 con 0x5629eacbdb60 2023-11-16T18:25:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.011+0000 7f7945b4db80 1 -- 172.21.15.2:0/1281080518 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5629eacb0b20 con 0x5629eacbdb60 2023-11-16T18:25:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.012+0000 7f7912ffd700 1 -- 172.21.15.2:0/1281080518 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7924024d90 con 0x5629eacbdb60 2023-11-16T18:25:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.012+0000 7f7945b4db80 10 client.25321 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.012+0000 7f7945b4db80 1 -- 172.21.15.2:0/1281080518 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5629eabd5190 con 0x5629eacbdb60 2023-11-16T18:25:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.012+0000 7f7912ffd700 1 -- 172.21.15.2:0/1281080518 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5629eacb0b20 con 0x5629eacbdb60 2023-11-16T18:25:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.012+0000 7f7912ffd700 1 -- 172.21.15.2:0/1281080518 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7924037070 con 0x5629eacbdb60 2023-11-16T18:25:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.012+0000 7f7945b4db80 10 client.25321 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.012+0000 7f7945b4db80 10 client.25321 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.012+0000 7f7945b4db80 20 client.25321 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.012+0000 7f7945b4db80 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5629eacdfec0 0x5629eace22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.012+0000 7f7945b4db80 4 client.25321 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:25:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.012+0000 7f7945b4db80 1 -- 172.21.15.2:0/1281080518 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5629eabd10a0 con 0x5629eacdfec0 2023-11-16T18:25:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.012+0000 7f79327fc700 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5629eacdfec0 0x5629eace22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.013+0000 7f79327fc700 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5629eacdfec0 0x5629eace22d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e71306b307f147e7 server_cookie=c2603db5ef690ed5 in_seq=0 out_seq=0 2023-11-16T18:25:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.013+0000 7f7912ffd700 10 client.25321 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7912ffd700 1 -- 172.21.15.2:0/1281080518 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5629eabd10a0 con 0x5629eacdfec0 2023-11-16T18:25:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7912ffd700 10 client.25321 handle_command_reply: tid=0 2023-11-16T18:25:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7945b4db80 1 client.25321 shutdown 2023-11-16T18:25:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7945b4db80 1 -- 172.21.15.2:0/1281080518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 msgr2=0x5629eacca0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7945b4db80 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacca0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7924000c00 tx=0x7f792401a050 comp rx=0 tx=0).stop 2023-11-16T18:25:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7945b4db80 1 -- 172.21.15.2:0/1281080518 shutdown_connections 2023-11-16T18:25:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7945b4db80 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5629eacdfec0 0x5629eace22d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7945b4db80 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacca0e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7945b4db80 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacc28b0 0x5629eacc7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7945b4db80 1 --2- 172.21.15.2:0/1281080518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacbe470 0x5629eacc72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7912ffd700 0 client.25321 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:14.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7945b4db80 1 -- 172.21.15.2:0/1281080518 >> 172.21.15.2:0/1281080518 conn(0x5629eacb3360 msgr2=0x5629eacc3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:14.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7945b4db80 1 -- 172.21.15.2:0/1281080518 shutdown_connections 2023-11-16T18:25:14.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7945b4db80 1 -- 172.21.15.2:0/1281080518 wait complete. 2023-11-16T18:25:14.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.014+0000 7f7945b4db80 20 client.25321 trim_cache size 0 max 16384 2023-11-16T18:25:14.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7945b4db80 1 Processor -- start 2023-11-16T18:25:14.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7945b4db80 1 -- start start 2023-11-16T18:25:14.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7945b4db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbdb60 0x5629eacdc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7945b4db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbe470 0x5629eacc9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7945b4db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 0x5629eacc6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7945b4db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629eacb0a10 con 0x5629eacbdb60 2023-11-16T18:25:14.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7945b4db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629eacc5650 con 0x5629eacc28b0 2023-11-16T18:25:14.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7945b4db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629eacba1c0 con 0x5629eacbe470 2023-11-16T18:25:14.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7932ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbdb60 0x5629eacdc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7932ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbdb60 0x5629eacdc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50580/0 (socket says 172.21.15.2:50580) 2023-11-16T18:25:14.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7932ffd700 1 -- 172.21.15.2:0/2635156215 learned_addr learned my addr 172.21.15.2:0/2635156215 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:14.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f79327fc700 1 --2- 172.21.15.2:0/2635156215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbe470 0x5629eacc9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f79337fe700 1 --2- 172.21.15.2:0/2635156215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 0x5629eacc6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7932ffd700 1 -- 172.21.15.2:0/2635156215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbe470 msgr2=0x5629eacc9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:14.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7932ffd700 1 --2- 172.21.15.2:0/2635156215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbe470 0x5629eacc9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7932ffd700 1 -- 172.21.15.2:0/2635156215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 msgr2=0x5629eacc6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:14.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.040+0000 7f7932ffd700 1 --2- 172.21.15.2:0/2635156215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 0x5629eacc6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.041+0000 7f7932ffd700 1 -- 172.21.15.2:0/2635156215 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629eabd4ee0 con 0x5629eacbdb60 2023-11-16T18:25:14.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.041+0000 7f7932ffd700 1 --2- 172.21.15.2:0/2635156215 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbdb60 0x5629eacdc330 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f792400b6f0 tx=0x7f7924005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1987349590fe0f8d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:14.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.041+0000 7f791a7fc700 1 -- 172.21.15.2:0/2635156215 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7924024c30 con 0x5629eacbdb60 2023-11-16T18:25:14.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.041+0000 7f791a7fc700 1 -- 172.21.15.2:0/2635156215 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7924028030 con 0x5629eacbdb60 2023-11-16T18:25:14.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.041+0000 7f791a7fc700 1 -- 172.21.15.2:0/2635156215 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7924005290 con 0x5629eacbdb60 2023-11-16T18:25:14.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.042+0000 7f7945b4db80 1 -- 172.21.15.2:0/2635156215 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbdb60 msgr2=0x5629eacdc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:14.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.042+0000 7f7945b4db80 1 --2- 172.21.15.2:0/2635156215 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbdb60 0x5629eacdc330 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f792400b6f0 tx=0x7f7924005f00 comp rx=0 tx=0).stop 2023-11-16T18:25:14.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.042+0000 7f7945b4db80 1 -- 172.21.15.2:0/2635156215 shutdown_connections 2023-11-16T18:25:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.042+0000 7f7945b4db80 1 --2- 172.21.15.2:0/2635156215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 0x5629eacc6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.042+0000 7f7945b4db80 1 --2- 172.21.15.2:0/2635156215 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbdb60 0x5629eacdc330 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.042+0000 7f7945b4db80 1 --2- 172.21.15.2:0/2635156215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbe470 0x5629eacc9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.042+0000 7f7945b4db80 1 -- 172.21.15.2:0/2635156215 >> 172.21.15.2:0/2635156215 conn(0x5629eacb3360 msgr2=0x5629eacb7550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.042+0000 7f7945b4db80 1 -- 172.21.15.2:0/2635156215 shutdown_connections 2023-11-16T18:25:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.042+0000 7f7945b4db80 1 -- 172.21.15.2:0/2635156215 wait complete. 2023-11-16T18:25:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.043+0000 7f7945b4db80 1 -- start start 2023-11-16T18:25:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.043+0000 7f7945b4db80 1 Processor -- start 2023-11-16T18:25:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.043+0000 7f7945b4db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacd7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.043+0000 7f7945b4db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbe470 0x5629eacd5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.043+0000 7f7945b4db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 0x5629eacd5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.043+0000 7f7945b4db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629eacd5de0 con 0x5629eacbe470 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.043+0000 7f7945b4db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629eacd5f50 con 0x5629eacc28b0 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.043+0000 7f7945b4db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629ead32e00 con 0x5629eacbdb60 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.043+0000 7f7945b4db80 10 client.0 _ll_register_callbacks cb 0x7f792400f770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.043+0000 7f79327fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbe470 0x5629eacd5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.043+0000 7f79327fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbe470 0x5629eacd5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50588/0 (socket says 172.21.15.2:50588) 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f79327fc700 1 -- 172.21.15.2:0/4197511541 learned_addr learned my addr 172.21.15.2:0/4197511541 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f7932ffd700 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacd7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f79337fe700 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 0x5629eacd5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f79337fe700 1 -- 172.21.15.2:0/4197511541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 msgr2=0x5629eacd7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f79337fe700 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacd7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f79337fe700 1 -- 172.21.15.2:0/4197511541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbe470 msgr2=0x5629eacd5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f79337fe700 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbe470 0x5629eacd5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f79337fe700 1 -- 172.21.15.2:0/4197511541 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5629eabd5190 con 0x5629eacc28b0 2023-11-16T18:25:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f7932ffd700 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacd7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:14.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f79337fe700 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 0x5629eacd5700 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f792c009790 tx=0x7f792c016620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ba00100fcc583ee server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:14.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f7912ffd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:25:14.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f7912ffd700 1 -- 172.21.15.2:0/4197511541 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792c016d10 con 0x5629eacc28b0 2023-11-16T18:25:14.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f7912ffd700 1 -- 172.21.15.2:0/4197511541 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f792c016e70 con 0x5629eacc28b0 2023-11-16T18:25:14.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.044+0000 7f7945b4db80 1 -- 172.21.15.2:0/4197511541 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79280a1dd0 con 0x5629eacc28b0 2023-11-16T18:25:14.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.045+0000 7f7912ffd700 1 -- 172.21.15.2:0/4197511541 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f792c004bb0 con 0x5629eacc28b0 2023-11-16T18:25:14.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.045+0000 7f7945b4db80 10 client.25352 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:14.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.045+0000 7f7945b4db80 1 -- 172.21.15.2:0/4197511541 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5629eaa25c60 con 0x5629eacc28b0 2023-11-16T18:25:14.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.045+0000 7f7912ffd700 1 -- 172.21.15.2:0/4197511541 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79280a1dd0 con 0x5629eacc28b0 2023-11-16T18:25:14.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.045+0000 7f7912ffd700 1 -- 172.21.15.2:0/4197511541 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f792c01e030 con 0x5629eacc28b0 2023-11-16T18:25:14.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.045+0000 7f7945b4db80 10 client.25352 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:14.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.045+0000 7f7945b4db80 10 client.25352 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:14.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.045+0000 7f7945b4db80 20 client.25352 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:14.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.045+0000 7f7945b4db80 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5629ead34630 0x5629ead36a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:14.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.045+0000 7f7945b4db80 4 client.25352 mds_command: new command op to 24473 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:25:14.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.045+0000 7f7945b4db80 1 -- 172.21.15.2:0/4197511541 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5629eabdd660 con 0x5629ead34630 2023-11-16T18:25:14.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.046+0000 7f7932ffd700 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5629ead34630 0x5629ead36a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:14.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.046+0000 7f7912ffd700 10 client.25352 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:14.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.046+0000 7f7932ffd700 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5629ead34630 0x5629ead36a40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d83c2a5b4b2fefa server_cookie=903e3b720a6ae474 in_seq=0 out_seq=0 2023-11-16T18:25:14.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.046+0000 7f7912ffd700 1 -- 172.21.15.2:0/4197511541 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5629eabdd660 con 0x5629ead34630 2023-11-16T18:25:14.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.046+0000 7f7912ffd700 10 client.25352 handle_command_reply: tid=0 2023-11-16T18:25:14.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.046+0000 7f7945b4db80 1 client.25352 shutdown 2023-11-16T18:25:14.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.047+0000 7f7945b4db80 1 -- 172.21.15.2:0/4197511541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 msgr2=0x5629eacd5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:14.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.047+0000 7f7945b4db80 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 0x5629eacd5700 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f792c009790 tx=0x7f792c016620 comp rx=0 tx=0).stop 2023-11-16T18:25:14.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.047+0000 7f7945b4db80 1 -- 172.21.15.2:0/4197511541 shutdown_connections 2023-11-16T18:25:14.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.047+0000 7f7945b4db80 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5629ead34630 0x5629ead36a40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.047+0000 7f7945b4db80 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629eacbdb60 0x5629eacd7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.047+0000 7f7945b4db80 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629eacc28b0 0x5629eacd5700 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.047+0000 7f7945b4db80 1 --2- 172.21.15.2:0/4197511541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5629eacbe470 0x5629eacd5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.047+0000 7f7945b4db80 1 -- 172.21.15.2:0/4197511541 >> 172.21.15.2:0/4197511541 conn(0x5629eacb3360 msgr2=0x5629eacc0970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:14.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.047+0000 7f7912ffd700 0 client.25352 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:14.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.047+0000 7f7945b4db80 1 -- 172.21.15.2:0/4197511541 shutdown_connections 2023-11-16T18:25:14.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.047+0000 7f7945b4db80 1 -- 172.21.15.2:0/4197511541 wait complete. 2023-11-16T18:25:14.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.047+0000 7f7945b4db80 20 client.25352 trim_cache size 0 max 16384 2023-11-16T18:25:14.055 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:25:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.048+0000 7f79399c1700 1 -- 172.21.15.2:0/4038977846 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f791c0a13e0 msgr2=0x7f791c0a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.048+0000 7f79399c1700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f791c0a13e0 0x7f791c0a3890 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f792400b2e0 tx=0x7f7924005fd0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.048+0000 7f79399c1700 1 -- 172.21.15.2:0/4038977846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79340ff2c0 msgr2=0x7f7934194690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.048+0000 7f79399c1700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79340ff2c0 0x7f7934194690 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f792800ea90 tx=0x7f792800eda0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.048+0000 7f79399c1700 1 -- 172.21.15.2:0/4038977846 shutdown_connections 2023-11-16T18:25:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.048+0000 7f79399c1700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f791c0a13e0 0x7f791c0a3890 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.048+0000 7f79399c1700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7934104920 0x7f7934198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.048+0000 7f79399c1700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79340fe910 0x7f7934194150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.048+0000 7f79399c1700 1 --2- 172.21.15.2:0/4038977846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79340ff2c0 0x7f7934194690 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:14.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.048+0000 7f79399c1700 1 -- 172.21.15.2:0/4038977846 >> 172.21.15.2:0/4038977846 conn(0x7f7934075880 msgr2=0x7f79340fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:14.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.048+0000 7f79399c1700 1 -- 172.21.15.2:0/4038977846 shutdown_connections 2023-11-16T18:25:14.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:14.048+0000 7f79399c1700 1 -- 172.21.15.2:0/4038977846 wait complete. 2023-11-16T18:25:14.065 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:13 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3752832232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:25:14.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:13 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3752832232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:25:14.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:13 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3752832232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:25:15.066 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T18:25:15.067 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T18:25:15.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.225+0000 7f796f21d700 1 -- 172.21.15.2:0/4059419795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7968100420 msgr2=0x7f796810bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.225+0000 7f796f21d700 1 --2- 172.21.15.2:0/4059419795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7968100420 0x7f796810bfd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7960009fc0 tx=0x7f796000b330 comp rx=0 tx=0).stop 2023-11-16T18:25:15.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.225+0000 7f796f21d700 1 -- 172.21.15.2:0/4059419795 shutdown_connections 2023-11-16T18:25:15.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.225+0000 7f796f21d700 1 --2- 172.21.15.2:0/4059419795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7968100420 0x7f796810bfd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.225+0000 7f796f21d700 1 --2- 172.21.15.2:0/4059419795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7968105250 0x7f7968105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.225+0000 7f796f21d700 1 --2- 172.21.15.2:0/4059419795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79680ffa70 0x7f79680ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.225+0000 7f796f21d700 1 -- 172.21.15.2:0/4059419795 >> 172.21.15.2:0/4059419795 conn(0x7f79680763a0 msgr2=0x7f79680767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:15.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.225+0000 7f796f21d700 1 -- 172.21.15.2:0/4059419795 shutdown_connections 2023-11-16T18:25:15.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.225+0000 7f796f21d700 1 -- 172.21.15.2:0/4059419795 wait complete. 2023-11-16T18:25:15.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.226+0000 7f796f21d700 1 Processor -- start 2023-11-16T18:25:15.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.226+0000 7f796f21d700 1 -- start start 2023-11-16T18:25:15.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.226+0000 7f796f21d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79680ffa70 0x7f7968197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.226+0000 7f796f21d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7968100420 0x7f7968197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.226+0000 7f796f21d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7968105250 0x7f79681a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.226+0000 7f796f21d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79681980d0 con 0x7f7968105250 2023-11-16T18:25:15.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.226+0000 7f796f21d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79681a2810 con 0x7f79680ffa70 2023-11-16T18:25:15.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.226+0000 7f796f21d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7968195040 con 0x7f7968100420 2023-11-16T18:25:15.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.226+0000 7f796d7ba700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7968105250 0x7f79681a22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.226+0000 7f796d7ba700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7968105250 0x7f79681a22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50600/0 (socket says 172.21.15.2:50600) 2023-11-16T18:25:15.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.226+0000 7f796d7ba700 1 -- 172.21.15.2:0/222156000 learned_addr learned my addr 172.21.15.2:0/222156000 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:15.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.226+0000 7f796cfb9700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79680ffa70 0x7f7968197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.226+0000 7f7967fff700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7968100420 0x7f7968197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.227+0000 7f796cfb9700 1 -- 172.21.15.2:0/222156000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7968100420 msgr2=0x7f7968197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.227+0000 7f796cfb9700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7968100420 0x7f7968197790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.227+0000 7f796cfb9700 1 -- 172.21.15.2:0/222156000 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7968105250 msgr2=0x7f79681a22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.227+0000 7f796cfb9700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7968105250 0x7f79681a22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.227+0000 7f796cfb9700 1 -- 172.21.15.2:0/222156000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79540095f0 con 0x7f79680ffa70 2023-11-16T18:25:15.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.227+0000 7f796d7ba700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7968105250 0x7f79681a22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:15.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.227+0000 7f7967fff700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7968100420 0x7f7968197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:15.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.227+0000 7f796cfb9700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79680ffa70 0x7f7968197250 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f795c00c970 tx=0x7f795c00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:15.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.227+0000 7f7965ffb700 1 -- 172.21.15.2:0/222156000 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c004a30 con 0x7f79680ffa70 2023-11-16T18:25:15.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.227+0000 7f796f21d700 1 -- 172.21.15.2:0/222156000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7960009c40 con 0x7f79680ffa70 2023-11-16T18:25:15.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.227+0000 7f7965ffb700 1 -- 172.21.15.2:0/222156000 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f795c0054e0 con 0x7f79680ffa70 2023-11-16T18:25:15.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.227+0000 7f7965ffb700 1 -- 172.21.15.2:0/222156000 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c00eae0 con 0x7f79680ffa70 2023-11-16T18:25:15.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.228+0000 7f796f21d700 1 -- 172.21.15.2:0/222156000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796810f1e0 con 0x7f79680ffa70 2023-11-16T18:25:15.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:14 smithi040 ceph-mon[100251]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 25 op/s 2023-11-16T18:25:15.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.230+0000 7f7965ffb700 1 -- 172.21.15.2:0/222156000 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f795c00ec80 con 0x7f79680ffa70 2023-11-16T18:25:15.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.231+0000 7f7965ffb700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f79580a13e0 0x7f79580a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.231+0000 7f7965ffb700 1 -- 172.21.15.2:0/222156000 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f795c0ccd30 con 0x7f79680ffa70 2023-11-16T18:25:15.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.231+0000 7f7967fff700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f79580a13e0 0x7f79580a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.232+0000 7f7967fff700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f79580a13e0 0x7f79580a3890 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7968068690 tx=0x7f7954009310 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:15.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.232+0000 7f797b3a9b80 1 Processor -- start 2023-11-16T18:25:15.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.232+0000 7f797b3a9b80 1 -- start start 2023-11-16T18:25:15.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f797b3a9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0abe0 0x5626efd0afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f797b3a9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0b4f0 0x5626efd0f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f797b3a9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0f930 0x5626efd0fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f797b3a9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626efc100e0 con 0x5626efd0f930 2023-11-16T18:25:15.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f797b3a9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626efc62920 con 0x5626efd0abe0 2023-11-16T18:25:15.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f797b3a9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626efc5bdb0 con 0x5626efd0b4f0 2023-11-16T18:25:15.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f796d7ba700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0f930 0x5626efd0fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f796d7ba700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0f930 0x5626efd0fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50608/0 (socket says 172.21.15.2:50608) 2023-11-16T18:25:15.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f796d7ba700 1 -- 172.21.15.2:0/2023973140 learned_addr learned my addr 172.21.15.2:0/2023973140 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:15.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f796cfb9700 1 --2- 172.21.15.2:0/2023973140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0abe0 0x5626efd0afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f796d7ba700 1 -- 172.21.15.2:0/2023973140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0b4f0 msgr2=0x5626efd0f3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T18:25:15.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f796d7ba700 1 --2- 172.21.15.2:0/2023973140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0b4f0 0x5626efd0f3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f796d7ba700 1 -- 172.21.15.2:0/2023973140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0abe0 msgr2=0x5626efd0afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f796d7ba700 1 --2- 172.21.15.2:0/2023973140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0abe0 0x5626efd0afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f796d7ba700 1 -- 172.21.15.2:0/2023973140 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626efadd6b0 con 0x5626efd0f930 2023-11-16T18:25:15.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.233+0000 7f796d7ba700 1 --2- 172.21.15.2:0/2023973140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0f930 0x5626efd0fd80 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7960009470 tx=0x7f79600094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d69ddaced377f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:15.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.234+0000 7f7952ffd700 1 -- 172.21.15.2:0/2023973140 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796000e040 con 0x5626efd0f930 2023-11-16T18:25:15.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.234+0000 7f7952ffd700 1 -- 172.21.15.2:0/2023973140 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7960009ac0 con 0x5626efd0f930 2023-11-16T18:25:15.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.234+0000 7f7952ffd700 1 -- 172.21.15.2:0/2023973140 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796001c6a0 con 0x5626efd0f930 2023-11-16T18:25:15.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.235+0000 7f797b3a9b80 1 -- 172.21.15.2:0/2023973140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0f930 msgr2=0x5626efd0fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.235+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/2023973140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0f930 0x5626efd0fd80 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7960009470 tx=0x7f79600094a0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.235+0000 7f797b3a9b80 1 -- 172.21.15.2:0/2023973140 shutdown_connections 2023-11-16T18:25:15.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.235+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/2023973140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0f930 0x5626efd0fd80 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.235+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/2023973140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0abe0 0x5626efd0afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.235+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/2023973140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0b4f0 0x5626efd0f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.235+0000 7f797b3a9b80 1 -- 172.21.15.2:0/2023973140 >> 172.21.15.2:0/2023973140 conn(0x5626efd003e0 msgr2=0x5626efd05bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:15.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.235+0000 7f797b3a9b80 1 -- 172.21.15.2:0/2023973140 shutdown_connections 2023-11-16T18:25:15.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.235+0000 7f797b3a9b80 1 -- 172.21.15.2:0/2023973140 wait complete. 2023-11-16T18:25:15.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.235+0000 7f797b3a9b80 1 -- start start 2023-11-16T18:25:15.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.235+0000 7f797b3a9b80 1 Processor -- start 2023-11-16T18:25:15.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f797b3a9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 0x5626efd17060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f797b3a9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0b4f0 0x5626efd142e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f797b3a9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0f930 0x5626efd148f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f797b3a9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626efbfd960 con 0x5626efd0abe0 2023-11-16T18:25:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f797b3a9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626efbfd600 con 0x5626efd0f930 2023-11-16T18:25:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f797b3a9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626efadf9a0 con 0x5626efd0b4f0 2023-11-16T18:25:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f797b3a9b80 10 client.0 _ll_register_callbacks cb 0x5626efc1b800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f796cfb9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 0x5626efd17060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f796cfb9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 0x5626efd17060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50612/0 (socket says 172.21.15.2:50612) 2023-11-16T18:25:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f796cfb9700 1 -- 172.21.15.2:0/3012329118 learned_addr learned my addr 172.21.15.2:0/3012329118 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f796cfb9700 1 -- 172.21.15.2:0/3012329118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0b4f0 msgr2=0x5626efd142e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T18:25:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f796cfb9700 1 --2- 172.21.15.2:0/3012329118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0b4f0 0x5626efd142e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f796cfb9700 1 -- 172.21.15.2:0/3012329118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0f930 msgr2=0x5626efd148f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f796cfb9700 1 --2- 172.21.15.2:0/3012329118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0f930 0x5626efd148f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.236+0000 7f796cfb9700 1 -- 172.21.15.2:0/3012329118 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796810c8d0 con 0x5626efd0abe0 2023-11-16T18:25:15.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.237+0000 7f796cfb9700 1 --2- 172.21.15.2:0/3012329118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 0x5626efd17060 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f795c0962b0 tx=0x7f795c0992d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5070cf42fc4135 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:15.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.237+0000 7f7947fff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:15.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.237+0000 7f7947fff700 1 -- 172.21.15.2:0/3012329118 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c0a1c90 con 0x5626efd0abe0 2023-11-16T18:25:15.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.237+0000 7f797b3a9b80 1 -- 172.21.15.2:0/3012329118 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626efc1e120 con 0x5626efd0abe0 2023-11-16T18:25:15.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.237+0000 7f7947fff700 1 -- 172.21.15.2:0/3012329118 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f795c0b90c0 con 0x5626efd0abe0 2023-11-16T18:25:15.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.237+0000 7f7947fff700 1 -- 172.21.15.2:0/3012329118 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f795c0b9280 con 0x5626efd0abe0 2023-11-16T18:25:15.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.237+0000 7f797b3a9b80 10 client.15777 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:15.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.237+0000 7f797b3a9b80 1 -- 172.21.15.2:0/3012329118 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5626efc22210 con 0x5626efd0abe0 2023-11-16T18:25:15.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.237+0000 7f7947fff700 1 -- 172.21.15.2:0/3012329118 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5626efc1e120 con 0x5626efd0abe0 2023-11-16T18:25:15.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.238+0000 7f7947fff700 1 -- 172.21.15.2:0/3012329118 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f795c0a1890 con 0x5626efd0abe0 2023-11-16T18:25:15.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.238+0000 7f797b3a9b80 10 client.15777 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:15.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.238+0000 7f797b3a9b80 10 client.15777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:15.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.238+0000 7f797b3a9b80 20 client.15777 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:15.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.238+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/3012329118 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626efd24610 0x5626efd26a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.238+0000 7f797b3a9b80 4 client.15777 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:25:15.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.238+0000 7f797b3a9b80 1 -- 172.21.15.2:0/3012329118 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626efc2a6e0 con 0x5626efd24610 2023-11-16T18:25:15.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.238+0000 7f7967fff700 1 --2- 172.21.15.2:0/3012329118 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626efd24610 0x5626efd26a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.238+0000 7f7947fff700 10 client.15777 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:15.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.238+0000 7f7967fff700 1 --2- 172.21.15.2:0/3012329118 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626efd24610 0x5626efd26a20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf2a279df478b2a9 server_cookie=12af80f2b5b7f410 in_seq=0 out_seq=0 2023-11-16T18:25:15.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.239+0000 7f7947fff700 1 -- 172.21.15.2:0/3012329118 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5626efc2a6e0 con 0x5626efd24610 2023-11-16T18:25:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.239+0000 7f7947fff700 10 client.15777 handle_command_reply: tid=0 2023-11-16T18:25:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.239+0000 7f797b3a9b80 1 client.15777 shutdown 2023-11-16T18:25:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.239+0000 7f797b3a9b80 1 -- 172.21.15.2:0/3012329118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 msgr2=0x5626efd17060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.239+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/3012329118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 0x5626efd17060 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f795c0962b0 tx=0x7f795c0992d0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.239+0000 7f797b3a9b80 1 -- 172.21.15.2:0/3012329118 shutdown_connections 2023-11-16T18:25:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.239+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/3012329118 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626efd24610 0x5626efd26a20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.239+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/3012329118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0f930 0x5626efd148f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.239+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/3012329118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 0x5626efd17060 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.239+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/3012329118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0b4f0 0x5626efd142e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.239+0000 7f797b3a9b80 1 -- 172.21.15.2:0/3012329118 >> 172.21.15.2:0/3012329118 conn(0x5626efd003e0 msgr2=0x5626efd10de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.239+0000 7f7947fff700 0 client.15777 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.240+0000 7f797b3a9b80 1 -- 172.21.15.2:0/3012329118 shutdown_connections 2023-11-16T18:25:15.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.240+0000 7f797b3a9b80 1 -- 172.21.15.2:0/3012329118 wait complete. 2023-11-16T18:25:15.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.240+0000 7f797b3a9b80 20 client.15777 trim_cache size 0 max 16384 2023-11-16T18:25:15.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.257+0000 7f797b3a9b80 1 Processor -- start 2023-11-16T18:25:15.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.257+0000 7f797b3a9b80 1 -- start start 2023-11-16T18:25:15.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.257+0000 7f797b3a9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0abe0 0x5626efd05d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f797b3a9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0b4f0 0x5626efd1b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f797b3a9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0f930 0x5626efd0a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f797b3a9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626efd158c0 con 0x5626efd0f930 2023-11-16T18:25:15.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f797b3a9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626efcfff60 con 0x5626efd0b4f0 2023-11-16T18:25:15.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f797b3a9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626efd12840 con 0x5626efd0abe0 2023-11-16T18:25:15.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f796cfb9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0abe0 0x5626efd05d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f796cfb9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0abe0 0x5626efd05d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:42148/0 (socket says 172.21.15.2:42148) 2023-11-16T18:25:15.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f796cfb9700 1 -- 172.21.15.2:0/971117290 learned_addr learned my addr 172.21.15.2:0/971117290 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:15.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f7967fff700 1 --2- 172.21.15.2:0/971117290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0b4f0 0x5626efd1b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f796d7ba700 1 --2- 172.21.15.2:0/971117290 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0f930 0x5626efd0a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f796cfb9700 1 -- 172.21.15.2:0/971117290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0b4f0 msgr2=0x5626efd1b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f796cfb9700 1 --2- 172.21.15.2:0/971117290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0b4f0 0x5626efd1b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f796cfb9700 1 -- 172.21.15.2:0/971117290 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0f930 msgr2=0x5626efd0a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f796cfb9700 1 --2- 172.21.15.2:0/971117290 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0f930 0x5626efd0a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f796cfb9700 1 -- 172.21.15.2:0/971117290 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626efd048a0 con 0x5626efd0abe0 2023-11-16T18:25:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f7967fff700 1 --2- 172.21.15.2:0/971117290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0b4f0 0x5626efd1b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:25:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f796d7ba700 1 --2- 172.21.15.2:0/971117290 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0f930 0x5626efd0a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:25:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.258+0000 7f796cfb9700 1 --2- 172.21.15.2:0/971117290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0abe0 0x5626efd05d00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f795c00cfd0 tx=0x7f795c097df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1d8109e78f8acee server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.259+0000 7f79537fe700 1 -- 172.21.15.2:0/971117290 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c0ca410 con 0x5626efd0abe0 2023-11-16T18:25:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.259+0000 7f79537fe700 1 -- 172.21.15.2:0/971117290 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f795c0a1c90 con 0x5626efd0abe0 2023-11-16T18:25:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.259+0000 7f79537fe700 1 -- 172.21.15.2:0/971117290 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c098d30 con 0x5626efd0abe0 2023-11-16T18:25:15.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.259+0000 7f797b3a9b80 1 -- 172.21.15.2:0/971117290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0abe0 msgr2=0x5626efd05d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.259+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/971117290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0abe0 0x5626efd05d00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f795c00cfd0 tx=0x7f795c097df0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.259+0000 7f797b3a9b80 1 -- 172.21.15.2:0/971117290 shutdown_connections 2023-11-16T18:25:15.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.259+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/971117290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0abe0 0x5626efd05d00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.259+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/971117290 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0f930 0x5626efd0a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.259+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/971117290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0b4f0 0x5626efd1b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.259+0000 7f797b3a9b80 1 -- 172.21.15.2:0/971117290 >> 172.21.15.2:0/971117290 conn(0x5626efd003e0 msgr2=0x5626efd099e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:15.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.260+0000 7f797b3a9b80 1 -- 172.21.15.2:0/971117290 shutdown_connections 2023-11-16T18:25:15.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.260+0000 7f797b3a9b80 1 -- 172.21.15.2:0/971117290 wait complete. 2023-11-16T18:25:15.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.260+0000 7f797b3a9b80 1 -- start start 2023-11-16T18:25:15.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.260+0000 7f797b3a9b80 1 Processor -- start 2023-11-16T18:25:15.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.260+0000 7f797b3a9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 0x5626efd180d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.260+0000 7f797b3a9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0b4f0 0x5626efd18610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.260+0000 7f797b3a9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0f930 0x5626efd18b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f797b3a9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626efd80d00 con 0x5626efd0abe0 2023-11-16T18:25:15.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f797b3a9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626efd80e40 con 0x5626efd0b4f0 2023-11-16T18:25:15.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f797b3a9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626efd80fb0 con 0x5626efd0f930 2023-11-16T18:25:15.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f797b3a9b80 10 client.0 _ll_register_callbacks cb 0x7f795c099610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:15.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f796cfb9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 0x5626efd180d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f796cfb9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 0x5626efd180d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50624/0 (socket says 172.21.15.2:50624) 2023-11-16T18:25:15.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f796cfb9700 1 -- 172.21.15.2:0/474900996 learned_addr learned my addr 172.21.15.2:0/474900996 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:15.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f796d7ba700 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0f930 0x5626efd18b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f796cfb9700 1 -- 172.21.15.2:0/474900996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0f930 msgr2=0x5626efd18b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f796cfb9700 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0f930 0x5626efd18b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f7967fff700 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0b4f0 0x5626efd18610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f796cfb9700 1 -- 172.21.15.2:0/474900996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0b4f0 msgr2=0x5626efd18610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f796cfb9700 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0b4f0 0x5626efd18610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f796cfb9700 1 -- 172.21.15.2:0/474900996 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626efc22210 con 0x5626efd0abe0 2023-11-16T18:25:15.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f796d7ba700 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0f930 0x5626efd18b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:25:15.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f7967fff700 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0b4f0 0x5626efd18610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:25:15.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f796cfb9700 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 0x5626efd180d0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f795c0ca710 tx=0x7f795c00a4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0e1d55291904c3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:15.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.261+0000 7f7947fff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:15.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f7947fff700 1 -- 172.21.15.2:0/474900996 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c098f30 con 0x5626efd0abe0 2023-11-16T18:25:15.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f7947fff700 1 -- 172.21.15.2:0/474900996 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f795c095040 con 0x5626efd0abe0 2023-11-16T18:25:15.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f797b3a9b80 1 -- 172.21.15.2:0/474900996 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626efcfdba0 con 0x5626efd0abe0 2023-11-16T18:25:15.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f7947fff700 1 -- 172.21.15.2:0/474900996 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f795c095200 con 0x5626efd0abe0 2023-11-16T18:25:15.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f797b3a9b80 10 client.15780 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:15.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f797b3a9b80 1 -- 172.21.15.2:0/474900996 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5626efa89e60 con 0x5626efd0abe0 2023-11-16T18:25:15.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f7947fff700 1 -- 172.21.15.2:0/474900996 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5626efcfdba0 con 0x5626efd0abe0 2023-11-16T18:25:15.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f7947fff700 1 -- 172.21.15.2:0/474900996 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f795c0987a0 con 0x5626efd0abe0 2023-11-16T18:25:15.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f797b3a9b80 10 client.15780 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:15.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f797b3a9b80 10 client.15780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:15.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f797b3a9b80 20 client.15780 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:15.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626efd82190 0x5626efd84570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f797b3a9b80 4 client.15780 mds_command: new command op to 24473 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T18:25:15.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.262+0000 7f797b3a9b80 1 -- 172.21.15.2:0/474900996 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5626efaaaff0 con 0x5626efd82190 2023-11-16T18:25:15.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.263+0000 7f7967fff700 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626efd82190 0x5626efd84570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.263+0000 7f7947fff700 10 client.15780 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:15.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.263+0000 7f7967fff700 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626efd82190 0x5626efd84570 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f6b00e916668459 server_cookie=8d96d93f9ec8a7a2 in_seq=0 out_seq=0 2023-11-16T18:25:15.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.264+0000 7f7947fff700 1 -- 172.21.15.2:0/474900996 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5626efaaaff0 con 0x5626efd82190 2023-11-16T18:25:15.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.264+0000 7f7947fff700 10 client.15780 handle_command_reply: tid=0 2023-11-16T18:25:15.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.264+0000 7f797b3a9b80 1 client.15780 shutdown 2023-11-16T18:25:15.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.265+0000 7f797b3a9b80 1 -- 172.21.15.2:0/474900996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 msgr2=0x5626efd180d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.265+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 0x5626efd180d0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f795c0ca710 tx=0x7f795c00a4e0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.265+0000 7f797b3a9b80 1 -- 172.21.15.2:0/474900996 shutdown_connections 2023-11-16T18:25:15.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.265+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626efd82190 0x5626efd84570 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.265+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626efd0f930 0x5626efd18b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.265+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626efd0abe0 0x5626efd180d0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.265+0000 7f797b3a9b80 1 --2- 172.21.15.2:0/474900996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626efd0b4f0 0x5626efd18610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.265+0000 7f797b3a9b80 1 -- 172.21.15.2:0/474900996 >> 172.21.15.2:0/474900996 conn(0x5626efd003e0 msgr2=0x5626efd0a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:15.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.265+0000 7f7947fff700 0 client.15780 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:15.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.265+0000 7f797b3a9b80 1 -- 172.21.15.2:0/474900996 shutdown_connections 2023-11-16T18:25:15.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.265+0000 7f797b3a9b80 1 -- 172.21.15.2:0/474900996 wait complete. 2023-11-16T18:25:15.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.265+0000 7f797b3a9b80 20 client.15780 trim_cache size 0 max 16384 2023-11-16T18:25:15.274 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:25:15.274 INFO:teuthology.orchestra.run.smithi002.stdout: "return_code": 0, 2023-11-16T18:25:15.274 INFO:teuthology.orchestra.run.smithi002.stdout: "scrub_tag": "c41edcb8-b745-49c0-ba9e-fd3c3f1ea629", 2023-11-16T18:25:15.274 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": "asynchronous" 2023-11-16T18:25:15.274 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:25:15.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.266+0000 7f796f21d700 1 -- 172.21.15.2:0/222156000 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f79580a13e0 msgr2=0x7f79580a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.266+0000 7f796f21d700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f79580a13e0 0x7f79580a3890 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7968068690 tx=0x7f7954009310 comp rx=0 tx=0).stop 2023-11-16T18:25:15.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.266+0000 7f796f21d700 1 -- 172.21.15.2:0/222156000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79680ffa70 msgr2=0x7f7968197250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.266+0000 7f796f21d700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79680ffa70 0x7f7968197250 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f795c00c970 tx=0x7f795c00cc80 comp rx=0 tx=0).stop 2023-11-16T18:25:15.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.266+0000 7f796f21d700 1 -- 172.21.15.2:0/222156000 shutdown_connections 2023-11-16T18:25:15.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.266+0000 7f796f21d700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f79580a13e0 0x7f79580a3890 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.266+0000 7f796f21d700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7968105250 0x7f79681a22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.266+0000 7f796f21d700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79680ffa70 0x7f7968197250 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.266+0000 7f796f21d700 1 --2- 172.21.15.2:0/222156000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7968100420 0x7f7968197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.266+0000 7f796f21d700 1 -- 172.21.15.2:0/222156000 >> 172.21.15.2:0/222156000 conn(0x7f79680763a0 msgr2=0x7f79680fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:15.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.266+0000 7f796f21d700 1 -- 172.21.15.2:0/222156000 shutdown_connections 2023-11-16T18:25:15.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.266+0000 7f796f21d700 1 -- 172.21.15.2:0/222156000 wait complete. 2023-11-16T18:25:15.282 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:14 smithi002 ceph-mon[98807]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 25 op/s 2023-11-16T18:25:15.283 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:25:15.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:14 smithi188 ceph-mon[100639]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 25 op/s 2023-11-16T18:25:15.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.414+0000 7f6e5ab93700 1 -- 172.21.15.2:0/2563750959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e540ff460 msgr2=0x7f6e540ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.414+0000 7f6e5ab93700 1 --2- 172.21.15.2:0/2563750959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e540ff460 0x7f6e540ff830 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6e480099c0 tx=0x7f6e48009cd0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.415+0000 7f6e5ab93700 1 -- 172.21.15.2:0/2563750959 shutdown_connections 2023-11-16T18:25:15.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.415+0000 7f6e5ab93700 1 --2- 172.21.15.2:0/2563750959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e541044a0 0x7f6e5410c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.415+0000 7f6e5ab93700 1 --2- 172.21.15.2:0/2563750959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e540ff460 0x7f6e540ff830 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.415+0000 7f6e5ab93700 1 --2- 172.21.15.2:0/2563750959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e540ffe00 0x7f6e54103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.415+0000 7f6e5ab93700 1 -- 172.21.15.2:0/2563750959 >> 172.21.15.2:0/2563750959 conn(0x7f6e540763a0 msgr2=0x7f6e540767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:15.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.415+0000 7f6e5ab93700 1 -- 172.21.15.2:0/2563750959 shutdown_connections 2023-11-16T18:25:15.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.415+0000 7f6e5ab93700 1 -- 172.21.15.2:0/2563750959 wait complete. 2023-11-16T18:25:15.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.415+0000 7f6e5ab93700 1 Processor -- start 2023-11-16T18:25:15.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.415+0000 7f6e5ab93700 1 -- start start 2023-11-16T18:25:15.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e5ab93700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e540ff460 0x7f6e541963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e5ab93700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e540ffe00 0x7f6e541968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e5ab93700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e541044a0 0x7f6e5419ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e5ab93700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5419b390 con 0x7f6e541044a0 2023-11-16T18:25:15.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e5ab93700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5419b500 con 0x7f6e540ffe00 2023-11-16T18:25:15.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e5ab93700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5419b6c0 con 0x7f6e540ff460 2023-11-16T18:25:15.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e59130700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e541044a0 0x7f6e5419ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e59130700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e541044a0 0x7f6e5419ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50642/0 (socket says 172.21.15.2:50642) 2023-11-16T18:25:15.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e59130700 1 -- 172.21.15.2:0/3966331868 learned_addr learned my addr 172.21.15.2:0/3966331868 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:15.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e5892f700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e540ff460 0x7f6e541963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e53fff700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e540ffe00 0x7f6e541968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e5892f700 1 -- 172.21.15.2:0/3966331868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e540ffe00 msgr2=0x7f6e541968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e5892f700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e540ffe00 0x7f6e541968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e5892f700 1 -- 172.21.15.2:0/3966331868 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e541044a0 msgr2=0x7f6e5419ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e5892f700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e541044a0 0x7f6e5419ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e5892f700 1 -- 172.21.15.2:0/3966331868 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c009c40 con 0x7f6e540ff460 2023-11-16T18:25:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.416+0000 7f6e53fff700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e540ffe00 0x7f6e541968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.417+0000 7f6e5892f700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e540ff460 0x7f6e541963b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6e480098e0 tx=0x7f6e4800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.417+0000 7f6e51ffb700 1 -- 172.21.15.2:0/3966331868 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4801d070 con 0x7f6e540ff460 2023-11-16T18:25:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.417+0000 7f6e5ab93700 1 -- 172.21.15.2:0/3966331868 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e480095f0 con 0x7f6e540ff460 2023-11-16T18:25:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.417+0000 7f6e51ffb700 1 -- 172.21.15.2:0/3966331868 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e48022470 con 0x7f6e540ff460 2023-11-16T18:25:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.417+0000 7f6e51ffb700 1 -- 172.21.15.2:0/3966331868 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e48017650 con 0x7f6e540ff460 2023-11-16T18:25:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.417+0000 7f6e5ab93700 1 -- 172.21.15.2:0/3966331868 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5410f260 con 0x7f6e540ff460 2023-11-16T18:25:15.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.420+0000 7f6e51ffb700 1 -- 172.21.15.2:0/3966331868 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f6e48022890 con 0x7f6e540ff460 2023-11-16T18:25:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.421+0000 7f6e51ffb700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6e440a13e0 0x7f6e440a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.421+0000 7f6e51ffb700 1 -- 172.21.15.2:0/3966331868 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6e480cdfa0 con 0x7f6e540ff460 2023-11-16T18:25:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.421+0000 7f6e53fff700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6e440a13e0 0x7f6e440a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.422+0000 7f6e66d1fb80 1 Processor -- start 2023-11-16T18:25:15.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.422+0000 7f6e53fff700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6e440a13e0 0x7f6e440a3890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6e541979c0 tx=0x7f6e40009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.422+0000 7f6e66d1fb80 1 -- start start 2023-11-16T18:25:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.422+0000 7f6e66d1fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77ef370 0x55fbf77ef740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.422+0000 7f6e66d1fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77efc80 0x55fbf77f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.422+0000 7f6e66d1fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77f40c0 0x55fbf77f4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.422+0000 7f6e66d1fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf76ec060 con 0x55fbf77ef370 2023-11-16T18:25:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.422+0000 7f6e66d1fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf773e8a0 con 0x55fbf77efc80 2023-11-16T18:25:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.422+0000 7f6e66d1fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf7737d30 con 0x55fbf77f40c0 2023-11-16T18:25:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.422+0000 7f6e5892f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77ef370 0x55fbf77ef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.423+0000 7f6e5892f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77ef370 0x55fbf77ef740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50646/0 (socket says 172.21.15.2:50646) 2023-11-16T18:25:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.423+0000 7f6e5892f700 1 -- 172.21.15.2:0/150066672 learned_addr learned my addr 172.21.15.2:0/150066672 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.423+0000 7f6e53fff700 1 --2- 172.21.15.2:0/150066672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77efc80 0x55fbf77f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.423+0000 7f6e59130700 1 --2- 172.21.15.2:0/150066672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77f40c0 0x55fbf77f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.423+0000 7f6e5892f700 1 -- 172.21.15.2:0/150066672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77f40c0 msgr2=0x55fbf77f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.423+0000 7f6e5892f700 1 --2- 172.21.15.2:0/150066672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77f40c0 0x55fbf77f4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.423+0000 7f6e5892f700 1 -- 172.21.15.2:0/150066672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77efc80 msgr2=0x55fbf77f3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.423+0000 7f6e5892f700 1 --2- 172.21.15.2:0/150066672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77efc80 0x55fbf77f3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.423+0000 7f6e5892f700 1 -- 172.21.15.2:0/150066672 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbf75b9630 con 0x55fbf77ef370 2023-11-16T18:25:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.423+0000 7f6e5892f700 1 --2- 172.21.15.2:0/150066672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77ef370 0x55fbf77ef740 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6e4809aa70 tx=0x7f6e4809a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41ba2f6ec6188b75 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:15.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.423+0000 7f6e3effd700 1 -- 172.21.15.2:0/150066672 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4809eb10 con 0x55fbf77ef370 2023-11-16T18:25:15.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.423+0000 7f6e3effd700 1 -- 172.21.15.2:0/150066672 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e480a2ea0 con 0x55fbf77ef370 2023-11-16T18:25:15.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.423+0000 7f6e3effd700 1 -- 172.21.15.2:0/150066672 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e480a2c70 con 0x55fbf77ef370 2023-11-16T18:25:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.424+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/150066672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77ef370 msgr2=0x55fbf77ef740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.424+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/150066672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77ef370 0x55fbf77ef740 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6e4809aa70 tx=0x7f6e4809a6a0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.424+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/150066672 shutdown_connections 2023-11-16T18:25:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.424+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/150066672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77f40c0 0x55fbf77f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.424+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/150066672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77ef370 0x55fbf77ef740 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.424+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/150066672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77efc80 0x55fbf77f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.424+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/150066672 >> 172.21.15.2:0/150066672 conn(0x55fbf77dc360 msgr2=0x55fbf77e1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.425+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/150066672 shutdown_connections 2023-11-16T18:25:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.425+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/150066672 wait complete. 2023-11-16T18:25:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.425+0000 7f6e66d1fb80 1 -- start start 2023-11-16T18:25:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.425+0000 7f6e66d1fb80 1 Processor -- start 2023-11-16T18:25:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e66d1fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77ef370 0x55fbf77fb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e66d1fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77efc80 0x55fbf77f8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e66d1fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77f40c0 0x55fbf77f8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e66d1fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf76d98e0 con 0x55fbf77efc80 2023-11-16T18:25:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e66d1fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf76d9580 con 0x55fbf77ef370 2023-11-16T18:25:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e66d1fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf75bb920 con 0x55fbf77f40c0 2023-11-16T18:25:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e66d1fb80 10 client.0 _ll_register_callbacks cb 0x55fbf76f7780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e5892f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77ef370 0x55fbf77fb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e5892f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77ef370 0x55fbf77fb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:41090/0 (socket says 172.21.15.2:41090) 2023-11-16T18:25:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e5892f700 1 -- 172.21.15.2:0/4169137118 learned_addr learned my addr 172.21.15.2:0/4169137118 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e53fff700 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77efc80 0x55fbf77f8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e59130700 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77f40c0 0x55fbf77f8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e5892f700 1 -- 172.21.15.2:0/4169137118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77f40c0 msgr2=0x55fbf77f8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e5892f700 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77f40c0 0x55fbf77f8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e5892f700 1 -- 172.21.15.2:0/4169137118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77efc80 msgr2=0x55fbf77f8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e5892f700 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77efc80 0x55fbf77f8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e5892f700 1 -- 172.21.15.2:0/4169137118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5410f260 con 0x55fbf77ef370 2023-11-16T18:25:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.426+0000 7f6e59130700 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77f40c0 0x55fbf77f8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:25:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.427+0000 7f6e5892f700 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77ef370 0x55fbf77fb690 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6e480a2970 tx=0x7f6e48005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ee21c53a1dc9435 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.427+0000 7f6e33fff700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:25:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.427+0000 7f6e33fff700 1 -- 172.21.15.2:0/4169137118 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4809a1e0 con 0x55fbf77ef370 2023-11-16T18:25:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.427+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/4169137118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fbf77d9b20 con 0x55fbf77ef370 2023-11-16T18:25:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.427+0000 7f6e33fff700 1 -- 172.21.15.2:0/4169137118 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6e480b3120 con 0x55fbf77ef370 2023-11-16T18:25:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.427+0000 7f6e33fff700 1 -- 172.21.15.2:0/4169137118 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e480a2a10 con 0x55fbf77ef370 2023-11-16T18:25:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.427+0000 7f6e66d1fb80 10 client.25376 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.427+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/4169137118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fbf76fe190 con 0x55fbf77ef370 2023-11-16T18:25:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.427+0000 7f6e33fff700 1 -- 172.21.15.2:0/4169137118 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fbf77d9b20 con 0x55fbf77ef370 2023-11-16T18:25:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.428+0000 7f6e33fff700 1 -- 172.21.15.2:0/4169137118 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6e4800fad0 con 0x55fbf77ef370 2023-11-16T18:25:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.428+0000 7f6e66d1fb80 10 client.25376 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:15.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.428+0000 7f6e66d1fb80 10 client.25376 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:15.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.428+0000 7f6e66d1fb80 20 client.25376 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:15.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.428+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fbf7808830 0x55fbf780ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.428+0000 7f6e66d1fb80 4 client.25376 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:25:15.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.428+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/4169137118 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fbf76fa0a0 con 0x55fbf7808830 2023-11-16T18:25:15.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.428+0000 7f6e53fff700 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fbf7808830 0x55fbf780ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.428+0000 7f6e33fff700 10 client.25376 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:15.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.428+0000 7f6e53fff700 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fbf7808830 0x55fbf780ac40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3364e7117d697793 server_cookie=78b87d97cd9ada03 in_seq=0 out_seq=0 2023-11-16T18:25:15.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.429+0000 7f6e33fff700 1 -- 172.21.15.2:0/4169137118 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55fbf76fa0a0 con 0x55fbf7808830 2023-11-16T18:25:15.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.429+0000 7f6e33fff700 10 client.25376 handle_command_reply: tid=0 2023-11-16T18:25:15.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.429+0000 7f6e66d1fb80 1 client.25376 shutdown 2023-11-16T18:25:15.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.429+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/4169137118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77ef370 msgr2=0x55fbf77fb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.429+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77ef370 0x55fbf77fb690 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6e480a2970 tx=0x7f6e48005c00 comp rx=0 tx=0).stop 2023-11-16T18:25:15.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.429+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/4169137118 shutdown_connections 2023-11-16T18:25:15.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.429+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fbf7808830 0x55fbf780ac40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.429+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77f40c0 0x55fbf77f8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.429+0000 7f6e33fff700 0 client.25376 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:15.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.429+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77ef370 0x55fbf77fb690 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.429+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/4169137118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77efc80 0x55fbf77f8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.429+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/4169137118 >> 172.21.15.2:0/4169137118 conn(0x55fbf77dc360 msgr2=0x55fbf77f5570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:15.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.430+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/4169137118 shutdown_connections 2023-11-16T18:25:15.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.430+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/4169137118 wait complete. 2023-11-16T18:25:15.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.430+0000 7f6e66d1fb80 20 client.25376 trim_cache size 0 max 16384 2023-11-16T18:25:15.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.462+0000 7f6e66d1fb80 1 Processor -- start 2023-11-16T18:25:15.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.462+0000 7f6e66d1fb80 1 -- start start 2023-11-16T18:25:15.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e66d1fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 0x55fbf77fb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e66d1fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77efc80 0x55fbf77ee170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e66d1fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77f40c0 0x55fbf77ee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e66d1fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf77f9f90 con 0x55fbf77f40c0 2023-11-16T18:25:15.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e66d1fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf77f0190 con 0x55fbf77efc80 2023-11-16T18:25:15.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e66d1fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf77f6f80 con 0x55fbf77ef370 2023-11-16T18:25:15.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e5892f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 0x55fbf77fb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e53fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77efc80 0x55fbf77ee170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e59130700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77f40c0 0x55fbf77ee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e5892f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 0x55fbf77fb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:42190/0 (socket says 172.21.15.2:42190) 2023-11-16T18:25:15.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e5892f700 1 -- 172.21.15.2:0/2168966484 learned_addr learned my addr 172.21.15.2:0/2168966484 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:15.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e5892f700 1 -- 172.21.15.2:0/2168966484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77efc80 msgr2=0x55fbf77ee170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e5892f700 1 --2- 172.21.15.2:0/2168966484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77efc80 0x55fbf77ee170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e5892f700 1 -- 172.21.15.2:0/2168966484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77f40c0 msgr2=0x55fbf77ee6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e5892f700 1 --2- 172.21.15.2:0/2168966484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77f40c0 0x55fbf77ee6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.463+0000 7f6e5892f700 1 -- 172.21.15.2:0/2168966484 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbf77f6770 con 0x55fbf77ef370 2023-11-16T18:25:15.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.464+0000 7f6e5892f700 1 --2- 172.21.15.2:0/2168966484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 0x55fbf77fb1c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6e4809a5b0 tx=0x7f6e48005c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1391ea28d3010094 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:15.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.464+0000 7f6e3f7fe700 1 -- 172.21.15.2:0/2168966484 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4809eb10 con 0x55fbf77ef370 2023-11-16T18:25:15.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.464+0000 7f6e3f7fe700 1 -- 172.21.15.2:0/2168966484 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e4809a1e0 con 0x55fbf77ef370 2023-11-16T18:25:15.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.464+0000 7f6e3f7fe700 1 -- 172.21.15.2:0/2168966484 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e480cb6e0 con 0x55fbf77ef370 2023-11-16T18:25:15.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.465+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/2168966484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 msgr2=0x55fbf77fb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.465+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/2168966484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 0x55fbf77fb1c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6e4809a5b0 tx=0x7f6e48005c70 comp rx=0 tx=0).stop 2023-11-16T18:25:15.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.465+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/2168966484 shutdown_connections 2023-11-16T18:25:15.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.465+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/2168966484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 0x55fbf77fb1c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.465+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/2168966484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77f40c0 0x55fbf77ee6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.465+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/2168966484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77efc80 0x55fbf77ee170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.465+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/2168966484 >> 172.21.15.2:0/2168966484 conn(0x55fbf77dc360 msgr2=0x55fbf77e0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:15.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.465+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/2168966484 shutdown_connections 2023-11-16T18:25:15.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.465+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/2168966484 wait complete. 2023-11-16T18:25:15.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.465+0000 7f6e66d1fb80 1 -- start start 2023-11-16T18:25:15.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.465+0000 7f6e66d1fb80 1 Processor -- start 2023-11-16T18:25:15.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.466+0000 7f6e66d1fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 0x55fbf7861840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.466+0000 7f6e66d1fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77efc80 0x55fbf7861d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.466+0000 7f6e66d1fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77f40c0 0x55fbf786c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.466+0000 7f6e66d1fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf786cc10 con 0x55fbf77efc80 2023-11-16T18:25:15.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.466+0000 7f6e66d1fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf786cd50 con 0x55fbf77f40c0 2023-11-16T18:25:15.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.466+0000 7f6e66d1fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbf786ce90 con 0x55fbf77ef370 2023-11-16T18:25:15.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.466+0000 7f6e66d1fb80 10 client.0 _ll_register_callbacks cb 0x7f6e4809a960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:15.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.466+0000 7f6e5892f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 0x55fbf7861840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.466+0000 7f6e5892f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 0x55fbf7861840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:42196/0 (socket says 172.21.15.2:42196) 2023-11-16T18:25:15.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.466+0000 7f6e5892f700 1 -- 172.21.15.2:0/1372174509 learned_addr learned my addr 172.21.15.2:0/1372174509 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:15.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.466+0000 7f6e59130700 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77f40c0 0x55fbf786c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.467+0000 7f6e53fff700 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77efc80 0x55fbf7861d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.467+0000 7f6e5892f700 1 -- 172.21.15.2:0/1372174509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77f40c0 msgr2=0x55fbf786c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.467+0000 7f6e5892f700 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77f40c0 0x55fbf786c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.467+0000 7f6e5892f700 1 -- 172.21.15.2:0/1372174509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77efc80 msgr2=0x55fbf7861d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.467+0000 7f6e5892f700 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77efc80 0x55fbf7861d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.467+0000 7f6e5892f700 1 -- 172.21.15.2:0/1372174509 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fbf76fe190 con 0x55fbf77ef370 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.467+0000 7f6e53fff700 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77efc80 0x55fbf7861d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.467+0000 7f6e5892f700 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 0x55fbf7861840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6e4800feb0 tx=0x7f6e48005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96a83eeaa4d6955d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.467+0000 7f6e33fff700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.467+0000 7f6e33fff700 1 -- 172.21.15.2:0/1372174509 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4809eb10 con 0x55fbf77ef370 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.467+0000 7f6e33fff700 1 -- 172.21.15.2:0/1372174509 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6e480a2a10 con 0x55fbf77ef370 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.467+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/1372174509 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e480a2ea0 con 0x55fbf77ef370 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.468+0000 7f6e33fff700 1 -- 172.21.15.2:0/1372174509 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e480a2bd0 con 0x55fbf77ef370 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.468+0000 7f6e66d1fb80 10 client.25354 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.468+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/1372174509 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fbf7565de0 con 0x55fbf77ef370 2023-11-16T18:25:15.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.468+0000 7f6e33fff700 1 -- 172.21.15.2:0/1372174509 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e480a2ea0 con 0x55fbf77ef370 2023-11-16T18:25:15.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.468+0000 7f6e33fff700 1 -- 172.21.15.2:0/1372174509 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6e480cbc00 con 0x55fbf77ef370 2023-11-16T18:25:15.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.468+0000 7f6e66d1fb80 10 client.25354 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:15.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.468+0000 7f6e66d1fb80 10 client.25354 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:15.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.468+0000 7f6e66d1fb80 20 client.25354 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:15.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.468+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fbf786e0f0 0x55fbf78704d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:15.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.468+0000 7f6e66d1fb80 4 client.25354 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:25:15.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.468+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/1372174509 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fbf7706660 con 0x55fbf786e0f0 2023-11-16T18:25:15.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e53fff700 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fbf786e0f0 0x55fbf78704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:15.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e33fff700 10 client.25354 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:15.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e53fff700 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fbf786e0f0 0x55fbf78704d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17ad2ca375e6ebb3 server_cookie=dd6f5eec7fc7e97d in_seq=0 out_seq=0 2023-11-16T18:25:15.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e33fff700 1 -- 172.21.15.2:0/1372174509 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55fbf7706660 con 0x55fbf786e0f0 2023-11-16T18:25:15.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e33fff700 10 client.25354 handle_command_reply: tid=0 2023-11-16T18:25:15.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e66d1fb80 1 client.25354 shutdown 2023-11-16T18:25:15.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/1372174509 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 msgr2=0x55fbf7861840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 0x55fbf7861840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6e4800feb0 tx=0x7f6e48005c00 comp rx=0 tx=0).stop 2023-11-16T18:25:15.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/1372174509 shutdown_connections 2023-11-16T18:25:15.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fbf786e0f0 0x55fbf78704d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fbf77ef370 0x55fbf7861840 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbf77f40c0 0x55fbf786c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e66d1fb80 1 --2- 172.21.15.2:0/1372174509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fbf77efc80 0x55fbf7861d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.469+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/1372174509 >> 172.21.15.2:0/1372174509 conn(0x55fbf77dc360 msgr2=0x55fbf77ee170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:15.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.470+0000 7f6e33fff700 0 client.25354 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:15.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.470+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/1372174509 shutdown_connections 2023-11-16T18:25:15.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.470+0000 7f6e66d1fb80 1 -- 172.21.15.2:0/1372174509 wait complete. 2023-11-16T18:25:15.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.470+0000 7f6e66d1fb80 20 client.25354 trim_cache size 0 max 16384 2023-11-16T18:25:15.497 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:25:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-16T18:25:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": { 2023-11-16T18:25:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: "c41edcb8-b745-49c0-ba9e-fd3c3f1ea629": { 2023-11-16T18:25:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/", 2023-11-16T18:25:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: "tag": "c41edcb8-b745-49c0-ba9e-fd3c3f1ea629", 2023-11-16T18:25:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: "options": "recursive,force" 2023-11-16T18:25:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:25:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:25:15.497 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:25:15.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.470+0000 7f6e5ab93700 1 -- 172.21.15.2:0/3966331868 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6e440a13e0 msgr2=0x7f6e440a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.470+0000 7f6e5ab93700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6e440a13e0 0x7f6e440a3890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6e541979c0 tx=0x7f6e40009260 comp rx=0 tx=0).stop 2023-11-16T18:25:15.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.470+0000 7f6e5ab93700 1 -- 172.21.15.2:0/3966331868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e540ff460 msgr2=0x7f6e541963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:15.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.470+0000 7f6e5ab93700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e540ff460 0x7f6e541963b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6e480098e0 tx=0x7f6e4800f690 comp rx=0 tx=0).stop 2023-11-16T18:25:15.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.471+0000 7f6e5ab93700 1 -- 172.21.15.2:0/3966331868 shutdown_connections 2023-11-16T18:25:15.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.471+0000 7f6e5ab93700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6e440a13e0 0x7f6e440a3890 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.471+0000 7f6e5ab93700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e540ff460 0x7f6e541963b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.471+0000 7f6e5ab93700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e541044a0 0x7f6e5419ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.471+0000 7f6e5ab93700 1 --2- 172.21.15.2:0/3966331868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e540ffe00 0x7f6e541968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:15.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.471+0000 7f6e5ab93700 1 -- 172.21.15.2:0/3966331868 >> 172.21.15.2:0/3966331868 conn(0x7f6e540763a0 msgr2=0x7f6e54068de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:15.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.471+0000 7f6e5ab93700 1 -- 172.21.15.2:0/3966331868 shutdown_connections 2023-11-16T18:25:15.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:15.471+0000 7f6e5ab93700 1 -- 172.21.15.2:0/3966331868 wait complete. 2023-11-16T18:25:15.500 INFO:tasks.cephfs.filesystem:scrub status for tag:c41edcb8-b745-49c0-ba9e-fd3c3f1ea629 - {'path': '/', 'tag': 'c41edcb8-b745-49c0-ba9e-fd3c3f1ea629', 'options': 'recursive,force'} 2023-11-16T18:25:16.233 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:15 smithi040 ceph-mon[100251]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2023-11-16T18:25:16.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:15 smithi188 ceph-mon[100639]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2023-11-16T18:25:16.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:15 smithi002 ceph-mon[98807]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2023-11-16T18:25:17.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:17 smithi188 ceph-mon[100639]: scrub queued for path: / 2023-11-16T18:25:17.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:17 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:25:17.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:17 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:25:17.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:17 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:25:17.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:17 smithi188 ceph-mon[100639]: scrub summary: idle 2023-11-16T18:25:17.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:17 smithi002 ceph-mon[98807]: scrub queued for path: / 2023-11-16T18:25:17.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:17 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:25:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:17 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:25:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:17 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:25:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:17 smithi002 ceph-mon[98807]: scrub summary: idle 2023-11-16T18:25:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:17 smithi040 ceph-mon[100251]: scrub queued for path: / 2023-11-16T18:25:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:17 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:25:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:17 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:25:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:17 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:25:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:17 smithi040 ceph-mon[100251]: scrub summary: idle 2023-11-16T18:25:18.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:18 smithi188 ceph-mon[100639]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2023-11-16T18:25:18.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:18 smithi002 ceph-mon[98807]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2023-11-16T18:25:18.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:18 smithi040 ceph-mon[100251]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2023-11-16T18:25:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.831+0000 7fc764088700 1 -- 172.21.15.2:0/3288442669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc75c0ff3c0 msgr2=0x7fc75c0ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.831+0000 7fc764088700 1 --2- 172.21.15.2:0/3288442669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc75c0ff3c0 0x7fc75c0ff790 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc7580099c0 tx=0x7fc758009cd0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.832+0000 7fc764088700 1 -- 172.21.15.2:0/3288442669 shutdown_connections 2023-11-16T18:25:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.832+0000 7fc764088700 1 --2- 172.21.15.2:0/3288442669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75c104400 0x7fc75c10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.832+0000 7fc764088700 1 --2- 172.21.15.2:0/3288442669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc75c0ff3c0 0x7fc75c0ff790 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.832+0000 7fc764088700 1 --2- 172.21.15.2:0/3288442669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc75c0ffd60 0x7fc75c103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.832+0000 7fc764088700 1 -- 172.21.15.2:0/3288442669 >> 172.21.15.2:0/3288442669 conn(0x7fc75c0763a0 msgr2=0x7fc75c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.832+0000 7fc764088700 1 -- 172.21.15.2:0/3288442669 shutdown_connections 2023-11-16T18:25:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.832+0000 7fc764088700 1 -- 172.21.15.2:0/3288442669 wait complete. 2023-11-16T18:25:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc764088700 1 Processor -- start 2023-11-16T18:25:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc764088700 1 -- start start 2023-11-16T18:25:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc764088700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc75c0ffd60 0x7fc75c1a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc764088700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc75c104400 0x7fc75c1a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc764088700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75c19ca80 0x7fc75c19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc764088700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c19d4f0 con 0x7fc75c0ffd60 2023-11-16T18:25:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc764088700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c19d630 con 0x7fc75c19ca80 2023-11-16T18:25:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc764088700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c19d7f0 con 0x7fc75c104400 2023-11-16T18:25:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc761e24700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc75c0ffd60 0x7fc75c1a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc761e24700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc75c0ffd60 0x7fc75c1a2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50680/0 (socket says 172.21.15.2:50680) 2023-11-16T18:25:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc761e24700 1 -- 172.21.15.2:0/2448996269 learned_addr learned my addr 172.21.15.2:0/2448996269 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc762625700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75c19ca80 0x7fc75c19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc761623700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc75c104400 0x7fc75c1a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc761e24700 1 -- 172.21.15.2:0/2448996269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc75c104400 msgr2=0x7fc75c1a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.833+0000 7fc761e24700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc75c104400 0x7fc75c1a2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.834+0000 7fc761e24700 1 -- 172.21.15.2:0/2448996269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75c19ca80 msgr2=0x7fc75c19cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.834+0000 7fc761e24700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75c19ca80 0x7fc75c19cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.834+0000 7fc761e24700 1 -- 172.21.15.2:0/2448996269 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7580095f0 con 0x7fc75c0ffd60 2023-11-16T18:25:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.834+0000 7fc762625700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75c19ca80 0x7fc75c19cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:25:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.834+0000 7fc761623700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc75c104400 0x7fc75c1a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:25:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.834+0000 7fc761e24700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc75c0ffd60 0x7fc75c1a2a10 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc758009990 tx=0x7fc7580040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.834+0000 7fc752ffd700 1 -- 172.21.15.2:0/2448996269 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758009e60 con 0x7fc75c0ffd60 2023-11-16T18:25:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.834+0000 7fc764088700 1 -- 172.21.15.2:0/2448996269 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc75c10e130 con 0x7fc75c0ffd60 2023-11-16T18:25:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.834+0000 7fc752ffd700 1 -- 172.21.15.2:0/2448996269 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc75800dc60 con 0x7fc75c0ffd60 2023-11-16T18:25:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.834+0000 7fc752ffd700 1 -- 172.21.15.2:0/2448996269 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758020e80 con 0x7fc75c0ffd60 2023-11-16T18:25:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.835+0000 7fc764088700 1 -- 172.21.15.2:0/2448996269 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75c10dfa0 con 0x7fc75c0ffd60 2023-11-16T18:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.839+0000 7fc752ffd700 1 -- 172.21.15.2:0/2448996269 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7fc758020460 con 0x7fc75c0ffd60 2023-11-16T18:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.839+0000 7fc770214b80 1 Processor -- start 2023-11-16T18:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.840+0000 7fc770214b80 1 -- start start 2023-11-16T18:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.840+0000 7fc752ffd700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc7480a13e0 0x7fc7480a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.840+0000 7fc770214b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5574801776b0 0x5574803211e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.840+0000 7fc770214b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321720 0x55748031b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.840+0000 7fc770214b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x55748031bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc770214b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557480221000 con 0x5574801776b0 2023-11-16T18:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc770214b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557480273840 con 0x557480321720 2023-11-16T18:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc770214b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55748026ccd0 con 0x557480321b90 2023-11-16T18:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc762625700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc7480a13e0 0x7fc7480a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc752ffd700 1 -- 172.21.15.2:0/2448996269 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc7580d1170 con 0x7fc75c0ffd60 2023-11-16T18:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc762625700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x55748031bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc762625700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x55748031bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:42218/0 (socket says 172.21.15.2:42218) 2023-11-16T18:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc761e24700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5574801776b0 0x5574803211e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc762625700 1 -- 172.21.15.2:0/1673500010 learned_addr learned my addr 172.21.15.2:0/1673500010 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc761623700 1 --2- 172.21.15.2:0/1673500010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321720 0x55748031b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc762625700 1 -- 172.21.15.2:0/1673500010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321720 msgr2=0x55748031b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc762625700 1 --2- 172.21.15.2:0/1673500010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321720 0x55748031b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc762625700 1 -- 172.21.15.2:0/1673500010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5574801776b0 msgr2=0x5574803211e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc762625700 1 --2- 172.21.15.2:0/1673500010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5574801776b0 0x5574803211e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.841+0000 7fc762625700 1 -- 172.21.15.2:0/1673500010 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574800ee5d0 con 0x557480321b90 2023-11-16T18:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.842+0000 7fc762625700 1 --2- 172.21.15.2:0/1673500010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x55748031bb00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc754007a80 tx=0x7fc754007d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc5e837e509f9312 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.842+0000 7fc747fff700 1 -- 172.21.15.2:0/1673500010 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75400e070 con 0x557480321b90 2023-11-16T18:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.842+0000 7fc747fff700 1 -- 172.21.15.2:0/1673500010 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc75400f040 con 0x557480321b90 2023-11-16T18:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.843+0000 7fc770214b80 1 -- 172.21.15.2:0/1673500010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 msgr2=0x55748031bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.843+0000 7fc762625700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc7480a13e0 0x7fc7480a3890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc75401e000 tx=0x7fc75401b000 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.843+0000 7fc770214b80 1 --2- 172.21.15.2:0/1673500010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x55748031bb00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc754007a80 tx=0x7fc754007d90 comp rx=0 tx=0).stop 2023-11-16T18:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.843+0000 7fc770214b80 1 -- 172.21.15.2:0/1673500010 shutdown_connections 2023-11-16T18:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.843+0000 7fc770214b80 1 --2- 172.21.15.2:0/1673500010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x55748031bb00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.843+0000 7fc770214b80 1 --2- 172.21.15.2:0/1673500010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5574801776b0 0x5574803211e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.843+0000 7fc770214b80 1 --2- 172.21.15.2:0/1673500010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321720 0x55748031b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.843+0000 7fc770214b80 1 -- 172.21.15.2:0/1673500010 >> 172.21.15.2:0/1673500010 conn(0x557480310f50 msgr2=0x557480316270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.843+0000 7fc770214b80 1 -- 172.21.15.2:0/1673500010 shutdown_connections 2023-11-16T18:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.843+0000 7fc770214b80 1 -- 172.21.15.2:0/1673500010 wait complete. 2023-11-16T18:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.844+0000 7fc770214b80 1 -- start start 2023-11-16T18:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.844+0000 7fc770214b80 1 Processor -- start 2023-11-16T18:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.844+0000 7fc770214b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574801776b0 0x557480327a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.844+0000 7fc770214b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 0x557480324c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.844+0000 7fc770214b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x557480325250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.844+0000 7fc770214b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55748020e880 con 0x557480321720 2023-11-16T18:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.844+0000 7fc770214b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55748020e520 con 0x5574801776b0 2023-11-16T18:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.844+0000 7fc770214b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574800f08c0 con 0x557480321b90 2023-11-16T18:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.844+0000 7fc770214b80 10 client.0 _ll_register_callbacks cb 0x55748022c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.844+0000 7fc761623700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 0x557480324c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.844+0000 7fc761623700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 0x557480324c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50696/0 (socket says 172.21.15.2:50696) 2023-11-16T18:25:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.844+0000 7fc761623700 1 -- 172.21.15.2:0/4209506205 learned_addr learned my addr 172.21.15.2:0/4209506205 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.844+0000 7fc761e24700 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574801776b0 0x557480327a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.845+0000 7fc762625700 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x557480325250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.845+0000 7fc761e24700 1 -- 172.21.15.2:0/4209506205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 msgr2=0x557480325250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.845+0000 7fc761e24700 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x557480325250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.845+0000 7fc761e24700 1 -- 172.21.15.2:0/4209506205 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 msgr2=0x557480324c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.845+0000 7fc761e24700 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 0x557480324c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.845+0000 7fc761e24700 1 -- 172.21.15.2:0/4209506205 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7540076b0 con 0x5574801776b0 2023-11-16T18:25:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.845+0000 7fc761623700 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 0x557480324c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.845+0000 7fc761e24700 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574801776b0 0x557480327a50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc7580cb1b0 tx=0x7fc7580209b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=333495a7b5c8f263 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.845+0000 7fc7457fa700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:25:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.845+0000 7fc7457fa700 1 -- 172.21.15.2:0/4209506205 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75809a450 con 0x5574801776b0 2023-11-16T18:25:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.845+0000 7fc7457fa700 1 -- 172.21.15.2:0/4209506205 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc75809aaf0 con 0x5574801776b0 2023-11-16T18:25:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.845+0000 7fc770214b80 1 -- 172.21.15.2:0/4209506205 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55748030e710 con 0x5574801776b0 2023-11-16T18:25:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.846+0000 7fc7457fa700 1 -- 172.21.15.2:0/4209506205 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc75809eb20 con 0x5574801776b0 2023-11-16T18:25:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.846+0000 7fc770214b80 10 client.25394 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.846+0000 7fc770214b80 1 -- 172.21.15.2:0/4209506205 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x557480233130 con 0x5574801776b0 2023-11-16T18:25:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.846+0000 7fc7457fa700 1 -- 172.21.15.2:0/4209506205 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55748030e710 con 0x5574801776b0 2023-11-16T18:25:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.846+0000 7fc7457fa700 1 -- 172.21.15.2:0/4209506205 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc7580cdbc0 con 0x5574801776b0 2023-11-16T18:25:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.846+0000 7fc770214b80 10 client.25394 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.846+0000 7fc770214b80 10 client.25394 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.846+0000 7fc770214b80 20 client.25394 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.846+0000 7fc770214b80 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557480345d50 0x557480348160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.846+0000 7fc770214b80 4 client.25394 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:25:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.846+0000 7fc770214b80 1 -- 172.21.15.2:0/4209506205 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55748022f040 con 0x557480345d50 2023-11-16T18:25:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.846+0000 7fc761623700 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557480345d50 0x557480348160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.847+0000 7fc7457fa700 10 client.25394 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.847+0000 7fc761623700 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557480345d50 0x557480348160 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c85d4b73a8bcfa4 server_cookie=1d2584299f7a11b1 in_seq=0 out_seq=0 2023-11-16T18:25:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc7457fa700 1 -- 172.21.15.2:0/4209506205 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55748022f040 con 0x557480345d50 2023-11-16T18:25:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc7457fa700 10 client.25394 handle_command_reply: tid=0 2023-11-16T18:25:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc770214b80 1 client.25394 shutdown 2023-11-16T18:25:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc770214b80 1 -- 172.21.15.2:0/4209506205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574801776b0 msgr2=0x557480327a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc770214b80 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574801776b0 0x557480327a50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc7580cb1b0 tx=0x7fc7580209b0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc770214b80 1 -- 172.21.15.2:0/4209506205 shutdown_connections 2023-11-16T18:25:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc770214b80 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557480345d50 0x557480348160 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc770214b80 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x557480325250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc770214b80 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574801776b0 0x557480327a50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc770214b80 1 --2- 172.21.15.2:0/4209506205 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 0x557480324c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc770214b80 1 -- 172.21.15.2:0/4209506205 >> 172.21.15.2:0/4209506205 conn(0x557480310f50 msgr2=0x5574803157e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc7457fa700 0 client.25394 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc770214b80 1 -- 172.21.15.2:0/4209506205 shutdown_connections 2023-11-16T18:25:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc770214b80 1 -- 172.21.15.2:0/4209506205 wait complete. 2023-11-16T18:25:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.848+0000 7fc770214b80 20 client.25394 trim_cache size 0 max 16384 2023-11-16T18:25:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.877+0000 7fc770214b80 1 Processor -- start 2023-11-16T18:25:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.877+0000 7fc770214b80 1 -- start start 2023-11-16T18:25:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.877+0000 7fc770214b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5574801776b0 0x55748033d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.877+0000 7fc770214b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 0x557480315f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.877+0000 7fc770214b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321b90 0x557480327580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.877+0000 7fc770214b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557480313040 con 0x557480321720 2023-11-16T18:25:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.877+0000 7fc770214b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557480326120 con 0x557480321b90 2023-11-16T18:25:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.877+0000 7fc770214b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574803231e0 con 0x5574801776b0 2023-11-16T18:25:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.877+0000 7fc761623700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 0x557480315f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.877+0000 7fc761623700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 0x557480315f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50702/0 (socket says 172.21.15.2:50702) 2023-11-16T18:25:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.877+0000 7fc761623700 1 -- 172.21.15.2:0/2097955720 learned_addr learned my addr 172.21.15.2:0/2097955720 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.877+0000 7fc761e24700 1 --2- 172.21.15.2:0/2097955720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5574801776b0 0x55748033d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.877+0000 7fc762625700 1 --2- 172.21.15.2:0/2097955720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321b90 0x557480327580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.878+0000 7fc761623700 1 -- 172.21.15.2:0/2097955720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5574801776b0 msgr2=0x55748033d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.878+0000 7fc761623700 1 --2- 172.21.15.2:0/2097955720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5574801776b0 0x55748033d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.878+0000 7fc761623700 1 -- 172.21.15.2:0/2097955720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321b90 msgr2=0x557480327580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.878+0000 7fc761623700 1 --2- 172.21.15.2:0/2097955720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321b90 0x557480327580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.878+0000 7fc761623700 1 -- 172.21.15.2:0/2097955720 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557480232e80 con 0x557480321720 2023-11-16T18:25:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.878+0000 7fc761e24700 1 --2- 172.21.15.2:0/2097955720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5574801776b0 0x55748033d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.878+0000 7fc762625700 1 --2- 172.21.15.2:0/2097955720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321b90 0x557480327580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:25:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.878+0000 7fc761623700 1 --2- 172.21.15.2:0/2097955720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 0x557480315f60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc74c003cc0 tx=0x7fc74c004220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925d186af26a319b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.878+0000 7fc750ff9700 1 -- 172.21.15.2:0/2097955720 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c017040 con 0x557480321720 2023-11-16T18:25:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.878+0000 7fc750ff9700 1 -- 172.21.15.2:0/2097955720 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc74c009a80 con 0x557480321720 2023-11-16T18:25:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.878+0000 7fc750ff9700 1 -- 172.21.15.2:0/2097955720 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c012a40 con 0x557480321720 2023-11-16T18:25:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.879+0000 7fc770214b80 1 -- 172.21.15.2:0/2097955720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 msgr2=0x557480315f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.879+0000 7fc770214b80 1 --2- 172.21.15.2:0/2097955720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 0x557480315f60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc74c003cc0 tx=0x7fc74c004220 comp rx=0 tx=0).stop 2023-11-16T18:25:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.879+0000 7fc770214b80 1 -- 172.21.15.2:0/2097955720 shutdown_connections 2023-11-16T18:25:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.879+0000 7fc770214b80 1 --2- 172.21.15.2:0/2097955720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5574801776b0 0x55748033d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.879+0000 7fc770214b80 1 --2- 172.21.15.2:0/2097955720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321b90 0x557480327580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.879+0000 7fc770214b80 1 --2- 172.21.15.2:0/2097955720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557480321720 0x557480315f60 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.879+0000 7fc770214b80 1 -- 172.21.15.2:0/2097955720 >> 172.21.15.2:0/2097955720 conn(0x557480310f50 msgr2=0x557480347d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.879+0000 7fc770214b80 1 -- 172.21.15.2:0/2097955720 shutdown_connections 2023-11-16T18:25:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.879+0000 7fc770214b80 1 -- 172.21.15.2:0/2097955720 wait complete. 2023-11-16T18:25:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.879+0000 7fc770214b80 1 -- start start 2023-11-16T18:25:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.880+0000 7fc770214b80 1 Processor -- start 2023-11-16T18:25:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.880+0000 7fc770214b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5574801776b0 0x55748031cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.880+0000 7fc770214b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321720 0x55748031d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.880+0000 7fc770214b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x55748031d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.880+0000 7fc770214b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55748031dc20 con 0x5574801776b0 2023-11-16T18:25:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.880+0000 7fc770214b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55748031dd90 con 0x557480321720 2023-11-16T18:25:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.880+0000 7fc770214b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55748031df50 con 0x557480321b90 2023-11-16T18:25:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.880+0000 7fc770214b80 10 client.0 _ll_register_callbacks cb 0x7fc758096e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.880+0000 7fc761e24700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5574801776b0 0x55748031cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.880+0000 7fc761e24700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5574801776b0 0x55748031cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50708/0 (socket says 172.21.15.2:50708) 2023-11-16T18:25:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.880+0000 7fc761e24700 1 -- 172.21.15.2:0/1781951961 learned_addr learned my addr 172.21.15.2:0/1781951961 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.880+0000 7fc762625700 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x55748031d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc761623700 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321720 0x55748031d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc761e24700 1 -- 172.21.15.2:0/1781951961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 msgr2=0x55748031d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc761e24700 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x55748031d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc761e24700 1 -- 172.21.15.2:0/1781951961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321720 msgr2=0x55748031d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc761e24700 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321720 0x55748031d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc761e24700 1 -- 172.21.15.2:0/1781951961 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557480233130 con 0x5574801776b0 2023-11-16T18:25:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc761623700 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321720 0x55748031d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:25:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc761e24700 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5574801776b0 0x55748031cc60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc75809aa80 tx=0x7fc758003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693e5fb4d702bbdf server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc7457fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc7457fa700 1 -- 172.21.15.2:0/1781951961 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7580cddc0 con 0x5574801776b0 2023-11-16T18:25:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc7457fa700 1 -- 172.21.15.2:0/1781951961 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc75809aaf0 con 0x5574801776b0 2023-11-16T18:25:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc770214b80 1 -- 172.21.15.2:0/1781951961 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc75400f040 con 0x5574801776b0 2023-11-16T18:25:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc7457fa700 1 -- 172.21.15.2:0/1781951961 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc75809a450 con 0x5574801776b0 2023-11-16T18:25:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc770214b80 10 client.15813 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.881+0000 7fc770214b80 1 -- 172.21.15.2:0/1781951961 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7fc74c003870 con 0x5574801776b0 2023-11-16T18:25:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.882+0000 7fc7457fa700 1 -- 172.21.15.2:0/1781951961 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc75400f040 con 0x5574801776b0 2023-11-16T18:25:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.882+0000 7fc7457fa700 1 -- 172.21.15.2:0/1781951961 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc7580cb520 con 0x5574801776b0 2023-11-16T18:25:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.882+0000 7fc770214b80 10 client.15813 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.882+0000 7fc770214b80 10 client.15813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.882+0000 7fc770214b80 20 client.15813 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.882+0000 7fc770214b80 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557480330790 0x557480332ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.882+0000 7fc770214b80 4 client.15813 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:25:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.882+0000 7fc770214b80 1 -- 172.21.15.2:0/1781951961 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55748023b600 con 0x557480330790 2023-11-16T18:25:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.882+0000 7fc761623700 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557480330790 0x557480332ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.883+0000 7fc761623700 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557480330790 0x557480332ba0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32e94cbb19b49693 server_cookie=6a010c222d849cce in_seq=0 out_seq=0 2023-11-16T18:25:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.883+0000 7fc7457fa700 10 client.15813 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.886+0000 7fc7457fa700 1 -- 172.21.15.2:0/1781951961 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55748023b600 con 0x557480330790 2023-11-16T18:25:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.886+0000 7fc7457fa700 10 client.15813 handle_command_reply: tid=0 2023-11-16T18:25:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.887+0000 7fc770214b80 1 client.15813 shutdown 2023-11-16T18:25:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.887+0000 7fc770214b80 1 -- 172.21.15.2:0/1781951961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5574801776b0 msgr2=0x55748031cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.887+0000 7fc770214b80 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5574801776b0 0x55748031cc60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc75809aa80 tx=0x7fc758003d60 comp rx=0 tx=0).stop 2023-11-16T18:25:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.887+0000 7fc770214b80 1 -- 172.21.15.2:0/1781951961 shutdown_connections 2023-11-16T18:25:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.887+0000 7fc770214b80 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557480330790 0x557480332ba0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.887+0000 7fc770214b80 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557480321b90 0x55748031d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.887+0000 7fc770214b80 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5574801776b0 0x55748031cc60 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.887+0000 7fc770214b80 1 --2- 172.21.15.2:0/1781951961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557480321720 0x55748031d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.887+0000 7fc770214b80 1 -- 172.21.15.2:0/1781951961 >> 172.21.15.2:0/1781951961 conn(0x557480310f50 msgr2=0x55748031eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.887+0000 7fc7457fa700 0 client.15813 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.887+0000 7fc770214b80 1 -- 172.21.15.2:0/1781951961 shutdown_connections 2023-11-16T18:25:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.887+0000 7fc770214b80 1 -- 172.21.15.2:0/1781951961 wait complete. 2023-11-16T18:25:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.887+0000 7fc770214b80 20 client.15813 trim_cache size 0 max 16384 2023-11-16T18:25:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:25:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:25:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:25:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:25:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.888+0000 7fc764088700 1 -- 172.21.15.2:0/2448996269 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc7480a13e0 msgr2=0x7fc7480a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.888+0000 7fc764088700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc7480a13e0 0x7fc7480a3890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc75401e000 tx=0x7fc75401b000 comp rx=0 tx=0).stop 2023-11-16T18:25:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.888+0000 7fc764088700 1 -- 172.21.15.2:0/2448996269 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc75c0ffd60 msgr2=0x7fc75c1a2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.888+0000 7fc764088700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc75c0ffd60 0x7fc75c1a2a10 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc758009990 tx=0x7fc7580040e0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.888+0000 7fc764088700 1 -- 172.21.15.2:0/2448996269 shutdown_connections 2023-11-16T18:25:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.888+0000 7fc764088700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc7480a13e0 0x7fc7480a3890 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.888+0000 7fc764088700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75c19ca80 0x7fc75c19cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.888+0000 7fc764088700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc75c0ffd60 0x7fc75c1a2a10 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.888+0000 7fc764088700 1 --2- 172.21.15.2:0/2448996269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc75c104400 0x7fc75c1a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.888+0000 7fc764088700 1 -- 172.21.15.2:0/2448996269 >> 172.21.15.2:0/2448996269 conn(0x7fc75c0763a0 msgr2=0x7fc75c0fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.888+0000 7fc764088700 1 -- 172.21.15.2:0/2448996269 shutdown_connections 2023-11-16T18:25:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:19.888+0000 7fc764088700 1 -- 172.21.15.2:0/2448996269 wait complete. 2023-11-16T18:25:20.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:20 smithi188 ceph-mon[100639]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2023-11-16T18:25:20.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:20 smithi002 ceph-mon[98807]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2023-11-16T18:25:20.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:20 smithi040 ceph-mon[100251]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2023-11-16T18:25:22.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:22 smithi188 ceph-mon[100639]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-11-16T18:25:22.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:22 smithi002 ceph-mon[98807]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-11-16T18:25:22.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:22 smithi040 ceph-mon[100251]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-11-16T18:25:24.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:24 smithi188 ceph-mon[100639]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2023-11-16T18:25:24.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:24 smithi002 ceph-mon[98807]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2023-11-16T18:25:24.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:24 smithi040 ceph-mon[100251]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2023-11-16T18:25:26.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:26 smithi188 ceph-mon[100639]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-11-16T18:25:26.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:26 smithi002 ceph-mon[98807]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-11-16T18:25:26.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:26 smithi040 ceph-mon[100251]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-11-16T18:25:28.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:28 smithi188 ceph-mon[100639]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2023-11-16T18:25:28.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:28 smithi002 ceph-mon[98807]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2023-11-16T18:25:28.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:28 smithi040 ceph-mon[100251]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2023-11-16T18:25:30.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:30 smithi188 ceph-mon[100639]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2023-11-16T18:25:30.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:30 smithi002 ceph-mon[98807]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2023-11-16T18:25:30.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:30 smithi040 ceph-mon[100251]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2023-11-16T18:25:32.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:32 smithi188 ceph-mon[100639]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T18:25:32.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:32 smithi002 ceph-mon[98807]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T18:25:32.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:32 smithi040 ceph-mon[100251]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T18:25:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.015+0000 7fd2504ef700 1 -- 172.21.15.2:0/1724944909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2480ffd60 msgr2=0x7fd248103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.015+0000 7fd2504ef700 1 --2- 172.21.15.2:0/1724944909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2480ffd60 0x7fd248103d20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd23c0099c0 tx=0x7fd23c009cd0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.015+0000 7fd2504ef700 1 -- 172.21.15.2:0/1724944909 shutdown_connections 2023-11-16T18:25:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.015+0000 7fd2504ef700 1 --2- 172.21.15.2:0/1724944909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2480ff3c0 0x7fd2480ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.015+0000 7fd2504ef700 1 --2- 172.21.15.2:0/1724944909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd248104400 0x7fd24810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.015+0000 7fd2504ef700 1 --2- 172.21.15.2:0/1724944909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2480ffd60 0x7fd248103d20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.015+0000 7fd2504ef700 1 -- 172.21.15.2:0/1724944909 >> 172.21.15.2:0/1724944909 conn(0x7fd2480763a0 msgr2=0x7fd2480767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.016+0000 7fd2504ef700 1 -- 172.21.15.2:0/1724944909 shutdown_connections 2023-11-16T18:25:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.016+0000 7fd2504ef700 1 -- 172.21.15.2:0/1724944909 wait complete. 2023-11-16T18:25:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.016+0000 7fd2504ef700 1 Processor -- start 2023-11-16T18:25:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.016+0000 7fd2504ef700 1 -- start start 2023-11-16T18:25:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.016+0000 7fd2504ef700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2480ff3c0 0x7fd248196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.016+0000 7fd2504ef700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2480ffd60 0x7fd248196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.016+0000 7fd2504ef700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd248104400 0x7fd24819ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd2504ef700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24819b320 con 0x7fd2480ff3c0 2023-11-16T18:25:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd2504ef700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24819b490 con 0x7fd2480ffd60 2023-11-16T18:25:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd2504ef700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24819b650 con 0x7fd248104400 2023-11-16T18:25:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd24e28b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2480ff3c0 0x7fd248196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd24e28b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2480ff3c0 0x7fd248196340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50722/0 (socket says 172.21.15.2:50722) 2023-11-16T18:25:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd24e28b700 1 -- 172.21.15.2:0/3101803947 learned_addr learned my addr 172.21.15.2:0/3101803947 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd24da8a700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2480ffd60 0x7fd248196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd24ea8c700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd248104400 0x7fd24819ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd24e28b700 1 -- 172.21.15.2:0/3101803947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd248104400 msgr2=0x7fd24819ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd24e28b700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd248104400 0x7fd24819ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd24e28b700 1 -- 172.21.15.2:0/3101803947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2480ffd60 msgr2=0x7fd248196880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd24e28b700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2480ffd60 0x7fd248196880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd24e28b700 1 -- 172.21.15.2:0/3101803947 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23c0095f0 con 0x7fd2480ff3c0 2023-11-16T18:25:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd24e28b700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2480ff3c0 0x7fd248196340 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd24400c9b0 tx=0x7fd24400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd23b7fe700 1 -- 172.21.15.2:0/3101803947 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd244007b00 con 0x7fd2480ff3c0 2023-11-16T18:25:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd2504ef700 1 -- 172.21.15.2:0/3101803947 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24810f180 con 0x7fd2480ff3c0 2023-11-16T18:25:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd23b7fe700 1 -- 172.21.15.2:0/3101803947 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd244007c60 con 0x7fd2480ff3c0 2023-11-16T18:25:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.017+0000 7fd23b7fe700 1 -- 172.21.15.2:0/3101803947 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd244018660 con 0x7fd2480ff3c0 2023-11-16T18:25:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.018+0000 7fd2504ef700 1 -- 172.21.15.2:0/3101803947 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd24810eff0 con 0x7fd2480ff3c0 2023-11-16T18:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.020+0000 7fd23b7fe700 1 -- 172.21.15.2:0/3101803947 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7fd2440187c0 con 0x7fd2480ff3c0 2023-11-16T18:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.021+0000 7fd23b7fe700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd2340a13e0 0x7fd2340a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.021+0000 7fd24da8a700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd2340a13e0 0x7fd2340a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.021+0000 7fd23b7fe700 1 -- 172.21.15.2:0/3101803947 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd2440cd220 con 0x7fd2480ff3c0 2023-11-16T18:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.021+0000 7fd24da8a700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd2340a13e0 0x7fd2340a3890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009f50 tx=0x7fd23c0051c0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.022+0000 7fd25c67bb80 1 Processor -- start 2023-11-16T18:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.022+0000 7fd25c67bb80 1 -- start start 2023-11-16T18:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.022+0000 7fd25c67bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 0x55c5ee7001e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.022+0000 7fd25c67bb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700720 0x55c5ee6fa560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.022+0000 7fd25c67bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700b90 0x55c5ee6fab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.022+0000 7fd25c67bb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ee600000 con 0x55c5ee700720 2023-11-16T18:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.022+0000 7fd25c67bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ee652840 con 0x55c5ee700b90 2023-11-16T18:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.022+0000 7fd25c67bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ee64bcd0 con 0x55c5ee5566b0 2023-11-16T18:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.022+0000 7fd24da8a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700720 0x55c5ee6fa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.022+0000 7fd24da8a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700720 0x55c5ee6fa560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50732/0 (socket says 172.21.15.2:50732) 2023-11-16T18:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.022+0000 7fd24da8a700 1 -- 172.21.15.2:0/3605579445 learned_addr learned my addr 172.21.15.2:0/3605579445 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.022+0000 7fd24e28b700 1 --2- 172.21.15.2:0/3605579445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 0x55c5ee7001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.022+0000 7fd24ea8c700 1 --2- 172.21.15.2:0/3605579445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700b90 0x55c5ee6fab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.023+0000 7fd24da8a700 1 -- 172.21.15.2:0/3605579445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 msgr2=0x55c5ee7001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.023+0000 7fd24da8a700 1 --2- 172.21.15.2:0/3605579445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 0x55c5ee7001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.023+0000 7fd24da8a700 1 -- 172.21.15.2:0/3605579445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700b90 msgr2=0x55c5ee6fab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.023+0000 7fd24da8a700 1 --2- 172.21.15.2:0/3605579445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700b90 0x55c5ee6fab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.023+0000 7fd24da8a700 1 -- 172.21.15.2:0/3605579445 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5ee4cd5d0 con 0x55c5ee700720 2023-11-16T18:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.023+0000 7fd24da8a700 1 --2- 172.21.15.2:0/3605579445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700720 0x55c5ee6fa560 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd23c0099c0 tx=0x7fd23c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2f0be70eb25bf6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:33.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.023+0000 7fd238ff9700 1 -- 172.21.15.2:0/3605579445 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c00bdf0 con 0x55c5ee700720 2023-11-16T18:25:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.023+0000 7fd238ff9700 1 -- 172.21.15.2:0/3605579445 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd23c01e040 con 0x55c5ee700720 2023-11-16T18:25:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.023+0000 7fd238ff9700 1 -- 172.21.15.2:0/3605579445 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c00f7e0 con 0x55c5ee700720 2023-11-16T18:25:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.024+0000 7fd25c67bb80 1 -- 172.21.15.2:0/3605579445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700720 msgr2=0x55c5ee6fa560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.024+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/3605579445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700720 0x55c5ee6fa560 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd23c0099c0 tx=0x7fd23c020000 comp rx=0 tx=0).stop 2023-11-16T18:25:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.024+0000 7fd25c67bb80 1 -- 172.21.15.2:0/3605579445 shutdown_connections 2023-11-16T18:25:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.024+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/3605579445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 0x55c5ee7001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.024+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/3605579445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700b90 0x55c5ee6fab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.024+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/3605579445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700720 0x55c5ee6fa560 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.024+0000 7fd25c67bb80 1 -- 172.21.15.2:0/3605579445 >> 172.21.15.2:0/3605579445 conn(0x55c5ee6eff50 msgr2=0x55c5ee6f5270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.024+0000 7fd25c67bb80 1 -- 172.21.15.2:0/3605579445 shutdown_connections 2023-11-16T18:25:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.024+0000 7fd25c67bb80 1 -- 172.21.15.2:0/3605579445 wait complete. 2023-11-16T18:25:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.024+0000 7fd25c67bb80 1 -- start start 2023-11-16T18:25:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd25c67bb80 1 Processor -- start 2023-11-16T18:25:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd25c67bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 0x55c5ee706a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd25c67bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700720 0x55c5ee703cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd25c67bb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee704340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd25c67bb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ee5ed880 con 0x55c5ee700b90 2023-11-16T18:25:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd25c67bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ee5ed520 con 0x55c5ee700720 2023-11-16T18:25:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd25c67bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ee4cf8c0 con 0x55c5ee5566b0 2023-11-16T18:25:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd25c67bb80 10 client.0 _ll_register_callbacks cb 0x55c5ee60b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd24ea8c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee704340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd24ea8c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee704340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50740/0 (socket says 172.21.15.2:50740) 2023-11-16T18:25:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd24ea8c700 1 -- 172.21.15.2:0/1514796646 learned_addr learned my addr 172.21.15.2:0/1514796646 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd24da8a700 1 --2- 172.21.15.2:0/1514796646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700720 0x55c5ee703cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd24e28b700 1 --2- 172.21.15.2:0/1514796646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 0x55c5ee706a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd24ea8c700 1 -- 172.21.15.2:0/1514796646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 msgr2=0x55c5ee706a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd24ea8c700 1 --2- 172.21.15.2:0/1514796646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 0x55c5ee706a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd24ea8c700 1 -- 172.21.15.2:0/1514796646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700720 msgr2=0x55c5ee703cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd24ea8c700 1 --2- 172.21.15.2:0/1514796646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700720 0x55c5ee703cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.025+0000 7fd24ea8c700 1 -- 172.21.15.2:0/1514796646 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd24810c810 con 0x55c5ee700b90 2023-11-16T18:25:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd24ea8c700 1 --2- 172.21.15.2:0/1514796646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee704340 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd240007af0 tx=0x7fd240007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a45ecaaa052a981 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd231ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd231ffb700 1 -- 172.21.15.2:0/1514796646 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240012b40 con 0x55c5ee700b90 2023-11-16T18:25:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd231ffb700 1 -- 172.21.15.2:0/1514796646 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd240012ca0 con 0x55c5ee700b90 2023-11-16T18:25:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd25c67bb80 1 -- 172.21.15.2:0/1514796646 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c5ee6ed710 con 0x55c5ee700b90 2023-11-16T18:25:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd231ffb700 1 -- 172.21.15.2:0/1514796646 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd240012e60 con 0x55c5ee700b90 2023-11-16T18:25:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd25c67bb80 10 client.15825 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd25c67bb80 1 -- 172.21.15.2:0/1514796646 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c5ee612130 con 0x55c5ee700b90 2023-11-16T18:25:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd231ffb700 1 -- 172.21.15.2:0/1514796646 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c5ee6ed710 con 0x55c5ee700b90 2023-11-16T18:25:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd231ffb700 1 -- 172.21.15.2:0/1514796646 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd240014040 con 0x55c5ee700b90 2023-11-16T18:25:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd25c67bb80 10 client.15825 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd25c67bb80 10 client.15825 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd25c67bb80 20 client.15825 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/1514796646 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c5ee714070 0x55c5ee716480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.026+0000 7fd25c67bb80 4 client.15825 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:25:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.027+0000 7fd25c67bb80 1 -- 172.21.15.2:0/1514796646 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c5ee60e040 con 0x55c5ee714070 2023-11-16T18:25:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.027+0000 7fd24e28b700 1 --2- 172.21.15.2:0/1514796646 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c5ee714070 0x55c5ee716480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.027+0000 7fd231ffb700 10 client.15825 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.027+0000 7fd24e28b700 1 --2- 172.21.15.2:0/1514796646 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c5ee714070 0x55c5ee716480 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc328b6383f3a79e server_cookie=3d3c36af83645f57 in_seq=0 out_seq=0 2023-11-16T18:25:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd231ffb700 1 -- 172.21.15.2:0/1514796646 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c5ee60e040 con 0x55c5ee714070 2023-11-16T18:25:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd231ffb700 10 client.15825 handle_command_reply: tid=0 2023-11-16T18:25:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd25c67bb80 1 client.15825 shutdown 2023-11-16T18:25:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd25c67bb80 1 -- 172.21.15.2:0/1514796646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 msgr2=0x55c5ee704340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/1514796646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee704340 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd240007af0 tx=0x7fd240007e00 comp rx=0 tx=0).stop 2023-11-16T18:25:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd25c67bb80 1 -- 172.21.15.2:0/1514796646 shutdown_connections 2023-11-16T18:25:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/1514796646 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c5ee714070 0x55c5ee716480 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/1514796646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 0x55c5ee706a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd231ffb700 0 client.15825 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/1514796646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee704340 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/1514796646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700720 0x55c5ee703cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd25c67bb80 1 -- 172.21.15.2:0/1514796646 >> 172.21.15.2:0/1514796646 conn(0x55c5ee6eff50 msgr2=0x55c5ee6f47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd25c67bb80 1 -- 172.21.15.2:0/1514796646 shutdown_connections 2023-11-16T18:25:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.028+0000 7fd25c67bb80 1 -- 172.21.15.2:0/1514796646 wait complete. 2023-11-16T18:25:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.029+0000 7fd25c67bb80 20 client.15825 trim_cache size 0 max 16384 2023-11-16T18:25:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.043+0000 7fd25c67bb80 1 Processor -- start 2023-11-16T18:25:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.043+0000 7fd25c67bb80 1 -- start start 2023-11-16T18:25:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.043+0000 7fd25c67bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 0x55c5ee6f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.043+0000 7fd25c67bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700720 0x55c5ee6fa560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.043+0000 7fd25c67bb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee6faaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.043+0000 7fd25c67bb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ee6f7050 con 0x55c5ee700b90 2023-11-16T18:25:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.043+0000 7fd25c67bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ee6efb70 con 0x55c5ee700720 2023-11-16T18:25:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.043+0000 7fd25c67bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ee6fe970 con 0x55c5ee5566b0 2023-11-16T18:25:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd24ea8c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee6faaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd24ea8c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee6faaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50748/0 (socket says 172.21.15.2:50748) 2023-11-16T18:25:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd24ea8c700 1 -- 172.21.15.2:0/439659629 learned_addr learned my addr 172.21.15.2:0/439659629 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd24e28b700 1 --2- 172.21.15.2:0/439659629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 0x55c5ee6f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd24da8a700 1 --2- 172.21.15.2:0/439659629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700720 0x55c5ee6fa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd24ea8c700 1 -- 172.21.15.2:0/439659629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 msgr2=0x55c5ee6f4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd24ea8c700 1 --2- 172.21.15.2:0/439659629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 0x55c5ee6f4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd24ea8c700 1 -- 172.21.15.2:0/439659629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700720 msgr2=0x55c5ee6fa560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd24ea8c700 1 --2- 172.21.15.2:0/439659629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700720 0x55c5ee6fa560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd24ea8c700 1 -- 172.21.15.2:0/439659629 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5ee6f16f0 con 0x55c5ee700b90 2023-11-16T18:25:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd24ea8c700 1 --2- 172.21.15.2:0/439659629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee6faaa0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd240015580 tx=0x7fd240014320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf5c8ff50eaabea server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd2397fa700 1 -- 172.21.15.2:0/439659629 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240012520 con 0x55c5ee700b90 2023-11-16T18:25:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd2397fa700 1 -- 172.21.15.2:0/439659629 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd24000f040 con 0x55c5ee700b90 2023-11-16T18:25:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.044+0000 7fd2397fa700 1 -- 172.21.15.2:0/439659629 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240015d60 con 0x55c5ee700b90 2023-11-16T18:25:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.045+0000 7fd25c67bb80 1 -- 172.21.15.2:0/439659629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 msgr2=0x55c5ee6faaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.045+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/439659629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee6faaa0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd240015580 tx=0x7fd240014320 comp rx=0 tx=0).stop 2023-11-16T18:25:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.045+0000 7fd25c67bb80 1 -- 172.21.15.2:0/439659629 shutdown_connections 2023-11-16T18:25:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.045+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/439659629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee5566b0 0x55c5ee6f4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.045+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/439659629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee6faaa0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.045+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/439659629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee700720 0x55c5ee6fa560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.045+0000 7fd25c67bb80 1 -- 172.21.15.2:0/439659629 >> 172.21.15.2:0/439659629 conn(0x55c5ee6eff50 msgr2=0x55c5ee6fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.045+0000 7fd25c67bb80 1 -- 172.21.15.2:0/439659629 shutdown_connections 2023-11-16T18:25:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.045+0000 7fd25c67bb80 1 -- 172.21.15.2:0/439659629 wait complete. 2023-11-16T18:25:33.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.045+0000 7fd25c67bb80 1 -- start start 2023-11-16T18:25:33.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.045+0000 7fd25c67bb80 1 Processor -- start 2023-11-16T18:25:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd25c67bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee5566b0 0x55c5ee709050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd25c67bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee700720 0x55c5ee709590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd25c67bb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee709ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd25c67bb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ee766f30 con 0x55c5ee700b90 2023-11-16T18:25:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd25c67bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ee767070 con 0x55c5ee5566b0 2023-11-16T18:25:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd25c67bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ee7671e0 con 0x55c5ee700720 2023-11-16T18:25:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd25c67bb80 10 client.0 _ll_register_callbacks cb 0x7fd240009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd24ea8c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee709ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd24ea8c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee709ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50754/0 (socket says 172.21.15.2:50754) 2023-11-16T18:25:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd24ea8c700 1 -- 172.21.15.2:0/4018131863 learned_addr learned my addr 172.21.15.2:0/4018131863 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd24e28b700 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee5566b0 0x55c5ee709050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd24da8a700 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee700720 0x55c5ee709590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd24ea8c700 1 -- 172.21.15.2:0/4018131863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee700720 msgr2=0x55c5ee709590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd24ea8c700 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee700720 0x55c5ee709590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd24ea8c700 1 -- 172.21.15.2:0/4018131863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee5566b0 msgr2=0x55c5ee709050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd24ea8c700 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee5566b0 0x55c5ee709050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd24ea8c700 1 -- 172.21.15.2:0/4018131863 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5ee612130 con 0x55c5ee700b90 2023-11-16T18:25:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd24da8a700 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee700720 0x55c5ee709590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:25:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd24ea8c700 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee709ad0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd240015300 tx=0x7fd240015970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc5d25135d1a1f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd231ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd231ffb700 1 -- 172.21.15.2:0/4018131863 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240004c60 con 0x55c5ee700b90 2023-11-16T18:25:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd231ffb700 1 -- 172.21.15.2:0/4018131863 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd240004dc0 con 0x55c5ee700b90 2023-11-16T18:25:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.046+0000 7fd25c67bb80 1 -- 172.21.15.2:0/4018131863 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd23c01e040 con 0x55c5ee700b90 2023-11-16T18:25:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.047+0000 7fd231ffb700 1 -- 172.21.15.2:0/4018131863 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd240015a30 con 0x55c5ee700b90 2023-11-16T18:25:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.047+0000 7fd25c67bb80 10 client.15831 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.047+0000 7fd25c67bb80 1 -- 172.21.15.2:0/4018131863 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c5ee479d80 con 0x55c5ee700b90 2023-11-16T18:25:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.047+0000 7fd231ffb700 1 -- 172.21.15.2:0/4018131863 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd23c01e040 con 0x55c5ee700b90 2023-11-16T18:25:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.047+0000 7fd231ffb700 1 -- 172.21.15.2:0/4018131863 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd240007b20 con 0x55c5ee700b90 2023-11-16T18:25:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.047+0000 7fd25c67bb80 10 client.15831 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.047+0000 7fd25c67bb80 10 client.15831 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.047+0000 7fd25c67bb80 20 client.15831 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.047+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c5ee768420 0x55c5ee76a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.047+0000 7fd25c67bb80 4 client.15831 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:25:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.047+0000 7fd25c67bb80 1 -- 172.21.15.2:0/4018131863 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c5ee61a600 con 0x55c5ee768420 2023-11-16T18:25:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.047+0000 7fd24e28b700 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c5ee768420 0x55c5ee76a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.048+0000 7fd24e28b700 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c5ee768420 0x55c5ee76a800 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4ddbfda41490b60 server_cookie=5339796577bd63ce in_seq=0 out_seq=0 2023-11-16T18:25:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.048+0000 7fd231ffb700 10 client.15831 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:33.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd231ffb700 1 -- 172.21.15.2:0/4018131863 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c5ee61a600 con 0x55c5ee768420 2023-11-16T18:25:33.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd231ffb700 10 client.15831 handle_command_reply: tid=0 2023-11-16T18:25:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd25c67bb80 1 client.15831 shutdown 2023-11-16T18:25:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd25c67bb80 1 -- 172.21.15.2:0/4018131863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 msgr2=0x55c5ee709ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee709ad0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd240015300 tx=0x7fd240015970 comp rx=0 tx=0).stop 2023-11-16T18:25:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd25c67bb80 1 -- 172.21.15.2:0/4018131863 shutdown_connections 2023-11-16T18:25:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c5ee768420 0x55c5ee76a800 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c5ee700b90 0x55c5ee709ad0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5ee5566b0 0x55c5ee709050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd25c67bb80 1 --2- 172.21.15.2:0/4018131863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5ee700720 0x55c5ee709590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd231ffb700 0 client.15831 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd25c67bb80 1 -- 172.21.15.2:0/4018131863 >> 172.21.15.2:0/4018131863 conn(0x55c5ee6eff50 msgr2=0x55c5ee700f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd25c67bb80 1 -- 172.21.15.2:0/4018131863 shutdown_connections 2023-11-16T18:25:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd25c67bb80 1 -- 172.21.15.2:0/4018131863 wait complete. 2023-11-16T18:25:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.056+0000 7fd25c67bb80 20 client.15831 trim_cache size 0 max 16384 2023-11-16T18:25:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:25:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:25:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:25:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:25:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.057+0000 7fd2504ef700 1 -- 172.21.15.2:0/3101803947 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd2340a13e0 msgr2=0x7fd2340a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.057+0000 7fd2504ef700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd2340a13e0 0x7fd2340a3890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009f50 tx=0x7fd23c0051c0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.057+0000 7fd2504ef700 1 -- 172.21.15.2:0/3101803947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2480ff3c0 msgr2=0x7fd248196340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.057+0000 7fd2504ef700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2480ff3c0 0x7fd248196340 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd24400c9b0 tx=0x7fd24400ccc0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.057+0000 7fd2504ef700 1 -- 172.21.15.2:0/3101803947 shutdown_connections 2023-11-16T18:25:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.057+0000 7fd2504ef700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd2340a13e0 0x7fd2340a3890 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.057+0000 7fd2504ef700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd248104400 0x7fd24819ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.057+0000 7fd2504ef700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2480ff3c0 0x7fd248196340 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.057+0000 7fd2504ef700 1 --2- 172.21.15.2:0/3101803947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2480ffd60 0x7fd248196880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.057+0000 7fd2504ef700 1 -- 172.21.15.2:0/3101803947 >> 172.21.15.2:0/3101803947 conn(0x7fd2480763a0 msgr2=0x7fd248068050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.057+0000 7fd2504ef700 1 -- 172.21.15.2:0/3101803947 shutdown_connections 2023-11-16T18:25:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:33.057+0000 7fd2504ef700 1 -- 172.21.15.2:0/3101803947 wait complete. 2023-11-16T18:25:34.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:34 smithi188 ceph-mon[100639]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2023-11-16T18:25:34.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:34 smithi002 ceph-mon[98807]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2023-11-16T18:25:34.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:34 smithi040 ceph-mon[100251]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2023-11-16T18:25:35.420 INFO:tasks.workunit.client.0.smithi002.stderr:Note: switching to '1c9565146c8647d4e00eb6c7ae626dc192680e7b'. 2023-11-16T18:25:35.420 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T18:25:35.420 INFO:tasks.workunit.client.0.smithi002.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-11-16T18:25:35.420 INFO:tasks.workunit.client.0.smithi002.stderr:changes and commit them, and you can discard any commits you make in this 2023-11-16T18:25:35.420 INFO:tasks.workunit.client.0.smithi002.stderr:state without impacting any branches by switching back to a branch. 2023-11-16T18:25:35.420 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T18:25:35.420 INFO:tasks.workunit.client.0.smithi002.stderr:If you want to create a new branch to retain commits you create, you may 2023-11-16T18:25:35.420 INFO:tasks.workunit.client.0.smithi002.stderr:do so (now or later) by using -c with the switch command. Example: 2023-11-16T18:25:35.420 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T18:25:35.420 INFO:tasks.workunit.client.0.smithi002.stderr: git switch -c 2023-11-16T18:25:35.420 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T18:25:35.421 INFO:tasks.workunit.client.0.smithi002.stderr:Or undo this operation with: 2023-11-16T18:25:35.421 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T18:25:35.421 INFO:tasks.workunit.client.0.smithi002.stderr: git switch - 2023-11-16T18:25:35.421 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T18:25:35.421 INFO:tasks.workunit.client.0.smithi002.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-11-16T18:25:35.421 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T18:25:35.421 INFO:tasks.workunit.client.0.smithi002.stderr:HEAD is now at 1c9565146c8 Other1 2023-11-16T18:25:35.423 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-11-16T18:25:35.451 INFO:tasks.workunit.client.0.smithi002.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-11-16T18:25:35.453 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-16T18:25:35.453 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-11-16T18:25:35.980 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-11-16T18:25:36.034 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-11-16T18:25:36.073 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-16T18:25:36.074 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-16T18:25:36.074 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-11-16T18:25:36.112 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-16T18:25:36.115 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:25:36.115 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-11-16T18:25:36.174 INFO:tasks.workunit:Running workunits matching fs/misc on client.0... 2023-11-16T18:25:36.175 INFO:tasks.workunit:Running workunit fs/misc/acl.sh... 2023-11-16T18:25:36.175 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/acl.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh 2023-11-16T18:25:36.238 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2023-11-16T18:25:36.238 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir -p testdir 2023-11-16T18:25:36.240 INFO:tasks.workunit.client.0.smithi002.stderr:+ cd testdir 2023-11-16T18:25:36.240 INFO:tasks.workunit.client.0.smithi002.stderr:+ set +e 2023-11-16T18:25:36.240 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfacl -d -m u:nobody:rw . 2023-11-16T18:25:36.254 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T18:25:36.254 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2023-11-16T18:25:36.254 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=0 2023-11-16T18:25:36.254 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 0 -lt 100 ']' 2023-11-16T18:25:36.255 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 0 + 1 2023-11-16T18:25:36.255 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=1 2023-11-16T18:25:36.255 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:36.257 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:36.258 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:36.262 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:36.262 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:36.262 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:36.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:36 smithi188 ceph-mon[100639]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2023-11-16T18:25:36.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:36 smithi002 ceph-mon[98807]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2023-11-16T18:25:36.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:36 smithi040 ceph-mon[100251]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2023-11-16T18:25:37.009 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:37.009 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:37.172 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:37.172 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:37.186 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:37.187 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:37.187 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 1 -lt 100 ']' 2023-11-16T18:25:37.187 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 1 + 1 2023-11-16T18:25:37.217 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=2 2023-11-16T18:25:37.217 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:37.263 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:37.264 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:37.264 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:37.264 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:37.265 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:37.734 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:37.734 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:37.839 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:37.839 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:37.853 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:37.853 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:37.853 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -lt 100 ']' 2023-11-16T18:25:37.853 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 2 + 1 2023-11-16T18:25:37.884 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=3 2023-11-16T18:25:37.884 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:37.909 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:37.909 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:37.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:37.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:37.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:38.234 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:38.234 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:38.347 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:38.347 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:38.355 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:38.355 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:38.355 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 3 -lt 100 ']' 2023-11-16T18:25:38.355 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 3 + 1 2023-11-16T18:25:38.384 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=4 2023-11-16T18:25:38.384 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:38.409 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:38.409 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:38.410 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:38.410 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:38.410 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:38.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:38 smithi188 ceph-mon[100639]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2023-11-16T18:25:38.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:38 smithi002 ceph-mon[98807]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2023-11-16T18:25:38.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:38 smithi040 ceph-mon[100251]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2023-11-16T18:25:38.742 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:38.742 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:38.872 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:38.872 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:38.878 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:38.879 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:38.879 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 4 -lt 100 ']' 2023-11-16T18:25:38.879 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 4 + 1 2023-11-16T18:25:38.901 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=5 2023-11-16T18:25:38.901 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:38.926 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:38.926 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:38.927 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:38.927 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:38.927 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:39.242 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:39.242 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:39.397 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:39.397 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:39.430 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:39.430 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:39.430 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 5 -lt 100 ']' 2023-11-16T18:25:39.430 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 5 + 1 2023-11-16T18:25:39.467 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=6 2023-11-16T18:25:39.467 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:39.493 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:39.493 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:39.493 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:39.493 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:39.493 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:39.842 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:39.842 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:39.963 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:39.963 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:39.978 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:39.978 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:39.978 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 6 -lt 100 ']' 2023-11-16T18:25:39.978 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 6 + 1 2023-11-16T18:25:40.017 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=7 2023-11-16T18:25:40.017 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:40.042 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:40.042 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:40.043 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:40.043 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:40.043 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:40.367 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:40.376 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:40.463 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:40.463 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:40.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:40.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:40.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 7 -lt 100 ']' 2023-11-16T18:25:40.481 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 7 + 1 2023-11-16T18:25:40.509 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=8 2023-11-16T18:25:40.510 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:40.533 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:40.534 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:40.535 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:40.535 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:40.535 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:40.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:40 smithi188 ceph-mon[100639]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2023-11-16T18:25:40.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:40 smithi002 ceph-mon[98807]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2023-11-16T18:25:40.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:40 smithi040 ceph-mon[100251]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2023-11-16T18:25:40.829 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:40.829 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:40.938 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:40.938 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:40.954 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:40.955 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:40.955 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 8 -lt 100 ']' 2023-11-16T18:25:40.955 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 8 + 1 2023-11-16T18:25:40.986 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=9 2023-11-16T18:25:40.986 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:41.019 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:41.020 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:41.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:41.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:41.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:41.400 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:41.400 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:41.530 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:41.530 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:41.546 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:41.546 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:41.547 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 9 -lt 100 ']' 2023-11-16T18:25:41.547 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 9 + 1 2023-11-16T18:25:41.575 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=10 2023-11-16T18:25:41.575 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:41.601 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:41.601 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:41.602 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:41.602 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:41.602 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:41.912 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:41.912 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:42.005 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:42.005 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:42.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:42.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:42.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 10 -lt 100 ']' 2023-11-16T18:25:42.020 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 10 + 1 2023-11-16T18:25:42.057 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=11 2023-11-16T18:25:42.057 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:42.079 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:42.079 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:42.079 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:42.080 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:42.080 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:42.383 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:42.383 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:42.479 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:42.480 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:42.495 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:42.495 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:42.495 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 11 -lt 100 ']' 2023-11-16T18:25:42.495 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 11 + 1 2023-11-16T18:25:42.525 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=12 2023-11-16T18:25:42.525 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:42.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:42 smithi188 ceph-mon[100639]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.9 KiB/s wr, 7 op/s 2023-11-16T18:25:42.550 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:42.550 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:42.551 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:42.551 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:42.551 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:42.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:42 smithi002 ceph-mon[98807]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.9 KiB/s wr, 7 op/s 2023-11-16T18:25:42.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:42 smithi040 ceph-mon[100251]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.9 KiB/s wr, 7 op/s 2023-11-16T18:25:42.883 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:42.883 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:43.005 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:43.006 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:43.012 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:43.012 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:43.012 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 12 -lt 100 ']' 2023-11-16T18:25:43.012 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 12 + 1 2023-11-16T18:25:43.042 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=13 2023-11-16T18:25:43.042 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:43.081 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:43.082 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:43.082 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:43.082 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:43.083 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:43.367 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:43.368 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:43.471 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:43.471 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:43.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:43.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:43.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 13 -lt 100 ']' 2023-11-16T18:25:43.479 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 13 + 1 2023-11-16T18:25:43.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=14 2023-11-16T18:25:43.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:43.531 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:43.531 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:43.532 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:43.532 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:43.533 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:43.900 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:43.900 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:44.013 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:44.013 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:44.022 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:44.022 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:44.022 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 14 -lt 100 ']' 2023-11-16T18:25:44.022 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 14 + 1 2023-11-16T18:25:44.050 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=15 2023-11-16T18:25:44.050 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:44.075 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:44.075 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:44.076 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:44.076 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:44.076 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:44.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:44 smithi002 ceph-mon[98807]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 10 op/s 2023-11-16T18:25:44.441 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:44.441 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:44.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:44 smithi188 ceph-mon[100639]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 10 op/s 2023-11-16T18:25:44.579 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:44.579 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:44.587 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:44.587 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:44.587 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 15 -lt 100 ']' 2023-11-16T18:25:44.588 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 15 + 1 2023-11-16T18:25:44.617 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=16 2023-11-16T18:25:44.617 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:44.646 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:44.647 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:44.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:44.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:44.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:44.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:44 smithi040 ceph-mon[100251]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 10 op/s 2023-11-16T18:25:44.974 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:44.975 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:45.071 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:45.071 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:45.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:45.089 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:45.089 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 16 -lt 100 ']' 2023-11-16T18:25:45.089 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 16 + 1 2023-11-16T18:25:45.116 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=17 2023-11-16T18:25:45.116 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:45.159 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:45.160 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:45.160 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:45.160 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:45.160 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:45.491 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:45.491 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:45.502 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:25:45.631 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:45.631 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:45.645 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:45.645 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:45.645 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 17 -lt 100 ']' 2023-11-16T18:25:45.645 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 17 + 1 2023-11-16T18:25:45.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=18 2023-11-16T18:25:45.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:45.697 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:45.697 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:45.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:45.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:45.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:46.193 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:46.193 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:46.354 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:46.354 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:46.391 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:46.391 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:46.391 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 18 -lt 100 ']' 2023-11-16T18:25:46.392 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 18 + 1 2023-11-16T18:25:46.437 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=19 2023-11-16T18:25:46.438 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:46.470 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:46.470 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:46.471 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:46.471 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:46.471 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:46.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:46 smithi188 ceph-mon[100639]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 7 op/s 2023-11-16T18:25:46.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:46 smithi002 ceph-mon[98807]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 7 op/s 2023-11-16T18:25:46.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:46 smithi040 ceph-mon[100251]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 7 op/s 2023-11-16T18:25:46.974 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:46.974 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:47.120 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:47.121 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:47.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:47.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:47.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 19 -lt 100 ']' 2023-11-16T18:25:47.127 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 19 + 1 2023-11-16T18:25:47.179 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=20 2023-11-16T18:25:47.179 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:47.228 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:47.229 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:47.229 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:47.229 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:47.230 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:47.808 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:47.808 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:48.012 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:48.012 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:48.036 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:48.037 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:48.037 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 20 -lt 100 ']' 2023-11-16T18:25:48.037 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 20 + 1 2023-11-16T18:25:48.082 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=21 2023-11-16T18:25:48.083 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:48.147 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:48.147 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:48.148 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:48.148 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:48.148 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.475+0000 7f941add1700 1 -- 172.21.15.2:0/526836899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94141015f0 msgr2=0x7f9414104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.475+0000 7f941add1700 1 --2- 172.21.15.2:0/526836899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94141015f0 0x7f9414104e80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9400009a40 tx=0x7f9400009d50 comp rx=0 tx=0).stop 2023-11-16T18:25:48.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.475+0000 7f4b35858700 1 -- 172.21.15.2:0/2625945969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b30105610 msgr2=0x7f4b3010d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.475+0000 7f4b35858700 1 --2- 172.21.15.2:0/2625945969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b30105610 0x7f4b3010d190 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4b2800b0d0 tx=0x7f4b2800b3e0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f4b35858700 1 -- 172.21.15.2:0/2625945969 shutdown_connections 2023-11-16T18:25:48.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f4b35858700 1 --2- 172.21.15.2:0/2625945969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b30105610 0x7f4b3010d190 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f4b35858700 1 --2- 172.21.15.2:0/2625945969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b30100c50 0x7f4b30101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f4b35858700 1 --2- 172.21.15.2:0/2625945969 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b301015f0 0x7f4b30104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f4b35858700 1 -- 172.21.15.2:0/2625945969 >> 172.21.15.2:0/2625945969 conn(0x7f4b300775c0 msgr2=0x7f4b300779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f941add1700 1 -- 172.21.15.2:0/526836899 shutdown_connections 2023-11-16T18:25:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f941add1700 1 --2- 172.21.15.2:0/526836899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9414100c50 0x7f9414101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f941add1700 1 --2- 172.21.15.2:0/526836899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9414105560 0x7f941410d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f941add1700 1 --2- 172.21.15.2:0/526836899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94141015f0 0x7f9414104e80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f941add1700 1 -- 172.21.15.2:0/526836899 >> 172.21.15.2:0/526836899 conn(0x7f9414077550 msgr2=0x7f9414077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:48.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f4b35858700 1 -- 172.21.15.2:0/2625945969 shutdown_connections 2023-11-16T18:25:48.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f4b35858700 1 -- 172.21.15.2:0/2625945969 wait complete. 2023-11-16T18:25:48.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f941add1700 1 -- 172.21.15.2:0/526836899 shutdown_connections 2023-11-16T18:25:48.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.476+0000 7f941add1700 1 -- 172.21.15.2:0/526836899 wait complete. 2023-11-16T18:25:48.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f941add1700 1 Processor -- start 2023-11-16T18:25:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f4b35858700 1 Processor -- start 2023-11-16T18:25:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f941add1700 1 -- start start 2023-11-16T18:25:48.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f4b35858700 1 -- start start 2023-11-16T18:25:48.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f941add1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9414100c50 0x7f94141941e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f941add1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94141015f0 0x7f9414194720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f941add1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9414105560 0x7f9414198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f941add1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94141991c0 con 0x7f9414105560 2023-11-16T18:25:48.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f941add1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9414199330 con 0x7f9414100c50 2023-11-16T18:25:48.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f941add1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94141994f0 con 0x7f94141015f0 2023-11-16T18:25:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f4b35858700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b30100c50 0x7f4b30196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f4b35858700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b301015f0 0x7f4b30196ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f4b35858700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b30105610 0x7f4b3019af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f941936e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9414105560 0x7f9414198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f4b35858700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3019b570 con 0x7f4b301015f0 2023-11-16T18:25:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f4b35858700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3019b6e0 con 0x7f4b30105610 2023-11-16T18:25:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f4b35858700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3019b8a0 con 0x7f4b30100c50 2023-11-16T18:25:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f941936e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9414105560 0x7f9414198b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50774/0 (socket says 172.21.15.2:50774) 2023-11-16T18:25:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f941936e700 1 -- 172.21.15.2:0/1173194400 learned_addr learned my addr 172.21.15.2:0/1173194400 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f9418b6d700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9414100c50 0x7f94141941e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f9413fff700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94141015f0 0x7f9414194720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f941936e700 1 -- 172.21.15.2:0/1173194400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94141015f0 msgr2=0x7f9414194720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f941936e700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94141015f0 0x7f9414194720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f941936e700 1 -- 172.21.15.2:0/1173194400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9414100c50 msgr2=0x7f94141941e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f941936e700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9414100c50 0x7f94141941e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f941936e700 1 -- 172.21.15.2:0/1173194400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94000095f0 con 0x7f9414105560 2023-11-16T18:25:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f4b2e7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b301015f0 0x7f4b30196ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f4b2e7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b301015f0 0x7f4b30196ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50778/0 (socket says 172.21.15.2:50778) 2023-11-16T18:25:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/1516505211 learned_addr learned my addr 172.21.15.2:0/1516505211 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f4b2effd700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b30100c50 0x7f4b30196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.477+0000 7f4b2f7fe700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b30105610 0x7f4b3019af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/1516505211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b30100c50 msgr2=0x7f4b30196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b30100c50 0x7f4b30196590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/1516505211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b30105610 msgr2=0x7f4b3019af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b30105610 0x7f4b3019af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/1516505211 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b28009d00 con 0x7f4b301015f0 2023-11-16T18:25:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f4b2f7fe700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b30105610 0x7f4b3019af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:25:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f941936e700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9414105560 0x7f9414198b80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f940c007ee0 tx=0x7f940c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:48.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b301015f0 0x7f4b30196ad0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f4b2400d980 tx=0x7f4b2400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f9411ffb700 1 -- 172.21.15.2:0/1173194400 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c00dd20 con 0x7f9414105560 2023-11-16T18:25:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f941add1700 1 -- 172.21.15.2:0/1173194400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f941410e1e0 con 0x7f9414105560 2023-11-16T18:25:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f9411ffb700 1 -- 172.21.15.2:0/1173194400 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f940c00f040 con 0x7f9414105560 2023-11-16T18:25:48.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f4b34856700 1 -- 172.21.15.2:0/1516505211 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b24009930 con 0x7f4b301015f0 2023-11-16T18:25:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f9411ffb700 1 -- 172.21.15.2:0/1173194400 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c0048a0 con 0x7f9414105560 2023-11-16T18:25:48.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f4b34856700 1 -- 172.21.15.2:0/1516505211 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b2400b5e0 con 0x7f4b301015f0 2023-11-16T18:25:48.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f4b35858700 1 -- 172.21.15.2:0/1516505211 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b301103d0 con 0x7f4b301015f0 2023-11-16T18:25:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f941add1700 1 -- 172.21.15.2:0/1173194400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941410e050 con 0x7f9414105560 2023-11-16T18:25:48.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f4b34856700 1 -- 172.21.15.2:0/1516505211 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2400f3c0 con 0x7f4b301015f0 2023-11-16T18:25:48.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.478+0000 7f4b35858700 1 -- 172.21.15.2:0/1516505211 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b30110240 con 0x7f4b301015f0 2023-11-16T18:25:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.481+0000 7f9411ffb700 1 -- 172.21.15.2:0/1173194400 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f940c003a00 con 0x7f9414105560 2023-11-16T18:25:48.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.481+0000 7f4b34856700 1 -- 172.21.15.2:0/1516505211 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f4b24009a90 con 0x7f4b301015f0 2023-11-16T18:25:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.481+0000 7f9411ffb700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f94040a13e0 0x7f94040a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.482+0000 7f9418b6d700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f94040a13e0 0x7f94040a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.482+0000 7f9411ffb700 1 -- 172.21.15.2:0/1173194400 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f940c0ccbb0 con 0x7f9414105560 2023-11-16T18:25:48.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.482+0000 7f4b34856700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f4b180a1460 0x7f4b180a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.482+0000 7f9418b6d700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f94040a13e0 0x7f94040a3890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9408007be0 tx=0x7f94080073d0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:48.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.482+0000 7f4b2effd700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f4b180a1460 0x7f4b180a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.482+0000 7f4b34856700 1 -- 172.21.15.2:0/1516505211 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f4b240ccd70 con 0x7f4b301015f0 2023-11-16T18:25:48.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.482+0000 7f4b2effd700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f4b180a1460 0x7f4b180a3910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4b20005950 tx=0x7f4b2000b410 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.528+0000 7f9426f5db80 1 Processor -- start 2023-11-16T18:25:48.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.528+0000 7f4b419e4b80 1 Processor -- start 2023-11-16T18:25:48.530 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.528+0000 7f9426f5db80 1 -- start start 2023-11-16T18:25:48.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.528+0000 7f4b419e4b80 1 -- start start 2023-11-16T18:25:48.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b419e4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69807f70 0x55bb69808340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b419e4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69808880 0x55bb6980c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9426f5db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5639381ab6b0 0x5639383551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9426f5db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355720 0x56393834f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b419e4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb6980d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b419e4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb69709060 con 0x55bb69807f70 2023-11-16T18:25:48.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b419e4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb6975b8a0 con 0x55bb6980ccc0 2023-11-16T18:25:48.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b419e4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb69754d30 con 0x55bb69808880 2023-11-16T18:25:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9426f5db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563938355b90 0x56393834fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9426f5db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563938255000 con 0x563938355720 2023-11-16T18:25:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9426f5db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639382a7840 con 0x563938355b90 2023-11-16T18:25:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9426f5db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639382a0cd0 con 0x5639381ab6b0 2023-11-16T18:25:48.533 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b2effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69807f70 0x55bb69808340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9413fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355720 0x56393834f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9413fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355720 0x56393834f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50792/0 (socket says 172.21.15.2:50792) 2023-11-16T18:25:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9413fff700 1 -- 172.21.15.2:0/805111410 learned_addr learned my addr 172.21.15.2:0/805111410 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:48.534 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b2effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69807f70 0x55bb69808340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50786/0 (socket says 172.21.15.2:50786) 2023-11-16T18:25:48.534 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b2effd700 1 -- 172.21.15.2:0/3352528985 learned_addr learned my addr 172.21.15.2:0/3352528985 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:48.534 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/3352528985 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69808880 0x55bb6980c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.534 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b2f7fe700 1 --2- 172.21.15.2:0/3352528985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb6980d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f941936e700 1 --2- 172.21.15.2:0/805111410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563938355b90 0x56393834fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9418b6d700 1 --2- 172.21.15.2:0/805111410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5639381ab6b0 0x5639383551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/3352528985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 msgr2=0x55bb6980d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/3352528985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb6980d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/3352528985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69807f70 msgr2=0x55bb69808340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/3352528985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69807f70 0x55bb69808340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9413fff700 1 -- 172.21.15.2:0/805111410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5639381ab6b0 msgr2=0x5639383551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9413fff700 1 --2- 172.21.15.2:0/805111410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5639381ab6b0 0x5639383551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9413fff700 1 -- 172.21.15.2:0/805111410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563938355b90 msgr2=0x56393834fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9413fff700 1 --2- 172.21.15.2:0/805111410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563938355b90 0x56393834fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f9413fff700 1 -- 172.21.15.2:0/805111410 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639381225d0 con 0x563938355720 2023-11-16T18:25:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/3352528985 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb695d6630 con 0x55bb69808880 2023-11-16T18:25:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.529+0000 7f4b2f7fe700 1 --2- 172.21.15.2:0/3352528985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb6980d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.530+0000 7f9413fff700 1 --2- 172.21.15.2:0/805111410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355720 0x56393834f560 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9400009a10 tx=0x7f9400005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c68a0c3208c0cc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:48.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.530+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/3352528985 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69808880 0x55bb6980c780 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4b240c9fd0 tx=0x7f4b24099260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2fd9d6e04e2972e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:48.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.530+0000 7f4b15ffb700 1 -- 172.21.15.2:0/3352528985 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2409d8b0 con 0x55bb69808880 2023-11-16T18:25:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.530+0000 7f93feffd700 1 -- 172.21.15.2:0/805111410 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9400010bc0 con 0x563938355720 2023-11-16T18:25:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.530+0000 7f93feffd700 1 -- 172.21.15.2:0/805111410 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9400005da0 con 0x563938355720 2023-11-16T18:25:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.530+0000 7f93feffd700 1 -- 172.21.15.2:0/805111410 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940000bcd0 con 0x563938355720 2023-11-16T18:25:48.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.530+0000 7f4b15ffb700 1 -- 172.21.15.2:0/3352528985 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b240a1cd0 con 0x55bb69808880 2023-11-16T18:25:48.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.530+0000 7f4b15ffb700 1 -- 172.21.15.2:0/3352528985 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b24098f70 con 0x55bb69808880 2023-11-16T18:25:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f9426f5db80 1 -- 172.21.15.2:0/805111410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355720 msgr2=0x56393834f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f9426f5db80 1 --2- 172.21.15.2:0/805111410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355720 0x56393834f560 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9400009a10 tx=0x7f9400005cb0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f4b419e4b80 1 -- 172.21.15.2:0/3352528985 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69808880 msgr2=0x55bb6980c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.540 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/3352528985 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69808880 0x55bb6980c780 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4b240c9fd0 tx=0x7f4b24099260 comp rx=0 tx=0).stop 2023-11-16T18:25:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f9426f5db80 1 -- 172.21.15.2:0/805111410 shutdown_connections 2023-11-16T18:25:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f9426f5db80 1 --2- 172.21.15.2:0/805111410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5639381ab6b0 0x5639383551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f9426f5db80 1 --2- 172.21.15.2:0/805111410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563938355b90 0x56393834fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f9426f5db80 1 --2- 172.21.15.2:0/805111410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355720 0x56393834f560 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f9426f5db80 1 -- 172.21.15.2:0/805111410 >> 172.21.15.2:0/805111410 conn(0x563938344f50 msgr2=0x56393834a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:48.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f4b419e4b80 1 -- 172.21.15.2:0/3352528985 shutdown_connections 2023-11-16T18:25:48.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/3352528985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb6980d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/3352528985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69807f70 0x55bb69808340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f9426f5db80 1 -- 172.21.15.2:0/805111410 shutdown_connections 2023-11-16T18:25:48.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/3352528985 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69808880 0x55bb6980c780 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.542 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f9426f5db80 1 -- 172.21.15.2:0/805111410 wait complete. 2023-11-16T18:25:48.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f4b419e4b80 1 -- 172.21.15.2:0/3352528985 >> 172.21.15.2:0/3352528985 conn(0x55bb697f9360 msgr2=0x55bb697feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:48.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f4b419e4b80 1 -- 172.21.15.2:0/3352528985 shutdown_connections 2023-11-16T18:25:48.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f4b419e4b80 1 -- 172.21.15.2:0/3352528985 wait complete. 2023-11-16T18:25:48.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f9426f5db80 1 -- start start 2023-11-16T18:25:48.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f9426f5db80 1 Processor -- start 2023-11-16T18:25:48.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.531+0000 7f4b419e4b80 1 -- start start 2023-11-16T18:25:48.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b419e4b80 1 Processor -- start 2023-11-16T18:25:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f9426f5db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639381ab6b0 0x56393835e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f9426f5db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355720 0x56393835ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f9426f5db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 0x563938358bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f9426f5db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563938242880 con 0x563938355b90 2023-11-16T18:25:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f9426f5db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563938242520 con 0x5639381ab6b0 2023-11-16T18:25:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f9426f5db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639381248c0 con 0x563938355720 2023-11-16T18:25:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f9426f5db80 10 client.0 _ll_register_callbacks cb 0x563938260720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f941936e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 0x563938358bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f941936e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 0x563938358bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50802/0 (socket says 172.21.15.2:50802) 2023-11-16T18:25:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f941936e700 1 -- 172.21.15.2:0/2242593454 learned_addr learned my addr 172.21.15.2:0/2242593454 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f9413fff700 1 --2- 172.21.15.2:0/2242593454 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355720 0x56393835ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f9418b6d700 1 --2- 172.21.15.2:0/2242593454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639381ab6b0 0x56393835e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b419e4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69807f70 0x55bb698144d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b419e4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 0x55bb69811750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b419e4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb69811e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b419e4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb696f68e0 con 0x55bb69808880 2023-11-16T18:25:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b419e4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb696f6580 con 0x55bb6980ccc0 2023-11-16T18:25:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b419e4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb695d8920 con 0x55bb69807f70 2023-11-16T18:25:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b419e4b80 10 client.0 _ll_register_callbacks cb 0x55bb69714780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b2e7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 0x55bb69811750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b2e7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 0x55bb69811750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50806/0 (socket says 172.21.15.2:50806) 2023-11-16T18:25:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b2effd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69807f70 0x55bb698144d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/1743432142 learned_addr learned my addr 172.21.15.2:0/1743432142 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f941936e700 1 -- 172.21.15.2:0/2242593454 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355720 msgr2=0x56393835ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f941936e700 1 --2- 172.21.15.2:0/2242593454 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355720 0x56393835ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f941936e700 1 -- 172.21.15.2:0/2242593454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639381ab6b0 msgr2=0x56393835e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f941936e700 1 --2- 172.21.15.2:0/2242593454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639381ab6b0 0x56393835e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f941936e700 1 -- 172.21.15.2:0/2242593454 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941410d9b0 con 0x563938355b90 2023-11-16T18:25:48.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b2f7fe700 1 --2- 172.21.15.2:0/1743432142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb69811e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f941936e700 1 --2- 172.21.15.2:0/2242593454 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 0x563938358bd0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f940c00d700 tx=0x7f940c0992a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520f85bd29edba67 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f93f3fff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:48.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:48 smithi188 ceph-mon[100639]: pgmap v31: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 10 op/s 2023-11-16T18:25:48.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/1743432142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69807f70 msgr2=0x55bb698144d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/1743432142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69807f70 0x55bb698144d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/1743432142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 msgr2=0x55bb69811e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.532+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/1743432142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb69811e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/1743432142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b30110240 con 0x55bb69808880 2023-11-16T18:25:48.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f93f3fff700 1 -- 172.21.15.2:0/2242593454 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c0a1ab0 con 0x563938355b90 2023-11-16T18:25:48.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f93f3fff700 1 -- 172.21.15.2:0/2242593454 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f940c009410 con 0x563938355b90 2023-11-16T18:25:48.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f9426f5db80 1 -- 172.21.15.2:0/2242593454 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563938342710 con 0x563938355b90 2023-11-16T18:25:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/1743432142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 0x55bb69811750 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4b240ca750 tx=0x7f4b2409d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1620dae3b5824a42 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f4b0effd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f4b0effd700 1 -- 172.21.15.2:0/1743432142 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b24010680 con 0x55bb69808880 2023-11-16T18:25:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f4b0effd700 1 -- 172.21.15.2:0/1743432142 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f4b24098d70 con 0x55bb69808880 2023-11-16T18:25:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f4b419e4b80 1 -- 172.21.15.2:0/1743432142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bb697f6b20 con 0x55bb69808880 2023-11-16T18:25:48.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f93f3fff700 1 -- 172.21.15.2:0/2242593454 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f940c0840b0 con 0x563938355b90 2023-11-16T18:25:48.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f9426f5db80 10 client.15852 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:48.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f9426f5db80 1 -- 172.21.15.2:0/2242593454 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x563938267130 con 0x563938355b90 2023-11-16T18:25:48.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f4b0effd700 1 -- 172.21.15.2:0/1743432142 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b24098f30 con 0x55bb69808880 2023-11-16T18:25:48.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f4b419e4b80 10 client.15855 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:48.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f4b419e4b80 1 -- 172.21.15.2:0/1743432142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bb6971b190 con 0x55bb69808880 2023-11-16T18:25:48.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f93f3fff700 1 -- 172.21.15.2:0/2242593454 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563938342710 con 0x563938355b90 2023-11-16T18:25:48.555 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f4b0effd700 1 -- 172.21.15.2:0/1743432142 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b240c9020 con 0x55bb69808880 2023-11-16T18:25:48.555 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f4b0effd700 1 -- 172.21.15.2:0/1743432142 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4b240ca420 con 0x55bb69808880 2023-11-16T18:25:48.555 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f4b419e4b80 10 client.15855 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:48.555 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f4b419e4b80 10 client.15855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:48.555 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f4b419e4b80 20 client.15855 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:48.555 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/1743432142 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bb698328a0 0x55bb69834c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.555 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f4b419e4b80 4 client.15855 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:25:48.556 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f4b419e4b80 1 -- 172.21.15.2:0/1743432142 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bb697170a0 con 0x55bb698328a0 2023-11-16T18:25:48.556 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f4b2effd700 1 --2- 172.21.15.2:0/1743432142 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bb698328a0 0x55bb69834c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f93f3fff700 1 -- 172.21.15.2:0/2242593454 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f940c098b50 con 0x563938355b90 2023-11-16T18:25:48.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f9426f5db80 10 client.15852 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:48.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f9426f5db80 10 client.15852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:48.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.533+0000 7f9426f5db80 20 client.15852 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f9426f5db80 1 --2- 172.21.15.2:0/2242593454 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x563938368ce0 0x56393836b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f9426f5db80 4 client.15852 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:25:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f9426f5db80 1 -- 172.21.15.2:0/2242593454 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563938263040 con 0x563938368ce0 2023-11-16T18:25:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f9418b6d700 1 --2- 172.21.15.2:0/2242593454 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x563938368ce0 0x56393836b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f9418b6d700 1 --2- 172.21.15.2:0/2242593454 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x563938368ce0 0x56393836b0f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b2c4864c3da62c9 server_cookie=81287a530c4e2505 in_seq=0 out_seq=0 2023-11-16T18:25:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.534+0000 7f93f3fff700 10 client.15852 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:48.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f4b0effd700 10 client.15855 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:48.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f4b2effd700 1 --2- 172.21.15.2:0/1743432142 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bb698328a0 0x55bb69834c80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81ee9172a366fa1 server_cookie=229cc545ba443d64 in_seq=0 out_seq=0 2023-11-16T18:25:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f93f3fff700 1 -- 172.21.15.2:0/2242593454 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563938263040 con 0x563938368ce0 2023-11-16T18:25:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f93f3fff700 10 client.15852 handle_command_reply: tid=0 2023-11-16T18:25:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f9426f5db80 1 client.15852 shutdown 2023-11-16T18:25:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f9426f5db80 1 -- 172.21.15.2:0/2242593454 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 msgr2=0x563938358bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f9426f5db80 1 --2- 172.21.15.2:0/2242593454 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 0x563938358bd0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f940c00d700 tx=0x7f940c0992a0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f9426f5db80 1 -- 172.21.15.2:0/2242593454 shutdown_connections 2023-11-16T18:25:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f9426f5db80 1 --2- 172.21.15.2:0/2242593454 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x563938368ce0 0x56393836b0f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f9426f5db80 1 --2- 172.21.15.2:0/2242593454 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 0x563938358bd0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f93f3fff700 0 client.15852 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f9426f5db80 1 --2- 172.21.15.2:0/2242593454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639381ab6b0 0x56393835e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f9426f5db80 1 --2- 172.21.15.2:0/2242593454 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355720 0x56393835ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.535+0000 7f9426f5db80 1 -- 172.21.15.2:0/2242593454 >> 172.21.15.2:0/2242593454 conn(0x563938344f50 msgr2=0x5639383497e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f9426f5db80 1 -- 172.21.15.2:0/2242593454 shutdown_connections 2023-11-16T18:25:48.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b0effd700 1 -- 172.21.15.2:0/1743432142 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55bb697170a0 con 0x55bb698328a0 2023-11-16T18:25:48.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b0effd700 10 client.15855 handle_command_reply: tid=0 2023-11-16T18:25:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f9426f5db80 1 -- 172.21.15.2:0/2242593454 wait complete. 2023-11-16T18:25:48.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b419e4b80 1 client.15855 shutdown 2023-11-16T18:25:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f9426f5db80 20 client.15852 trim_cache size 0 max 16384 2023-11-16T18:25:48.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b419e4b80 1 -- 172.21.15.2:0/1743432142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 msgr2=0x55bb69811750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/1743432142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 0x55bb69811750 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4b240ca750 tx=0x7f4b2409d920 comp rx=0 tx=0).stop 2023-11-16T18:25:48.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b419e4b80 1 -- 172.21.15.2:0/1743432142 shutdown_connections 2023-11-16T18:25:48.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/1743432142 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bb698328a0 0x55bb69834c80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/1743432142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69807f70 0x55bb698144d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/1743432142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb69811e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b0effd700 0 client.15855 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:48.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/1743432142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 0x55bb69811750 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b419e4b80 1 -- 172.21.15.2:0/1743432142 >> 172.21.15.2:0/1743432142 conn(0x55bb697f9360 msgr2=0x55bb6980e170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:48.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b419e4b80 1 -- 172.21.15.2:0/1743432142 shutdown_connections 2023-11-16T18:25:48.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b419e4b80 1 -- 172.21.15.2:0/1743432142 wait complete. 2023-11-16T18:25:48.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.536+0000 7f4b419e4b80 20 client.15855 trim_cache size 0 max 16384 2023-11-16T18:25:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f9426f5db80 1 Processor -- start 2023-11-16T18:25:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f9426f5db80 1 -- start start 2023-11-16T18:25:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f9426f5db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639381ab6b0 0x56393834f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f9426f5db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355720 0x563938349f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f9426f5db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 0x563938355f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f9426f5db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56393834faa0 con 0x563938355b90 2023-11-16T18:25:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f9426f5db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56393835a170 con 0x5639381ab6b0 2023-11-16T18:25:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f9426f5db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56393834af50 con 0x563938355720 2023-11-16T18:25:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f941936e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 0x563938355f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f941936e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 0x563938355f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50818/0 (socket says 172.21.15.2:50818) 2023-11-16T18:25:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f941936e700 1 -- 172.21.15.2:0/3926696398 learned_addr learned my addr 172.21.15.2:0/3926696398 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f9418b6d700 1 --2- 172.21.15.2:0/3926696398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639381ab6b0 0x56393834f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f9413fff700 1 --2- 172.21.15.2:0/3926696398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355720 0x563938349f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f941936e700 1 -- 172.21.15.2:0/3926696398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355720 msgr2=0x563938349f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f941936e700 1 --2- 172.21.15.2:0/3926696398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355720 0x563938349f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f941936e700 1 -- 172.21.15.2:0/3926696398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639381ab6b0 msgr2=0x56393834f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f941936e700 1 --2- 172.21.15.2:0/3926696398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639381ab6b0 0x56393834f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f941936e700 1 -- 172.21.15.2:0/3926696398 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563938266e80 con 0x563938355b90 2023-11-16T18:25:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.551+0000 7f9418b6d700 1 --2- 172.21.15.2:0/3926696398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639381ab6b0 0x56393834f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:25:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.552+0000 7f941936e700 1 --2- 172.21.15.2:0/3926696398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 0x563938355f60 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f940c095b60 tx=0x7f940c00dfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c76ed75b5e4755f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.552+0000 7f93ff7fe700 1 -- 172.21.15.2:0/3926696398 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c009410 con 0x563938355b90 2023-11-16T18:25:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.552+0000 7f93ff7fe700 1 -- 172.21.15.2:0/3926696398 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f940c0c3e90 con 0x563938355b90 2023-11-16T18:25:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.552+0000 7f93ff7fe700 1 -- 172.21.15.2:0/3926696398 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c0998c0 con 0x563938355b90 2023-11-16T18:25:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.553+0000 7f9426f5db80 1 -- 172.21.15.2:0/3926696398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 msgr2=0x563938355f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.553+0000 7f9426f5db80 1 --2- 172.21.15.2:0/3926696398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 0x563938355f60 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f940c095b60 tx=0x7f940c00dfd0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.553+0000 7f9426f5db80 1 -- 172.21.15.2:0/3926696398 shutdown_connections 2023-11-16T18:25:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.553+0000 7f9426f5db80 1 --2- 172.21.15.2:0/3926696398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563938355b90 0x563938355f60 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.553+0000 7f9426f5db80 1 --2- 172.21.15.2:0/3926696398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5639381ab6b0 0x56393834f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.553+0000 7f9426f5db80 1 --2- 172.21.15.2:0/3926696398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355720 0x563938349f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.553+0000 7f9426f5db80 1 -- 172.21.15.2:0/3926696398 >> 172.21.15.2:0/3926696398 conn(0x563938344f50 msgr2=0x563938359500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.553+0000 7f9426f5db80 1 -- 172.21.15.2:0/3926696398 shutdown_connections 2023-11-16T18:25:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.553+0000 7f9426f5db80 1 -- 172.21.15.2:0/3926696398 wait complete. 2023-11-16T18:25:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.553+0000 7f9426f5db80 1 -- start start 2023-11-16T18:25:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.553+0000 7f9426f5db80 1 Processor -- start 2023-11-16T18:25:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9426f5db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639381ab6b0 0x5639383b5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9426f5db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563938355720 0x5639383b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9426f5db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355b90 0x563938361640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9426f5db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563938361c60 con 0x5639381ab6b0 2023-11-16T18:25:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9426f5db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563938361da0 con 0x563938355720 2023-11-16T18:25:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9426f5db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x563938361f60 con 0x563938355b90 2023-11-16T18:25:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9426f5db80 10 client.0 _ll_register_callbacks cb 0x7f940c099430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9418b6d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639381ab6b0 0x5639383b5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9418b6d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639381ab6b0 0x5639383b5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50820/0 (socket says 172.21.15.2:50820) 2023-11-16T18:25:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9418b6d700 1 -- 172.21.15.2:0/2420787791 learned_addr learned my addr 172.21.15.2:0/2420787791 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f941936e700 1 --2- 172.21.15.2:0/2420787791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355b90 0x563938361640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9413fff700 1 --2- 172.21.15.2:0/2420787791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563938355720 0x5639383b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9418b6d700 1 -- 172.21.15.2:0/2420787791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355b90 msgr2=0x563938361640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9418b6d700 1 --2- 172.21.15.2:0/2420787791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355b90 0x563938361640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9418b6d700 1 -- 172.21.15.2:0/2420787791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563938355720 msgr2=0x5639383b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9418b6d700 1 --2- 172.21.15.2:0/2420787791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563938355720 0x5639383b63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.554+0000 7f9418b6d700 1 -- 172.21.15.2:0/2420787791 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563938267130 con 0x5639381ab6b0 2023-11-16T18:25:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f9418b6d700 1 --2- 172.21.15.2:0/2420787791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639381ab6b0 0x5639383b5ea0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9408010b80 tx=0x7f94080047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa5fc4b7616e89f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f93f3fff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f93f3fff700 1 -- 172.21.15.2:0/2420787791 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408027610 con 0x5639381ab6b0 2023-11-16T18:25:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f93f3fff700 1 -- 172.21.15.2:0/2420787791 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f9408017040 con 0x5639381ab6b0 2023-11-16T18:25:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f9426f5db80 1 -- 172.21.15.2:0/2420787791 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9400005da0 con 0x5639381ab6b0 2023-11-16T18:25:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f93f3fff700 1 -- 172.21.15.2:0/2420787791 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94080277f0 con 0x5639381ab6b0 2023-11-16T18:25:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f9426f5db80 10 client.15861 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f9426f5db80 1 -- 172.21.15.2:0/2420787791 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5639380b7c00 con 0x5639381ab6b0 2023-11-16T18:25:48.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f4b419e4b80 1 Processor -- start 2023-11-16T18:25:48.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f4b419e4b80 1 -- start start 2023-11-16T18:25:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f93f3fff700 1 -- 172.21.15.2:0/2420787791 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9400005da0 con 0x5639381ab6b0 2023-11-16T18:25:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f93f3fff700 1 -- 172.21.15.2:0/2420787791 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9408011440 con 0x5639381ab6b0 2023-11-16T18:25:48.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f4b419e4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69807f70 0x55bb69814000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f4b419e4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69808880 0x55bb69814540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f4b419e4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb697fe560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f4b419e4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb6980fab0 con 0x55bb69807f70 2023-11-16T18:25:48.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f4b419e4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb698001c0 con 0x55bb6980ccc0 2023-11-16T18:25:48.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f4b419e4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb698352e0 con 0x55bb69808880 2023-11-16T18:25:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f9426f5db80 10 client.15861 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f9426f5db80 10 client.15861 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f9426f5db80 20 client.15861 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f9426f5db80 1 --2- 172.21.15.2:0/2420787791 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5639383bdcc0 0x5639383c00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f9426f5db80 4 client.15861 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:25:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.555+0000 7f9426f5db80 1 -- 172.21.15.2:0/2420787791 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56393826f600 con 0x5639383bdcc0 2023-11-16T18:25:48.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b2effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69807f70 0x55bb69814000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b2effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69807f70 0x55bb69814000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50826/0 (socket says 172.21.15.2:50826) 2023-11-16T18:25:48.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b2effd700 1 -- 172.21.15.2:0/3218860453 learned_addr learned my addr 172.21.15.2:0/3218860453 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:48.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b2f7fe700 1 --2- 172.21.15.2:0/3218860453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb697fe560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/3218860453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69808880 0x55bb69814540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f9413fff700 1 --2- 172.21.15.2:0/2420787791 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5639383bdcc0 0x5639383c00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b2effd700 1 -- 172.21.15.2:0/3218860453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69808880 msgr2=0x55bb69814540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b2effd700 1 --2- 172.21.15.2:0/3218860453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69808880 0x55bb69814540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b2effd700 1 -- 172.21.15.2:0/3218860453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 msgr2=0x55bb697fe560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b2effd700 1 --2- 172.21.15.2:0/3218860453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb697fe560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b2effd700 1 -- 172.21.15.2:0/3218860453 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb6971aee0 con 0x55bb69807f70 2023-11-16T18:25:48.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b2f7fe700 1 --2- 172.21.15.2:0/3218860453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb697fe560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:25:48.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/3218860453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69808880 0x55bb69814540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:25:48.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b2effd700 1 --2- 172.21.15.2:0/3218860453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69807f70 0x55bb69814000 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x55bb69835550 tx=0x7f4b20013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9165e557abd2727a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:48.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f93f3fff700 10 client.15861 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:48.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f9413fff700 1 --2- 172.21.15.2:0/2420787791 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5639383bdcc0 0x5639383c00d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b131a640a06fd317 server_cookie=32ef035b07cf40d9 in_seq=0 out_seq=0 2023-11-16T18:25:48.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b167fc700 1 -- 172.21.15.2:0/3218860453 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b20024040 con 0x55bb69807f70 2023-11-16T18:25:48.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b167fc700 1 -- 172.21.15.2:0/3218860453 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b20015070 con 0x55bb69807f70 2023-11-16T18:25:48.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.556+0000 7f4b167fc700 1 -- 172.21.15.2:0/3218860453 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b20004a90 con 0x55bb69807f70 2023-11-16T18:25:48.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.557+0000 7f4b419e4b80 1 -- 172.21.15.2:0/3218860453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69807f70 msgr2=0x55bb69814000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.557+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/3218860453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69807f70 0x55bb69814000 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x55bb69835550 tx=0x7f4b20013000 comp rx=0 tx=0).stop 2023-11-16T18:25:48.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.557+0000 7f4b419e4b80 1 -- 172.21.15.2:0/3218860453 shutdown_connections 2023-11-16T18:25:48.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.557+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/3218860453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb697fe560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.557+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/3218860453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69807f70 0x55bb69814000 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.557+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/3218860453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69808880 0x55bb69814540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.557+0000 7f4b419e4b80 1 -- 172.21.15.2:0/3218860453 >> 172.21.15.2:0/3218860453 conn(0x55bb697f9360 msgr2=0x55bb6982efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.557+0000 7f4b419e4b80 1 -- 172.21.15.2:0/3218860453 shutdown_connections 2023-11-16T18:25:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.557+0000 7f4b419e4b80 1 -- 172.21.15.2:0/3218860453 wait complete. 2023-11-16T18:25:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.557+0000 7f4b419e4b80 1 -- start start 2023-11-16T18:25:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b419e4b80 1 Processor -- start 2023-11-16T18:25:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b419e4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69807f70 0x55bb69818e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b419e4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 0x55bb698173a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b419e4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb69817980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b419e4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb69818110 con 0x55bb69808880 2023-11-16T18:25:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b419e4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb69886700 con 0x55bb6980ccc0 2023-11-16T18:25:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b419e4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb69886840 con 0x55bb69807f70 2023-11-16T18:25:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b419e4b80 10 client.0 _ll_register_callbacks cb 0x7f4b2000bdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b2e7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 0x55bb698173a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b2e7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 0x55bb698173a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50836/0 (socket says 172.21.15.2:50836) 2023-11-16T18:25:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/860450220 learned_addr learned my addr 172.21.15.2:0/860450220 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b2f7fe700 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb69817980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b2effd700 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69807f70 0x55bb69818e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/860450220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69807f70 msgr2=0x55bb69818e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69807f70 0x55bb69818e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/860450220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 msgr2=0x55bb69817980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb69817980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b2e7fc700 1 -- 172.21.15.2:0/860450220 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b20010e60 con 0x55bb69808880 2023-11-16T18:25:48.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.558+0000 7f4b2effd700 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69807f70 0x55bb69818e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:25:48.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b2e7fc700 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 0x55bb698173a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4b24095d80 tx=0x7f4b240108b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a844f6bd6e350512 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:48.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b0effd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:48.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b0effd700 1 -- 172.21.15.2:0/860450220 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b240991e0 con 0x55bb69808880 2023-11-16T18:25:48.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b0effd700 1 -- 172.21.15.2:0/860450220 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f4b240b9180 con 0x55bb69808880 2023-11-16T18:25:48.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b419e4b80 1 -- 172.21.15.2:0/860450220 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4b240a1cd0 con 0x55bb69808880 2023-11-16T18:25:48.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b0effd700 1 -- 172.21.15.2:0/860450220 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b240c9f10 con 0x55bb69808880 2023-11-16T18:25:48.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b419e4b80 10 client.15867 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:48.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b419e4b80 1 -- 172.21.15.2:0/860450220 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bb6956bc60 con 0x55bb69808880 2023-11-16T18:25:48.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b0effd700 1 -- 172.21.15.2:0/860450220 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b240a1cd0 con 0x55bb69808880 2023-11-16T18:25:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b0effd700 1 -- 172.21.15.2:0/860450220 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4b240987e0 con 0x55bb69808880 2023-11-16T18:25:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b419e4b80 10 client.15867 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b419e4b80 10 client.15867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:25:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b419e4b80 20 client.15867 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bb69887a10 0x55bb69889df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b419e4b80 4 client.15867 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:25:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.559+0000 7f4b419e4b80 1 -- 172.21.15.2:0/860450220 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55bb69723660 con 0x55bb69887a10 2023-11-16T18:25:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.560+0000 7f4b2effd700 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bb69887a10 0x55bb69889df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.560+0000 7f4b2effd700 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bb69887a10 0x55bb69889df0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc94207d6f74ff1d server_cookie=fafb558f2972fd90 in_seq=0 out_seq=0 2023-11-16T18:25:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.560+0000 7f4b0effd700 10 client.15867 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:48.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f93f3fff700 1 -- 172.21.15.2:0/2420787791 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56393826f600 con 0x5639383bdcc0 2023-11-16T18:25:48.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f93f3fff700 10 client.15861 handle_command_reply: tid=0 2023-11-16T18:25:48.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f9426f5db80 1 client.15861 shutdown 2023-11-16T18:25:48.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b0effd700 1 -- 172.21.15.2:0/860450220 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55bb69723660 con 0x55bb69887a10 2023-11-16T18:25:48.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b0effd700 10 client.15867 handle_command_reply: tid=0 2023-11-16T18:25:48.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b419e4b80 1 client.15867 shutdown 2023-11-16T18:25:48.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f9426f5db80 1 -- 172.21.15.2:0/2420787791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639381ab6b0 msgr2=0x5639383b5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f9426f5db80 1 --2- 172.21.15.2:0/2420787791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639381ab6b0 0x5639383b5ea0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9408010b80 tx=0x7f94080047c0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f9426f5db80 1 -- 172.21.15.2:0/2420787791 shutdown_connections 2023-11-16T18:25:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f9426f5db80 1 --2- 172.21.15.2:0/2420787791 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5639383bdcc0 0x5639383c00d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f9426f5db80 1 --2- 172.21.15.2:0/2420787791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563938355b90 0x563938361640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f93f3fff700 0 client.15861 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f9426f5db80 1 --2- 172.21.15.2:0/2420787791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639381ab6b0 0x5639383b5ea0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f9426f5db80 1 --2- 172.21.15.2:0/2420787791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563938355720 0x5639383b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f9426f5db80 1 -- 172.21.15.2:0/2420787791 >> 172.21.15.2:0/2420787791 conn(0x563938344f50 msgr2=0x563938355f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b419e4b80 1 -- 172.21.15.2:0/860450220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 msgr2=0x55bb698173a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 0x55bb698173a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4b24095d80 tx=0x7f4b240108b0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b419e4b80 1 -- 172.21.15.2:0/860450220 shutdown_connections 2023-11-16T18:25:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bb69887a10 0x55bb69889df0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bb69807f70 0x55bb69818e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb6980ccc0 0x55bb69817980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b0effd700 0 client.15867 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:25:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b419e4b80 1 --2- 172.21.15.2:0/860450220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bb69808880 0x55bb698173a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b419e4b80 1 -- 172.21.15.2:0/860450220 >> 172.21.15.2:0/860450220 conn(0x55bb697f9360 msgr2=0x55bb6980bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:48.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f9426f5db80 1 -- 172.21.15.2:0/2420787791 shutdown_connections 2023-11-16T18:25:48.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f9426f5db80 1 -- 172.21.15.2:0/2420787791 wait complete. 2023-11-16T18:25:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b419e4b80 1 -- 172.21.15.2:0/860450220 shutdown_connections 2023-11-16T18:25:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b419e4b80 1 -- 172.21.15.2:0/860450220 wait complete. 2023-11-16T18:25:48.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f9426f5db80 20 client.15861 trim_cache size 0 max 16384 2023-11-16T18:25:48.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:25:48.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:25:48.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:25:48.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:25:48.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.582+0000 7f4b419e4b80 20 client.15867 trim_cache size 0 max 16384 2023-11-16T18:25:48.593 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:25:48.593 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "no active scrubs running", 2023-11-16T18:25:48.593 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": {} 2023-11-16T18:25:48.593 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:25:48.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f941add1700 1 -- 172.21.15.2:0/1173194400 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f94040a13e0 msgr2=0x7f94040a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f941add1700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f94040a13e0 0x7f94040a3890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9408007be0 tx=0x7f94080073d0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f941add1700 1 -- 172.21.15.2:0/1173194400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9414105560 msgr2=0x7f9414198b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f941add1700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9414105560 0x7f9414198b80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f940c007ee0 tx=0x7f940c00d3b0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f4b35858700 1 -- 172.21.15.2:0/1516505211 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f4b180a1460 msgr2=0x7f4b180a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f4b35858700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f4b180a1460 0x7f4b180a3910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4b20005950 tx=0x7f4b2000b410 comp rx=0 tx=0).stop 2023-11-16T18:25:48.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f4b35858700 1 -- 172.21.15.2:0/1516505211 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b301015f0 msgr2=0x7f4b30196ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:48.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f4b35858700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b301015f0 0x7f4b30196ad0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f4b2400d980 tx=0x7f4b2400dc90 comp rx=0 tx=0).stop 2023-11-16T18:25:48.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f941add1700 1 -- 172.21.15.2:0/1173194400 shutdown_connections 2023-11-16T18:25:48.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f941add1700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f94040a13e0 0x7f94040a3890 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f941add1700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9414105560 0x7f9414198b80 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f941add1700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9414100c50 0x7f94141941e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f941add1700 1 --2- 172.21.15.2:0/1173194400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94141015f0 0x7f9414194720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f4b35858700 1 -- 172.21.15.2:0/1516505211 shutdown_connections 2023-11-16T18:25:48.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f4b35858700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f4b180a1460 0x7f4b180a3910 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f4b35858700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b30100c50 0x7f4b30196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f4b35858700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b30105610 0x7f4b3019af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f4b35858700 1 --2- 172.21.15.2:0/1516505211 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b301015f0 0x7f4b30196ad0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:48.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f4b35858700 1 -- 172.21.15.2:0/1516505211 >> 172.21.15.2:0/1516505211 conn(0x7f4b300775c0 msgr2=0x7f4b300fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:48.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f941add1700 1 -- 172.21.15.2:0/1173194400 >> 172.21.15.2:0/1173194400 conn(0x7f9414077550 msgr2=0x7f94140fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:48.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f941add1700 1 -- 172.21.15.2:0/1173194400 shutdown_connections 2023-11-16T18:25:48.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f4b35858700 1 -- 172.21.15.2:0/1516505211 shutdown_connections 2023-11-16T18:25:48.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f941add1700 1 -- 172.21.15.2:0/1173194400 wait complete. 2023-11-16T18:25:48.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:48.583+0000 7f4b35858700 1 -- 172.21.15.2:0/1516505211 wait complete. 2023-11-16T18:25:48.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:48 smithi002 ceph-mon[98807]: pgmap v31: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 10 op/s 2023-11-16T18:25:48.655 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T18:25:48.655 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:25:48.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:48 smithi040 ceph-mon[100251]: pgmap v31: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 10 op/s 2023-11-16T18:25:48.774 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:48.774 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:48.854 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:48.854 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:48.868 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:48.868 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:48.868 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 21 -lt 100 ']' 2023-11-16T18:25:48.868 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 21 + 1 2023-11-16T18:25:48.912 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=22 2023-11-16T18:25:48.912 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:48.921 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:48.921 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:48.921 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:48.922 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:48.922 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:49.266 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:49.266 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:49.346 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:49.346 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:49.394 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:49.394 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:49.394 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 22 -lt 100 ']' 2023-11-16T18:25:49.394 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 22 + 1 2023-11-16T18:25:49.454 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=23 2023-11-16T18:25:49.454 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:49.486 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:49.486 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:49.509 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:49.509 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:49.509 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:49.957 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:49.957 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:50.228 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:50.228 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:50.239 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:50.240 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:50.240 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 23 -lt 100 ']' 2023-11-16T18:25:50.240 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 23 + 1 2023-11-16T18:25:50.321 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=24 2023-11-16T18:25:50.321 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:50.379 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:50.379 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:50.380 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:50.381 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:50.381 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:50.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:50 smithi188 ceph-mon[100639]: pgmap v32: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-11-16T18:25:50.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:50 smithi002 ceph-mon[98807]: pgmap v32: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-11-16T18:25:50.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:50 smithi040 ceph-mon[100251]: pgmap v32: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-11-16T18:25:50.890 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:50.890 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:51.095 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:51.095 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:51.111 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:51.111 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:51.111 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 24 -lt 100 ']' 2023-11-16T18:25:51.112 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 24 + 1 2023-11-16T18:25:51.228 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=25 2023-11-16T18:25:51.229 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:51.285 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:51.285 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:51.286 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:51.286 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:51.286 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:51.534 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.533+0000 7f9560460700 1 -- 172.21.15.2:0/2219186476 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9558105630 msgr2=0x7f955810d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:51.534 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.533+0000 7f9560460700 1 --2- 172.21.15.2:0/2219186476 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9558105630 0x7f955810d200 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f955000b0d0 tx=0x7f955000b3e0 comp rx=0 tx=0).stop 2023-11-16T18:25:51.534 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.533+0000 7f9560460700 1 -- 172.21.15.2:0/2219186476 shutdown_connections 2023-11-16T18:25:51.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.533+0000 7f9560460700 1 --2- 172.21.15.2:0/2219186476 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9558105630 0x7f955810d200 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:51.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.533+0000 7f9560460700 1 --2- 172.21.15.2:0/2219186476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95581005f0 0x7f95581009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:51.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.533+0000 7f9560460700 1 --2- 172.21.15.2:0/2219186476 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9558100f90 0x7f9558104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:51.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.533+0000 7f9560460700 1 -- 172.21.15.2:0/2219186476 >> 172.21.15.2:0/2219186476 conn(0x7f9558077140 msgr2=0x7f9558077540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:51.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.533+0000 7f9560460700 1 -- 172.21.15.2:0/2219186476 shutdown_connections 2023-11-16T18:25:51.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.533+0000 7f9560460700 1 -- 172.21.15.2:0/2219186476 wait complete. 2023-11-16T18:25:51.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.533+0000 7f9560460700 1 Processor -- start 2023-11-16T18:25:51.535 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f9560460700 1 -- start start 2023-11-16T18:25:51.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f9560460700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95581005f0 0x7f9558196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:51.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f9560460700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9558100f90 0x7f9558196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:51.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f9560460700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9558105630 0x7f955819aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:51.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f9560460700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955819b520 con 0x7f95581005f0 2023-11-16T18:25:51.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f9560460700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955819b690 con 0x7f9558100f90 2023-11-16T18:25:51.537 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f9560460700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955819b850 con 0x7f9558105630 2023-11-16T18:25:51.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f955e1fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95581005f0 0x7f9558196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:51.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f955e1fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95581005f0 0x7f9558196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50848/0 (socket says 172.21.15.2:50848) 2023-11-16T18:25:51.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f955e1fc700 1 -- 172.21.15.2:0/3597482740 learned_addr learned my addr 172.21.15.2:0/3597482740 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:51.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f955d9fb700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9558100f90 0x7f9558196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:51.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f955e9fd700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9558105630 0x7f955819aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:51.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f955e1fc700 1 -- 172.21.15.2:0/3597482740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9558105630 msgr2=0x7f955819aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:51.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f955e1fc700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9558105630 0x7f955819aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:51.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f955e1fc700 1 -- 172.21.15.2:0/3597482740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9558100f90 msgr2=0x7f9558196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:51.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f955e1fc700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9558100f90 0x7f9558196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:51.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f955e1fc700 1 -- 172.21.15.2:0/3597482740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9550009d00 con 0x7f95581005f0 2023-11-16T18:25:51.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.534+0000 7f955e1fc700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95581005f0 0x7f9558196540 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f955400b530 tx=0x7f955400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:51.538 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.535+0000 7f954f7fe700 1 -- 172.21.15.2:0/3597482740 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554011830 con 0x7f95581005f0 2023-11-16T18:25:51.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.535+0000 7f954f7fe700 1 -- 172.21.15.2:0/3597482740 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9554011e50 con 0x7f95581005f0 2023-11-16T18:25:51.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.535+0000 7f9560460700 1 -- 172.21.15.2:0/3597482740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9558110520 con 0x7f95581005f0 2023-11-16T18:25:51.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.535+0000 7f954f7fe700 1 -- 172.21.15.2:0/3597482740 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955400f2e0 con 0x7f95581005f0 2023-11-16T18:25:51.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.535+0000 7f9560460700 1 -- 172.21.15.2:0/3597482740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9558110390 con 0x7f95581005f0 2023-11-16T18:25:51.539 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.536+0000 7f9560460700 1 -- 172.21.15.2:0/3597482740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95581106b0 con 0x7f95581005f0 2023-11-16T18:25:51.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.537+0000 7f954f7fe700 1 -- 172.21.15.2:0/3597482740 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f9554010580 con 0x7f95581005f0 2023-11-16T18:25:51.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.538+0000 7f954f7fe700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f95440a13e0 0x7f95440a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:51.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.538+0000 7f954f7fe700 1 -- 172.21.15.2:0/3597482740 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f95540cd350 con 0x7f95581005f0 2023-11-16T18:25:51.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.538+0000 7f955d9fb700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f95440a13e0 0x7f95440a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:51.541 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.539+0000 7f955d9fb700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f95440a13e0 0x7f95440a3890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9558197b50 tx=0x7f9548008040 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:51.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.542+0000 7f954f7fe700 1 -- 172.21.15.2:0/3597482740 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9554096250 con 0x7f95581005f0 2023-11-16T18:25:51.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.798+0000 7f9560460700 1 -- 172.21.15.2:0/3597482740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9558067af0 con 0x7f95581005f0 2023-11-16T18:25:51.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.798+0000 7f954f7fe700 1 -- 172.21.15.2:0/3597482740 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10774 (secure 0 0 0) 0x7f9554099870 con 0x7f95581005f0 2023-11-16T18:25:51.799 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:25:51.800 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:23:07.909835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:25:51.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.801+0000 7f9560460700 1 -- 172.21.15.2:0/3597482740 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f95440a13e0 msgr2=0x7f95440a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:51.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.801+0000 7f9560460700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f95440a13e0 0x7f95440a3890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9558197b50 tx=0x7f9548008040 comp rx=0 tx=0).stop 2023-11-16T18:25:51.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.801+0000 7f9560460700 1 -- 172.21.15.2:0/3597482740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95581005f0 msgr2=0x7f9558196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:51.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.801+0000 7f9560460700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95581005f0 0x7f9558196540 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f955400b530 tx=0x7f955400b8f0 comp rx=0 tx=0).stop 2023-11-16T18:25:51.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.801+0000 7f9560460700 1 -- 172.21.15.2:0/3597482740 shutdown_connections 2023-11-16T18:25:51.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.801+0000 7f9560460700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f95440a13e0 0x7f95440a3890 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:51.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.801+0000 7f9560460700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9558105630 0x7f955819aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:51.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.801+0000 7f9560460700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95581005f0 0x7f9558196540 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:51.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.801+0000 7f9560460700 1 --2- 172.21.15.2:0/3597482740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9558100f90 0x7f9558196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:51.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.801+0000 7f9560460700 1 -- 172.21.15.2:0/3597482740 >> 172.21.15.2:0/3597482740 conn(0x7f9558077140 msgr2=0x7f95580fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:51.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.801+0000 7f9560460700 1 -- 172.21.15.2:0/3597482740 shutdown_connections 2023-11-16T18:25:51.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:51.801+0000 7f9560460700 1 -- 172.21.15.2:0/3597482740 wait complete. 2023-11-16T18:25:51.803 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T18:25:51.907 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:51.907 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:51.914 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-11-16T18:25:52.104 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:52.104 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:52.109 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:52.109 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:52.109 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 25 -lt 100 ']' 2023-11-16T18:25:52.110 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 25 + 1 2023-11-16T18:25:52.153 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=26 2023-11-16T18:25:52.153 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:52.171 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:52.171 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:52.172 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:52.172 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:52.172 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:52.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:52 smithi188 ceph-mon[100639]: pgmap v33: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-11-16T18:25:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:52 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3597482740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:25:52.548 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:52 smithi002 ceph-mon[98807]: pgmap v33: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-11-16T18:25:52.548 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:52 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3597482740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:25:52.598 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:52.598 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:52.720 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:52.720 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:52.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:52 smithi040 ceph-mon[100251]: pgmap v33: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-11-16T18:25:52.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:52 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3597482740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:25:52.751 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:52.751 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:52.751 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 26 -lt 100 ']' 2023-11-16T18:25:52.751 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 26 + 1 2023-11-16T18:25:52.803 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=27 2023-11-16T18:25:52.803 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:52.826 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:52.826 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:52.827 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:52.827 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:52.827 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:53.331 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:53.332 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:53.536 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:53.536 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:53.561 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:53.561 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:53.561 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 27 -lt 100 ']' 2023-11-16T18:25:53.561 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 27 + 1 2023-11-16T18:25:53.645 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=28 2023-11-16T18:25:53.645 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:53.707 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:53.707 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:53.708 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:53.708 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:53.708 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:54.231 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:54.232 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:54.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:54 smithi002 ceph-mon[98807]: pgmap v34: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 13 op/s 2023-11-16T18:25:54.386 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:54.386 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:54.411 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:54.411 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:54.411 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 28 -lt 100 ']' 2023-11-16T18:25:54.411 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 28 + 1 2023-11-16T18:25:54.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=29 2023-11-16T18:25:54.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:54.519 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:54.519 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:54.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:54.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:54.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:54.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:54 smithi188 ceph-mon[100639]: pgmap v34: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 13 op/s 2023-11-16T18:25:54.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:54 smithi040 ceph-mon[100251]: pgmap v34: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 13 op/s 2023-11-16T18:25:54.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.857+0000 7f5701649700 1 -- 172.21.15.2:0/1635115037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0ff410 msgr2=0x7f56fc0ff7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:54.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.857+0000 7f5701649700 1 --2- 172.21.15.2:0/1635115037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0ff410 0x7f56fc0ff7e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0099c0 tx=0x7f56ec009cd0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.858+0000 7f5701649700 1 -- 172.21.15.2:0/1635115037 shutdown_connections 2023-11-16T18:25:54.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.858+0000 7f5701649700 1 --2- 172.21.15.2:0/1635115037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc104450 0x7f56fc10c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.858+0000 7f5701649700 1 --2- 172.21.15.2:0/1635115037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0ff410 0x7f56fc0ff7e0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.858+0000 7f5701649700 1 --2- 172.21.15.2:0/1635115037 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56fc0ffdb0 0x7f56fc103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.858+0000 7f5701649700 1 -- 172.21.15.2:0/1635115037 >> 172.21.15.2:0/1635115037 conn(0x7f56fc0763a0 msgr2=0x7f56fc0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:54.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.858+0000 7f5701649700 1 -- 172.21.15.2:0/1635115037 shutdown_connections 2023-11-16T18:25:54.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.858+0000 7f5701649700 1 -- 172.21.15.2:0/1635115037 wait complete. 2023-11-16T18:25:54.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.858+0000 7f5701649700 1 Processor -- start 2023-11-16T18:25:54.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f5701649700 1 -- start start 2023-11-16T18:25:54.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f5701649700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56fc0ff410 0x7f56fc100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f5701649700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0ffdb0 0x7f56fc101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f5701649700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc104450 0x7f56fc1a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f5701649700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc101b80 con 0x7f56fc0ffdb0 2023-11-16T18:25:54.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f5701649700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc1a4ab0 con 0x7f56fc104450 2023-11-16T18:25:54.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f5701649700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc1a4bf0 con 0x7f56fc0ff410 2023-11-16T18:25:54.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f56fa7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0ffdb0 0x7f56fc101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f56fa7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0ffdb0 0x7f56fc101350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50864/0 (socket says 172.21.15.2:50864) 2023-11-16T18:25:54.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f56fa7fc700 1 -- 172.21.15.2:0/2344005706 learned_addr learned my addr 172.21.15.2:0/2344005706 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:54.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f56fb7fe700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc104450 0x7f56fc1a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f56faffd700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56fc0ff410 0x7f56fc100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f56fa7fc700 1 -- 172.21.15.2:0/2344005706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56fc0ff410 msgr2=0x7f56fc100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:54.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f56fa7fc700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56fc0ff410 0x7f56fc100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f56fa7fc700 1 -- 172.21.15.2:0/2344005706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc104450 msgr2=0x7f56fc1a4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:54.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.859+0000 7f56fa7fc700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc104450 0x7f56fc1a4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.860+0000 7f56fa7fc700 1 -- 172.21.15.2:0/2344005706 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec0095f0 con 0x7f56fc0ffdb0 2023-11-16T18:25:54.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.860+0000 7f56fa7fc700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0ffdb0 0x7f56fc101350 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f56f000e940 tx=0x7f56f000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:54.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.860+0000 7f56e3fff700 1 -- 172.21.15.2:0/2344005706 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f0009730 con 0x7f56fc0ffdb0 2023-11-16T18:25:54.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.860+0000 7f56e3fff700 1 -- 172.21.15.2:0/2344005706 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56f0010430 con 0x7f56fc0ffdb0 2023-11-16T18:25:54.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.860+0000 7f5701649700 1 -- 172.21.15.2:0/2344005706 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc10f210 con 0x7f56fc0ffdb0 2023-11-16T18:25:54.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.860+0000 7f56e3fff700 1 -- 172.21.15.2:0/2344005706 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f000cea0 con 0x7f56fc0ffdb0 2023-11-16T18:25:54.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.860+0000 7f5701649700 1 -- 172.21.15.2:0/2344005706 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56fc10f080 con 0x7f56fc0ffdb0 2023-11-16T18:25:54.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.862+0000 7f56e3fff700 1 -- 172.21.15.2:0/2344005706 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f56f0010810 con 0x7f56fc0ffdb0 2023-11-16T18:25:54.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.863+0000 7f56e3fff700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f56e40a1460 0x7f56e40a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.863+0000 7f56e3fff700 1 -- 172.21.15.2:0/2344005706 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f56f0014070 con 0x7f56fc0ffdb0 2023-11-16T18:25:54.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.863+0000 7f56faffd700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f56e40a1460 0x7f56e40a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.864+0000 7f56faffd700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f56e40a1460 0x7f56e40a3910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009990 tx=0x7f56ec005fb0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:54.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.942+0000 7f570d7d5b80 1 Processor -- start 2023-11-16T18:25:54.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.942+0000 7f570d7d5b80 1 -- start start 2023-11-16T18:25:54.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.942+0000 7f570d7d5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f85f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.942+0000 7f570d7d5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 0x55fc98f8a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.942+0000 7f570d7d5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f8a8b0 0x55fc98f8ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.942+0000 7f570d7d5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc98e8b060 con 0x55fc98f85b60 2023-11-16T18:25:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.942+0000 7f570d7d5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc98edd8a0 con 0x55fc98f8a8b0 2023-11-16T18:25:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.942+0000 7f570d7d5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc98ed6d30 con 0x55fc98f86470 2023-11-16T18:25:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.942+0000 7f56faffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f85f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.942+0000 7f56faffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f85f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50870/0 (socket says 172.21.15.2:50870) 2023-11-16T18:25:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.942+0000 7f56faffd700 1 -- 172.21.15.2:0/4270354605 learned_addr learned my addr 172.21.15.2:0/4270354605 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.942+0000 7f56fa7fc700 1 --2- 172.21.15.2:0/4270354605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 0x55fc98f8a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.943+0000 7f56fb7fe700 1 --2- 172.21.15.2:0/4270354605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f8a8b0 0x55fc98f8ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.943+0000 7f56faffd700 1 -- 172.21.15.2:0/4270354605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 msgr2=0x55fc98f8a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.943+0000 7f56faffd700 1 --2- 172.21.15.2:0/4270354605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 0x55fc98f8a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.943+0000 7f56faffd700 1 -- 172.21.15.2:0/4270354605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f8a8b0 msgr2=0x55fc98f8ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:54.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.943+0000 7f56faffd700 1 --2- 172.21.15.2:0/4270354605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f8a8b0 0x55fc98f8ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.943+0000 7f56faffd700 1 -- 172.21.15.2:0/4270354605 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc98d58630 con 0x55fc98f85b60 2023-11-16T18:25:54.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.943+0000 7f56faffd700 1 --2- 172.21.15.2:0/4270354605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f85f30 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0099c0 tx=0x7f56ec01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960ba19fd313fdb4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:54.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.943+0000 7f56e17fa700 1 -- 172.21.15.2:0/4270354605 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec00b990 con 0x55fc98f85b60 2023-11-16T18:25:54.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.943+0000 7f56e17fa700 1 -- 172.21.15.2:0/4270354605 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56ec01e040 con 0x55fc98f85b60 2023-11-16T18:25:54.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.943+0000 7f56e17fa700 1 -- 172.21.15.2:0/4270354605 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec004800 con 0x55fc98f85b60 2023-11-16T18:25:54.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.944+0000 7f570d7d5b80 1 -- 172.21.15.2:0/4270354605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 msgr2=0x55fc98f85f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:54.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.944+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/4270354605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f85f30 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0099c0 tx=0x7f56ec01f000 comp rx=0 tx=0).stop 2023-11-16T18:25:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.944+0000 7f570d7d5b80 1 -- 172.21.15.2:0/4270354605 shutdown_connections 2023-11-16T18:25:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.944+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/4270354605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f8a8b0 0x55fc98f8ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.944+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/4270354605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f85f30 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.944+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/4270354605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 0x55fc98f8a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.944+0000 7f570d7d5b80 1 -- 172.21.15.2:0/4270354605 >> 172.21.15.2:0/4270354605 conn(0x55fc98f7b360 msgr2=0x55fc98f80b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.944+0000 7f570d7d5b80 1 -- 172.21.15.2:0/4270354605 shutdown_connections 2023-11-16T18:25:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.944+0000 7f570d7d5b80 1 -- 172.21.15.2:0/4270354605 wait complete. 2023-11-16T18:25:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.944+0000 7f570d7d5b80 1 -- start start 2023-11-16T18:25:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f570d7d5b80 1 Processor -- start 2023-11-16T18:25:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f570d7d5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f920a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f570d7d5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 0x55fc98f8f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f570d7d5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f8a8b0 0x55fc98f8fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f570d7d5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc98e788e0 con 0x55fc98f85b60 2023-11-16T18:25:54.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f570d7d5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc98e78580 con 0x55fc98f8a8b0 2023-11-16T18:25:54.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f570d7d5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc98d5a920 con 0x55fc98f86470 2023-11-16T18:25:54.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f570d7d5b80 10 client.0 _ll_register_callbacks cb 0x55fc98e96780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:54.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f56faffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f920a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f56faffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f920a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50876/0 (socket says 172.21.15.2:50876) 2023-11-16T18:25:54.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f56faffd700 1 -- 172.21.15.2:0/1751800133 learned_addr learned my addr 172.21.15.2:0/1751800133 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:54.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f56fa7fc700 1 --2- 172.21.15.2:0/1751800133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 0x55fc98f8f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f56fb7fe700 1 --2- 172.21.15.2:0/1751800133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f8a8b0 0x55fc98f8fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f56faffd700 1 -- 172.21.15.2:0/1751800133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 msgr2=0x55fc98f8f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:54.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.945+0000 7f56faffd700 1 --2- 172.21.15.2:0/1751800133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 0x55fc98f8f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.946+0000 7f56faffd700 1 -- 172.21.15.2:0/1751800133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f8a8b0 msgr2=0x55fc98f8fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.946+0000 7f56faffd700 1 --2- 172.21.15.2:0/1751800133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f8a8b0 0x55fc98f8fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.946+0000 7f56faffd700 1 -- 172.21.15.2:0/1751800133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56fc10c860 con 0x55fc98f85b60 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.946+0000 7f56faffd700 1 --2- 172.21.15.2:0/1751800133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f920a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0098e0 tx=0x7f56ec00b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed6b17126dc6e884 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.946+0000 7f56deff5700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.946+0000 7f56deff5700 1 -- 172.21.15.2:0/1751800133 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec00fa20 con 0x55fc98f85b60 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.946+0000 7f56deff5700 1 -- 172.21.15.2:0/1751800133 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f56ec00fb80 con 0x55fc98f85b60 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.946+0000 7f570d7d5b80 1 -- 172.21.15.2:0/1751800133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fc98f78b20 con 0x55fc98f85b60 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.946+0000 7f56deff5700 1 -- 172.21.15.2:0/1751800133 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56ec00fd40 con 0x55fc98f85b60 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.946+0000 7f570d7d5b80 10 client.15885 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.946+0000 7f570d7d5b80 1 -- 172.21.15.2:0/1751800133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fc98e9d190 con 0x55fc98f85b60 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.946+0000 7f56deff5700 1 -- 172.21.15.2:0/1751800133 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fc98f78b20 con 0x55fc98f85b60 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.947+0000 7f56deff5700 1 -- 172.21.15.2:0/1751800133 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f56ec03a880 con 0x55fc98f85b60 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.947+0000 7f570d7d5b80 10 client.15885 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.947+0000 7f570d7d5b80 10 client.15885 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2023-11-16T18:25:54.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.947+0000 7f570d7d5b80 20 client.15885 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:54.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.947+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/1751800133 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55fc98f9f4b0 0x55fc98fa1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.947+0000 7f570d7d5b80 4 client.15885 mds_command: new command op to 24446 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:25:54.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.947+0000 7f570d7d5b80 1 -- 172.21.15.2:0/1751800133 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fc98e990a0 con 0x55fc98f9f4b0 2023-11-16T18:25:54.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.947+0000 7f56fa7fc700 1 --2- 172.21.15.2:0/1751800133 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55fc98f9f4b0 0x55fc98fa1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.948+0000 7f56fa7fc700 1 --2- 172.21.15.2:0/1751800133 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55fc98f9f4b0 0x55fc98fa1890 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=34dab5c9b23fa3e8 server_cookie=fd2f1fe64cf601f8 in_seq=0 out_seq=0 2023-11-16T18:25:54.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.948+0000 7f56deff5700 10 client.15885 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:25:54.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.948+0000 7f56deff5700 1 -- 172.21.15.2:0/1751800133 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55fc98e990a0 con 0x55fc98f9f4b0 2023-11-16T18:25:54.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.948+0000 7f56deff5700 10 client.15885 handle_command_reply: tid=0 2023-11-16T18:25:54.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.948+0000 7f570d7d5b80 1 client.15885 shutdown 2023-11-16T18:25:54.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.948+0000 7f570d7d5b80 1 -- 172.21.15.2:0/1751800133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 msgr2=0x55fc98f920a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:54.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.948+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/1751800133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f920a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0098e0 tx=0x7f56ec00b5c0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.949+0000 7f570d7d5b80 1 -- 172.21.15.2:0/1751800133 shutdown_connections 2023-11-16T18:25:54.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.949+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/1751800133 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55fc98f9f4b0 0x55fc98fa1890 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.949+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/1751800133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f8a8b0 0x55fc98f8fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.949+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/1751800133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f920a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.949+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/1751800133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 0x55fc98f8f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.949+0000 7f56deff5700 0 client.15885 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:25:54.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.949+0000 7f570d7d5b80 1 -- 172.21.15.2:0/1751800133 >> 172.21.15.2:0/1751800133 conn(0x55fc98f7b360 msgr2=0x55fc98f8bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:54.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.949+0000 7f570d7d5b80 1 -- 172.21.15.2:0/1751800133 shutdown_connections 2023-11-16T18:25:54.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.949+0000 7f570d7d5b80 1 -- 172.21.15.2:0/1751800133 wait complete. 2023-11-16T18:25:54.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.949+0000 7f570d7d5b80 20 client.15885 trim_cache size 0 max 16384 2023-11-16T18:25:54.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.962+0000 7f570d7d5b80 1 Processor -- start 2023-11-16T18:25:54.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.962+0000 7f570d7d5b80 1 -- start start 2023-11-16T18:25:54.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f570d7d5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f85b60 0x55fc98f91c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f570d7d5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 0x55fc98f88b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f570d7d5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f8a8b0 0x55fc98f80560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f570d7d5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc98f82350 con 0x55fc98f8a8b0 2023-11-16T18:25:54.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f570d7d5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc98f90900 con 0x55fc98f85b60 2023-11-16T18:25:54.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f570d7d5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc98f89560 con 0x55fc98f86470 2023-11-16T18:25:54.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f56fb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f8a8b0 0x55fc98f80560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f56fb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f8a8b0 0x55fc98f80560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50888/0 (socket says 172.21.15.2:50888) 2023-11-16T18:25:54.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f56fb7fe700 1 -- 172.21.15.2:0/2970260072 learned_addr learned my addr 172.21.15.2:0/2970260072 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:54.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f56faffd700 1 --2- 172.21.15.2:0/2970260072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f85b60 0x55fc98f91c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f56fa7fc700 1 --2- 172.21.15.2:0/2970260072 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 0x55fc98f88b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f56fb7fe700 1 -- 172.21.15.2:0/2970260072 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 msgr2=0x55fc98f88b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:54.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f56fb7fe700 1 --2- 172.21.15.2:0/2970260072 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 0x55fc98f88b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f56fb7fe700 1 -- 172.21.15.2:0/2970260072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f85b60 msgr2=0x55fc98f91c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:54.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f56fb7fe700 1 --2- 172.21.15.2:0/2970260072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f85b60 0x55fc98f91c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f56fb7fe700 1 -- 172.21.15.2:0/2970260072 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc98e9cee0 con 0x55fc98f8a8b0 2023-11-16T18:25:54.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f56faffd700 1 --2- 172.21.15.2:0/2970260072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f85b60 0x55fc98f91c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:25:54.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.963+0000 7f56fb7fe700 1 --2- 172.21.15.2:0/2970260072 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f8a8b0 0x55fc98f80560 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f56f4003fb0 tx=0x7f56f4009940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670c3ba90c14bab server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:54.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.964+0000 7f56e1ffb700 1 -- 172.21.15.2:0/2970260072 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f4004510 con 0x55fc98f8a8b0 2023-11-16T18:25:54.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.964+0000 7f56e1ffb700 1 -- 172.21.15.2:0/2970260072 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56f400f040 con 0x55fc98f8a8b0 2023-11-16T18:25:54.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.964+0000 7f56e1ffb700 1 -- 172.21.15.2:0/2970260072 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f4016690 con 0x55fc98f8a8b0 2023-11-16T18:25:54.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.964+0000 7f570d7d5b80 1 -- 172.21.15.2:0/2970260072 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f8a8b0 msgr2=0x55fc98f80560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:54.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.964+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/2970260072 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f8a8b0 0x55fc98f80560 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f56f4003fb0 tx=0x7f56f4009940 comp rx=0 tx=0).stop 2023-11-16T18:25:54.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.964+0000 7f570d7d5b80 1 -- 172.21.15.2:0/2970260072 shutdown_connections 2023-11-16T18:25:54.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.964+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/2970260072 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f8a8b0 0x55fc98f80560 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.964+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/2970260072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f85b60 0x55fc98f91c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.964+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/2970260072 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f86470 0x55fc98f88b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.964+0000 7f570d7d5b80 1 -- 172.21.15.2:0/2970260072 >> 172.21.15.2:0/2970260072 conn(0x55fc98f7b360 msgr2=0x55fc98f9bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:54.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.964+0000 7f570d7d5b80 1 -- 172.21.15.2:0/2970260072 shutdown_connections 2023-11-16T18:25:54.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.964+0000 7f570d7d5b80 1 -- 172.21.15.2:0/2970260072 wait complete. 2023-11-16T18:25:54.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f570d7d5b80 1 -- start start 2023-11-16T18:25:54.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f570d7d5b80 1 Processor -- start 2023-11-16T18:25:54.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f570d7d5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f94b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f570d7d5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f86470 0x55fc98f95060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f570d7d5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f8a8b0 0x55fc98ff2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f570d7d5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc98f957e0 con 0x55fc98f85b60 2023-11-16T18:25:54.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f570d7d5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc98ff2880 con 0x55fc98f86470 2023-11-16T18:25:54.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f570d7d5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc98ff29f0 con 0x55fc98f8a8b0 2023-11-16T18:25:54.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f570d7d5b80 10 client.0 _ll_register_callbacks cb 0x7f56f0010f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:54.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f56faffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f94b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f56faffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f94b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50890/0 (socket says 172.21.15.2:50890) 2023-11-16T18:25:54.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f56faffd700 1 -- 172.21.15.2:0/3043768889 learned_addr learned my addr 172.21.15.2:0/3043768889 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:54.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f56fb7fe700 1 --2- 172.21.15.2:0/3043768889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f8a8b0 0x55fc98ff2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.965+0000 7f56fa7fc700 1 --2- 172.21.15.2:0/3043768889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f86470 0x55fc98f95060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f56faffd700 1 -- 172.21.15.2:0/3043768889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f8a8b0 msgr2=0x55fc98ff2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:54.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f56faffd700 1 --2- 172.21.15.2:0/3043768889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f8a8b0 0x55fc98ff2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f56faffd700 1 -- 172.21.15.2:0/3043768889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f86470 msgr2=0x55fc98f95060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:54.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f56faffd700 1 --2- 172.21.15.2:0/3043768889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f86470 0x55fc98f95060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f56faffd700 1 -- 172.21.15.2:0/3043768889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f4003c90 con 0x55fc98f85b60 2023-11-16T18:25:54.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f56faffd700 1 --2- 172.21.15.2:0/3043768889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f94b20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f56ec003ab0 tx=0x7f56ec03a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe09e996019c67d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:54.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f56deff5700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:54.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f56deff5700 1 -- 172.21.15.2:0/3043768889 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec03a980 con 0x55fc98f85b60 2023-11-16T18:25:54.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f56deff5700 1 -- 172.21.15.2:0/3043768889 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f56ec00f780 con 0x55fc98f85b60 2023-11-16T18:25:54.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f570d7d5b80 1 -- 172.21.15.2:0/3043768889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f56ec01e040 con 0x55fc98f85b60 2023-11-16T18:25:54.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f56deff5700 1 -- 172.21.15.2:0/3043768889 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56ec00f940 con 0x55fc98f85b60 2023-11-16T18:25:54.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f570d7d5b80 10 client.15891 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:54.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f570d7d5b80 1 -- 172.21.15.2:0/3043768889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fc98cedc60 con 0x55fc98f85b60 2023-11-16T18:25:54.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.966+0000 7f56deff5700 1 -- 172.21.15.2:0/3043768889 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56ec04d080 con 0x55fc98f85b60 2023-11-16T18:25:54.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.967+0000 7f56deff5700 1 -- 172.21.15.2:0/3043768889 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f56ec030550 con 0x55fc98f85b60 2023-11-16T18:25:54.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.967+0000 7f570d7d5b80 10 client.15891 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:54.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.967+0000 7f570d7d5b80 10 client.15891 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2023-11-16T18:25:54.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.967+0000 7f570d7d5b80 20 client.15891 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:54.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.967+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/3043768889 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55fc98ff3cb0 0x55fc98ff6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:54.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.967+0000 7f570d7d5b80 4 client.15891 mds_command: new command op to 24446 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:25:54.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.967+0000 7f570d7d5b80 1 -- 172.21.15.2:0/3043768889 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fc98ea5660 con 0x55fc98ff3cb0 2023-11-16T18:25:54.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.967+0000 7f56fa7fc700 1 --2- 172.21.15.2:0/3043768889 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55fc98ff3cb0 0x55fc98ff6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:54.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f56fa7fc700 1 --2- 172.21.15.2:0/3043768889 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55fc98ff3cb0 0x55fc98ff6090 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9512e90cc4bb6937 server_cookie=576746401cce6ec0 in_seq=0 out_seq=0 2023-11-16T18:25:54.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f56deff5700 10 client.15891 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:25:54.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f56deff5700 1 -- 172.21.15.2:0/3043768889 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fc98ea5660 con 0x55fc98ff3cb0 2023-11-16T18:25:54.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f56deff5700 10 client.15891 handle_command_reply: tid=0 2023-11-16T18:25:54.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f570d7d5b80 1 client.15891 shutdown 2023-11-16T18:25:54.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f570d7d5b80 1 -- 172.21.15.2:0/3043768889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 msgr2=0x55fc98f94b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:54.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/3043768889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f94b20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f56ec003ab0 tx=0x7f56ec03a480 comp rx=0 tx=0).stop 2023-11-16T18:25:54.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f570d7d5b80 1 -- 172.21.15.2:0/3043768889 shutdown_connections 2023-11-16T18:25:54.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/3043768889 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55fc98ff3cb0 0x55fc98ff6090 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/3043768889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98f8a8b0 0x55fc98ff2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/3043768889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fc98f85b60 0x55fc98f94b20 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f570d7d5b80 1 --2- 172.21.15.2:0/3043768889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc98f86470 0x55fc98f95060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f570d7d5b80 1 -- 172.21.15.2:0/3043768889 >> 172.21.15.2:0/3043768889 conn(0x55fc98f7b360 msgr2=0x55fc98f88b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:54.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f56deff5700 0 client.15891 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:25:54.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f570d7d5b80 1 -- 172.21.15.2:0/3043768889 shutdown_connections 2023-11-16T18:25:54.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f570d7d5b80 1 -- 172.21.15.2:0/3043768889 wait complete. 2023-11-16T18:25:54.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.968+0000 7f570d7d5b80 20 client.15891 trim_cache size 0 max 16384 2023-11-16T18:25:54.976 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:25:54.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.969+0000 7f5701649700 1 -- 172.21.15.2:0/2344005706 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f56e40a1460 msgr2=0x7f56e40a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:54.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.969+0000 7f5701649700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f56e40a1460 0x7f56e40a3910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009990 tx=0x7f56ec005fb0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.969+0000 7f5701649700 1 -- 172.21.15.2:0/2344005706 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0ffdb0 msgr2=0x7f56fc101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:54.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.969+0000 7f5701649700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0ffdb0 0x7f56fc101350 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f56f000e940 tx=0x7f56f000ec50 comp rx=0 tx=0).stop 2023-11-16T18:25:54.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.969+0000 7f5701649700 1 -- 172.21.15.2:0/2344005706 shutdown_connections 2023-11-16T18:25:54.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.969+0000 7f5701649700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f56e40a1460 0x7f56e40a3910 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.969+0000 7f5701649700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56fc0ff410 0x7f56fc100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.969+0000 7f5701649700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc104450 0x7f56fc1a4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.969+0000 7f5701649700 1 --2- 172.21.15.2:0/2344005706 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0ffdb0 0x7f56fc101350 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:54.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.969+0000 7f5701649700 1 -- 172.21.15.2:0/2344005706 >> 172.21.15.2:0/2344005706 conn(0x7f56fc0763a0 msgr2=0x7f56fc102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:54.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.969+0000 7f5701649700 1 -- 172.21.15.2:0/2344005706 shutdown_connections 2023-11-16T18:25:54.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:54.969+0000 7f5701649700 1 -- 172.21.15.2:0/2344005706 wait complete. 2023-11-16T18:25:55.024 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-11-16T18:25:55.148 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:55.148 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:55.237 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:55.237 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:55.269 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:55.269 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:55.270 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 29 -lt 100 ']' 2023-11-16T18:25:55.270 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 29 + 1 2023-11-16T18:25:55.311 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=30 2023-11-16T18:25:55.311 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:55.319 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:55.320 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:55.321 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:55.321 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:55.321 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:55.689 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:55.690 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:55.819 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:55.819 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:55.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:55.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:55.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 30 -lt 100 ']' 2023-11-16T18:25:55.842 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 30 + 1 2023-11-16T18:25:55.886 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=31 2023-11-16T18:25:55.886 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:55.909 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:55.909 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:55.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:55.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:55.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:56.416 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:56.417 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:56.594 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:56.594 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:56.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:56 smithi002 ceph-mon[98807]: pgmap v35: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 11 op/s 2023-11-16T18:25:56.671 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:56.671 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:56.671 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 31 -lt 100 ']' 2023-11-16T18:25:56.671 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 31 + 1 2023-11-16T18:25:56.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:56 smithi040 ceph-mon[100251]: pgmap v35: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 11 op/s 2023-11-16T18:25:56.736 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=32 2023-11-16T18:25:56.736 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:56.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:56 smithi188 ceph-mon[100639]: pgmap v35: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 11 op/s 2023-11-16T18:25:56.809 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:56.809 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:56.810 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:56.810 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:56.810 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:57.381 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:57.381 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:57.536 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:57.536 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:57.550 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:57.550 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:57.550 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 32 -lt 100 ']' 2023-11-16T18:25:57.550 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 32 + 1 2023-11-16T18:25:57.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:57 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:25:57.606 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=33 2023-11-16T18:25:57.606 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:57.618 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:57.619 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:57.619 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:57.619 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:57.620 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:57 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:25:57.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:57 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:25:57.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.964+0000 7f18ec68c700 1 -- 172.21.15.2:0/1067382336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e40ffa70 msgr2=0x7f18e40ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:57.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.964+0000 7f18ec68c700 1 --2- 172.21.15.2:0/1067382336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e40ffa70 0x7f18e40ffee0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009990 tx=0x7f18d8009ca0 comp rx=0 tx=0).stop 2023-11-16T18:25:57.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.964+0000 7f18ec68c700 1 -- 172.21.15.2:0/1067382336 shutdown_connections 2023-11-16T18:25:57.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.964+0000 7f18ec68c700 1 --2- 172.21.15.2:0/1067382336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e4100420 0x7f18e410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:57.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.964+0000 7f18ec68c700 1 --2- 172.21.15.2:0/1067382336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e4105250 0x7f18e4105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:57.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.964+0000 7f18ec68c700 1 --2- 172.21.15.2:0/1067382336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e40ffa70 0x7f18e40ffee0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:57.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.964+0000 7f18ec68c700 1 -- 172.21.15.2:0/1067382336 >> 172.21.15.2:0/1067382336 conn(0x7f18e40763a0 msgr2=0x7f18e40767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:57.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.964+0000 7f18ec68c700 1 -- 172.21.15.2:0/1067382336 shutdown_connections 2023-11-16T18:25:57.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.964+0000 7f18ec68c700 1 -- 172.21.15.2:0/1067382336 wait complete. 2023-11-16T18:25:57.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18ec68c700 1 Processor -- start 2023-11-16T18:25:57.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18ec68c700 1 -- start start 2023-11-16T18:25:57.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18ec68c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e40ffa70 0x7f18e4100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:57.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18ec68c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e4100420 0x7f18e41013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:57.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18ec68c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e4105250 0x7f18e4101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:57.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18ec68c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e41a47e0 con 0x7f18e4105250 2023-11-16T18:25:57.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18ec68c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e41a4950 con 0x7f18e4100420 2023-11-16T18:25:57.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18ec68c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e41a4b10 con 0x7f18e40ffa70 2023-11-16T18:25:57.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18eac29700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e4105250 0x7f18e4101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:57.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18eac29700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e4105250 0x7f18e4101930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50908/0 (socket says 172.21.15.2:50908) 2023-11-16T18:25:57.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18eac29700 1 -- 172.21.15.2:0/2979325548 learned_addr learned my addr 172.21.15.2:0/2979325548 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:57.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18e9c27700 1 --2- 172.21.15.2:0/2979325548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e4100420 0x7f18e41013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:57.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18eac29700 1 -- 172.21.15.2:0/2979325548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e40ffa70 msgr2=0x7f18e4100e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:57.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18eac29700 1 --2- 172.21.15.2:0/2979325548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e40ffa70 0x7f18e4100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:57.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18eac29700 1 -- 172.21.15.2:0/2979325548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e4100420 msgr2=0x7f18e41013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:57.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18eac29700 1 --2- 172.21.15.2:0/2979325548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e4100420 0x7f18e41013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:57.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.965+0000 7f18eac29700 1 -- 172.21.15.2:0/2979325548 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d80095f0 con 0x7f18e4105250 2023-11-16T18:25:57.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.966+0000 7f18eac29700 1 --2- 172.21.15.2:0/2979325548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e4105250 0x7f18e4101930 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f18dc007d70 tx=0x7f18dc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:57.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.966+0000 7f18d77fe700 1 -- 172.21.15.2:0/2979325548 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc00ad40 con 0x7f18e4105250 2023-11-16T18:25:57.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.966+0000 7f18ec68c700 1 -- 172.21.15.2:0/2979325548 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e410f230 con 0x7f18e4105250 2023-11-16T18:25:57.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.966+0000 7f18d77fe700 1 -- 172.21.15.2:0/2979325548 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18dc0128e0 con 0x7f18e4105250 2023-11-16T18:25:57.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.966+0000 7f18d77fe700 1 -- 172.21.15.2:0/2979325548 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc011780 con 0x7f18e4105250 2023-11-16T18:25:57.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.966+0000 7f18ec68c700 1 -- 172.21.15.2:0/2979325548 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e410f0a0 con 0x7f18e4105250 2023-11-16T18:25:57.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.968+0000 7f18d77fe700 1 -- 172.21.15.2:0/2979325548 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f18dc012a50 con 0x7f18e4105250 2023-11-16T18:25:57.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.968+0000 7f18d77fe700 1 --2- 172.21.15.2:0/2979325548 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f18d00a13e0 0x7f18d00a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:57.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.969+0000 7f18ea428700 1 --2- 172.21.15.2:0/2979325548 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f18d00a13e0 0x7f18d00a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:57.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.969+0000 7f18d77fe700 1 -- 172.21.15.2:0/2979325548 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f18dc0cd650 con 0x7f18e4105250 2023-11-16T18:25:57.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:57.969+0000 7f18ea428700 1 --2- 172.21.15.2:0/2979325548 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f18d00a13e0 0x7f18d00a3890 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f18e0005fd0 tx=0x7f18e0005ee0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:58.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.084+0000 7f18f8818b80 1 Processor -- start 2023-11-16T18:25:58.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.084+0000 7f18f8818b80 1 -- start start 2023-11-16T18:25:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18f8818b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0daf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18f8818b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0df370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18f8818b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 0x5637ed0dfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18f8818b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ecfe0060 con 0x5637ed0dab60 2023-11-16T18:25:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18f8818b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ed0328a0 con 0x5637ed0df8b0 2023-11-16T18:25:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18f8818b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ed02bd30 con 0x5637ed0db470 2023-11-16T18:25:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18ea428700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0daf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18ea428700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0daf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50912/0 (socket says 172.21.15.2:50912) 2023-11-16T18:25:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18ea428700 1 -- 172.21.15.2:0/1129904414 learned_addr learned my addr 172.21.15.2:0/1129904414 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18e9c27700 1 --2- 172.21.15.2:0/1129904414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0df370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18eac29700 1 --2- 172.21.15.2:0/1129904414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 0x5637ed0dfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18ea428700 1 -- 172.21.15.2:0/1129904414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 msgr2=0x5637ed0df370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18ea428700 1 --2- 172.21.15.2:0/1129904414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0df370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18ea428700 1 -- 172.21.15.2:0/1129904414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 msgr2=0x5637ed0dfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18ea428700 1 --2- 172.21.15.2:0/1129904414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 0x5637ed0dfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.085+0000 7f18ea428700 1 -- 172.21.15.2:0/1129904414 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637ecead630 con 0x5637ed0dab60 2023-11-16T18:25:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.086+0000 7f18ea428700 1 --2- 172.21.15.2:0/1129904414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0daf30 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f18e000dbe0 tx=0x7f18e0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c38a2cc83b00ffd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.086+0000 7f18d4ff9700 1 -- 172.21.15.2:0/1129904414 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0010af0 con 0x5637ed0dab60 2023-11-16T18:25:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.086+0000 7f18d4ff9700 1 -- 172.21.15.2:0/1129904414 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18e0010c50 con 0x5637ed0dab60 2023-11-16T18:25:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.086+0000 7f18d4ff9700 1 -- 172.21.15.2:0/1129904414 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e000f620 con 0x5637ed0dab60 2023-11-16T18:25:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.086+0000 7f18f8818b80 1 -- 172.21.15.2:0/1129904414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 msgr2=0x5637ed0daf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.086+0000 7f18f8818b80 1 --2- 172.21.15.2:0/1129904414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0daf30 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f18e000dbe0 tx=0x7f18e0010000 comp rx=0 tx=0).stop 2023-11-16T18:25:58.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 -- 172.21.15.2:0/1129904414 shutdown_connections 2023-11-16T18:25:58.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 --2- 172.21.15.2:0/1129904414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 0x5637ed0dfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 --2- 172.21.15.2:0/1129904414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0daf30 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 --2- 172.21.15.2:0/1129904414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0df370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 -- 172.21.15.2:0/1129904414 >> 172.21.15.2:0/1129904414 conn(0x5637ed0d0360 msgr2=0x5637ed0d5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:58.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 -- 172.21.15.2:0/1129904414 shutdown_connections 2023-11-16T18:25:58.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 -- 172.21.15.2:0/1129904414 wait complete. 2023-11-16T18:25:58.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 -- start start 2023-11-16T18:25:58.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 Processor -- start 2023-11-16T18:25:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0e7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0e4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 0x5637ed0e4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ecfcd8e0 con 0x5637ed0dab60 2023-11-16T18:25:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ecfcd580 con 0x5637ed0df8b0 2023-11-16T18:25:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637eceaf920 con 0x5637ed0db470 2023-11-16T18:25:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18f8818b80 10 client.0 _ll_register_callbacks cb 0x5637ecfeb780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18ea428700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0e7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.087+0000 7f18ea428700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0e7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50918/0 (socket says 172.21.15.2:50918) 2023-11-16T18:25:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18ea428700 1 -- 172.21.15.2:0/418946968 learned_addr learned my addr 172.21.15.2:0/418946968 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18eac29700 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 0x5637ed0e4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18e9c27700 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0e4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18ea428700 1 -- 172.21.15.2:0/418946968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 msgr2=0x5637ed0e4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18ea428700 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0e4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18ea428700 1 -- 172.21.15.2:0/418946968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 msgr2=0x5637ed0e4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18ea428700 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 0x5637ed0e4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18ea428700 1 -- 172.21.15.2:0/418946968 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e000d890 con 0x5637ed0dab60 2023-11-16T18:25:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18eac29700 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 0x5637ed0e4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18ea428700 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0e7050 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f18e000b330 tx=0x7f18e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b678386b43a219ef server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18cdffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18cdffb700 1 -- 172.21.15.2:0/418946968 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e001d640 con 0x5637ed0dab60 2023-11-16T18:25:58.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18cdffb700 1 -- 172.21.15.2:0/418946968 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f18e002d070 con 0x5637ed0dab60 2023-11-16T18:25:58.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18f8818b80 1 -- 172.21.15.2:0/418946968 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5637ed0cdb20 con 0x5637ed0dab60 2023-11-16T18:25:58.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18cdffb700 1 -- 172.21.15.2:0/418946968 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18e001d7a0 con 0x5637ed0dab60 2023-11-16T18:25:58.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18f8818b80 10 client.15903 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:58.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.088+0000 7f18f8818b80 1 -- 172.21.15.2:0/418946968 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5637ecff2190 con 0x5637ed0dab60 2023-11-16T18:25:58.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.089+0000 7f18cdffb700 1 -- 172.21.15.2:0/418946968 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5637ed0cdb20 con 0x5637ed0dab60 2023-11-16T18:25:58.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.089+0000 7f18cdffb700 1 -- 172.21.15.2:0/418946968 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f18e0026400 con 0x5637ed0dab60 2023-11-16T18:25:58.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.089+0000 7f18f8818b80 10 client.15903 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:58.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.089+0000 7f18f8818b80 10 client.15903 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-11-16T18:25:58.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.089+0000 7f18f8818b80 20 client.15903 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:58.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.089+0000 7f18f8818b80 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x5637ed0f4650 0x5637ed0f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.089+0000 7f18f8818b80 4 client.15903 mds_command: new command op to 24460 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:25:58.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.089+0000 7f18f8818b80 1 -- 172.21.15.2:0/418946968 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5637ecfee0a0 con 0x5637ed0f4650 2023-11-16T18:25:58.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.089+0000 7f18e9c27700 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x5637ed0f4650 0x5637ed0f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18cdffb700 10 client.15903 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:25:58.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18e9c27700 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x5637ed0f4650 0x5637ed0f6a60 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a1842cc4d320312a server_cookie=c411eee7da72df06 in_seq=0 out_seq=0 2023-11-16T18:25:58.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18cdffb700 1 -- 172.21.15.2:0/418946968 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5637ecfee0a0 con 0x5637ed0f4650 2023-11-16T18:25:58.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18cdffb700 10 client.15903 handle_command_reply: tid=0 2023-11-16T18:25:58.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18f8818b80 1 client.15903 shutdown 2023-11-16T18:25:58.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18f8818b80 1 -- 172.21.15.2:0/418946968 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 msgr2=0x5637ed0e7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:58.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18f8818b80 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0e7050 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f18e000b330 tx=0x7f18e0005880 comp rx=0 tx=0).stop 2023-11-16T18:25:58.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18f8818b80 1 -- 172.21.15.2:0/418946968 shutdown_connections 2023-11-16T18:25:58.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18f8818b80 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x5637ed0f4650 0x5637ed0f6a60 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18f8818b80 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 0x5637ed0e4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18f8818b80 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0e7050 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18cdffb700 0 client.15903 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:25:58.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18f8818b80 1 --2- 172.21.15.2:0/418946968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0e4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.090+0000 7f18f8818b80 1 -- 172.21.15.2:0/418946968 >> 172.21.15.2:0/418946968 conn(0x5637ed0d0360 msgr2=0x5637ed0e0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:58.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.091+0000 7f18f8818b80 1 -- 172.21.15.2:0/418946968 shutdown_connections 2023-11-16T18:25:58.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.091+0000 7f18f8818b80 1 -- 172.21.15.2:0/418946968 wait complete. 2023-11-16T18:25:58.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.091+0000 7f18f8818b80 20 client.15903 trim_cache size 0 max 16384 2023-11-16T18:25:58.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18f8818b80 1 Processor -- start 2023-11-16T18:25:58.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18f8818b80 1 -- start start 2023-11-16T18:25:58.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18f8818b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0dab60 0x5637ed0eb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18f8818b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0d4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18f8818b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0df8b0 0x5637ed0e6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18f8818b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ed0cfee0 con 0x5637ed0df8b0 2023-11-16T18:25:58.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18f8818b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ed0e27c0 con 0x5637ed0dab60 2023-11-16T18:25:58.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18f8818b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ed0de3e0 con 0x5637ed0db470 2023-11-16T18:25:58.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18eac29700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0df8b0 0x5637ed0e6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18eac29700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0df8b0 0x5637ed0e6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50930/0 (socket says 172.21.15.2:50930) 2023-11-16T18:25:58.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18eac29700 1 -- 172.21.15.2:0/1798711635 learned_addr learned my addr 172.21.15.2:0/1798711635 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:58.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18ea428700 1 --2- 172.21.15.2:0/1798711635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0dab60 0x5637ed0eb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18e9c27700 1 --2- 172.21.15.2:0/1798711635 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0d4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18eac29700 1 -- 172.21.15.2:0/1798711635 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 msgr2=0x5637ed0d4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:58.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18eac29700 1 --2- 172.21.15.2:0/1798711635 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0d4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18eac29700 1 -- 172.21.15.2:0/1798711635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0dab60 msgr2=0x5637ed0eb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:58.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18eac29700 1 --2- 172.21.15.2:0/1798711635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0dab60 0x5637ed0eb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.104+0000 7f18eac29700 1 -- 172.21.15.2:0/1798711635 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637ecff1ee0 con 0x5637ed0df8b0 2023-11-16T18:25:58.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.105+0000 7f18ea428700 1 --2- 172.21.15.2:0/1798711635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0dab60 0x5637ed0eb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:25:58.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.105+0000 7f18eac29700 1 --2- 172.21.15.2:0/1798711635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0df8b0 0x5637ed0e6b80 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f18dc099ec0 tx=0x7f18dc007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8eff502dfdc6c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:58.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.105+0000 7f18d57fa700 1 -- 172.21.15.2:0/1798711635 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc09e240 con 0x5637ed0df8b0 2023-11-16T18:25:58.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.105+0000 7f18d57fa700 1 -- 172.21.15.2:0/1798711635 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18dc0a2660 con 0x5637ed0df8b0 2023-11-16T18:25:58.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.105+0000 7f18d57fa700 1 -- 172.21.15.2:0/1798711635 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc0a21d0 con 0x5637ed0df8b0 2023-11-16T18:25:58.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.105+0000 7f18f8818b80 1 -- 172.21.15.2:0/1798711635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0df8b0 msgr2=0x5637ed0e6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:58.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.105+0000 7f18f8818b80 1 --2- 172.21.15.2:0/1798711635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0df8b0 0x5637ed0e6b80 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f18dc099ec0 tx=0x7f18dc007f70 comp rx=0 tx=0).stop 2023-11-16T18:25:58.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 -- 172.21.15.2:0/1798711635 shutdown_connections 2023-11-16T18:25:58.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 --2- 172.21.15.2:0/1798711635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0df8b0 0x5637ed0e6b80 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 --2- 172.21.15.2:0/1798711635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0dab60 0x5637ed0eb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 --2- 172.21.15.2:0/1798711635 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0d4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 -- 172.21.15.2:0/1798711635 >> 172.21.15.2:0/1798711635 conn(0x5637ed0d0360 msgr2=0x5637ed0fb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:58.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 -- 172.21.15.2:0/1798711635 shutdown_connections 2023-11-16T18:25:58.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 -- 172.21.15.2:0/1798711635 wait complete. 2023-11-16T18:25:58.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 -- start start 2023-11-16T18:25:58.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 Processor -- start 2023-11-16T18:25:58.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0e7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0e8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 0x5637ed147500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ed147a40 con 0x5637ed0dab60 2023-11-16T18:25:58.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ed147b80 con 0x5637ed0df8b0 2023-11-16T18:25:58.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ed147cc0 con 0x5637ed0db470 2023-11-16T18:25:58.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18f8818b80 10 client.0 _ll_register_callbacks cb 0x7f18dc099fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:25:58.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18ea428700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0e7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18ea428700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0e7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50932/0 (socket says 172.21.15.2:50932) 2023-11-16T18:25:58.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18ea428700 1 -- 172.21.15.2:0/2123688166 learned_addr learned my addr 172.21.15.2:0/2123688166 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:25:58.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.106+0000 7f18e9c27700 1 --2- 172.21.15.2:0/2123688166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0e8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18eac29700 1 --2- 172.21.15.2:0/2123688166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 0x5637ed147500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18ea428700 1 -- 172.21.15.2:0/2123688166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 msgr2=0x5637ed0e8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:58.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18ea428700 1 --2- 172.21.15.2:0/2123688166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0e8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18ea428700 1 -- 172.21.15.2:0/2123688166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 msgr2=0x5637ed147500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:25:58.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18ea428700 1 --2- 172.21.15.2:0/2123688166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 0x5637ed147500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18ea428700 1 -- 172.21.15.2:0/2123688166 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5637ecff2190 con 0x5637ed0dab60 2023-11-16T18:25:58.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18ea428700 1 --2- 172.21.15.2:0/2123688166 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0e7b40 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f18e000b680 tx=0x7f18e0026af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d69d2f2e7897db92 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:25:58.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18cdffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:25:58.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18cdffb700 1 -- 172.21.15.2:0/2123688166 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e001d640 con 0x5637ed0dab60 2023-11-16T18:25:58.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18f8818b80 1 -- 172.21.15.2:0/2123688166 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f18e0010c50 con 0x5637ed0dab60 2023-11-16T18:25:58.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18cdffb700 1 -- 172.21.15.2:0/2123688166 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f18e002d040 con 0x5637ed0dab60 2023-11-16T18:25:58.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18cdffb700 1 -- 172.21.15.2:0/2123688166 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18e001d7a0 con 0x5637ed0dab60 2023-11-16T18:25:58.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18f8818b80 10 client.15909 fetch_fsmap learned FSMap version 24 2023-11-16T18:25:58.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18f8818b80 1 -- 172.21.15.2:0/2123688166 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f18e410f0a0 con 0x5637ed0dab60 2023-11-16T18:25:58.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.107+0000 7f18cdffb700 1 -- 172.21.15.2:0/2123688166 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18e0010c50 con 0x5637ed0dab60 2023-11-16T18:25:58.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.108+0000 7f18cdffb700 1 -- 172.21.15.2:0/2123688166 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f18e0032ce0 con 0x5637ed0dab60 2023-11-16T18:25:58.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.108+0000 7f18f8818b80 10 client.15909 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:25:58.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.108+0000 7f18f8818b80 10 client.15909 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-11-16T18:25:58.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.108+0000 7f18f8818b80 20 client.15909 populate_metadata read hostname 'smithi002' 2023-11-16T18:25:58.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.108+0000 7f18f8818b80 1 --2- 172.21.15.2:0/2123688166 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x5637ed148f20 0x5637ed14b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:25:58.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.108+0000 7f18f8818b80 4 client.15909 mds_command: new command op to 24460 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:25:58.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.108+0000 7f18f8818b80 1 -- 172.21.15.2:0/2123688166 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5637ecffa660 con 0x5637ed148f20 2023-11-16T18:25:58.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.108+0000 7f18e9c27700 1 --2- 172.21.15.2:0/2123688166 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x5637ed148f20 0x5637ed14b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:25:58.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.108+0000 7f18cdffb700 10 client.15909 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:25:58.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.108+0000 7f18e9c27700 1 --2- 172.21.15.2:0/2123688166 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x5637ed148f20 0x5637ed14b330 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db7649acf7e8449c server_cookie=ef0ba1280cd91d3 in_seq=0 out_seq=0 2023-11-16T18:25:58.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18cdffb700 1 -- 172.21.15.2:0/2123688166 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5637ecffa660 con 0x5637ed148f20 2023-11-16T18:25:58.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18cdffb700 10 client.15909 handle_command_reply: tid=0 2023-11-16T18:25:58.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18f8818b80 1 client.15909 shutdown 2023-11-16T18:25:58.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18f8818b80 1 -- 172.21.15.2:0/2123688166 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 msgr2=0x5637ed0e7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:58.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18f8818b80 1 --2- 172.21.15.2:0/2123688166 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0e7b40 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f18e000b680 tx=0x7f18e0026af0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18f8818b80 1 -- 172.21.15.2:0/2123688166 shutdown_connections 2023-11-16T18:25:58.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18f8818b80 1 --2- 172.21.15.2:0/2123688166 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x5637ed148f20 0x5637ed14b330 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18f8818b80 1 --2- 172.21.15.2:0/2123688166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ed0df8b0 0x5637ed147500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18f8818b80 1 --2- 172.21.15.2:0/2123688166 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637ed0dab60 0x5637ed0e7b40 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18cdffb700 0 client.15909 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:25:58.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18f8818b80 1 --2- 172.21.15.2:0/2123688166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637ed0db470 0x5637ed0e8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18f8818b80 1 -- 172.21.15.2:0/2123688166 >> 172.21.15.2:0/2123688166 conn(0x5637ed0d0360 msgr2=0x5637ed0dd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:58.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18f8818b80 1 -- 172.21.15.2:0/2123688166 shutdown_connections 2023-11-16T18:25:58.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18f8818b80 1 -- 172.21.15.2:0/2123688166 wait complete. 2023-11-16T18:25:58.118 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18f8818b80 20 client.15909 trim_cache size 0 max 16384 2023-11-16T18:25:58.118 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:25:58.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18ec68c700 1 -- 172.21.15.2:0/2979325548 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f18d00a13e0 msgr2=0x7f18d00a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:58.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18ec68c700 1 --2- 172.21.15.2:0/2979325548 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f18d00a13e0 0x7f18d00a3890 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f18e0005fd0 tx=0x7f18e0005ee0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18ec68c700 1 -- 172.21.15.2:0/2979325548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e4105250 msgr2=0x7f18e4101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:25:58.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.109+0000 7f18ec68c700 1 --2- 172.21.15.2:0/2979325548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e4105250 0x7f18e4101930 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f18dc007d70 tx=0x7f18dc00a340 comp rx=0 tx=0).stop 2023-11-16T18:25:58.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.110+0000 7f18ec68c700 1 -- 172.21.15.2:0/2979325548 shutdown_connections 2023-11-16T18:25:58.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.110+0000 7f18ec68c700 1 --2- 172.21.15.2:0/2979325548 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f18d00a13e0 0x7f18d00a3890 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.110+0000 7f18ec68c700 1 --2- 172.21.15.2:0/2979325548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18e40ffa70 0x7f18e4100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.110+0000 7f18ec68c700 1 --2- 172.21.15.2:0/2979325548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e4105250 0x7f18e4101930 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.110+0000 7f18ec68c700 1 --2- 172.21.15.2:0/2979325548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e4100420 0x7f18e41013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:25:58.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.110+0000 7f18ec68c700 1 -- 172.21.15.2:0/2979325548 >> 172.21.15.2:0/2979325548 conn(0x7f18e40763a0 msgr2=0x7f18e40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:25:58.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.110+0000 7f18ec68c700 1 -- 172.21.15.2:0/2979325548 shutdown_connections 2023-11-16T18:25:58.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:25:58.110+0000 7f18ec68c700 1 -- 172.21.15.2:0/2979325548 wait complete. 2023-11-16T18:25:58.256 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:58.256 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:58.258 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T18:25:58.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:58.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:58.510 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:58.511 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:58.511 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 33 -lt 100 ']' 2023-11-16T18:25:58.511 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 33 + 1 2023-11-16T18:25:58.569 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=34 2023-11-16T18:25:58.569 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:58.577 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:58.578 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:58.578 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:58.579 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:58.579 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:25:58.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:25:58 smithi002 ceph-mon[98807]: pgmap v36: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 15 op/s 2023-11-16T18:25:58.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:25:58 smithi040 ceph-mon[100251]: pgmap v36: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 15 op/s 2023-11-16T18:25:58.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:25:58 smithi188 ceph-mon[100639]: pgmap v36: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 15 op/s 2023-11-16T18:25:59.231 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:59.231 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:59.386 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:25:59.386 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:25:59.417 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:59.417 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:25:59.417 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 34 -lt 100 ']' 2023-11-16T18:25:59.417 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 34 + 1 2023-11-16T18:25:59.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=35 2023-11-16T18:25:59.479 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:25:59.508 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:25:59.508 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:25:59.509 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:25:59.509 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:25:59.509 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:00.222 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:00.222 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:00.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:00 smithi002 ceph-mon[98807]: pgmap v37: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2023-11-16T18:26:00.627 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:00.627 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:00.685 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:00.685 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:00.685 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 35 -lt 100 ']' 2023-11-16T18:26:00.685 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 35 + 1 2023-11-16T18:26:00.718 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=36 2023-11-16T18:26:00.719 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:00.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:00 smithi040 ceph-mon[100251]: pgmap v37: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2023-11-16T18:26:00.747 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:00.747 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:00.748 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:00.748 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:00.748 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:00.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:00 smithi188 ceph-mon[100639]: pgmap v37: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2023-11-16T18:26:01.922 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:01.922 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:02.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:02.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:02.143 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:02.143 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:02.143 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 36 -lt 100 ']' 2023-11-16T18:26:02.143 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 36 + 1 2023-11-16T18:26:02.193 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=37 2023-11-16T18:26:02.193 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:02.288 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:02.288 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:02.289 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:02.289 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:02.289 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:02.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.556+0000 7f9836ae5700 1 -- 172.21.15.2:0/1987278218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98301055c0 msgr2=0x7f983010d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:02.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.556+0000 7f9836ae5700 1 --2- 172.21.15.2:0/1987278218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98301055c0 0x7f983010d140 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f982800b0d0 tx=0x7f982800b3e0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.556+0000 7f9836ae5700 1 -- 172.21.15.2:0/1987278218 shutdown_connections 2023-11-16T18:26:02.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.556+0000 7f9836ae5700 1 --2- 172.21.15.2:0/1987278218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98301055c0 0x7f983010d140 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.556+0000 7f9836ae5700 1 --2- 172.21.15.2:0/1987278218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9830100cc0 0x7f9830101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.556+0000 7f9836ae5700 1 --2- 172.21.15.2:0/1987278218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9830101660 0x7f9830104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.556+0000 7f9836ae5700 1 -- 172.21.15.2:0/1987278218 >> 172.21.15.2:0/1987278218 conn(0x7f9830077560 msgr2=0x7f9830077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:02.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.556+0000 7f9836ae5700 1 -- 172.21.15.2:0/1987278218 shutdown_connections 2023-11-16T18:26:02.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.556+0000 7f9836ae5700 1 -- 172.21.15.2:0/1987278218 wait complete. 2023-11-16T18:26:02.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.557+0000 7f9836ae5700 1 Processor -- start 2023-11-16T18:26:02.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.557+0000 7f9836ae5700 1 -- start start 2023-11-16T18:26:02.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.557+0000 7f9836ae5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9830100cc0 0x7f9830196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.557+0000 7f9836ae5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9830101660 0x7f9830196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.557+0000 7f9836ae5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98301055c0 0x7f983019aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.557+0000 7f9836ae5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983019b520 con 0x7f9830101660 2023-11-16T18:26:02.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.557+0000 7f9836ae5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983019b690 con 0x7f9830100cc0 2023-11-16T18:26:02.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.557+0000 7f9836ae5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983019b850 con 0x7f98301055c0 2023-11-16T18:26:02.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.557+0000 7f982ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9830101660 0x7f9830196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.557+0000 7f982ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9830101660 0x7f9830196a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50948/0 (socket says 172.21.15.2:50948) 2023-11-16T18:26:02.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.557+0000 7f982ffff700 1 -- 172.21.15.2:0/1341290310 learned_addr learned my addr 172.21.15.2:0/1341290310 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:02.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.557+0000 7f9834881700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9830100cc0 0x7f9830196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.557+0000 7f9835082700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98301055c0 0x7f983019aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.558+0000 7f982ffff700 1 -- 172.21.15.2:0/1341290310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98301055c0 msgr2=0x7f983019aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:02.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.558+0000 7f982ffff700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98301055c0 0x7f983019aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.558+0000 7f982ffff700 1 -- 172.21.15.2:0/1341290310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9830100cc0 msgr2=0x7f9830196540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:02.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.558+0000 7f982ffff700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9830100cc0 0x7f9830196540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.558+0000 7f982ffff700 1 -- 172.21.15.2:0/1341290310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9828009d00 con 0x7f9830101660 2023-11-16T18:26:02.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.558+0000 7f9835082700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98301055c0 0x7f983019aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:26:02.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.558+0000 7f982ffff700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9830101660 0x7f9830196a80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f982400e910 tx=0x7f982400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:02.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.558+0000 7f982dffb700 1 -- 172.21.15.2:0/1341290310 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982400cbd0 con 0x7f9830101660 2023-11-16T18:26:02.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.558+0000 7f982dffb700 1 -- 172.21.15.2:0/1341290310 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f982400ee60 con 0x7f9830101660 2023-11-16T18:26:02.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.558+0000 7f982dffb700 1 -- 172.21.15.2:0/1341290310 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98240055c0 con 0x7f9830101660 2023-11-16T18:26:02.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.558+0000 7f9836ae5700 1 -- 172.21.15.2:0/1341290310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9830110390 con 0x7f9830101660 2023-11-16T18:26:02.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.559+0000 7f9836ae5700 1 -- 172.21.15.2:0/1341290310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9830110200 con 0x7f9830101660 2023-11-16T18:26:02.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.560+0000 7f982dffb700 1 -- 172.21.15.2:0/1341290310 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f9824004750 con 0x7f9830101660 2023-11-16T18:26:02.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.561+0000 7f982dffb700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f98180a1460 0x7f98180a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.561+0000 7f982dffb700 1 -- 172.21.15.2:0/1341290310 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f9824014070 con 0x7f9830101660 2023-11-16T18:26:02.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.561+0000 7f9834881700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f98180a1460 0x7f98180a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.562+0000 7f9834881700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f98180a1460 0x7f98180a3910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9820009c60 tx=0x7f9820009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:02.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:02 smithi002 ceph-mon[98807]: pgmap v38: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2023-11-16T18:26:02.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.613+0000 7f9842c71b80 1 Processor -- start 2023-11-16T18:26:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.613+0000 7f9842c71b80 1 -- start start 2023-11-16T18:26:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.613+0000 7f9842c71b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd39f0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.613+0000 7f9842c71b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd39f5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.613+0000 7f9842c71b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f58b0 0x55bbd39f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.613+0000 7f9842c71b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbd38f6060 con 0x55bbd39f0b60 2023-11-16T18:26:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.613+0000 7f9842c71b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbd39488a0 con 0x55bbd39f58b0 2023-11-16T18:26:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.613+0000 7f9842c71b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbd3941d30 con 0x55bbd39f1470 2023-11-16T18:26:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f9834881700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd39f0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f9834881700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd39f0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50954/0 (socket says 172.21.15.2:50954) 2023-11-16T18:26:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f9834881700 1 -- 172.21.15.2:0/3070218366 learned_addr learned my addr 172.21.15.2:0/3070218366 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:02.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f982ffff700 1 --2- 172.21.15.2:0/3070218366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd39f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f9835082700 1 --2- 172.21.15.2:0/3070218366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f58b0 0x55bbd39f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f9834881700 1 -- 172.21.15.2:0/3070218366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 msgr2=0x55bbd39f5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:02.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f9834881700 1 --2- 172.21.15.2:0/3070218366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd39f5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f9834881700 1 -- 172.21.15.2:0/3070218366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f58b0 msgr2=0x55bbd39f5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:02.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f9834881700 1 --2- 172.21.15.2:0/3070218366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f58b0 0x55bbd39f5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f9834881700 1 -- 172.21.15.2:0/3070218366 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbd37c3630 con 0x55bbd39f0b60 2023-11-16T18:26:02.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f9835082700 1 --2- 172.21.15.2:0/3070218366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f58b0 0x55bbd39f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:02.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f982ffff700 1 --2- 172.21.15.2:0/3070218366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd39f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:02.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f9834881700 1 --2- 172.21.15.2:0/3070218366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd39f0f30 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f982000e4e0 tx=0x7f9820014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303b068ca56a7ee2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:02.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.614+0000 7f9816ffd700 1 -- 172.21.15.2:0/3070218366 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820026430 con 0x55bbd39f0b60 2023-11-16T18:26:02.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.615+0000 7f9816ffd700 1 -- 172.21.15.2:0/3070218366 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9820021070 con 0x55bbd39f0b60 2023-11-16T18:26:02.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.615+0000 7f9816ffd700 1 -- 172.21.15.2:0/3070218366 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982001d700 con 0x55bbd39f0b60 2023-11-16T18:26:02.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.615+0000 7f9842c71b80 1 -- 172.21.15.2:0/3070218366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 msgr2=0x55bbd39f0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:02.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.615+0000 7f9842c71b80 1 --2- 172.21.15.2:0/3070218366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd39f0f30 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f982000e4e0 tx=0x7f9820014000 comp rx=0 tx=0).stop 2023-11-16T18:26:02.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.615+0000 7f9842c71b80 1 -- 172.21.15.2:0/3070218366 shutdown_connections 2023-11-16T18:26:02.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.615+0000 7f9842c71b80 1 --2- 172.21.15.2:0/3070218366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f58b0 0x55bbd39f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.615+0000 7f9842c71b80 1 --2- 172.21.15.2:0/3070218366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd39f0f30 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.615+0000 7f9842c71b80 1 --2- 172.21.15.2:0/3070218366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd39f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.615+0000 7f9842c71b80 1 -- 172.21.15.2:0/3070218366 >> 172.21.15.2:0/3070218366 conn(0x55bbd39e6360 msgr2=0x55bbd39ebb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.615+0000 7f9842c71b80 1 -- 172.21.15.2:0/3070218366 shutdown_connections 2023-11-16T18:26:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.615+0000 7f9842c71b80 1 -- 172.21.15.2:0/3070218366 wait complete. 2023-11-16T18:26:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.616+0000 7f9842c71b80 1 -- start start 2023-11-16T18:26:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.616+0000 7f9842c71b80 1 Processor -- start 2023-11-16T18:26:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.616+0000 7f9842c71b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f0b60 0x55bbd39fcff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.616+0000 7f9842c71b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd39fa270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.616+0000 7f9842c71b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f58b0 0x55bbd39fa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.616+0000 7f9842c71b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbd38e38e0 con 0x55bbd39f58b0 2023-11-16T18:26:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.616+0000 7f9842c71b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbd38e3580 con 0x55bbd39f0b60 2023-11-16T18:26:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.616+0000 7f9842c71b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbd37c5920 con 0x55bbd39f1470 2023-11-16T18:26:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.616+0000 7f9842c71b80 10 client.0 _ll_register_callbacks cb 0x55bbd3901780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f9835082700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f58b0 0x55bbd39fa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f9835082700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f58b0 0x55bbd39fa8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50964/0 (socket says 172.21.15.2:50964) 2023-11-16T18:26:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f9834881700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f0b60 0x55bbd39fcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f9835082700 1 -- 172.21.15.2:0/2350389131 learned_addr learned my addr 172.21.15.2:0/2350389131 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f982ffff700 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd39fa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f9835082700 1 -- 172.21.15.2:0/2350389131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 msgr2=0x55bbd39fa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f9835082700 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd39fa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f9835082700 1 -- 172.21.15.2:0/2350389131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f0b60 msgr2=0x55bbd39fcff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f9835082700 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f0b60 0x55bbd39fcff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f9835082700 1 -- 172.21.15.2:0/2350389131 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9820017070 con 0x55bbd39f58b0 2023-11-16T18:26:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f982ffff700 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd39fa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f9834881700 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f0b60 0x55bbd39fcff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:02.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f9835082700 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f58b0 0x55bbd39fa8a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9828009610 tx=0x7f982800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4957e2be2c7016c0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:02.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f980ffff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:02.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f980ffff700 1 -- 172.21.15.2:0/2350389131 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982800e040 con 0x55bbd39f58b0 2023-11-16T18:26:02.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f980ffff700 1 -- 172.21.15.2:0/2350389131 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f98280074e0 con 0x55bbd39f58b0 2023-11-16T18:26:02.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f9842c71b80 1 -- 172.21.15.2:0/2350389131 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bbd39e3b20 con 0x55bbd39f58b0 2023-11-16T18:26:02.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f980ffff700 1 -- 172.21.15.2:0/2350389131 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9828004e60 con 0x55bbd39f58b0 2023-11-16T18:26:02.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.617+0000 7f9842c71b80 10 client.15921 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:02.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.618+0000 7f9842c71b80 1 -- 172.21.15.2:0/2350389131 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bbd3908190 con 0x55bbd39f58b0 2023-11-16T18:26:02.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.618+0000 7f980ffff700 1 -- 172.21.15.2:0/2350389131 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bbd39e3b20 con 0x55bbd39f58b0 2023-11-16T18:26:02.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.618+0000 7f980ffff700 1 -- 172.21.15.2:0/2350389131 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9828019050 con 0x55bbd39f58b0 2023-11-16T18:26:02.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.618+0000 7f9842c71b80 10 client.15921 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:02.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.618+0000 7f9842c71b80 10 client.15921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:02.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.618+0000 7f9842c71b80 20 client.15921 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:02.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.618+0000 7f9842c71b80 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bbd3a0ec10 0x55bbd3a11020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.618+0000 7f9842c71b80 4 client.15921 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:02.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.618+0000 7f9842c71b80 1 -- 172.21.15.2:0/2350389131 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bbd39040a0 con 0x55bbd3a0ec10 2023-11-16T18:26:02.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.618+0000 7f9834881700 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bbd3a0ec10 0x55bbd3a11020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.619+0000 7f980ffff700 10 client.15921 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:02.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.619+0000 7f9834881700 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bbd3a0ec10 0x55bbd3a11020 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d921c2a50a850f5 server_cookie=d01c01f8fd1e62ef in_seq=0 out_seq=0 2023-11-16T18:26:02.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f980ffff700 1 -- 172.21.15.2:0/2350389131 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55bbd39040a0 con 0x55bbd3a0ec10 2023-11-16T18:26:02.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f980ffff700 10 client.15921 handle_command_reply: tid=0 2023-11-16T18:26:02.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f9842c71b80 1 client.15921 shutdown 2023-11-16T18:26:02.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f9842c71b80 1 -- 172.21.15.2:0/2350389131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f58b0 msgr2=0x55bbd39fa8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:02.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f9842c71b80 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f58b0 0x55bbd39fa8a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9828009610 tx=0x7f982800cd50 comp rx=0 tx=0).stop 2023-11-16T18:26:02.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f9842c71b80 1 -- 172.21.15.2:0/2350389131 shutdown_connections 2023-11-16T18:26:02.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f9842c71b80 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bbd3a0ec10 0x55bbd3a11020 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f9842c71b80 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f58b0 0x55bbd39fa8a0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f980ffff700 0 client.15921 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:02.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f9842c71b80 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f0b60 0x55bbd39fcff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f9842c71b80 1 --2- 172.21.15.2:0/2350389131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd39fa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f9842c71b80 1 -- 172.21.15.2:0/2350389131 >> 172.21.15.2:0/2350389131 conn(0x55bbd39e6360 msgr2=0x55bbd39f6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:02.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f9842c71b80 1 -- 172.21.15.2:0/2350389131 shutdown_connections 2023-11-16T18:26:02.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f9842c71b80 1 -- 172.21.15.2:0/2350389131 wait complete. 2023-11-16T18:26:02.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.620+0000 7f9842c71b80 20 client.15921 trim_cache size 0 max 16384 2023-11-16T18:26:02.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.635+0000 7f9842c71b80 1 Processor -- start 2023-11-16T18:26:02.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.635+0000 7f9842c71b80 1 -- start start 2023-11-16T18:26:02.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9842c71b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd39f9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9842c71b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd3a0b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9842c71b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f58b0 0x55bbd39fcb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9842c71b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbd39fb850 con 0x55bbd39f0b60 2023-11-16T18:26:02.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9842c71b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbd39f4420 con 0x55bbd39f58b0 2023-11-16T18:26:02.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9842c71b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbd39f1900 con 0x55bbd39f1470 2023-11-16T18:26:02.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9834881700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd39f9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9834881700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd39f9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50968/0 (socket says 172.21.15.2:50968) 2023-11-16T18:26:02.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9834881700 1 -- 172.21.15.2:0/170353931 learned_addr learned my addr 172.21.15.2:0/170353931 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:02.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9835082700 1 --2- 172.21.15.2:0/170353931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f58b0 0x55bbd39fcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f982ffff700 1 --2- 172.21.15.2:0/170353931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd3a0b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9834881700 1 -- 172.21.15.2:0/170353931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 msgr2=0x55bbd3a0b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:02.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9834881700 1 --2- 172.21.15.2:0/170353931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd3a0b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9834881700 1 -- 172.21.15.2:0/170353931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f58b0 msgr2=0x55bbd39fcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:02.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9834881700 1 --2- 172.21.15.2:0/170353931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f58b0 0x55bbd39fcb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9834881700 1 -- 172.21.15.2:0/170353931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbd3907ee0 con 0x55bbd39f0b60 2023-11-16T18:26:02.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f982ffff700 1 --2- 172.21.15.2:0/170353931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd3a0b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:02.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.636+0000 7f9834881700 1 --2- 172.21.15.2:0/170353931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd39f9a30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f982000e4b0 tx=0x7f982001cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce13b240a5ebec04 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:02.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.637+0000 7f98177fe700 1 -- 172.21.15.2:0/170353931 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820033ae0 con 0x55bbd39f0b60 2023-11-16T18:26:02.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.637+0000 7f98177fe700 1 -- 172.21.15.2:0/170353931 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98200335e0 con 0x55bbd39f0b60 2023-11-16T18:26:02.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.637+0000 7f98177fe700 1 -- 172.21.15.2:0/170353931 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98200038e0 con 0x55bbd39f0b60 2023-11-16T18:26:02.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.637+0000 7f9842c71b80 1 -- 172.21.15.2:0/170353931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 msgr2=0x55bbd39f9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:02.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.637+0000 7f9842c71b80 1 --2- 172.21.15.2:0/170353931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd39f9a30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f982000e4b0 tx=0x7f982001cee0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.637+0000 7f9842c71b80 1 -- 172.21.15.2:0/170353931 shutdown_connections 2023-11-16T18:26:02.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.637+0000 7f9842c71b80 1 --2- 172.21.15.2:0/170353931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f58b0 0x55bbd39fcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.637+0000 7f9842c71b80 1 --2- 172.21.15.2:0/170353931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd39f9a30 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.637+0000 7f9842c71b80 1 --2- 172.21.15.2:0/170353931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f1470 0x55bbd3a0b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.637+0000 7f9842c71b80 1 -- 172.21.15.2:0/170353931 >> 172.21.15.2:0/170353931 conn(0x55bbd39e6360 msgr2=0x55bbd39f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:02.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.637+0000 7f9842c71b80 1 -- 172.21.15.2:0/170353931 shutdown_connections 2023-11-16T18:26:02.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.638+0000 7f9842c71b80 1 -- 172.21.15.2:0/170353931 wait complete. 2023-11-16T18:26:02.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.638+0000 7f9842c71b80 1 -- start start 2023-11-16T18:26:02.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.638+0000 7f9842c71b80 1 Processor -- start 2023-11-16T18:26:02.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.638+0000 7f9842c71b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd3a03e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.638+0000 7f9842c71b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f1470 0x55bbd3a04360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.638+0000 7f9842c71b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f58b0 0x55bbd3a048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.638+0000 7f9842c71b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbd3a61ba0 con 0x55bbd39f0b60 2023-11-16T18:26:02.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.638+0000 7f9842c71b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbd3a61d10 con 0x55bbd39f1470 2023-11-16T18:26:02.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.638+0000 7f9842c71b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbd3a61ed0 con 0x55bbd39f58b0 2023-11-16T18:26:02.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.638+0000 7f9842c71b80 10 client.0 _ll_register_callbacks cb 0x7f9820005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:02.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.638+0000 7f9834881700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd3a03e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f9834881700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd3a03e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50974/0 (socket says 172.21.15.2:50974) 2023-11-16T18:26:02.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f9834881700 1 -- 172.21.15.2:0/2930010024 learned_addr learned my addr 172.21.15.2:0/2930010024 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:02.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f9835082700 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f58b0 0x55bbd3a048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f982ffff700 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f1470 0x55bbd3a04360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f9834881700 1 -- 172.21.15.2:0/2930010024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f58b0 msgr2=0x55bbd3a048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:02.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f9834881700 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f58b0 0x55bbd3a048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f9834881700 1 -- 172.21.15.2:0/2930010024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f1470 msgr2=0x55bbd3a04360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:02.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f9834881700 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f1470 0x55bbd3a04360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f9834881700 1 -- 172.21.15.2:0/2930010024 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982002a040 con 0x55bbd39f0b60 2023-11-16T18:26:02.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f9835082700 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f58b0 0x55bbd3a048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:02.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f9834881700 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd3a03e20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f982001ceb0 tx=0x7f98200331b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5a14c83eafbdf7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:02.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f980ffff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:02.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f980ffff700 1 -- 172.21.15.2:0/2930010024 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98200333e0 con 0x55bbd39f0b60 2023-11-16T18:26:02.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f980ffff700 1 -- 172.21.15.2:0/2930010024 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f9820033ae0 con 0x55bbd39f0b60 2023-11-16T18:26:02.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.639+0000 7f9842c71b80 1 -- 172.21.15.2:0/2930010024 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9820021070 con 0x55bbd39f0b60 2023-11-16T18:26:02.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.640+0000 7f980ffff700 1 -- 172.21.15.2:0/2930010024 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9820033ca0 con 0x55bbd39f0b60 2023-11-16T18:26:02.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.640+0000 7f9842c71b80 10 client.15927 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:02.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.640+0000 7f9842c71b80 1 -- 172.21.15.2:0/2930010024 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bbd3758c60 con 0x55bbd39f0b60 2023-11-16T18:26:02.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.640+0000 7f980ffff700 1 -- 172.21.15.2:0/2930010024 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9820046020 con 0x55bbd39f0b60 2023-11-16T18:26:02.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.640+0000 7f980ffff700 1 -- 172.21.15.2:0/2930010024 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9820043070 con 0x55bbd39f0b60 2023-11-16T18:26:02.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.640+0000 7f9842c71b80 10 client.15927 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:02.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.640+0000 7f9842c71b80 10 client.15927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:02.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.640+0000 7f9842c71b80 20 client.15927 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:02.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.640+0000 7f9842c71b80 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bbd3a0ec10 0x55bbd3a65280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:02.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.640+0000 7f9842c71b80 4 client.15927 mds_command: new command op to 24473 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:26:02.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.640+0000 7f9842c71b80 1 -- 172.21.15.2:0/2930010024 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bbd3910660 con 0x55bbd3a0ec10 2023-11-16T18:26:02.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.640+0000 7f982ffff700 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bbd3a0ec10 0x55bbd3a65280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:02.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f980ffff700 10 client.15927 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:02.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f982ffff700 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bbd3a0ec10 0x55bbd3a65280 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4800efe04b064734 server_cookie=c2e5fad885daa47d in_seq=0 out_seq=0 2023-11-16T18:26:02.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f980ffff700 1 -- 172.21.15.2:0/2930010024 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bbd3910660 con 0x55bbd3a0ec10 2023-11-16T18:26:02.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f980ffff700 10 client.15927 handle_command_reply: tid=0 2023-11-16T18:26:02.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f9842c71b80 1 client.15927 shutdown 2023-11-16T18:26:02.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f9842c71b80 1 -- 172.21.15.2:0/2930010024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 msgr2=0x55bbd3a03e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:02.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f9842c71b80 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd3a03e20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f982001ceb0 tx=0x7f98200331b0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f9842c71b80 1 -- 172.21.15.2:0/2930010024 shutdown_connections 2023-11-16T18:26:02.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f9842c71b80 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bbd3a0ec10 0x55bbd3a65280 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f9842c71b80 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbd39f58b0 0x55bbd3a048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f980ffff700 0 client.15927 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:02.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f9842c71b80 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bbd39f0b60 0x55bbd3a03e20 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f9842c71b80 1 --2- 172.21.15.2:0/2930010024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbd39f1470 0x55bbd3a04360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f9842c71b80 1 -- 172.21.15.2:0/2930010024 >> 172.21.15.2:0/2930010024 conn(0x55bbd39e6360 msgr2=0x55bbd3a0a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:02.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f9842c71b80 1 -- 172.21.15.2:0/2930010024 shutdown_connections 2023-11-16T18:26:02.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f9842c71b80 1 -- 172.21.15.2:0/2930010024 wait complete. 2023-11-16T18:26:02.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.641+0000 7f9842c71b80 20 client.15927 trim_cache size 0 max 16384 2023-11-16T18:26:02.654 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:26:02.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.642+0000 7f9836ae5700 1 -- 172.21.15.2:0/1341290310 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f98180a1460 msgr2=0x7f98180a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:02.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.642+0000 7f9836ae5700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f98180a1460 0x7f98180a3910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9820009c60 tx=0x7f9820009260 comp rx=0 tx=0).stop 2023-11-16T18:26:02.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.642+0000 7f9836ae5700 1 -- 172.21.15.2:0/1341290310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9830101660 msgr2=0x7f9830196a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:02.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.642+0000 7f9836ae5700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9830101660 0x7f9830196a80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f982400e910 tx=0x7f982400ec20 comp rx=0 tx=0).stop 2023-11-16T18:26:02.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.642+0000 7f9836ae5700 1 -- 172.21.15.2:0/1341290310 shutdown_connections 2023-11-16T18:26:02.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.642+0000 7f9836ae5700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f98180a1460 0x7f98180a3910 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.642+0000 7f9836ae5700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98301055c0 0x7f983019aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.642+0000 7f9836ae5700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9830100cc0 0x7f9830196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.642+0000 7f9836ae5700 1 --2- 172.21.15.2:0/1341290310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9830101660 0x7f9830196a80 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:02.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.642+0000 7f9836ae5700 1 -- 172.21.15.2:0/1341290310 >> 172.21.15.2:0/1341290310 conn(0x7f9830077560 msgr2=0x7f9830103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:02.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.642+0000 7f9836ae5700 1 -- 172.21.15.2:0/1341290310 shutdown_connections 2023-11-16T18:26:02.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:02.642+0000 7f9836ae5700 1 -- 172.21.15.2:0/1341290310 wait complete. 2023-11-16T18:26:02.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:02 smithi040 ceph-mon[100251]: pgmap v38: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2023-11-16T18:26:02.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:02 smithi188 ceph-mon[100639]: pgmap v38: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2023-11-16T18:26:03.063 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:03.063 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:03.225 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:03.225 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:03.236 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:03.236 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:03.236 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 37 -lt 100 ']' 2023-11-16T18:26:03.236 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 37 + 1 2023-11-16T18:26:03.272 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=38 2023-11-16T18:26:03.272 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:03.310 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:03.310 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:03.311 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:03.311 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:03.311 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:03.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:03 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:26:03.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:03 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:26:03.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:03 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:26:03.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:03 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:26:03.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:03 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:26:03.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:03 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:26:03.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:03 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:26:03.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:03 smithi002 ceph-mon[98807]: pgmap v39: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 46 KiB/s wr, 27 op/s 2023-11-16T18:26:03.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:03 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:26:03.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:03 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:26:03.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:03 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:26:03.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:03 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:26:03.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:03 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:26:03.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:03 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:26:03.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:03 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:26:03.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:03 smithi040 ceph-mon[100251]: pgmap v39: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 46 KiB/s wr, 27 op/s 2023-11-16T18:26:03.738 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T18:26:03.813 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T18:26:03.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:03 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:26:03.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:03 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:26:03.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:03 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:26:03.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:03 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:26:03.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:03 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:26:03.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:03 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:26:03.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:03 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:26:03.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:03 smithi188 ceph-mon[100639]: pgmap v39: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 46 KiB/s wr, 27 op/s 2023-11-16T18:26:03.947 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:03.947 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:04.052 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:04.052 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:04.074 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:04.075 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:04.075 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 38 -lt 100 ']' 2023-11-16T18:26:04.075 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 38 + 1 2023-11-16T18:26:04.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=39 2023-11-16T18:26:04.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:04.158 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:04.158 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:04.159 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:04.159 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:04.159 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.522+0000 7fe6801d5700 1 -- 172.21.15.2:0/1077106115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe678105580 msgr2=0x7fe67810d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.522+0000 7fe6801d5700 1 --2- 172.21.15.2:0/1077106115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe678105580 0x7fe67810d100 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe67000b0d0 tx=0x7fe67000b3e0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.522+0000 7fe6801d5700 1 -- 172.21.15.2:0/1077106115 shutdown_connections 2023-11-16T18:26:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.522+0000 7fe6801d5700 1 --2- 172.21.15.2:0/1077106115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6780fe9e0 0x7fe6780fedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.522+0000 7fe6801d5700 1 --2- 172.21.15.2:0/1077106115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe678105580 0x7fe67810d100 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.522+0000 7fe6801d5700 1 --2- 172.21.15.2:0/1077106115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6780ff380 0x7fe678104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.522+0000 7fe6801d5700 1 -- 172.21.15.2:0/1077106115 >> 172.21.15.2:0/1077106115 conn(0x7fe678075810 msgr2=0x7fe678075c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.522+0000 7fe6801d5700 1 -- 172.21.15.2:0/1077106115 shutdown_connections 2023-11-16T18:26:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.522+0000 7fe6801d5700 1 -- 172.21.15.2:0/1077106115 wait complete. 2023-11-16T18:26:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe6801d5700 1 Processor -- start 2023-11-16T18:26:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe6801d5700 1 -- start start 2023-11-16T18:26:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe6801d5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6780fe9e0 0x7fe678196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe6801d5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6780ff380 0x7fe6781969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe6801d5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe678105580 0x7fe67819adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe6801d5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67819b3f0 con 0x7fe6780fe9e0 2023-11-16T18:26:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe6801d5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67819b560 con 0x7fe6780ff380 2023-11-16T18:26:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe6801d5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67819b720 con 0x7fe678105580 2023-11-16T18:26:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe67df71700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6780fe9e0 0x7fe678196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe67df71700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6780fe9e0 0x7fe678196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50988/0 (socket says 172.21.15.2:50988) 2023-11-16T18:26:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe67df71700 1 -- 172.21.15.2:0/3668679313 learned_addr learned my addr 172.21.15.2:0/3668679313 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe67e772700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe678105580 0x7fe67819adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe67d770700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6780ff380 0x7fe6781969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe67df71700 1 -- 172.21.15.2:0/3668679313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe678105580 msgr2=0x7fe67819adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe67df71700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe678105580 0x7fe67819adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe67df71700 1 -- 172.21.15.2:0/3668679313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6780ff380 msgr2=0x7fe6781969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe67df71700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6780ff380 0x7fe6781969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe67df71700 1 -- 172.21.15.2:0/3668679313 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe670009d00 con 0x7fe6780fe9e0 2023-11-16T18:26:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.523+0000 7fe67df71700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6780fe9e0 0x7fe678196480 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe67400b530 tx=0x7fe67400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.524+0000 7fe66effd700 1 -- 172.21.15.2:0/3668679313 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe674011830 con 0x7fe6780fe9e0 2023-11-16T18:26:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.524+0000 7fe66effd700 1 -- 172.21.15.2:0/3668679313 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe674011e50 con 0x7fe6780fe9e0 2023-11-16T18:26:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.524+0000 7fe6801d5700 1 -- 172.21.15.2:0/3668679313 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe678110370 con 0x7fe6780fe9e0 2023-11-16T18:26:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.524+0000 7fe66effd700 1 -- 172.21.15.2:0/3668679313 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67400f2e0 con 0x7fe6780fe9e0 2023-11-16T18:26:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.524+0000 7fe6801d5700 1 -- 172.21.15.2:0/3668679313 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6781101e0 con 0x7fe6780fe9e0 2023-11-16T18:26:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.526+0000 7fe66effd700 1 -- 172.21.15.2:0/3668679313 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7fe674010380 con 0x7fe6780fe9e0 2023-11-16T18:26:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.526+0000 7fe66effd700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe6640a1460 0x7fe6640a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.527+0000 7fe67d770700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe6640a1460 0x7fe6640a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.527+0000 7fe66effd700 1 -- 172.21.15.2:0/3668679313 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe6740ccc60 con 0x7fe6780fe9e0 2023-11-16T18:26:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.527+0000 7fe67d770700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe6640a1460 0x7fe6640a3910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe668005ea0 tx=0x7fe668005e30 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.606+0000 7fe68c361b80 1 Processor -- start 2023-11-16T18:26:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.606+0000 7fe68c361b80 1 -- start start 2023-11-16T18:26:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe68c361b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 0x55fa6dd0f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe68c361b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 0x55fa6dd09560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe68c361b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd09b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe68c361b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6dc0f000 con 0x55fa6dd0fb90 2023-11-16T18:26:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe68c361b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6dc61840 con 0x55fa6db656b0 2023-11-16T18:26:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe68c361b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6dc5acd0 con 0x55fa6dd0f720 2023-11-16T18:26:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe67e772700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd09b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe67e772700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd09b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51000/0 (socket says 172.21.15.2:51000) 2023-11-16T18:26:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe67e772700 1 -- 172.21.15.2:0/3594456399 learned_addr learned my addr 172.21.15.2:0/3594456399 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe67df71700 1 --2- 172.21.15.2:0/3594456399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 0x55fa6dd0f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe67d770700 1 --2- 172.21.15.2:0/3594456399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 0x55fa6dd09560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe67e772700 1 -- 172.21.15.2:0/3594456399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 msgr2=0x55fa6dd09560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe67e772700 1 --2- 172.21.15.2:0/3594456399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 0x55fa6dd09560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe67e772700 1 -- 172.21.15.2:0/3594456399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 msgr2=0x55fa6dd0f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe67e772700 1 --2- 172.21.15.2:0/3594456399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 0x55fa6dd0f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe67e772700 1 -- 172.21.15.2:0/3594456399 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa6dadc5d0 con 0x55fa6dd0fb90 2023-11-16T18:26:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe67df71700 1 --2- 172.21.15.2:0/3594456399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 0x55fa6dd0f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.607+0000 7fe67e772700 1 --2- 172.21.15.2:0/3594456399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd09b00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe670009150 tx=0x7fe670009180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fbdf0fcff3adaff server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.608+0000 7fe663fff700 1 -- 172.21.15.2:0/3594456399 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67000e070 con 0x55fa6dd0fb90 2023-11-16T18:26:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.608+0000 7fe663fff700 1 -- 172.21.15.2:0/3594456399 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6700079a0 con 0x55fa6dd0fb90 2023-11-16T18:26:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.608+0000 7fe663fff700 1 -- 172.21.15.2:0/3594456399 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6700054e0 con 0x55fa6dd0fb90 2023-11-16T18:26:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.608+0000 7fe68c361b80 1 -- 172.21.15.2:0/3594456399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 msgr2=0x55fa6dd09b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.608+0000 7fe68c361b80 1 --2- 172.21.15.2:0/3594456399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd09b00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe670009150 tx=0x7fe670009180 comp rx=0 tx=0).stop 2023-11-16T18:26:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.608+0000 7fe68c361b80 1 -- 172.21.15.2:0/3594456399 shutdown_connections 2023-11-16T18:26:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.608+0000 7fe68c361b80 1 --2- 172.21.15.2:0/3594456399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd09b00 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.608+0000 7fe68c361b80 1 --2- 172.21.15.2:0/3594456399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 0x55fa6dd0f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.608+0000 7fe68c361b80 1 --2- 172.21.15.2:0/3594456399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 0x55fa6dd09560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.608+0000 7fe68c361b80 1 -- 172.21.15.2:0/3594456399 >> 172.21.15.2:0/3594456399 conn(0x55fa6dcfef50 msgr2=0x55fa6dd04270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.608+0000 7fe68c361b80 1 -- 172.21.15.2:0/3594456399 shutdown_connections 2023-11-16T18:26:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.608+0000 7fe68c361b80 1 -- 172.21.15.2:0/3594456399 wait complete. 2023-11-16T18:26:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.609+0000 7fe68c361b80 1 -- start start 2023-11-16T18:26:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.609+0000 7fe68c361b80 1 Processor -- start 2023-11-16T18:26:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.609+0000 7fe68c361b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 0x55fa6dd18a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.609+0000 7fe68c361b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0f720 0x55fa6dd18f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.609+0000 7fe68c361b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd12ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.609+0000 7fe68c361b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6dbfc880 con 0x55fa6dd0f720 2023-11-16T18:26:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.609+0000 7fe68c361b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6dbfc520 con 0x55fa6db656b0 2023-11-16T18:26:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.609+0000 7fe68c361b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6dade8c0 con 0x55fa6dd0fb90 2023-11-16T18:26:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.609+0000 7fe68c361b80 10 client.0 _ll_register_callbacks cb 0x55fa6dc1a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe67d770700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0f720 0x55fa6dd18f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe67d770700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0f720 0x55fa6dd18f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51004/0 (socket says 172.21.15.2:51004) 2023-11-16T18:26:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe67d770700 1 -- 172.21.15.2:0/3496510083 learned_addr learned my addr 172.21.15.2:0/3496510083 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe67df71700 1 --2- 172.21.15.2:0/3496510083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 0x55fa6dd18a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe67e772700 1 --2- 172.21.15.2:0/3496510083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd12ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe67d770700 1 -- 172.21.15.2:0/3496510083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0fb90 msgr2=0x55fa6dd12ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe67d770700 1 --2- 172.21.15.2:0/3496510083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd12ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe67d770700 1 -- 172.21.15.2:0/3496510083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 msgr2=0x55fa6dd18a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe67d770700 1 --2- 172.21.15.2:0/3496510083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 0x55fa6dd18a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe67d770700 1 -- 172.21.15.2:0/3496510083 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67810d940 con 0x55fa6dd0f720 2023-11-16T18:26:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe67d770700 1 --2- 172.21.15.2:0/3496510083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0f720 0x55fa6dd18f50 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe66800a9e0 tx=0x7fe6680047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=496ae9c3ee883090 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe6617fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe6617fa700 1 -- 172.21.15.2:0/3496510083 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66801e840 con 0x55fa6dd0f720 2023-11-16T18:26:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe6617fa700 1 -- 172.21.15.2:0/3496510083 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe66801e9a0 con 0x55fa6dd0f720 2023-11-16T18:26:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe68c361b80 1 -- 172.21.15.2:0/3496510083 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fa6dcfc710 con 0x55fa6dd0f720 2023-11-16T18:26:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe6617fa700 1 -- 172.21.15.2:0/3496510083 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe66801eb60 con 0x55fa6dd0f720 2023-11-16T18:26:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe68c361b80 10 client.15939 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.610+0000 7fe68c361b80 1 -- 172.21.15.2:0/3496510083 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fa6dc21130 con 0x55fa6dd0f720 2023-11-16T18:26:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.611+0000 7fe6617fa700 1 -- 172.21.15.2:0/3496510083 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fa6dcfc710 con 0x55fa6dd0f720 2023-11-16T18:26:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.611+0000 7fe6617fa700 1 -- 172.21.15.2:0/3496510083 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe668024020 con 0x55fa6dd0f720 2023-11-16T18:26:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.611+0000 7fe68c361b80 10 client.15939 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.611+0000 7fe68c361b80 10 client.15939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.611+0000 7fe68c361b80 20 client.15939 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.611+0000 7fe68c361b80 1 --2- 172.21.15.2:0/3496510083 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fa6dd22f10 0x55fa6dd25320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.611+0000 7fe68c361b80 4 client.15939 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.611+0000 7fe68c361b80 1 -- 172.21.15.2:0/3496510083 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fa6dc1d040 con 0x55fa6dd22f10 2023-11-16T18:26:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.611+0000 7fe67df71700 1 --2- 172.21.15.2:0/3496510083 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fa6dd22f10 0x55fa6dd25320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.612+0000 7fe67df71700 1 --2- 172.21.15.2:0/3496510083 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fa6dd22f10 0x55fa6dd25320 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e8e5efdefc432e3 server_cookie=e1591baf3cf046df in_seq=0 out_seq=0 2023-11-16T18:26:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.612+0000 7fe6617fa700 10 client.15939 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.612+0000 7fe6617fa700 1 -- 172.21.15.2:0/3496510083 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55fa6dc1d040 con 0x55fa6dd22f10 2023-11-16T18:26:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.612+0000 7fe6617fa700 10 client.15939 handle_command_reply: tid=0 2023-11-16T18:26:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.612+0000 7fe68c361b80 1 client.15939 shutdown 2023-11-16T18:26:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.612+0000 7fe68c361b80 1 -- 172.21.15.2:0/3496510083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0f720 msgr2=0x55fa6dd18f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.612+0000 7fe68c361b80 1 --2- 172.21.15.2:0/3496510083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0f720 0x55fa6dd18f50 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe66800a9e0 tx=0x7fe6680047c0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.613+0000 7fe68c361b80 1 -- 172.21.15.2:0/3496510083 shutdown_connections 2023-11-16T18:26:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.613+0000 7fe68c361b80 1 --2- 172.21.15.2:0/3496510083 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fa6dd22f10 0x55fa6dd25320 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.613+0000 7fe68c361b80 1 --2- 172.21.15.2:0/3496510083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd12ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.613+0000 7fe68c361b80 1 --2- 172.21.15.2:0/3496510083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 0x55fa6dd18a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.613+0000 7fe6617fa700 0 client.15939 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.613+0000 7fe68c361b80 1 --2- 172.21.15.2:0/3496510083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0f720 0x55fa6dd18f50 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.613+0000 7fe68c361b80 1 -- 172.21.15.2:0/3496510083 >> 172.21.15.2:0/3496510083 conn(0x55fa6dcfef50 msgr2=0x55fa6dd037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.613+0000 7fe68c361b80 1 -- 172.21.15.2:0/3496510083 shutdown_connections 2023-11-16T18:26:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.613+0000 7fe68c361b80 1 -- 172.21.15.2:0/3496510083 wait complete. 2023-11-16T18:26:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.613+0000 7fe68c361b80 20 client.15939 trim_cache size 0 max 16384 2023-11-16T18:26:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe68c361b80 1 Processor -- start 2023-11-16T18:26:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe68c361b80 1 -- start start 2023-11-16T18:26:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe68c361b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6db656b0 0x55fa6dd12c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe68c361b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 0x55fa6dd03f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe68c361b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd09560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe68c361b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6dd0a2a0 con 0x55fa6db656b0 2023-11-16T18:26:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe68c361b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6dd0f1e0 con 0x55fa6dd0fb90 2023-11-16T18:26:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe68c361b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6dd131c0 con 0x55fa6dd0f720 2023-11-16T18:26:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe67df71700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6db656b0 0x55fa6dd12c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe67df71700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6db656b0 0x55fa6dd12c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51010/0 (socket says 172.21.15.2:51010) 2023-11-16T18:26:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe67df71700 1 -- 172.21.15.2:0/2244041358 learned_addr learned my addr 172.21.15.2:0/2244041358 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe67d770700 1 --2- 172.21.15.2:0/2244041358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 0x55fa6dd03f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe67e772700 1 --2- 172.21.15.2:0/2244041358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd09560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe67df71700 1 -- 172.21.15.2:0/2244041358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 msgr2=0x55fa6dd03f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe67df71700 1 --2- 172.21.15.2:0/2244041358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 0x55fa6dd03f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe67df71700 1 -- 172.21.15.2:0/2244041358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6dd0fb90 msgr2=0x55fa6dd09560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe67df71700 1 --2- 172.21.15.2:0/2244041358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd09560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe67df71700 1 -- 172.21.15.2:0/2244041358 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa6dd006f0 con 0x55fa6db656b0 2023-11-16T18:26:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe67d770700 1 --2- 172.21.15.2:0/2244041358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 0x55fa6dd03f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:26:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.629+0000 7fe67df71700 1 --2- 172.21.15.2:0/2244041358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6db656b0 0x55fa6dd12c50 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe674095e50 tx=0x7fe674004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cbdb96124c8b9a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.630+0000 7fe66cff9700 1 -- 172.21.15.2:0/2244041358 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6740a1da0 con 0x55fa6db656b0 2023-11-16T18:26:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.630+0000 7fe66cff9700 1 -- 172.21.15.2:0/2244041358 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe674097a90 con 0x55fa6db656b0 2023-11-16T18:26:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.630+0000 7fe66cff9700 1 -- 172.21.15.2:0/2244041358 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe674098e40 con 0x55fa6db656b0 2023-11-16T18:26:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.630+0000 7fe68c361b80 1 -- 172.21.15.2:0/2244041358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6db656b0 msgr2=0x55fa6dd12c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.630+0000 7fe68c361b80 1 --2- 172.21.15.2:0/2244041358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6db656b0 0x55fa6dd12c50 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe674095e50 tx=0x7fe674004860 comp rx=0 tx=0).stop 2023-11-16T18:26:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.630+0000 7fe68c361b80 1 -- 172.21.15.2:0/2244041358 shutdown_connections 2023-11-16T18:26:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.630+0000 7fe68c361b80 1 --2- 172.21.15.2:0/2244041358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd09560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.630+0000 7fe68c361b80 1 --2- 172.21.15.2:0/2244041358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6db656b0 0x55fa6dd12c50 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.630+0000 7fe68c361b80 1 --2- 172.21.15.2:0/2244041358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 0x55fa6dd03f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.630+0000 7fe68c361b80 1 -- 172.21.15.2:0/2244041358 >> 172.21.15.2:0/2244041358 conn(0x55fa6dcfef50 msgr2=0x55fa6dd12540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.630+0000 7fe68c361b80 1 -- 172.21.15.2:0/2244041358 shutdown_connections 2023-11-16T18:26:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.630+0000 7fe68c361b80 1 -- 172.21.15.2:0/2244041358 wait complete. 2023-11-16T18:26:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.631+0000 7fe68c361b80 1 -- start start 2023-11-16T18:26:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.631+0000 7fe68c361b80 1 Processor -- start 2023-11-16T18:26:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.631+0000 7fe68c361b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 0x55fa6dd165c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.631+0000 7fe68c361b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 0x55fa6dd16b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.631+0000 7fe68c361b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd17040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.631+0000 7fe68c361b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6dd75dd0 con 0x55fa6dd0fb90 2023-11-16T18:26:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.631+0000 7fe68c361b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6dd75f10 con 0x55fa6db656b0 2023-11-16T18:26:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.631+0000 7fe68c361b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa6dd76080 con 0x55fa6dd0f720 2023-11-16T18:26:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.631+0000 7fe68c361b80 10 client.0 _ll_register_callbacks cb 0x7fe66800ed60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe67e772700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd17040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe67e772700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd17040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51020/0 (socket says 172.21.15.2:51020) 2023-11-16T18:26:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe67e772700 1 -- 172.21.15.2:0/2016602014 learned_addr learned my addr 172.21.15.2:0/2016602014 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe67df71700 1 --2- 172.21.15.2:0/2016602014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 0x55fa6dd165c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe67d770700 1 --2- 172.21.15.2:0/2016602014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 0x55fa6dd16b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe67e772700 1 -- 172.21.15.2:0/2016602014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 msgr2=0x55fa6dd16b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe67e772700 1 --2- 172.21.15.2:0/2016602014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 0x55fa6dd16b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe67e772700 1 -- 172.21.15.2:0/2016602014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 msgr2=0x55fa6dd165c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe67e772700 1 --2- 172.21.15.2:0/2016602014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 0x55fa6dd165c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe67e772700 1 -- 172.21.15.2:0/2016602014 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6781101e0 con 0x55fa6dd0fb90 2023-11-16T18:26:04.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe67e772700 1 --2- 172.21.15.2:0/2016602014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd17040 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe670009bf0 tx=0x7fe670004c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f0eaa969b9f146 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:04.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe6617fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:04.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe6617fa700 1 -- 172.21.15.2:0/2016602014 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67000e040 con 0x55fa6dd0fb90 2023-11-16T18:26:04.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe6617fa700 1 -- 172.21.15.2:0/2016602014 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe67002c070 con 0x55fa6dd0fb90 2023-11-16T18:26:04.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe68c361b80 1 -- 172.21.15.2:0/2016602014 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6700079a0 con 0x55fa6dd0fb90 2023-11-16T18:26:04.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe6617fa700 1 -- 172.21.15.2:0/2016602014 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe670009450 con 0x55fa6dd0fb90 2023-11-16T18:26:04.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe68c361b80 10 client.15945 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.632+0000 7fe68c361b80 1 -- 172.21.15.2:0/2016602014 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fa6da88d80 con 0x55fa6dd0fb90 2023-11-16T18:26:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.633+0000 7fe6617fa700 1 -- 172.21.15.2:0/2016602014 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6700079a0 con 0x55fa6dd0fb90 2023-11-16T18:26:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.633+0000 7fe6617fa700 1 -- 172.21.15.2:0/2016602014 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe670019050 con 0x55fa6dd0fb90 2023-11-16T18:26:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.633+0000 7fe68c361b80 10 client.15945 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.633+0000 7fe68c361b80 10 client.15945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.633+0000 7fe68c361b80 20 client.15945 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.633+0000 7fe68c361b80 1 --2- 172.21.15.2:0/2016602014 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fa6dd77350 0x55fa6dd79730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.633+0000 7fe68c361b80 4 client.15945 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:26:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.633+0000 7fe68c361b80 1 -- 172.21.15.2:0/2016602014 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fa6dc29600 con 0x55fa6dd77350 2023-11-16T18:26:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.633+0000 7fe67df71700 1 --2- 172.21.15.2:0/2016602014 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fa6dd77350 0x55fa6dd79730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.633+0000 7fe67df71700 1 --2- 172.21.15.2:0/2016602014 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fa6dd77350 0x55fa6dd79730 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16b1a8ab78aadd01 server_cookie=8b737477e3fb6d46 in_seq=0 out_seq=0 2023-11-16T18:26:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.633+0000 7fe6617fa700 10 client.15945 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.662+0000 7fe6617fa700 1 -- 172.21.15.2:0/2016602014 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fa6dc29600 con 0x55fa6dd77350 2023-11-16T18:26:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.662+0000 7fe6617fa700 10 client.15945 handle_command_reply: tid=0 2023-11-16T18:26:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.662+0000 7fe68c361b80 1 client.15945 shutdown 2023-11-16T18:26:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.662+0000 7fe68c361b80 1 -- 172.21.15.2:0/2016602014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 msgr2=0x55fa6dd17040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.662+0000 7fe68c361b80 1 --2- 172.21.15.2:0/2016602014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd17040 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe670009bf0 tx=0x7fe670004c40 comp rx=0 tx=0).stop 2023-11-16T18:26:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.662+0000 7fe68c361b80 1 -- 172.21.15.2:0/2016602014 shutdown_connections 2023-11-16T18:26:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe68c361b80 1 --2- 172.21.15.2:0/2016602014 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55fa6dd77350 0x55fa6dd79730 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe68c361b80 1 --2- 172.21.15.2:0/2016602014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55fa6dd0fb90 0x55fa6dd17040 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe68c361b80 1 --2- 172.21.15.2:0/2016602014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa6db656b0 0x55fa6dd165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe68c361b80 1 --2- 172.21.15.2:0/2016602014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa6dd0f720 0x55fa6dd16b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe6617fa700 0 client.15945 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe68c361b80 1 -- 172.21.15.2:0/2016602014 >> 172.21.15.2:0/2016602014 conn(0x55fa6dcfef50 msgr2=0x55fa6dd0ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe68c361b80 1 -- 172.21.15.2:0/2016602014 shutdown_connections 2023-11-16T18:26:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe68c361b80 1 -- 172.21.15.2:0/2016602014 wait complete. 2023-11-16T18:26:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe68c361b80 20 client.15945 trim_cache size 0 max 16384 2023-11-16T18:26:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:26:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:26:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:26:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:26:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe6801d5700 1 -- 172.21.15.2:0/3668679313 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe6640a1460 msgr2=0x7fe6640a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe6801d5700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe6640a1460 0x7fe6640a3910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe668005ea0 tx=0x7fe668005e30 comp rx=0 tx=0).stop 2023-11-16T18:26:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe6801d5700 1 -- 172.21.15.2:0/3668679313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6780fe9e0 msgr2=0x7fe678196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe6801d5700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6780fe9e0 0x7fe678196480 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe67400b530 tx=0x7fe67400b840 comp rx=0 tx=0).stop 2023-11-16T18:26:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe6801d5700 1 -- 172.21.15.2:0/3668679313 shutdown_connections 2023-11-16T18:26:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe6801d5700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe6640a1460 0x7fe6640a3910 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe6801d5700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe678105580 0x7fe67819adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe6801d5700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6780fe9e0 0x7fe678196480 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe6801d5700 1 --2- 172.21.15.2:0/3668679313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6780ff380 0x7fe6781969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe6801d5700 1 -- 172.21.15.2:0/3668679313 >> 172.21.15.2:0/3668679313 conn(0x7fe678075810 msgr2=0x7fe6780fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe6801d5700 1 -- 172.21.15.2:0/3668679313 shutdown_connections 2023-11-16T18:26:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:04.663+0000 7fe6801d5700 1 -- 172.21.15.2:0/3668679313 wait complete. 2023-11-16T18:26:04.881 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:04.881 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:05.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:05.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:05.409 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:05.409 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:05.409 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 39 -lt 100 ']' 2023-11-16T18:26:05.409 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 39 + 1 2023-11-16T18:26:05.530 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=40 2023-11-16T18:26:05.530 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:05.583 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:05.583 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:05.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:05.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:05.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:06.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:06 smithi188 ceph-mon[100639]: pgmap v40: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 36 KiB/s wr, 19 op/s 2023-11-16T18:26:06.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:06 smithi002 ceph-mon[98807]: pgmap v40: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 36 KiB/s wr, 19 op/s 2023-11-16T18:26:06.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:06 smithi040 ceph-mon[100251]: pgmap v40: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 36 KiB/s wr, 19 op/s 2023-11-16T18:26:07.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:07 smithi002 ceph-mon[98807]: pgmap v41: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 47 KiB/s wr, 28 op/s 2023-11-16T18:26:07.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:07 smithi188 ceph-mon[100639]: pgmap v41: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 47 KiB/s wr, 28 op/s 2023-11-16T18:26:07.679 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:07.680 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:07.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:07 smithi040 ceph-mon[100251]: pgmap v41: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 47 KiB/s wr, 28 op/s 2023-11-16T18:26:08.084 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:08.084 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:08.182 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:08.182 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:08.182 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 40 -lt 100 ']' 2023-11-16T18:26:08.183 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 40 + 1 2023-11-16T18:26:08.259 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=41 2023-11-16T18:26:08.259 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:08.355 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:08.355 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:08.357 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:08.358 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:08.358 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:09.596 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:09.596 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:09.826 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:09.826 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:09.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:09.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:09.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 41 -lt 100 ']' 2023-11-16T18:26:09.842 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 41 + 1 2023-11-16T18:26:09.896 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=42 2023-11-16T18:26:09.896 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:09.926 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:09.926 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:09.927 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:09.927 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:09.927 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:10.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:10 smithi188 ceph-mon[100639]: pgmap v42: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 39 KiB/s wr, 24 op/s 2023-11-16T18:26:10.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:10 smithi002 ceph-mon[98807]: pgmap v42: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 39 KiB/s wr, 24 op/s 2023-11-16T18:26:10.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:10 smithi040 ceph-mon[100251]: pgmap v42: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 39 KiB/s wr, 24 op/s 2023-11-16T18:26:10.654 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:10.654 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:10.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.696+0000 7fe25e895700 1 -- 172.21.15.2:0/2575793860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2581007c0 msgr2=0x7fe258100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:10.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.696+0000 7fe25e895700 1 --2- 172.21.15.2:0/2575793860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2581007c0 0x7fe258100b90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe2480099c0 tx=0x7fe248009cd0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.696+0000 7fe25e895700 1 -- 172.21.15.2:0/2575793860 shutdown_connections 2023-11-16T18:26:10.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.696+0000 7fe25e895700 1 --2- 172.21.15.2:0/2575793860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe258109d70 0x7fe25810c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.696+0000 7fe25e895700 1 --2- 172.21.15.2:0/2575793860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2581007c0 0x7fe258100b90 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.696+0000 7fe25e895700 1 --2- 172.21.15.2:0/2575793860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2581010d0 0x7fe258109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.696+0000 7fe25e895700 1 -- 172.21.15.2:0/2575793860 >> 172.21.15.2:0/2575793860 conn(0x7fe258074c60 msgr2=0x7fe258075060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:10.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.696+0000 7fe25e895700 1 -- 172.21.15.2:0/2575793860 shutdown_connections 2023-11-16T18:26:10.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.696+0000 7fe25e895700 1 -- 172.21.15.2:0/2575793860 wait complete. 2023-11-16T18:26:10.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.697+0000 7fe25e895700 1 Processor -- start 2023-11-16T18:26:10.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.697+0000 7fe25e895700 1 -- start start 2023-11-16T18:26:10.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe25e895700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2581007c0 0x7fe258196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe25e895700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2581010d0 0x7fe258196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe25e895700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe258109d70 0x7fe25819afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe25e895700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25819b5e0 con 0x7fe2581010d0 2023-11-16T18:26:10.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe25e895700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25819b750 con 0x7fe258109d70 2023-11-16T18:26:10.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe25e895700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25819b910 con 0x7fe2581007c0 2023-11-16T18:26:10.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe2577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2581010d0 0x7fe258196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe2577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2581010d0 0x7fe258196b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51030/0 (socket says 172.21.15.2:51030) 2023-11-16T18:26:10.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe2577fe700 1 -- 172.21.15.2:0/2012697029 learned_addr learned my addr 172.21.15.2:0/2012697029 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:10.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe25ce32700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe258109d70 0x7fe25819afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe257fff700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2581007c0 0x7fe258196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe2577fe700 1 -- 172.21.15.2:0/2012697029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2581007c0 msgr2=0x7fe258196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe2577fe700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2581007c0 0x7fe258196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe2577fe700 1 -- 172.21.15.2:0/2012697029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe258109d70 msgr2=0x7fe25819afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe2577fe700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe258109d70 0x7fe25819afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.698+0000 7fe2577fe700 1 -- 172.21.15.2:0/2012697029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2480095f0 con 0x7fe2581010d0 2023-11-16T18:26:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.699+0000 7fe2577fe700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2581010d0 0x7fe258196b40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00e9a0 tx=0x7fe24c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:10.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.699+0000 7fe2557fa700 1 -- 172.21.15.2:0/2012697029 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c009790 con 0x7fe2581010d0 2023-11-16T18:26:10.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.699+0000 7fe2557fa700 1 -- 172.21.15.2:0/2012697029 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe24c010430 con 0x7fe2581010d0 2023-11-16T18:26:10.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.699+0000 7fe25e895700 1 -- 172.21.15.2:0/2012697029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25810f460 con 0x7fe2581010d0 2023-11-16T18:26:10.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.699+0000 7fe2557fa700 1 -- 172.21.15.2:0/2012697029 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c018680 con 0x7fe2581010d0 2023-11-16T18:26:10.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.699+0000 7fe25e895700 1 -- 172.21.15.2:0/2012697029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25810f2d0 con 0x7fe2581010d0 2023-11-16T18:26:10.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.701+0000 7fe2557fa700 1 -- 172.21.15.2:0/2012697029 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7fe24c0105a0 con 0x7fe2581010d0 2023-11-16T18:26:10.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.702+0000 7fe2557fa700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe2400a13e0 0x7fe2400a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.702+0000 7fe2557fa700 1 -- 172.21.15.2:0/2012697029 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe24c014070 con 0x7fe2581010d0 2023-11-16T18:26:10.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.702+0000 7fe257fff700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe2400a13e0 0x7fe2400a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.703+0000 7fe257fff700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe2400a13e0 0x7fe2400a3890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe248009fd0 tx=0x7fe248005fb0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:10.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.816+0000 7fe26aa21b80 1 Processor -- start 2023-11-16T18:26:10.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.816+0000 7fe26aa21b80 1 -- start start 2023-11-16T18:26:10.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.816+0000 7fe26aa21b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abdbe0 0x5630e1abdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.816+0000 7fe26aa21b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 0x5630e1ac23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe26aa21b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe26aa21b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630e19c30e0 con 0x5630e1abe4f0 2023-11-16T18:26:10.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe26aa21b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630e1a15920 con 0x5630e1abdbe0 2023-11-16T18:26:10.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe26aa21b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630e1a0edb0 con 0x5630e1ac2930 2023-11-16T18:26:10.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe2577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 0x5630e1ac23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe257fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abdbe0 0x5630e1abdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe2577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 0x5630e1ac23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51038/0 (socket says 172.21.15.2:51038) 2023-11-16T18:26:10.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe257fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abdbe0 0x5630e1abdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:41474/0 (socket says 172.21.15.2:41474) 2023-11-16T18:26:10.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe2577fe700 1 -- 172.21.15.2:0/2423896978 learned_addr learned my addr 172.21.15.2:0/2423896978 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:10.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe25ce32700 1 --2- 172.21.15.2:0/2423896978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe2577fe700 1 -- 172.21.15.2:0/2423896978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 msgr2=0x5630e1ac2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:10.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe2577fe700 1 --2- 172.21.15.2:0/2423896978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe2577fe700 1 -- 172.21.15.2:0/2423896978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abdbe0 msgr2=0x5630e1abdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:10.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe2577fe700 1 --2- 172.21.15.2:0/2423896978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abdbe0 0x5630e1abdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe2577fe700 1 -- 172.21.15.2:0/2423896978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630e18906b0 con 0x5630e1abe4f0 2023-11-16T18:26:10.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe25ce32700 1 --2- 172.21.15.2:0/2423896978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:10.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe257fff700 1 --2- 172.21.15.2:0/2423896978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abdbe0 0x5630e1abdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:10.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.817+0000 7fe2577fe700 1 --2- 172.21.15.2:0/2423896978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 0x5630e1ac23f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe24c099f90 tx=0x7fe24c099a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7474050b88201785 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:10.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.818+0000 7fe23e7fc700 1 -- 172.21.15.2:0/2423896978 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c09e030 con 0x5630e1abe4f0 2023-11-16T18:26:10.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.818+0000 7fe23e7fc700 1 -- 172.21.15.2:0/2423896978 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe24c0a2450 con 0x5630e1abe4f0 2023-11-16T18:26:10.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.818+0000 7fe23e7fc700 1 -- 172.21.15.2:0/2423896978 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c0a1fc0 con 0x5630e1abe4f0 2023-11-16T18:26:10.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.818+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2423896978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 msgr2=0x5630e1ac23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:10.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.818+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2423896978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 0x5630e1ac23f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe24c099f90 tx=0x7fe24c099a50 comp rx=0 tx=0).stop 2023-11-16T18:26:10.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.819+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2423896978 shutdown_connections 2023-11-16T18:26:10.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.819+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2423896978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.819+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2423896978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abdbe0 0x5630e1abdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.819+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2423896978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 0x5630e1ac23f0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.819+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2423896978 >> 172.21.15.2:0/2423896978 conn(0x5630e1ab33e0 msgr2=0x5630e1ab8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.819+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2423896978 shutdown_connections 2023-11-16T18:26:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.819+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2423896978 wait complete. 2023-11-16T18:26:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.819+0000 7fe26aa21b80 1 -- start start 2023-11-16T18:26:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.820+0000 7fe26aa21b80 1 Processor -- start 2023-11-16T18:26:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.820+0000 7fe26aa21b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 0x5630e1aca140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.820+0000 7fe26aa21b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abe4f0 0x5630e1ac7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.820+0000 7fe26aa21b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.820+0000 7fe26aa21b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630e19b0960 con 0x5630e1abdbe0 2023-11-16T18:26:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.820+0000 7fe26aa21b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630e19b0600 con 0x5630e1abe4f0 2023-11-16T18:26:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.820+0000 7fe26aa21b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630e18929a0 con 0x5630e1ac2930 2023-11-16T18:26:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.820+0000 7fe26aa21b80 10 client.0 _ll_register_callbacks cb 0x5630e19ce800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.820+0000 7fe257fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 0x5630e1aca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.820+0000 7fe257fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 0x5630e1aca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51042/0 (socket says 172.21.15.2:51042) 2023-11-16T18:26:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.820+0000 7fe257fff700 1 -- 172.21.15.2:0/3784814099 learned_addr learned my addr 172.21.15.2:0/3784814099 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.820+0000 7fe25ce32700 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.820+0000 7fe2577fe700 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abe4f0 0x5630e1ac7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.821+0000 7fe257fff700 1 -- 172.21.15.2:0/3784814099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 msgr2=0x5630e1ac7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.821+0000 7fe257fff700 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.821+0000 7fe257fff700 1 -- 172.21.15.2:0/3784814099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abe4f0 msgr2=0x5630e1ac7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.821+0000 7fe257fff700 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abe4f0 0x5630e1ac7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.821+0000 7fe257fff700 1 -- 172.21.15.2:0/3784814099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25810f2d0 con 0x5630e1abdbe0 2023-11-16T18:26:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.821+0000 7fe25ce32700 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.821+0000 7fe2577fe700 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abe4f0 0x5630e1ac7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.821+0000 7fe257fff700 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 0x5630e1aca140 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe2480099c0 tx=0x7fe248009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753c066f9aa9bf1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.821+0000 7fe2377fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.821+0000 7fe2377fe700 1 -- 172.21.15.2:0/3784814099 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24802c040 con 0x5630e1abdbe0 2023-11-16T18:26:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.821+0000 7fe2377fe700 1 -- 172.21.15.2:0/3784814099 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe24800be70 con 0x5630e1abdbe0 2023-11-16T18:26:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.821+0000 7fe26aa21b80 1 -- 172.21.15.2:0/3784814099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5630e1ab0ba0 con 0x5630e1abdbe0 2023-11-16T18:26:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.822+0000 7fe2377fe700 1 -- 172.21.15.2:0/3784814099 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe248004460 con 0x5630e1abdbe0 2023-11-16T18:26:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.822+0000 7fe26aa21b80 10 client.15957 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.822+0000 7fe26aa21b80 1 -- 172.21.15.2:0/3784814099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5630e19d5210 con 0x5630e1abdbe0 2023-11-16T18:26:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.822+0000 7fe2377fe700 1 -- 172.21.15.2:0/3784814099 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5630e1ab0ba0 con 0x5630e1abdbe0 2023-11-16T18:26:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.822+0000 7fe2377fe700 1 -- 172.21.15.2:0/3784814099 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe248026430 con 0x5630e1abdbe0 2023-11-16T18:26:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.822+0000 7fe26aa21b80 10 client.15957 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.822+0000 7fe26aa21b80 10 client.15957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.822+0000 7fe26aa21b80 20 client.15957 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.822+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5630e1ad79e0 0x5630e1ad9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.822+0000 7fe26aa21b80 4 client.15957 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.822+0000 7fe26aa21b80 1 -- 172.21.15.2:0/3784814099 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5630e19d1120 con 0x5630e1ad79e0 2023-11-16T18:26:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.822+0000 7fe2577fe700 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5630e1ad79e0 0x5630e1ad9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.823+0000 7fe2377fe700 10 client.15957 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.823+0000 7fe2577fe700 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5630e1ad79e0 0x5630e1ad9df0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c57c8683cf1afa server_cookie=84c7e968ae3c4573 in_seq=0 out_seq=0 2023-11-16T18:26:10.831 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:10.831 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:10.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe2377fe700 1 -- 172.21.15.2:0/3784814099 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5630e19d1120 con 0x5630e1ad79e0 2023-11-16T18:26:10.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe2377fe700 10 client.15957 handle_command_reply: tid=0 2023-11-16T18:26:10.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe26aa21b80 1 client.15957 shutdown 2023-11-16T18:26:10.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe26aa21b80 1 -- 172.21.15.2:0/3784814099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 msgr2=0x5630e1aca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:10.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 0x5630e1aca140 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe2480099c0 tx=0x7fe248009cd0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe26aa21b80 1 -- 172.21.15.2:0/3784814099 shutdown_connections 2023-11-16T18:26:10.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5630e1ad79e0 0x5630e1ad9df0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 0x5630e1aca140 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/3784814099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abe4f0 0x5630e1ac7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe2377fe700 0 client.15957 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:10.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe26aa21b80 1 -- 172.21.15.2:0/3784814099 >> 172.21.15.2:0/3784814099 conn(0x5630e1ab33e0 msgr2=0x5630e1ab7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:10.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe26aa21b80 1 -- 172.21.15.2:0/3784814099 shutdown_connections 2023-11-16T18:26:10.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe26aa21b80 1 -- 172.21.15.2:0/3784814099 wait complete. 2023-11-16T18:26:10.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.824+0000 7fe26aa21b80 20 client.15957 trim_cache size 0 max 16384 2023-11-16T18:26:10.833 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:10.833 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:10.833 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 42 -lt 100 ']' 2023-11-16T18:26:10.833 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 42 + 1 2023-11-16T18:26:10.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.842+0000 7fe26aa21b80 1 Processor -- start 2023-11-16T18:26:10.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.842+0000 7fe26aa21b80 1 -- start start 2023-11-16T18:26:10.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.842+0000 7fe26aa21b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 0x5630e1ac2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.842+0000 7fe26aa21b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abe4f0 0x5630e1abd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.842+0000 7fe26aa21b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.842+0000 7fe26aa21b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630e1abea40 con 0x5630e1abdbe0 2023-11-16T18:26:10.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.842+0000 7fe26aa21b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630e1abec40 con 0x5630e1abe4f0 2023-11-16T18:26:10.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.842+0000 7fe26aa21b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630e1ac89d0 con 0x5630e1ac2930 2023-11-16T18:26:10.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe257fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 0x5630e1ac2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe257fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 0x5630e1ac2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51050/0 (socket says 172.21.15.2:51050) 2023-11-16T18:26:10.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe257fff700 1 -- 172.21.15.2:0/2983111527 learned_addr learned my addr 172.21.15.2:0/2983111527 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:10.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe25ce32700 1 --2- 172.21.15.2:0/2983111527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe2577fe700 1 --2- 172.21.15.2:0/2983111527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abe4f0 0x5630e1abd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe257fff700 1 -- 172.21.15.2:0/2983111527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 msgr2=0x5630e1ac9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:10.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe257fff700 1 --2- 172.21.15.2:0/2983111527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe257fff700 1 -- 172.21.15.2:0/2983111527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abe4f0 msgr2=0x5630e1abd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:10.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe257fff700 1 --2- 172.21.15.2:0/2983111527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abe4f0 0x5630e1abd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe257fff700 1 -- 172.21.15.2:0/2983111527 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630e19d4f60 con 0x5630e1abdbe0 2023-11-16T18:26:10.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe2577fe700 1 --2- 172.21.15.2:0/2983111527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abe4f0 0x5630e1abd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:10.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe257fff700 1 --2- 172.21.15.2:0/2983111527 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 0x5630e1ac2200 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe248005a60 tx=0x7fe24800b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14e2a5ea04a540f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:10.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe23effd700 1 -- 172.21.15.2:0/2983111527 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24802c050 con 0x5630e1abdbe0 2023-11-16T18:26:10.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe23effd700 1 -- 172.21.15.2:0/2983111527 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe24801e040 con 0x5630e1abdbe0 2023-11-16T18:26:10.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.843+0000 7fe23effd700 1 -- 172.21.15.2:0/2983111527 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24800f460 con 0x5630e1abdbe0 2023-11-16T18:26:10.852 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.844+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2983111527 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 msgr2=0x5630e1ac2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:10.852 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.844+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2983111527 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 0x5630e1ac2200 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe248005a60 tx=0x7fe24800b560 comp rx=0 tx=0).stop 2023-11-16T18:26:10.852 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.844+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2983111527 shutdown_connections 2023-11-16T18:26:10.852 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.844+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2983111527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1ac2930 0x5630e1ac9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.852 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.844+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2983111527 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abdbe0 0x5630e1ac2200 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.852 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.844+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2983111527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1abe4f0 0x5630e1abd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.852 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.844+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2983111527 >> 172.21.15.2:0/2983111527 conn(0x5630e1ab33e0 msgr2=0x5630e1abc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:10.852 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.844+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2983111527 shutdown_connections 2023-11-16T18:26:10.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.844+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2983111527 wait complete. 2023-11-16T18:26:10.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.844+0000 7fe26aa21b80 1 -- start start 2023-11-16T18:26:10.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.845+0000 7fe26aa21b80 1 Processor -- start 2023-11-16T18:26:10.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.845+0000 7fe26aa21b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1abdbe0 0x5630e1b24b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.845+0000 7fe26aa21b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 0x5630e1b250a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.845+0000 7fe26aa21b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1ac2930 0x5630e1ac67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.845+0000 7fe26aa21b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630e1ac6e40 con 0x5630e1abe4f0 2023-11-16T18:26:10.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.845+0000 7fe26aa21b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630e1ac6fb0 con 0x5630e1ac2930 2023-11-16T18:26:10.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.845+0000 7fe26aa21b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630e1ac7170 con 0x5630e1abdbe0 2023-11-16T18:26:10.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe26aa21b80 10 client.0 _ll_register_callbacks cb 0x7fe248003e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:10.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe2577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 0x5630e1b250a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe2577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 0x5630e1b250a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51058/0 (socket says 172.21.15.2:51058) 2023-11-16T18:26:10.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe2577fe700 1 -- 172.21.15.2:0/2653724137 learned_addr learned my addr 172.21.15.2:0/2653724137 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:10.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe257fff700 1 --2- 172.21.15.2:0/2653724137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1abdbe0 0x5630e1b24b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe25ce32700 1 --2- 172.21.15.2:0/2653724137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1ac2930 0x5630e1ac67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe2577fe700 1 -- 172.21.15.2:0/2653724137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1abdbe0 msgr2=0x5630e1b24b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:10.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe2577fe700 1 --2- 172.21.15.2:0/2653724137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1abdbe0 0x5630e1b24b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe2577fe700 1 -- 172.21.15.2:0/2653724137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1ac2930 msgr2=0x5630e1ac67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:10.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe2577fe700 1 --2- 172.21.15.2:0/2653724137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1ac2930 0x5630e1ac67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe2577fe700 1 -- 172.21.15.2:0/2653724137 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630e19d5210 con 0x5630e1abe4f0 2023-11-16T18:26:10.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe2577fe700 1 --2- 172.21.15.2:0/2653724137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 0x5630e1b250a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe24c099d00 tx=0x7fe24c0092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66f9cf871969c3e5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:10.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe2377fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:10.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe2377fe700 1 -- 172.21.15.2:0/2653724137 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c09e030 con 0x5630e1abe4f0 2023-11-16T18:26:10.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe2377fe700 1 -- 172.21.15.2:0/2653724137 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe24c0a1fc0 con 0x5630e1abe4f0 2023-11-16T18:26:10.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.846+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2653724137 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe24c0a2450 con 0x5630e1abe4f0 2023-11-16T18:26:10.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.847+0000 7fe2377fe700 1 -- 172.21.15.2:0/2653724137 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe24c0a2180 con 0x5630e1abe4f0 2023-11-16T18:26:10.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.847+0000 7fe26aa21b80 10 client.15963 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:10.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.847+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2653724137 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5630e1825ce0 con 0x5630e1abe4f0 2023-11-16T18:26:10.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.847+0000 7fe2377fe700 1 -- 172.21.15.2:0/2653724137 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe24c0a2450 con 0x5630e1abe4f0 2023-11-16T18:26:10.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.847+0000 7fe2377fe700 1 -- 172.21.15.2:0/2653724137 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe24c0caa30 con 0x5630e1abe4f0 2023-11-16T18:26:10.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.847+0000 7fe26aa21b80 10 client.15963 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:10.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.847+0000 7fe26aa21b80 10 client.15963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:10.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.847+0000 7fe26aa21b80 20 client.15963 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:10.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.847+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2653724137 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5630e1b2d800 0x5630e1b2fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:10.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.847+0000 7fe26aa21b80 4 client.15963 mds_command: new command op to 24473 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T18:26:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.847+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2653724137 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5630e19dd6e0 con 0x5630e1b2d800 2023-11-16T18:26:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.847+0000 7fe257fff700 1 --2- 172.21.15.2:0/2653724137 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5630e1b2d800 0x5630e1b2fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.848+0000 7fe257fff700 1 --2- 172.21.15.2:0/2653724137 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5630e1b2d800 0x5630e1b2fc10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4d7702204ab9d1c server_cookie=b9ca83b855f44899 in_seq=0 out_seq=0 2023-11-16T18:26:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.848+0000 7fe2377fe700 10 client.15963 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.849+0000 7fe2377fe700 1 -- 172.21.15.2:0/2653724137 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5630e19dd6e0 con 0x5630e1b2d800 2023-11-16T18:26:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.849+0000 7fe2377fe700 10 client.15963 handle_command_reply: tid=0 2023-11-16T18:26:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.849+0000 7fe26aa21b80 1 client.15963 shutdown 2023-11-16T18:26:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2653724137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 msgr2=0x5630e1b250a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2653724137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 0x5630e1b250a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe24c099d00 tx=0x7fe24c0092d0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2653724137 shutdown_connections 2023-11-16T18:26:10.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2653724137 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5630e1b2d800 0x5630e1b2fc10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2653724137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5630e1abdbe0 0x5630e1b24b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2653724137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5630e1ac2930 0x5630e1ac67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe26aa21b80 1 --2- 172.21.15.2:0/2653724137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5630e1abe4f0 0x5630e1b250a0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2653724137 >> 172.21.15.2:0/2653724137 conn(0x5630e1ab33e0 msgr2=0x5630e1abc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe2377fe700 0 client.15963 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2653724137 shutdown_connections 2023-11-16T18:26:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe26aa21b80 1 -- 172.21.15.2:0/2653724137 wait complete. 2023-11-16T18:26:10.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe26aa21b80 20 client.15963 trim_cache size 0 max 16384 2023-11-16T18:26:10.857 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:26:10.858 INFO:teuthology.orchestra.run.smithi002.stdout: "return_code": 0, 2023-11-16T18:26:10.858 INFO:teuthology.orchestra.run.smithi002.stdout: "scrub_tag": "3818f4d3-3e6d-437f-96a2-9507719f2231", 2023-11-16T18:26:10.858 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": "asynchronous" 2023-11-16T18:26:10.858 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:26:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe25e895700 1 -- 172.21.15.2:0/2012697029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe2400a13e0 msgr2=0x7fe2400a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe25e895700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe2400a13e0 0x7fe2400a3890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe248009fd0 tx=0x7fe248005fb0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe25e895700 1 -- 172.21.15.2:0/2012697029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2581010d0 msgr2=0x7fe258196b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.850+0000 7fe25e895700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2581010d0 0x7fe258196b40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00e9a0 tx=0x7fe24c00ecb0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.851+0000 7fe25e895700 1 -- 172.21.15.2:0/2012697029 shutdown_connections 2023-11-16T18:26:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.851+0000 7fe25e895700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe2400a13e0 0x7fe2400a3890 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.851+0000 7fe25e895700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2581007c0 0x7fe258196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.851+0000 7fe25e895700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe258109d70 0x7fe25819afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.851+0000 7fe25e895700 1 --2- 172.21.15.2:0/2012697029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2581010d0 0x7fe258196b40 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.851+0000 7fe25e895700 1 -- 172.21.15.2:0/2012697029 >> 172.21.15.2:0/2012697029 conn(0x7fe258074c60 msgr2=0x7fe2580fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:10.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.851+0000 7fe25e895700 1 -- 172.21.15.2:0/2012697029 shutdown_connections 2023-11-16T18:26:10.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:10.851+0000 7fe25e895700 1 -- 172.21.15.2:0/2012697029 wait complete. 2023-11-16T18:26:10.862 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=43 2023-11-16T18:26:10.862 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:10.900 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:10.900 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:10.901 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:10.901 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:10.901 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:10.910 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:26:11.329 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:11.329 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:11.413 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:11.413 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:11.423 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:11.423 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:11.423 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 43 -lt 100 ']' 2023-11-16T18:26:11.423 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 43 + 1 2023-11-16T18:26:11.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=44 2023-11-16T18:26:11.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:11.557 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:11.557 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:11.558 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:11.558 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:11.558 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:11.940 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:11.940 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:12.092 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:12.092 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:12.099 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:12.099 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:12.099 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 44 -lt 100 ']' 2023-11-16T18:26:12.099 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 44 + 1 2023-11-16T18:26:12.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:12 smithi002 ceph-mon[98807]: scrub queued for path: / 2023-11-16T18:26:12.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:12 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:26:12.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:12 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:26:12.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:12 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:26:12.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:12 smithi002 ceph-mon[98807]: pgmap v43: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 39 KiB/s wr, 24 op/s 2023-11-16T18:26:12.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:12 smithi002 ceph-mon[98807]: scrub summary: idle 2023-11-16T18:26:12.170 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=45 2023-11-16T18:26:12.171 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:12.250 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:12.251 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:12.251 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:12.251 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:12.252 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:12 smithi040 ceph-mon[100251]: scrub queued for path: / 2023-11-16T18:26:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:12 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:26:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:12 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:26:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:12 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:26:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:12 smithi040 ceph-mon[100251]: pgmap v43: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 39 KiB/s wr, 24 op/s 2023-11-16T18:26:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:12 smithi040 ceph-mon[100251]: scrub summary: idle 2023-11-16T18:26:12.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:12 smithi188 ceph-mon[100639]: scrub queued for path: / 2023-11-16T18:26:12.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:12 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:26:12.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:12 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:26:12.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:12 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:26:12.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:12 smithi188 ceph-mon[100639]: pgmap v43: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 39 KiB/s wr, 24 op/s 2023-11-16T18:26:12.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:12 smithi188 ceph-mon[100639]: scrub summary: idle 2023-11-16T18:26:13.070 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:13.071 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:13.259 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:13.259 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:13.267 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:13.267 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:13.267 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 45 -lt 100 ']' 2023-11-16T18:26:13.268 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 45 + 1 2023-11-16T18:26:13.308 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=46 2023-11-16T18:26:13.308 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:13.350 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:13.351 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:13.351 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:13.351 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:13.352 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.687+0000 7fc145d2b700 1 -- 172.21.15.2:0/3413651079 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1400fea80 msgr2=0x7fc1400fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.687+0000 7fc145d2b700 1 --2- 172.21.15.2:0/3413651079 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1400fea80 0x7fc1400fee50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fc1300099c0 tx=0x7fc130009cd0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.687+0000 7fc145d2b700 1 -- 172.21.15.2:0/3413651079 shutdown_connections 2023-11-16T18:26:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.687+0000 7fc145d2b700 1 --2- 172.21.15.2:0/3413651079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1401055d0 0x7fc14010d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.687+0000 7fc145d2b700 1 --2- 172.21.15.2:0/3413651079 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1400fea80 0x7fc1400fee50 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.687+0000 7fc145d2b700 1 --2- 172.21.15.2:0/3413651079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1400ff420 0x7fc140104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.687+0000 7fc145d2b700 1 -- 172.21.15.2:0/3413651079 >> 172.21.15.2:0/3413651079 conn(0x7fc140075810 msgr2=0x7fc140075c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.687+0000 7fc145d2b700 1 -- 172.21.15.2:0/3413651079 shutdown_connections 2023-11-16T18:26:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.687+0000 7fc145d2b700 1 -- 172.21.15.2:0/3413651079 wait complete. 2023-11-16T18:26:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.688+0000 7fc145d2b700 1 Processor -- start 2023-11-16T18:26:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.688+0000 7fc145d2b700 1 -- start start 2023-11-16T18:26:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.688+0000 7fc145d2b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1400fea80 0x7fc1400ff9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.688+0000 7fc145d2b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1400ff420 0x7fc1400fff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.688+0000 7fc145d2b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1401055d0 0x7fc1401a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.688+0000 7fc145d2b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc140100760 con 0x7fc1400ff420 2023-11-16T18:26:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.688+0000 7fc145d2b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1401a4af0 con 0x7fc1401055d0 2023-11-16T18:26:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.688+0000 7fc145d2b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1401a4c30 con 0x7fc1400fea80 2023-11-16T18:26:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.688+0000 7fc13effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1400ff420 0x7fc1400fff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.688+0000 7fc13effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1400ff420 0x7fc1400fff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51072/0 (socket says 172.21.15.2:51072) 2023-11-16T18:26:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.688+0000 7fc13effd700 1 -- 172.21.15.2:0/2833674832 learned_addr learned my addr 172.21.15.2:0/2833674832 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.689+0000 7fc13ffff700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1401055d0 0x7fc1401a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.689+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1400fea80 0x7fc1400ff9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.689+0000 7fc13effd700 1 -- 172.21.15.2:0/2833674832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1400fea80 msgr2=0x7fc1400ff9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.689+0000 7fc13effd700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1400fea80 0x7fc1400ff9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.689+0000 7fc13effd700 1 -- 172.21.15.2:0/2833674832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1401055d0 msgr2=0x7fc1401a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.689+0000 7fc13effd700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1401055d0 0x7fc1401a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.689+0000 7fc13effd700 1 -- 172.21.15.2:0/2833674832 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1300095f0 con 0x7fc1400ff420 2023-11-16T18:26:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.689+0000 7fc13effd700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1400ff420 0x7fc1400fff30 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc12800b520 tx=0x7fc12800b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.689+0000 7fc13cff9700 1 -- 172.21.15.2:0/2833674832 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128011830 con 0x7fc1400ff420 2023-11-16T18:26:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.689+0000 7fc13cff9700 1 -- 172.21.15.2:0/2833674832 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc128011e50 con 0x7fc1400ff420 2023-11-16T18:26:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.689+0000 7fc145d2b700 1 -- 172.21.15.2:0/2833674832 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1401103d0 con 0x7fc1400ff420 2023-11-16T18:26:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.689+0000 7fc13cff9700 1 -- 172.21.15.2:0/2833674832 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128010de0 con 0x7fc1400ff420 2023-11-16T18:26:13.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.689+0000 7fc145d2b700 1 -- 172.21.15.2:0/2833674832 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc140110240 con 0x7fc1400ff420 2023-11-16T18:26:13.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.692+0000 7fc13cff9700 1 -- 172.21.15.2:0/2833674832 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7fc128010480 con 0x7fc1400ff420 2023-11-16T18:26:13.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.692+0000 7fc13cff9700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc12c0a1460 0x7fc12c0a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.692+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc12c0a1460 0x7fc12c0a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.693+0000 7fc13cff9700 1 -- 172.21.15.2:0/2833674832 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc1280ccb70 con 0x7fc1400ff420 2023-11-16T18:26:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.693+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc12c0a1460 0x7fc12c0a3910 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc130009990 tx=0x7fc130000bc0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:13.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.737+0000 7fc151eb7b80 1 Processor -- start 2023-11-16T18:26:13.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.737+0000 7fc151eb7b80 1 -- start start 2023-11-16T18:26:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.737+0000 7fc151eb7b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c1db60 0x5651b0c1df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.737+0000 7fc151eb7b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1e470 0x5651b0c22370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.737+0000 7fc151eb7b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c228b0 0x5651b0c22d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc151eb7b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b0b23060 con 0x5651b0c228b0 2023-11-16T18:26:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc151eb7b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b0b758a0 con 0x5651b0c1e470 2023-11-16T18:26:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc151eb7b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b0b6ed30 con 0x5651b0c1db60 2023-11-16T18:26:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc13ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c228b0 0x5651b0c22d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc13ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c228b0 0x5651b0c22d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51082/0 (socket says 172.21.15.2:51082) 2023-11-16T18:26:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc13ffff700 1 -- 172.21.15.2:0/618553947 learned_addr learned my addr 172.21.15.2:0/618553947 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/618553947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c1db60 0x5651b0c1df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc13effd700 1 --2- 172.21.15.2:0/618553947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1e470 0x5651b0c22370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc13ffff700 1 -- 172.21.15.2:0/618553947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c1db60 msgr2=0x5651b0c1df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:13.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc13ffff700 1 --2- 172.21.15.2:0/618553947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c1db60 0x5651b0c1df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc13ffff700 1 -- 172.21.15.2:0/618553947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1e470 msgr2=0x5651b0c22370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:13.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc13ffff700 1 --2- 172.21.15.2:0/618553947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1e470 0x5651b0c22370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc13ffff700 1 -- 172.21.15.2:0/618553947 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651b09f0630 con 0x5651b0c228b0 2023-11-16T18:26:13.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc13ffff700 1 --2- 172.21.15.2:0/618553947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c228b0 0x5651b0c22d00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc138003fc0 tx=0x7fc138009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67af4ff4ae99702 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:13.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.738+0000 7fc135ffb700 1 -- 172.21.15.2:0/618553947 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138019070 con 0x5651b0c228b0 2023-11-16T18:26:13.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.739+0000 7fc135ffb700 1 -- 172.21.15.2:0/618553947 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc13800f040 con 0x5651b0c228b0 2023-11-16T18:26:13.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.739+0000 7fc135ffb700 1 -- 172.21.15.2:0/618553947 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138007a50 con 0x5651b0c228b0 2023-11-16T18:26:13.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.739+0000 7fc151eb7b80 1 -- 172.21.15.2:0/618553947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c228b0 msgr2=0x5651b0c22d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.739+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/618553947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c228b0 0x5651b0c22d00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc138003fc0 tx=0x7fc138009ba0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.739+0000 7fc151eb7b80 1 -- 172.21.15.2:0/618553947 shutdown_connections 2023-11-16T18:26:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.739+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/618553947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c1db60 0x5651b0c1df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.739+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/618553947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c228b0 0x5651b0c22d00 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.739+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/618553947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1e470 0x5651b0c22370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.739+0000 7fc151eb7b80 1 -- 172.21.15.2:0/618553947 >> 172.21.15.2:0/618553947 conn(0x5651b0c13360 msgr2=0x5651b0c18b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.739+0000 7fc151eb7b80 1 -- 172.21.15.2:0/618553947 shutdown_connections 2023-11-16T18:26:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.739+0000 7fc151eb7b80 1 -- 172.21.15.2:0/618553947 wait complete. 2023-11-16T18:26:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.740+0000 7fc151eb7b80 1 -- start start 2023-11-16T18:26:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.740+0000 7fc151eb7b80 1 Processor -- start 2023-11-16T18:26:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc151eb7b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1db60 0x5651b0c2a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc151eb7b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 0x5651b0c27250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc151eb7b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 0x5651b0c27860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc151eb7b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b0b108e0 con 0x5651b0c1e470 2023-11-16T18:26:13.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc151eb7b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b0b10580 con 0x5651b0c1db60 2023-11-16T18:26:13.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc151eb7b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b09f2920 con 0x5651b0c228b0 2023-11-16T18:26:13.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc151eb7b80 10 client.0 _ll_register_callbacks cb 0x5651b0b2e780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:13.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc13effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 0x5651b0c27250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc13effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 0x5651b0c27250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51086/0 (socket says 172.21.15.2:51086) 2023-11-16T18:26:13.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc13effd700 1 -- 172.21.15.2:0/3569846229 learned_addr learned my addr 172.21.15.2:0/3569846229 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:13.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/3569846229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1db60 0x5651b0c2a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc13effd700 1 -- 172.21.15.2:0/3569846229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 msgr2=0x5651b0c27860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:13.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc13effd700 1 --2- 172.21.15.2:0/3569846229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 0x5651b0c27860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc13effd700 1 -- 172.21.15.2:0/3569846229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1db60 msgr2=0x5651b0c2a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc13effd700 1 --2- 172.21.15.2:0/3569846229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1db60 0x5651b0c2a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc13effd700 1 -- 172.21.15.2:0/3569846229 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc138003bf0 con 0x5651b0c1e470 2023-11-16T18:26:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc13effd700 1 --2- 172.21.15.2:0/3569846229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 0x5651b0c27250 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc128009470 tx=0x7fc128099250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a89616d3892b68 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.741+0000 7fc11effd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc11effd700 1 -- 172.21.15.2:0/3569846229 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1280b1070 con 0x5651b0c1e470 2023-11-16T18:26:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc11effd700 1 -- 172.21.15.2:0/3569846229 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc12809d7c0 con 0x5651b0c1e470 2023-11-16T18:26:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc151eb7b80 1 -- 172.21.15.2:0/3569846229 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5651b0c10b20 con 0x5651b0c1e470 2023-11-16T18:26:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc11effd700 1 -- 172.21.15.2:0/3569846229 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1280a1be0 con 0x5651b0c1e470 2023-11-16T18:26:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc151eb7b80 10 client.15975 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc151eb7b80 1 -- 172.21.15.2:0/3569846229 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5651b0b35190 con 0x5651b0c1e470 2023-11-16T18:26:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc11effd700 1 -- 172.21.15.2:0/3569846229 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc1280c7020 con 0x5651b0c1e470 2023-11-16T18:26:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc11effd700 1 -- 172.21.15.2:0/3569846229 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc1280a1750 con 0x5651b0c1e470 2023-11-16T18:26:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc151eb7b80 10 client.15975 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc151eb7b80 10 client.15975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc151eb7b80 20 client.15975 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/3569846229 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5651b0c377c0 0x5651b0c39bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc151eb7b80 4 client.15975 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.742+0000 7fc151eb7b80 1 -- 172.21.15.2:0/3569846229 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5651b0b310a0 con 0x5651b0c377c0 2023-11-16T18:26:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.743+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/3569846229 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5651b0c377c0 0x5651b0c39bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.743+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/3569846229 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5651b0c377c0 0x5651b0c39bd0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c64dc4eb1e4288d4 server_cookie=354e4e00fa8dc944 in_seq=0 out_seq=0 2023-11-16T18:26:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.743+0000 7fc11effd700 10 client.15975 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc11effd700 1 -- 172.21.15.2:0/3569846229 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5651b0b310a0 con 0x5651b0c377c0 2023-11-16T18:26:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc11effd700 10 client.15975 handle_command_reply: tid=0 2023-11-16T18:26:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc151eb7b80 1 client.15975 shutdown 2023-11-16T18:26:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc151eb7b80 1 -- 172.21.15.2:0/3569846229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 msgr2=0x5651b0c27250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/3569846229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 0x5651b0c27250 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc128009470 tx=0x7fc128099250 comp rx=0 tx=0).stop 2023-11-16T18:26:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc151eb7b80 1 -- 172.21.15.2:0/3569846229 shutdown_connections 2023-11-16T18:26:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/3569846229 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5651b0c377c0 0x5651b0c39bd0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/3569846229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 0x5651b0c27860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/3569846229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1db60 0x5651b0c2a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc11effd700 0 client.15975 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/3569846229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 0x5651b0c27250 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc151eb7b80 1 -- 172.21.15.2:0/3569846229 >> 172.21.15.2:0/3569846229 conn(0x5651b0c13360 msgr2=0x5651b0c23d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc151eb7b80 1 -- 172.21.15.2:0/3569846229 shutdown_connections 2023-11-16T18:26:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc151eb7b80 1 -- 172.21.15.2:0/3569846229 wait complete. 2023-11-16T18:26:13.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.745+0000 7fc151eb7b80 20 client.15975 trim_cache size 0 max 16384 2023-11-16T18:26:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.762+0000 7fc151eb7b80 1 Processor -- start 2023-11-16T18:26:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.762+0000 7fc151eb7b80 1 -- start start 2023-11-16T18:26:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.762+0000 7fc151eb7b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1db60 0x5651b0c18cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.762+0000 7fc151eb7b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1e470 0x5651b0c2e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.762+0000 7fc151eb7b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 0x5651b0c29b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.762+0000 7fc151eb7b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b0c257d0 con 0x5651b0c1db60 2023-11-16T18:26:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.762+0000 7fc151eb7b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b0c213c0 con 0x5651b0c1e470 2023-11-16T18:26:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.762+0000 7fc151eb7b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b0c2eee0 con 0x5651b0c228b0 2023-11-16T18:26:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.762+0000 7fc13f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1db60 0x5651b0c18cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.762+0000 7fc13f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1db60 0x5651b0c18cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51092/0 (socket says 172.21.15.2:51092) 2023-11-16T18:26:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.762+0000 7fc13f7fe700 1 -- 172.21.15.2:0/1538005330 learned_addr learned my addr 172.21.15.2:0/1538005330 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.763+0000 7fc13effd700 1 --2- 172.21.15.2:0/1538005330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1e470 0x5651b0c2e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.763+0000 7fc13ffff700 1 --2- 172.21.15.2:0/1538005330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 0x5651b0c29b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.763+0000 7fc13f7fe700 1 -- 172.21.15.2:0/1538005330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 msgr2=0x5651b0c29b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:13.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.763+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/1538005330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 0x5651b0c29b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.763+0000 7fc13f7fe700 1 -- 172.21.15.2:0/1538005330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1e470 msgr2=0x5651b0c2e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:13.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.763+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/1538005330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1e470 0x5651b0c2e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.763+0000 7fc13f7fe700 1 -- 172.21.15.2:0/1538005330 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651b0b34ee0 con 0x5651b0c1db60 2023-11-16T18:26:13.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.763+0000 7fc13ffff700 1 --2- 172.21.15.2:0/1538005330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 0x5651b0c29b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:13.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.763+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/1538005330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1db60 0x5651b0c18cc0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc13000b580 tx=0x7fc130020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef81dd252fb7a381 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:13.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.763+0000 7fc1367fc700 1 -- 172.21.15.2:0/1538005330 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13002d040 con 0x5651b0c1db60 2023-11-16T18:26:13.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.763+0000 7fc1367fc700 1 -- 172.21.15.2:0/1538005330 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc13002e040 con 0x5651b0c1db60 2023-11-16T18:26:13.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.763+0000 7fc1367fc700 1 -- 172.21.15.2:0/1538005330 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc130003b30 con 0x5651b0c1db60 2023-11-16T18:26:13.771 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.764+0000 7fc151eb7b80 1 -- 172.21.15.2:0/1538005330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1db60 msgr2=0x5651b0c18cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.764+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/1538005330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1db60 0x5651b0c18cc0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc13000b580 tx=0x7fc130020000 comp rx=0 tx=0).stop 2023-11-16T18:26:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.764+0000 7fc151eb7b80 1 -- 172.21.15.2:0/1538005330 shutdown_connections 2023-11-16T18:26:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.764+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/1538005330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 0x5651b0c29b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.764+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/1538005330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1db60 0x5651b0c18cc0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.764+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/1538005330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1e470 0x5651b0c2e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.764+0000 7fc151eb7b80 1 -- 172.21.15.2:0/1538005330 >> 172.21.15.2:0/1538005330 conn(0x5651b0c13360 msgr2=0x5651b0c24c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.764+0000 7fc151eb7b80 1 -- 172.21.15.2:0/1538005330 shutdown_connections 2023-11-16T18:26:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.764+0000 7fc151eb7b80 1 -- 172.21.15.2:0/1538005330 wait complete. 2023-11-16T18:26:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.764+0000 7fc151eb7b80 1 -- start start 2023-11-16T18:26:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc151eb7b80 1 Processor -- start 2023-11-16T18:26:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc151eb7b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1db60 0x5651b0c2e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc151eb7b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 0x5651b0c2c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc151eb7b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 0x5651b0c2ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc151eb7b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b0c2d410 con 0x5651b0c1e470 2023-11-16T18:26:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc151eb7b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b0c2d580 con 0x5651b0c1db60 2023-11-16T18:26:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc151eb7b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b0c2d740 con 0x5651b0c228b0 2023-11-16T18:26:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc151eb7b80 10 client.0 _ll_register_callbacks cb 0x7fc1280996e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc13effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 0x5651b0c2c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc13effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 0x5651b0c2c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51100/0 (socket says 172.21.15.2:51100) 2023-11-16T18:26:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc13effd700 1 -- 172.21.15.2:0/2864429431 learned_addr learned my addr 172.21.15.2:0/2864429431 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc13ffff700 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 0x5651b0c2ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1db60 0x5651b0c2e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc13effd700 1 -- 172.21.15.2:0/2864429431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 msgr2=0x5651b0c2ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc13effd700 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 0x5651b0c2ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.765+0000 7fc13effd700 1 -- 172.21.15.2:0/2864429431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1db60 msgr2=0x5651b0c2e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.766+0000 7fc13effd700 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1db60 0x5651b0c2e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.766+0000 7fc13effd700 1 -- 172.21.15.2:0/2864429431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14010d970 con 0x5651b0c1e470 2023-11-16T18:26:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.766+0000 7fc13ffff700 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 0x5651b0c2ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.766+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1db60 0x5651b0c2e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.766+0000 7fc13effd700 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 0x5651b0c2c990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc128000c00 tx=0x7fc128098710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9a8a8c83934850 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.766+0000 7fc11effd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.766+0000 7fc11effd700 1 -- 172.21.15.2:0/2864429431 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1280b1070 con 0x5651b0c1e470 2023-11-16T18:26:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.766+0000 7fc151eb7b80 1 -- 172.21.15.2:0/2864429431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc13800f040 con 0x5651b0c1e470 2023-11-16T18:26:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.766+0000 7fc11effd700 1 -- 172.21.15.2:0/2864429431 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc1280c8080 con 0x5651b0c1e470 2023-11-16T18:26:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.766+0000 7fc11effd700 1 -- 172.21.15.2:0/2864429431 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1280c9ff0 con 0x5651b0c1e470 2023-11-16T18:26:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.766+0000 7fc151eb7b80 10 client.15981 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:13.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.766+0000 7fc151eb7b80 1 -- 172.21.15.2:0/2864429431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5651b0985c60 con 0x5651b0c1e470 2023-11-16T18:26:13.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.766+0000 7fc11effd700 1 -- 172.21.15.2:0/2864429431 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc13800f040 con 0x5651b0c1e470 2023-11-16T18:26:13.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.767+0000 7fc11effd700 1 -- 172.21.15.2:0/2864429431 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc128099bc0 con 0x5651b0c1e470 2023-11-16T18:26:13.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.767+0000 7fc151eb7b80 10 client.15981 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:13.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.767+0000 7fc151eb7b80 10 client.15981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:13.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.767+0000 7fc151eb7b80 20 client.15981 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:13.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.767+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5651b0c8c910 0x5651b0c8ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:13.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.767+0000 7fc151eb7b80 4 client.15981 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:26:13.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.767+0000 7fc151eb7b80 1 -- 172.21.15.2:0/2864429431 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5651b0b3d660 con 0x5651b0c8c910 2023-11-16T18:26:13.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.767+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5651b0c8c910 0x5651b0c8ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:13.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.767+0000 7fc11effd700 10 client.15981 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:13.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.767+0000 7fc13f7fe700 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5651b0c8c910 0x5651b0c8ed20 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44079bfc0f065739 server_cookie=377eb0215ec0ca68 in_seq=0 out_seq=0 2023-11-16T18:26:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc11effd700 1 -- 172.21.15.2:0/2864429431 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5651b0b3d660 con 0x5651b0c8c910 2023-11-16T18:26:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc11effd700 10 client.15981 handle_command_reply: tid=0 2023-11-16T18:26:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc151eb7b80 1 client.15981 shutdown 2023-11-16T18:26:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc151eb7b80 1 -- 172.21.15.2:0/2864429431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 msgr2=0x5651b0c2c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 0x5651b0c2c990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc128000c00 tx=0x7fc128098710 comp rx=0 tx=0).stop 2023-11-16T18:26:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc151eb7b80 1 -- 172.21.15.2:0/2864429431 shutdown_connections 2023-11-16T18:26:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5651b0c8c910 0x5651b0c8ed20 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5651b0c228b0 0x5651b0c2ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651b0c1db60 0x5651b0c2e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc151eb7b80 1 --2- 172.21.15.2:0/2864429431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5651b0c1e470 0x5651b0c2c990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc151eb7b80 1 -- 172.21.15.2:0/2864429431 >> 172.21.15.2:0/2864429431 conn(0x5651b0c13360 msgr2=0x5651b0c208b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc11effd700 0 client.15981 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc151eb7b80 1 -- 172.21.15.2:0/2864429431 shutdown_connections 2023-11-16T18:26:13.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc151eb7b80 1 -- 172.21.15.2:0/2864429431 wait complete. 2023-11-16T18:26:13.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.768+0000 7fc151eb7b80 20 client.15981 trim_cache size 0 max 16384 2023-11-16T18:26:13.777 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:26:13.777 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "no active scrubs running", 2023-11-16T18:26:13.777 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": {} 2023-11-16T18:26:13.777 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:26:13.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.769+0000 7fc145d2b700 1 -- 172.21.15.2:0/2833674832 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc12c0a1460 msgr2=0x7fc12c0a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:13.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.769+0000 7fc145d2b700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc12c0a1460 0x7fc12c0a3910 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc130009990 tx=0x7fc130000bc0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.769+0000 7fc145d2b700 1 -- 172.21.15.2:0/2833674832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1400ff420 msgr2=0x7fc1400fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:13.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.769+0000 7fc145d2b700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1400ff420 0x7fc1400fff30 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc12800b520 tx=0x7fc12800b830 comp rx=0 tx=0).stop 2023-11-16T18:26:13.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.769+0000 7fc145d2b700 1 -- 172.21.15.2:0/2833674832 shutdown_connections 2023-11-16T18:26:13.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.769+0000 7fc145d2b700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc12c0a1460 0x7fc12c0a3910 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.769+0000 7fc145d2b700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1400fea80 0x7fc1400ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.769+0000 7fc145d2b700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1401055d0 0x7fc1401a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.769+0000 7fc145d2b700 1 --2- 172.21.15.2:0/2833674832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1400ff420 0x7fc1400fff30 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:13.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.769+0000 7fc145d2b700 1 -- 172.21.15.2:0/2833674832 >> 172.21.15.2:0/2833674832 conn(0x7fc140075810 msgr2=0x7fc1400fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:13.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.769+0000 7fc145d2b700 1 -- 172.21.15.2:0/2833674832 shutdown_connections 2023-11-16T18:26:13.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:13.769+0000 7fc145d2b700 1 -- 172.21.15.2:0/2833674832 wait complete. 2023-11-16T18:26:13.903 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:13.904 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:13.906 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T18:26:13.906 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:26:14.117 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:14.119 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:14.122 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:14.123 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:14.123 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 46 -lt 100 ']' 2023-11-16T18:26:14.123 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 46 + 1 2023-11-16T18:26:14.162 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=47 2023-11-16T18:26:14.162 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:14.192 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:14.192 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:14.193 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:14.193 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:14.193 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:14.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:14 smithi002 ceph-mon[98807]: pgmap v44: 129 pgs: 129 active+clean; 1.9 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 46 KiB/s wr, 29 op/s 2023-11-16T18:26:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:14 smithi040 ceph-mon[100251]: pgmap v44: 129 pgs: 129 active+clean; 1.9 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 46 KiB/s wr, 29 op/s 2023-11-16T18:26:14.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:14 smithi188 ceph-mon[100639]: pgmap v44: 129 pgs: 129 active+clean; 1.9 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 46 KiB/s wr, 29 op/s 2023-11-16T18:26:14.645 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:14.645 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:14.766 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:14.766 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:14.784 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:14.784 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:14.784 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 47 -lt 100 ']' 2023-11-16T18:26:14.784 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 47 + 1 2023-11-16T18:26:14.862 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=48 2023-11-16T18:26:14.862 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:14.923 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:14.923 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:14.924 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:14.924 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:14.924 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:15.445 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:15.445 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:15.549 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:15.549 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:15.556 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:15.557 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:15.557 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 48 -lt 100 ']' 2023-11-16T18:26:15.557 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 48 + 1 2023-11-16T18:26:15.603 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=49 2023-11-16T18:26:15.604 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:15.646 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:15.646 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:15.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:15.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:15.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:16.219 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:16.220 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:16.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:16 smithi002 ceph-mon[98807]: pgmap v45: 129 pgs: 129 active+clean; 1.9 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2023-11-16T18:26:16.458 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:16.458 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:16.466 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:16.467 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:16.467 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 49 -lt 100 ']' 2023-11-16T18:26:16.467 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 49 + 1 2023-11-16T18:26:16.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:16 smithi040 ceph-mon[100251]: pgmap v45: 129 pgs: 129 active+clean; 1.9 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2023-11-16T18:26:16.494 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=50 2023-11-16T18:26:16.495 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:16.532 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:16.532 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:16.532 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:16.533 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:16.533 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:16.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:16 smithi188 ceph-mon[100639]: pgmap v45: 129 pgs: 129 active+clean; 1.9 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2023-11-16T18:26:16.931 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.929+0000 7f2561e29700 1 -- 172.21.15.2:0/1016824693 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f255c10a8b0 msgr2=0x7f255c10ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:16.931 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.929+0000 7f2561e29700 1 --2- 172.21.15.2:0/1016824693 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f255c10a8b0 0x7f255c10ccd0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f255400b0a0 tx=0x7f255400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:26:16.931 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.929+0000 7f2561e29700 1 -- 172.21.15.2:0/1016824693 shutdown_connections 2023-11-16T18:26:16.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.929+0000 7f2561e29700 1 --2- 172.21.15.2:0/1016824693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f255c101390 0x7f255c101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:16.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.929+0000 7f2561e29700 1 --2- 172.21.15.2:0/1016824693 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f255c10a8b0 0x7f255c10ccd0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:16.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.929+0000 7f2561e29700 1 --2- 172.21.15.2:0/1016824693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f255c101ca0 0x7f255c10a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:16.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.929+0000 7f2561e29700 1 -- 172.21.15.2:0/1016824693 >> 172.21.15.2:0/1016824693 conn(0x7f255c077060 msgr2=0x7f255c077460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:16.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.929+0000 7f2561e29700 1 -- 172.21.15.2:0/1016824693 shutdown_connections 2023-11-16T18:26:16.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.929+0000 7f2561e29700 1 -- 172.21.15.2:0/1016824693 wait complete. 2023-11-16T18:26:16.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.930+0000 7f2561e29700 1 Processor -- start 2023-11-16T18:26:16.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.930+0000 7f2561e29700 1 -- start start 2023-11-16T18:26:16.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f2561e29700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f255c101390 0x7f255c1964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:16.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f2561e29700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f255c101ca0 0x7f255c196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:16.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f2561e29700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f255c10a8b0 0x7f255c19ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:16.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f2561e29700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255c19b4b0 con 0x7f255c101390 2023-11-16T18:26:16.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f2561e29700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255c19b620 con 0x7f255c10a8b0 2023-11-16T18:26:16.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f2561e29700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255c19b7e0 con 0x7f255c101ca0 2023-11-16T18:26:16.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f255b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f255c101390 0x7f255c1964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:16.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f255b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f255c101390 0x7f255c1964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51112/0 (socket says 172.21.15.2:51112) 2023-11-16T18:26:16.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f255b7fe700 1 -- 172.21.15.2:0/507877337 learned_addr learned my addr 172.21.15.2:0/507877337 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:16.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f2552ffd700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f255c101ca0 0x7f255c196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:16.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f255bfff700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f255c10a8b0 0x7f255c19ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:16.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f255b7fe700 1 -- 172.21.15.2:0/507877337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f255c101ca0 msgr2=0x7f255c196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:16.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f255b7fe700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f255c101ca0 0x7f255c196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:16.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f255b7fe700 1 -- 172.21.15.2:0/507877337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f255c10a8b0 msgr2=0x7f255c19ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:16.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f255b7fe700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f255c10a8b0 0x7f255c19ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:16.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f255b7fe700 1 -- 172.21.15.2:0/507877337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2554009d00 con 0x7f255c101390 2023-11-16T18:26:16.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.931+0000 7f255b7fe700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f255c101390 0x7f255c1964d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f254c00d950 tx=0x7f254c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:16.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.932+0000 7f25597fa700 1 -- 172.21.15.2:0/507877337 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c009b40 con 0x7f255c101390 2023-11-16T18:26:16.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.932+0000 7f25597fa700 1 -- 172.21.15.2:0/507877337 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f254c009ca0 con 0x7f255c101390 2023-11-16T18:26:16.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.932+0000 7f2561e29700 1 -- 172.21.15.2:0/507877337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f255c1102f0 con 0x7f255c101390 2023-11-16T18:26:16.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.932+0000 7f25597fa700 1 -- 172.21.15.2:0/507877337 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c00f620 con 0x7f255c101390 2023-11-16T18:26:16.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.932+0000 7f2561e29700 1 -- 172.21.15.2:0/507877337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f255c110160 con 0x7f255c101390 2023-11-16T18:26:16.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.933+0000 7f2561e29700 1 -- 172.21.15.2:0/507877337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f255c110480 con 0x7f255c101390 2023-11-16T18:26:16.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.935+0000 7f25597fa700 1 -- 172.21.15.2:0/507877337 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f254c010460 con 0x7f255c101390 2023-11-16T18:26:16.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.935+0000 7f25597fa700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f25480a1460 0x7f25480a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:16.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.935+0000 7f25597fa700 1 -- 172.21.15.2:0/507877337 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f254c0cd670 con 0x7f255c101390 2023-11-16T18:26:16.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.935+0000 7f2552ffd700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f25480a1460 0x7f25480a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:16.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.936+0000 7f2552ffd700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f25480a1460 0x7f25480a3910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f255c197ae0 tx=0x7f2544009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:16.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:16.939+0000 7f25597fa700 1 -- 172.21.15.2:0/507877337 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f254c0964c0 con 0x7f255c101390 2023-11-16T18:26:17.069 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:17.070 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:17.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.201+0000 7f2561e29700 1 -- 172.21.15.2:0/507877337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f255c067af0 con 0x7f255c101390 2023-11-16T18:26:17.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.202+0000 7f25597fa700 1 -- 172.21.15.2:0/507877337 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10774 (secure 0 0 0) 0x7f254c016090 con 0x7f255c101390 2023-11-16T18:26:17.203 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:26:17.203 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:23:07.909835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:26:17.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.204+0000 7f2561e29700 1 -- 172.21.15.2:0/507877337 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f25480a1460 msgr2=0x7f25480a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:17.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.204+0000 7f2561e29700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f25480a1460 0x7f25480a3910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f255c197ae0 tx=0x7f2544009260 comp rx=0 tx=0).stop 2023-11-16T18:26:17.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.204+0000 7f2561e29700 1 -- 172.21.15.2:0/507877337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f255c101390 msgr2=0x7f255c1964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:17.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.204+0000 7f2561e29700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f255c101390 0x7f255c1964d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f254c00d950 tx=0x7f254c00dd10 comp rx=0 tx=0).stop 2023-11-16T18:26:17.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.204+0000 7f2561e29700 1 -- 172.21.15.2:0/507877337 shutdown_connections 2023-11-16T18:26:17.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.204+0000 7f2561e29700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f25480a1460 0x7f25480a3910 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:17.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.204+0000 7f2561e29700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f255c10a8b0 0x7f255c19ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:17.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.204+0000 7f2561e29700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f255c101390 0x7f255c1964d0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:17.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.204+0000 7f2561e29700 1 --2- 172.21.15.2:0/507877337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f255c101ca0 0x7f255c196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:17.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.204+0000 7f2561e29700 1 -- 172.21.15.2:0/507877337 >> 172.21.15.2:0/507877337 conn(0x7f255c077060 msgr2=0x7f255c0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:17.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.204+0000 7f2561e29700 1 -- 172.21.15.2:0/507877337 shutdown_connections 2023-11-16T18:26:17.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:17.204+0000 7f2561e29700 1 -- 172.21.15.2:0/507877337 wait complete. 2023-11-16T18:26:17.207 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T18:26:17.300 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:17.300 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:17.318 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-11-16T18:26:17.322 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:17.322 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:17.322 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 50 -lt 100 ']' 2023-11-16T18:26:17.323 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 50 + 1 2023-11-16T18:26:17.374 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=51 2023-11-16T18:26:17.374 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:17.391 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:17.391 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:17.392 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:17.392 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:17.392 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:17.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:17 smithi188 ceph-mon[100639]: pgmap v46: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 17 op/s 2023-11-16T18:26:17.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:17 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/507877337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:26:17.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:17 smithi002 ceph-mon[98807]: pgmap v46: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 17 op/s 2023-11-16T18:26:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:17 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/507877337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:26:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:17 smithi040 ceph-mon[100251]: pgmap v46: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 17 op/s 2023-11-16T18:26:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:17 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/507877337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:26:17.802 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:17.802 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:17.891 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:17.891 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:17.896 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:17.897 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:17.897 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 51 -lt 100 ']' 2023-11-16T18:26:17.897 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 51 + 1 2023-11-16T18:26:17.933 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=52 2023-11-16T18:26:17.933 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:17.985 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:17.985 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:17.986 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:17.986 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:17.986 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:18.486 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:18.486 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:18.640 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:18.641 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:18.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:18.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:18.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 52 -lt 100 ']' 2023-11-16T18:26:18.647 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 52 + 1 2023-11-16T18:26:18.699 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=53 2023-11-16T18:26:18.699 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:18.752 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:18.753 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:18.753 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:18.754 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:18.754 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:19.503 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:19.503 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:19.801 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:19.802 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:19.817 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:19.817 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:19.817 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 53 -lt 100 ']' 2023-11-16T18:26:19.817 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 53 + 1 2023-11-16T18:26:19.861 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=54 2023-11-16T18:26:19.861 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:19.928 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:19.928 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:19.929 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:19.930 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:19.930 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:20.331 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.330+0000 7f8e185f8700 1 -- 172.21.15.2:0/1431377368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e101055d0 msgr2=0x7f8e1010d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.331 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.330+0000 7f8e185f8700 1 --2- 172.21.15.2:0/1431377368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e101055d0 0x7f8e1010d130 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009f90 tx=0x7f8e0800b330 comp rx=0 tx=0).stop 2023-11-16T18:26:20.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.330+0000 7fe9239d0700 1 -- 172.21.15.2:0/2438738388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe91c10a810 msgr2=0x7fe91c10cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.330+0000 7fe9239d0700 1 --2- 172.21.15.2:0/2438738388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe91c10a810 0x7fe91c10cc30 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe91400b0a0 tx=0x7fe91400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.330+0000 7f8e185f8700 1 -- 172.21.15.2:0/1431377368 shutdown_connections 2023-11-16T18:26:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.330+0000 7f8e185f8700 1 --2- 172.21.15.2:0/1431377368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e100fea80 0x7f8e100fee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.330+0000 7f8e185f8700 1 --2- 172.21.15.2:0/1431377368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e101055d0 0x7f8e1010d130 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.330+0000 7f8e185f8700 1 --2- 172.21.15.2:0/1431377368 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e100ff420 0x7f8e10104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.330+0000 7f8e185f8700 1 -- 172.21.15.2:0/1431377368 >> 172.21.15.2:0/1431377368 conn(0x7f8e10075810 msgr2=0x7f8e10075c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.330+0000 7fe9239d0700 1 -- 172.21.15.2:0/2438738388 shutdown_connections 2023-11-16T18:26:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.330+0000 7fe9239d0700 1 --2- 172.21.15.2:0/2438738388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe91c10a810 0x7fe91c10cc30 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.330+0000 7fe9239d0700 1 --2- 172.21.15.2:0/2438738388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe91c101440 0x7fe91c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.330+0000 7fe9239d0700 1 --2- 172.21.15.2:0/2438738388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe91c101d50 0x7fe91c10a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.330+0000 7fe9239d0700 1 -- 172.21.15.2:0/2438738388 >> 172.21.15.2:0/2438738388 conn(0x7fe91c0763a0 msgr2=0x7fe91c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.330+0000 7f8e185f8700 1 -- 172.21.15.2:0/1431377368 shutdown_connections 2023-11-16T18:26:20.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.330+0000 7f8e185f8700 1 -- 172.21.15.2:0/1431377368 wait complete. 2023-11-16T18:26:20.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.330+0000 7fe9239d0700 1 -- 172.21.15.2:0/2438738388 shutdown_connections 2023-11-16T18:26:20.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.330+0000 7fe9239d0700 1 -- 172.21.15.2:0/2438738388 wait complete. 2023-11-16T18:26:20.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.331+0000 7fe9239d0700 1 Processor -- start 2023-11-16T18:26:20.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.331+0000 7fe9239d0700 1 -- start start 2023-11-16T18:26:20.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.331+0000 7f8e185f8700 1 Processor -- start 2023-11-16T18:26:20.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.332+0000 7f8e185f8700 1 -- start start 2023-11-16T18:26:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe9239d0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe91c101440 0x7fe91c196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe9239d0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe91c101d50 0x7fe91c196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe9239d0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe91c10a810 0x7fe91c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe9239d0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c19b400 con 0x7fe91c101440 2023-11-16T18:26:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe9239d0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c19b570 con 0x7fe91c10a810 2023-11-16T18:26:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe9239d0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c19b730 con 0x7fe91c101d50 2023-11-16T18:26:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe92176c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe91c101440 0x7fe91c196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe92176c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe91c101440 0x7fe91c196420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51130/0 (socket says 172.21.15.2:51130) 2023-11-16T18:26:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe92176c700 1 -- 172.21.15.2:0/3767241010 learned_addr learned my addr 172.21.15.2:0/3767241010 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:20.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.332+0000 7f8e185f8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100fea80 0x7f8e101965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.332+0000 7f8e185f8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e100ff420 0x7f8e10196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.332+0000 7f8e185f8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e101055d0 0x7f8e1019af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe920f6b700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe91c101d50 0x7fe91c196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe921f6d700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe91c10a810 0x7fe91c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.332+0000 7f8e185f8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1019b5c0 con 0x7f8e100ff420 2023-11-16T18:26:20.338 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.332+0000 7f8e185f8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1019b730 con 0x7f8e100fea80 2023-11-16T18:26:20.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.332+0000 7f8e185f8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1019b8f0 con 0x7f8e101055d0 2023-11-16T18:26:20.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.332+0000 7f8e16394700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100fea80 0x7f8e101965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.332+0000 7f8e15b93700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e100ff420 0x7f8e10196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.332+0000 7f8e16394700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100fea80 0x7f8e101965e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:41574/0 (socket says 172.21.15.2:41574) 2023-11-16T18:26:20.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.332+0000 7f8e16394700 1 -- 172.21.15.2:0/1662722334 learned_addr learned my addr 172.21.15.2:0/1662722334 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:20.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.332+0000 7f8e16b95700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e101055d0 0x7f8e1019af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe920f6b700 1 -- 172.21.15.2:0/3767241010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe91c10a810 msgr2=0x7fe91c19adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe920f6b700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe91c10a810 0x7fe91c19adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe920f6b700 1 -- 172.21.15.2:0/3767241010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe91c101440 msgr2=0x7fe91c196420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.332+0000 7fe920f6b700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe91c101440 0x7fe91c196420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.333+0000 7fe920f6b700 1 -- 172.21.15.2:0/3767241010 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe918009520 con 0x7fe91c101d50 2023-11-16T18:26:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.333+0000 7fe921f6d700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe91c10a810 0x7fe91c19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:20.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.333+0000 7f8e16394700 1 -- 172.21.15.2:0/1662722334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e101055d0 msgr2=0x7f8e1019af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.333+0000 7f8e16394700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e101055d0 0x7f8e1019af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.333+0000 7f8e16394700 1 -- 172.21.15.2:0/1662722334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e100ff420 msgr2=0x7f8e10196b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.333+0000 7f8e16394700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e100ff420 0x7f8e10196b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.333+0000 7f8e16394700 1 -- 172.21.15.2:0/1662722334 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e000095f0 con 0x7f8e100fea80 2023-11-16T18:26:20.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.333+0000 7f8e16b95700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e101055d0 0x7f8e1019af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:20.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.333+0000 7fe920f6b700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe91c101d50 0x7fe91c196960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00e9f0 tx=0x7fe90c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:20.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.333+0000 7fe9127fc700 1 -- 172.21.15.2:0/3767241010 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c00eee0 con 0x7fe91c101d50 2023-11-16T18:26:20.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.333+0000 7fe9239d0700 1 -- 172.21.15.2:0/3767241010 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe914009d00 con 0x7fe91c101d50 2023-11-16T18:26:20.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.333+0000 7fe9127fc700 1 -- 172.21.15.2:0/3767241010 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe90c00cbd0 con 0x7fe91c101d50 2023-11-16T18:26:20.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.333+0000 7fe9127fc700 1 -- 172.21.15.2:0/3767241010 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c018680 con 0x7fe91c101d50 2023-11-16T18:26:20.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.333+0000 7fe9239d0700 1 -- 172.21.15.2:0/3767241010 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91c110260 con 0x7fe91c101d50 2023-11-16T18:26:20.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.333+0000 7f8e16394700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100fea80 0x7f8e101965e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00e9a0 tx=0x7f8e0c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.333+0000 7f8e077fe700 1 -- 172.21.15.2:0/1662722334 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c009790 con 0x7f8e100fea80 2023-11-16T18:26:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.333+0000 7f8e077fe700 1 -- 172.21.15.2:0/1662722334 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e0c010430 con 0x7f8e100fea80 2023-11-16T18:26:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.333+0000 7f8e077fe700 1 -- 172.21.15.2:0/1662722334 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c018680 con 0x7f8e100fea80 2023-11-16T18:26:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.333+0000 7f8e185f8700 1 -- 172.21.15.2:0/1662722334 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e08009c40 con 0x7f8e100fea80 2023-11-16T18:26:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.334+0000 7f8e185f8700 1 -- 172.21.15.2:0/1662722334 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e10110390 con 0x7f8e100fea80 2023-11-16T18:26:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.336+0000 7fe9127fc700 1 -- 172.21.15.2:0/3767241010 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7fe90c00cd40 con 0x7fe91c101d50 2023-11-16T18:26:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.336+0000 7f8e077fe700 1 -- 172.21.15.2:0/1662722334 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f8e0c0105a0 con 0x7f8e100fea80 2023-11-16T18:26:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.336+0000 7fe9127fc700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe9080a13e0 0x7fe9080a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.337+0000 7fe92176c700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe9080a13e0 0x7fe9080a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.337+0000 7fe9127fc700 1 -- 172.21.15.2:0/3767241010 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe90c014070 con 0x7fe91c101d50 2023-11-16T18:26:20.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.337+0000 7f8e077fe700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f8dfc0a1460 0x7f8dfc0a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.337+0000 7f8e15b93700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f8dfc0a1460 0x7f8dfc0a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.337+0000 7fe92176c700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe9080a13e0 0x7fe9080a3890 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe918009ef0 tx=0x7fe918009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:20.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.337+0000 7f8e077fe700 1 -- 172.21.15.2:0/1662722334 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f8e0c014070 con 0x7f8e100fea80 2023-11-16T18:26:20.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.338+0000 7f8e15b93700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f8dfc0a1460 0x7f8dfc0a3910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8e10197bf0 tx=0x7f8e00009310 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:20.348 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:20 smithi002 ceph-mon[98807]: pgmap v47: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-11-16T18:26:20.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.414+0000 7f8e24784b80 1 Processor -- start 2023-11-16T18:26:20.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.414+0000 7fe92fb5cb80 1 Processor -- start 2023-11-16T18:26:20.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.414+0000 7fe92fb5cb80 1 -- start start 2023-11-16T18:26:20.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.415+0000 7f8e24784b80 1 -- start start 2023-11-16T18:26:20.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.415+0000 7fe92fb5cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a7011e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.415+0000 7fe92fb5cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701720 0x555e0a6fb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.415+0000 7fe92fb5cb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701b90 0x555e0a6fbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.415+0000 7fe92fb5cb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0a601000 con 0x555e0a701b90 2023-11-16T18:26:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.415+0000 7fe92fb5cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0a653840 con 0x555e0a701720 2023-11-16T18:26:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.415+0000 7fe92fb5cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0a64ccd0 con 0x555e0a5576b0 2023-11-16T18:26:20.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.415+0000 7f8e24784b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f22bb60 0x55a02f22bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.415+0000 7f8e24784b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f230370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.415+0000 7f8e24784b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f2308b0 0x55a02f230d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.415+0000 7f8e24784b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02f131060 con 0x55a02f22c470 2023-11-16T18:26:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.415+0000 7f8e24784b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02f1838a0 con 0x55a02f2308b0 2023-11-16T18:26:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.415+0000 7f8e24784b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02f17cd30 con 0x55a02f22bb60 2023-11-16T18:26:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.416+0000 7f8e16394700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f22bb60 0x55a02f22bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.416+0000 7f8e16394700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f22bb60 0x55a02f22bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:42682/0 (socket says 172.21.15.2:42682) 2023-11-16T18:26:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.416+0000 7f8e16394700 1 -- 172.21.15.2:0/681771227 learned_addr learned my addr 172.21.15.2:0/681771227 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:20.421 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.415+0000 7fe92176c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a7011e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.416+0000 7f8e15b93700 1 --2- 172.21.15.2:0/681771227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f230370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.416+0000 7f8e16b95700 1 --2- 172.21.15.2:0/681771227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f2308b0 0x55a02f230d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.416+0000 7fe920f6b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701720 0x555e0a6fb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.416+0000 7fe920f6b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701720 0x555e0a6fb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:41588/0 (socket says 172.21.15.2:41588) 2023-11-16T18:26:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.416+0000 7fe920f6b700 1 -- 172.21.15.2:0/6498725 learned_addr learned my addr 172.21.15.2:0/6498725 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.416+0000 7fe921f6d700 1 --2- 172.21.15.2:0/6498725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701b90 0x555e0a6fbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.416+0000 7f8e16394700 1 -- 172.21.15.2:0/681771227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f2308b0 msgr2=0x55a02f230d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.416+0000 7f8e16394700 1 --2- 172.21.15.2:0/681771227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f2308b0 0x55a02f230d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.416+0000 7f8e16394700 1 -- 172.21.15.2:0/681771227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 msgr2=0x55a02f230370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.416+0000 7f8e16394700 1 --2- 172.21.15.2:0/681771227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f230370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.416+0000 7f8e16394700 1 -- 172.21.15.2:0/681771227 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a02effe630 con 0x55a02f22bb60 2023-11-16T18:26:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.416+0000 7fe920f6b700 1 -- 172.21.15.2:0/6498725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 msgr2=0x555e0a7011e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.416+0000 7fe920f6b700 1 --2- 172.21.15.2:0/6498725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a7011e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.416+0000 7fe920f6b700 1 -- 172.21.15.2:0/6498725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701b90 msgr2=0x555e0a6fbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.416+0000 7fe920f6b700 1 --2- 172.21.15.2:0/6498725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701b90 0x555e0a6fbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.417+0000 7fe920f6b700 1 -- 172.21.15.2:0/6498725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e0a4ce5d0 con 0x555e0a701720 2023-11-16T18:26:20.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.417+0000 7f8e16394700 1 --2- 172.21.15.2:0/681771227 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f22bb60 0x55a02f22bf30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0964a0 tx=0x7f8e0c099bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fe6db21badfdb55 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:20.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.417+0000 7fe920f6b700 1 --2- 172.21.15.2:0/6498725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701720 0x555e0a6fb560 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe90c099c60 tx=0x7fe90c0998c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a2f85d3633a5356 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:20.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.417+0000 7f8e04ff9700 1 -- 172.21.15.2:0/681771227 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c099fa0 con 0x55a02f22bb60 2023-11-16T18:26:20.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.417+0000 7f8e04ff9700 1 -- 172.21.15.2:0/681771227 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e0c09dfd0 con 0x55a02f22bb60 2023-11-16T18:26:20.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.417+0000 7f8e04ff9700 1 -- 172.21.15.2:0/681771227 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c0996f0 con 0x55a02f22bb60 2023-11-16T18:26:20.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.417+0000 7fe9077fe700 1 -- 172.21.15.2:0/6498725 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c09df90 con 0x555e0a701720 2023-11-16T18:26:20.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.417+0000 7fe9077fe700 1 -- 172.21.15.2:0/6498725 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe90c0a23b0 con 0x555e0a701720 2023-11-16T18:26:20.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.417+0000 7fe9077fe700 1 -- 172.21.15.2:0/6498725 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c0a21a0 con 0x555e0a701720 2023-11-16T18:26:20.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.418+0000 7f8e24784b80 1 -- 172.21.15.2:0/681771227 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f22bb60 msgr2=0x55a02f22bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.418+0000 7f8e24784b80 1 --2- 172.21.15.2:0/681771227 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f22bb60 0x55a02f22bf30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0964a0 tx=0x7f8e0c099bb0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.418+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/6498725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701720 msgr2=0x555e0a6fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.418+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/6498725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701720 0x555e0a6fb560 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe90c099c60 tx=0x7fe90c0998c0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.418+0000 7f8e24784b80 1 -- 172.21.15.2:0/681771227 shutdown_connections 2023-11-16T18:26:20.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.418+0000 7f8e24784b80 1 --2- 172.21.15.2:0/681771227 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f22bb60 0x55a02f22bf30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.418+0000 7f8e24784b80 1 --2- 172.21.15.2:0/681771227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f2308b0 0x55a02f230d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.418+0000 7f8e24784b80 1 --2- 172.21.15.2:0/681771227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f230370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.418+0000 7f8e24784b80 1 -- 172.21.15.2:0/681771227 >> 172.21.15.2:0/681771227 conn(0x55a02f221360 msgr2=0x55a02f226b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.418+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/6498725 shutdown_connections 2023-11-16T18:26:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.418+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/6498725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a7011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.418+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/6498725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701b90 0x555e0a6fbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.418+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/6498725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701720 0x555e0a6fb560 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.418+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/6498725 >> 172.21.15.2:0/6498725 conn(0x555e0a6f0f50 msgr2=0x555e0a6f6270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:20.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.418+0000 7f8e24784b80 1 -- 172.21.15.2:0/681771227 shutdown_connections 2023-11-16T18:26:20.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.418+0000 7f8e24784b80 1 -- 172.21.15.2:0/681771227 wait complete. 2023-11-16T18:26:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.418+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/6498725 shutdown_connections 2023-11-16T18:26:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.418+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/6498725 wait complete. 2023-11-16T18:26:20.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.448+0000 7f8e24784b80 1 -- start start 2023-11-16T18:26:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.448+0000 7fe92fb5cb80 1 -- start start 2023-11-16T18:26:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.449+0000 7fe92fb5cb80 1 Processor -- start 2023-11-16T18:26:20.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.449+0000 7f8e24784b80 1 Processor -- start 2023-11-16T18:26:20.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.449+0000 7f8e24784b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f22bb60 0x55a02f238070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.449+0000 7f8e24784b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f2352f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.449+0000 7f8e24784b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f2308b0 0x55a02f235990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.449+0000 7f8e24784b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02f11e8e0 con 0x55a02f22c470 2023-11-16T18:26:20.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.449+0000 7f8e24784b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02f11e580 con 0x55a02f22bb60 2023-11-16T18:26:20.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.449+0000 7f8e24784b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02f000920 con 0x55a02f2308b0 2023-11-16T18:26:20.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.449+0000 7f8e24784b80 10 client.0 _ll_register_callbacks cb 0x55a02f13c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:20.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.449+0000 7f8e15b93700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f2352f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.449+0000 7f8e15b93700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f2352f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51158/0 (socket says 172.21.15.2:51158) 2023-11-16T18:26:20.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.449+0000 7f8e15b93700 1 -- 172.21.15.2:0/1900593503 learned_addr learned my addr 172.21.15.2:0/1900593503 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:20.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.449+0000 7f8e16b95700 1 --2- 172.21.15.2:0/1900593503 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f2308b0 0x55a02f235990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe92fb5cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a713300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe92fb5cb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 0x555e0a713840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.455 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.450+0000 7f8e16394700 1 --2- 172.21.15.2:0/1900593503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f22bb60 0x55a02f238070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe92fb5cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a70d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe92fb5cb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0a5ee880 con 0x555e0a701720 2023-11-16T18:26:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe92fb5cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0a5ee520 con 0x555e0a701b90 2023-11-16T18:26:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe92fb5cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0a4d08c0 con 0x555e0a5576b0 2023-11-16T18:26:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe92fb5cb80 10 client.0 _ll_register_callbacks cb 0x555e0a60c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:20.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.450+0000 7f8e15b93700 1 -- 172.21.15.2:0/1900593503 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f2308b0 msgr2=0x55a02f235990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.450+0000 7f8e15b93700 1 --2- 172.21.15.2:0/1900593503 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f2308b0 0x55a02f235990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.450+0000 7f8e15b93700 1 -- 172.21.15.2:0/1900593503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f22bb60 msgr2=0x55a02f238070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.450+0000 7f8e15b93700 1 --2- 172.21.15.2:0/1900593503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f22bb60 0x55a02f238070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.450+0000 7f8e15b93700 1 -- 172.21.15.2:0/1900593503 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e10110390 con 0x55a02f22c470 2023-11-16T18:26:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe920f6b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 0x555e0a713840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe920f6b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 0x555e0a713840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51164/0 (socket says 172.21.15.2:51164) 2023-11-16T18:26:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe920f6b700 1 -- 172.21.15.2:0/2951958371 learned_addr learned my addr 172.21.15.2:0/2951958371 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe921f6d700 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a70d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe92176c700 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a713300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.450+0000 7f8e15b93700 1 --2- 172.21.15.2:0/1900593503 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f2352f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8e00003930 tx=0x7f8e000047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f524e3223ce81d11 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:20.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.450+0000 7f8df9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe920f6b700 1 -- 172.21.15.2:0/2951958371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 msgr2=0x555e0a713300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe920f6b700 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a713300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe920f6b700 1 -- 172.21.15.2:0/2951958371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 msgr2=0x555e0a70d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe920f6b700 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a70d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe920f6b700 1 -- 172.21.15.2:0/2951958371 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91c110260 con 0x555e0a701720 2023-11-16T18:26:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.450+0000 7fe92176c700 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a713300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:20.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.450+0000 7f8df9ffb700 1 -- 172.21.15.2:0/1900593503 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00022070 con 0x55a02f22c470 2023-11-16T18:26:20.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.450+0000 7f8e24784b80 1 -- 172.21.15.2:0/1900593503 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a02f21eb20 con 0x55a02f22c470 2023-11-16T18:26:20.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.450+0000 7f8df9ffb700 1 -- 172.21.15.2:0/1900593503 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f8e0000e800 con 0x55a02f22c470 2023-11-16T18:26:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.451+0000 7fe920f6b700 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 0x555e0a713840 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00c5b0 tx=0x7fe90c0ca650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e64cffedfb032e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.451+0000 7fe904ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:20.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.451+0000 7f8df9ffb700 1 -- 172.21.15.2:0/1900593503 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e0000e9c0 con 0x55a02f22c470 2023-11-16T18:26:20.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.451+0000 7f8e24784b80 10 client.16008 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:20.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.451+0000 7f8e24784b80 1 -- 172.21.15.2:0/1900593503 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a02f143190 con 0x55a02f22c470 2023-11-16T18:26:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.451+0000 7fe904ff9700 1 -- 172.21.15.2:0/2951958371 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c0a1f20 con 0x555e0a701720 2023-11-16T18:26:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.451+0000 7fe904ff9700 1 -- 172.21.15.2:0/2951958371 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe90c0b82c0 con 0x555e0a701720 2023-11-16T18:26:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.451+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/2951958371 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e0a6ee710 con 0x555e0a701720 2023-11-16T18:26:20.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.451+0000 7f8df9ffb700 1 -- 172.21.15.2:0/1900593503 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a02f21eb20 con 0x55a02f22c470 2023-11-16T18:26:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.451+0000 7fe904ff9700 1 -- 172.21.15.2:0/2951958371 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe90c0ca0d0 con 0x555e0a701720 2023-11-16T18:26:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.451+0000 7fe92fb5cb80 10 client.16011 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.451+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/2951958371 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x555e0a613130 con 0x555e0a701720 2023-11-16T18:26:20.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.451+0000 7f8df9ffb700 1 -- 172.21.15.2:0/1900593503 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8e00027be0 con 0x55a02f22c470 2023-11-16T18:26:20.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.451+0000 7f8e24784b80 10 client.16008 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:20.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.451+0000 7f8e24784b80 10 client.16008 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2023-11-16T18:26:20.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.451+0000 7f8e24784b80 20 client.16008 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:20.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.451+0000 7f8e24784b80 1 --2- 172.21.15.2:0/1900593503 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a02f249ca0 0x55a02f24c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.451+0000 7f8e24784b80 4 client.16008 mds_command: new command op to 24446 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:20.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.451+0000 7f8e24784b80 1 -- 172.21.15.2:0/1900593503 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a02f13f0a0 con 0x55a02f249ca0 2023-11-16T18:26:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.451+0000 7fe904ff9700 1 -- 172.21.15.2:0/2951958371 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555e0a6ee710 con 0x555e0a701720 2023-11-16T18:26:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.452+0000 7fe904ff9700 1 -- 172.21.15.2:0/2951958371 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe90c098ec0 con 0x555e0a701720 2023-11-16T18:26:20.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.452+0000 7f8e16394700 1 --2- 172.21.15.2:0/1900593503 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a02f249ca0 0x55a02f24c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.452+0000 7fe92fb5cb80 10 client.16011 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.452+0000 7fe92fb5cb80 10 client.16011 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.452+0000 7fe92fb5cb80 20 client.16011 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.452+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555e0a71d3d0 0x555e0a71f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.452+0000 7fe92fb5cb80 4 client.16011 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.452+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/2951958371 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e0a60f040 con 0x555e0a71d3d0 2023-11-16T18:26:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.452+0000 7fe92176c700 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555e0a71d3d0 0x555e0a71f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.452+0000 7f8e16394700 1 --2- 172.21.15.2:0/1900593503 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a02f249ca0 0x55a02f24c0b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d158eb9a04474fbe server_cookie=74f7c64e1a1afcc5 in_seq=0 out_seq=0 2023-11-16T18:26:20.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.452+0000 7f8df9ffb700 10 client.16008 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:26:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.452+0000 7fe904ff9700 10 client.16011 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:20.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.452+0000 7fe92176c700 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555e0a71d3d0 0x555e0a71f7e0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50cc07b342ec6ddd server_cookie=ab52a163ce7b4e2f in_seq=0 out_seq=0 2023-11-16T18:26:20.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8df9ffb700 1 -- 172.21.15.2:0/1900593503 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a02f13f0a0 con 0x55a02f249ca0 2023-11-16T18:26:20.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8df9ffb700 10 client.16008 handle_command_reply: tid=0 2023-11-16T18:26:20.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8e24784b80 1 client.16008 shutdown 2023-11-16T18:26:20.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8e24784b80 1 -- 172.21.15.2:0/1900593503 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 msgr2=0x55a02f2352f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8e24784b80 1 --2- 172.21.15.2:0/1900593503 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f2352f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8e00003930 tx=0x7f8e000047c0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.453+0000 7fe904ff9700 1 -- 172.21.15.2:0/2951958371 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555e0a60f040 con 0x555e0a71d3d0 2023-11-16T18:26:20.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.453+0000 7fe904ff9700 10 client.16011 handle_command_reply: tid=0 2023-11-16T18:26:20.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8e24784b80 1 -- 172.21.15.2:0/1900593503 shutdown_connections 2023-11-16T18:26:20.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8e24784b80 1 --2- 172.21.15.2:0/1900593503 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a02f249ca0 0x55a02f24c0b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8e24784b80 1 --2- 172.21.15.2:0/1900593503 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f2308b0 0x55a02f235990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8e24784b80 1 --2- 172.21.15.2:0/1900593503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f22bb60 0x55a02f238070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8df9ffb700 0 client.16008 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:26:20.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8e24784b80 1 --2- 172.21.15.2:0/1900593503 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f2352f0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8e24784b80 1 -- 172.21.15.2:0/1900593503 >> 172.21.15.2:0/1900593503 conn(0x55a02f221360 msgr2=0x55a02f225de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:20.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.453+0000 7fe92fb5cb80 1 client.16011 shutdown 2023-11-16T18:26:20.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8e24784b80 1 -- 172.21.15.2:0/1900593503 shutdown_connections 2023-11-16T18:26:20.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.453+0000 7f8e24784b80 1 -- 172.21.15.2:0/1900593503 wait complete. 2023-11-16T18:26:20.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.453+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/2951958371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 msgr2=0x555e0a713840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.453+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 0x555e0a713840 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00c5b0 tx=0x7fe90c0ca650 comp rx=0 tx=0).stop 2023-11-16T18:26:20.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.454+0000 7f8e24784b80 20 client.16008 trim_cache size 0 max 16384 2023-11-16T18:26:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.454+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/2951958371 shutdown_connections 2023-11-16T18:26:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.454+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555e0a71d3d0 0x555e0a71f7e0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.454+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a713300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.454+0000 7fe904ff9700 0 client.16011 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.454+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a70d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.454+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/2951958371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 0x555e0a713840 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.454+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/2951958371 >> 172.21.15.2:0/2951958371 conn(0x555e0a6f0f50 msgr2=0x555e0a6f57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.454+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/2951958371 shutdown_connections 2023-11-16T18:26:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.454+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/2951958371 wait complete. 2023-11-16T18:26:20.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.454+0000 7fe92fb5cb80 20 client.16011 trim_cache size 0 max 16384 2023-11-16T18:26:20.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.467+0000 7f8e24784b80 1 Processor -- start 2023-11-16T18:26:20.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.467+0000 7f8e24784b80 1 -- start start 2023-11-16T18:26:20.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.467+0000 7f8e24784b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22bb60 0x55a02f22f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.467+0000 7f8e24784b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f22c470 0x55a02f237c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.467+0000 7f8e24784b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f2308b0 0x55a02f226560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e24784b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02f227810 con 0x55a02f22bb60 2023-11-16T18:26:20.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e24784b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02f220ee0 con 0x55a02f2308b0 2023-11-16T18:26:20.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e24784b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02f236900 con 0x55a02f22c470 2023-11-16T18:26:20.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e16394700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22bb60 0x55a02f22f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e16394700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22bb60 0x55a02f22f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51172/0 (socket says 172.21.15.2:51172) 2023-11-16T18:26:20.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e16394700 1 -- 172.21.15.2:0/4094960539 learned_addr learned my addr 172.21.15.2:0/4094960539 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:20.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e15b93700 1 --2- 172.21.15.2:0/4094960539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f22c470 0x55a02f237c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e16b95700 1 --2- 172.21.15.2:0/4094960539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f2308b0 0x55a02f226560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e16394700 1 -- 172.21.15.2:0/4094960539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f22c470 msgr2=0x55a02f237c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e16394700 1 --2- 172.21.15.2:0/4094960539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f22c470 0x55a02f237c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e16394700 1 -- 172.21.15.2:0/4094960539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f2308b0 msgr2=0x55a02f226560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e16394700 1 --2- 172.21.15.2:0/4094960539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f2308b0 0x55a02f226560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e16394700 1 -- 172.21.15.2:0/4094960539 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a02f142ee0 con 0x55a02f22bb60 2023-11-16T18:26:20.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e16b95700 1 --2- 172.21.15.2:0/4094960539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f2308b0 0x55a02f226560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:26:20.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e16394700 1 --2- 172.21.15.2:0/4094960539 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22bb60 0x55a02f22f1a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00e970 tx=0x7f8e0c095cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53dbb67cabe5d21a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:20.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e057fa700 1 -- 172.21.15.2:0/4094960539 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c0983c0 con 0x55a02f22bb60 2023-11-16T18:26:20.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e057fa700 1 -- 172.21.15.2:0/4094960539 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e0c0ca630 con 0x55a02f22bb60 2023-11-16T18:26:20.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.468+0000 7f8e057fa700 1 -- 172.21.15.2:0/4094960539 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c0a21c0 con 0x55a02f22bb60 2023-11-16T18:26:20.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.469+0000 7f8e24784b80 1 -- 172.21.15.2:0/4094960539 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22bb60 msgr2=0x55a02f22f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.469+0000 7f8e24784b80 1 --2- 172.21.15.2:0/4094960539 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22bb60 0x55a02f22f1a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00e970 tx=0x7f8e0c095cb0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.469+0000 7f8e24784b80 1 -- 172.21.15.2:0/4094960539 shutdown_connections 2023-11-16T18:26:20.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.469+0000 7f8e24784b80 1 --2- 172.21.15.2:0/4094960539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f2308b0 0x55a02f226560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.469+0000 7f8e24784b80 1 --2- 172.21.15.2:0/4094960539 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22bb60 0x55a02f22f1a0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.469+0000 7f8e24784b80 1 --2- 172.21.15.2:0/4094960539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f22c470 0x55a02f237c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.469+0000 7f8e24784b80 1 -- 172.21.15.2:0/4094960539 >> 172.21.15.2:0/4094960539 conn(0x55a02f221360 msgr2=0x55a02f22fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:20.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.469+0000 7f8e24784b80 1 -- 172.21.15.2:0/4094960539 shutdown_connections 2023-11-16T18:26:20.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.469+0000 7f8e24784b80 1 -- 172.21.15.2:0/4094960539 wait complete. 2023-11-16T18:26:20.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e24784b80 1 -- start start 2023-11-16T18:26:20.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e24784b80 1 Processor -- start 2023-11-16T18:26:20.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e24784b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f22bb60 0x55a02f23cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e24784b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f23d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e24784b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f2308b0 0x55a02f296b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e24784b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02f297100 con 0x55a02f22c470 2023-11-16T18:26:20.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e24784b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02f297270 con 0x55a02f22bb60 2023-11-16T18:26:20.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e24784b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02f297430 con 0x55a02f2308b0 2023-11-16T18:26:20.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e24784b80 10 client.0 _ll_register_callbacks cb 0x7f8e00004530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.470+0000 7fe92fb5cb80 1 Processor -- start 2023-11-16T18:26:20.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e15b93700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f23d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.470+0000 7fe92fb5cb80 1 -- start start 2023-11-16T18:26:20.491 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:20 smithi040 ceph-mon[100251]: pgmap v47: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-11-16T18:26:20.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e15b93700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f23d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51180/0 (socket says 172.21.15.2:51180) 2023-11-16T18:26:20.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e15b93700 1 -- 172.21.15.2:0/911865883 learned_addr learned my addr 172.21.15.2:0/911865883 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:20.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e16b95700 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f2308b0 0x55a02f296b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e16394700 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f22bb60 0x55a02f23cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e15b93700 1 -- 172.21.15.2:0/911865883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f2308b0 msgr2=0x55a02f296b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e15b93700 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f2308b0 0x55a02f296b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e15b93700 1 -- 172.21.15.2:0/911865883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f22bb60 msgr2=0x55a02f23cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.470+0000 7fe92fb5cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a700690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e15b93700 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f22bb60 0x55a02f23cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.470+0000 7f8e15b93700 1 -- 172.21.15.2:0/911865883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a02effe630 con 0x55a02f22c470 2023-11-16T18:26:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe92fb5cb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 0x555e0a6f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.471+0000 7f8e16b95700 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f2308b0 0x55a02f296b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe92fb5cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a6fdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe92fb5cb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0a6f0b70 con 0x555e0a701720 2023-11-16T18:26:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe92fb5cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0a71fe40 con 0x555e0a701b90 2023-11-16T18:26:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe92fb5cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0a7032c0 con 0x555e0a5576b0 2023-11-16T18:26:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe920f6b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 0x555e0a6f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe92176c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a700690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe920f6b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 0x555e0a6f5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51188/0 (socket says 172.21.15.2:51188) 2023-11-16T18:26:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe92176c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a700690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:42714/0 (socket says 172.21.15.2:42714) 2023-11-16T18:26:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe920f6b700 1 -- 172.21.15.2:0/1485415734 learned_addr learned my addr 172.21.15.2:0/1485415734 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:20.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.471+0000 7f8e15b93700 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f23d460 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009d40 tx=0x7f8e00003fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92675db3b6d693c0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe921f6d700 1 --2- 172.21.15.2:0/1485415734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a6fdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.471+0000 7f8df9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:20.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.471+0000 7f8df9ffb700 1 -- 172.21.15.2:0/911865883 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00022030 con 0x55a02f22c470 2023-11-16T18:26:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe920f6b700 1 -- 172.21.15.2:0/1485415734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 msgr2=0x555e0a700690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe920f6b700 1 --2- 172.21.15.2:0/1485415734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a700690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.471+0000 7f8e24784b80 1 -- 172.21.15.2:0/911865883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e0c09dfd0 con 0x55a02f22c470 2023-11-16T18:26:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe920f6b700 1 -- 172.21.15.2:0/1485415734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 msgr2=0x555e0a6fdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.471+0000 7f8df9ffb700 1 -- 172.21.15.2:0/911865883 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f8e00023070 con 0x55a02f22c470 2023-11-16T18:26:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe920f6b700 1 --2- 172.21.15.2:0/1485415734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a6fdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe920f6b700 1 -- 172.21.15.2:0/1485415734 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e0a6f26f0 con 0x555e0a701720 2023-11-16T18:26:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe921f6d700 1 --2- 172.21.15.2:0/1485415734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a6fdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:20.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.471+0000 7f8df9ffb700 1 -- 172.21.15.2:0/911865883 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e00011ab0 con 0x55a02f22c470 2023-11-16T18:26:20.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.471+0000 7f8e24784b80 10 client.16017 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:20.545 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.471+0000 7f8e24784b80 1 -- 172.21.15.2:0/911865883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a02ef93c60 con 0x55a02f22c470 2023-11-16T18:26:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.471+0000 7fe920f6b700 1 --2- 172.21.15.2:0/1485415734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 0x555e0a6f5f60 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0969d0 tx=0x7fe90c0963d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75ffc177fc19e2b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:20.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.472+0000 7f8df9ffb700 1 -- 172.21.15.2:0/911865883 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e0c09dfd0 con 0x55a02f22c470 2023-11-16T18:26:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.472+0000 7fe907fff700 1 -- 172.21.15.2:0/1485415734 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c0cac00 con 0x555e0a701720 2023-11-16T18:26:20.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.472+0000 7f8df9ffb700 1 -- 172.21.15.2:0/911865883 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8e00027970 con 0x55a02f22c470 2023-11-16T18:26:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.472+0000 7fe907fff700 1 -- 172.21.15.2:0/1485415734 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe90c0998c0 con 0x555e0a701720 2023-11-16T18:26:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.472+0000 7fe907fff700 1 -- 172.21.15.2:0/1485415734 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c098ec0 con 0x555e0a701720 2023-11-16T18:26:20.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.472+0000 7f8e24784b80 10 client.16017 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:20.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.472+0000 7f8e24784b80 10 client.16017 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2023-11-16T18:26:20.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.472+0000 7f8e24784b80 20 client.16017 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:20.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.472+0000 7f8e24784b80 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a02f29eb40 0x55a02f2a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.472+0000 7f8e24784b80 4 client.16017 mds_command: new command op to 24446 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:26:20.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.472+0000 7f8e24784b80 1 -- 172.21.15.2:0/911865883 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a02f14b660 con 0x55a02f29eb40 2023-11-16T18:26:20.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.472+0000 7f8e16394700 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a02f29eb40 0x55a02f2a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8df9ffb700 10 client.16017 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:26:20.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8e16394700 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a02f29eb40 0x55a02f2a0f50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=99eae500b1181eb7 server_cookie=ab325d25423b987d in_seq=0 out_seq=0 2023-11-16T18:26:20.551 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:20 smithi188 ceph-mon[100639]: pgmap v47: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-11-16T18:26:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.473+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/1485415734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 msgr2=0x555e0a6f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.473+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/1485415734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 0x555e0a6f5f60 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0969d0 tx=0x7fe90c0963d0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.473+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/1485415734 shutdown_connections 2023-11-16T18:26:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.473+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/1485415734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a5576b0 0x555e0a700690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.473+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/1485415734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a6fdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.473+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/1485415734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a701720 0x555e0a6f5f60 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.473+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/1485415734 >> 172.21.15.2:0/1485415734 conn(0x555e0a6f0f50 msgr2=0x555e0a70de80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.473+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/1485415734 shutdown_connections 2023-11-16T18:26:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.473+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/1485415734 wait complete. 2023-11-16T18:26:20.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8df9ffb700 1 -- 172.21.15.2:0/911865883 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a02f14b660 con 0x55a02f29eb40 2023-11-16T18:26:20.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8df9ffb700 10 client.16017 handle_command_reply: tid=0 2023-11-16T18:26:20.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8e24784b80 1 client.16017 shutdown 2023-11-16T18:26:20.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8e24784b80 1 -- 172.21.15.2:0/911865883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 msgr2=0x55a02f23d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8e24784b80 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f23d460 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009d40 tx=0x7f8e00003fd0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8e24784b80 1 -- 172.21.15.2:0/911865883 shutdown_connections 2023-11-16T18:26:20.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8e24784b80 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55a02f29eb40 0x55a02f2a0f50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8e24784b80 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a02f2308b0 0x55a02f296b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8e24784b80 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02f22bb60 0x55a02f23cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8e24784b80 1 --2- 172.21.15.2:0/911865883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a02f22c470 0x55a02f23d460 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8e24784b80 1 -- 172.21.15.2:0/911865883 >> 172.21.15.2:0/911865883 conn(0x55a02f221360 msgr2=0x55a02f22a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:20.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8df9ffb700 0 client.16017 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:26:20.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8e24784b80 1 -- 172.21.15.2:0/911865883 shutdown_connections 2023-11-16T18:26:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.473+0000 7fe92fb5cb80 1 -- start start 2023-11-16T18:26:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.473+0000 7fe92fb5cb80 1 Processor -- start 2023-11-16T18:26:20.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8e24784b80 1 -- 172.21.15.2:0/911865883 wait complete. 2023-11-16T18:26:20.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.473+0000 7f8e24784b80 20 client.16017 trim_cache size 0 max 16384 2023-11-16T18:26:20.578 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:26:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92fb5cb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a5576b0 0x555e0a710f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92fb5cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a701720 0x555e0a711490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92fb5cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a711a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92fb5cb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0a770450 con 0x555e0a5576b0 2023-11-16T18:26:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92fb5cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0a7705c0 con 0x555e0a701b90 2023-11-16T18:26:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92fb5cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0a770780 con 0x555e0a701720 2023-11-16T18:26:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92fb5cb80 10 client.0 _ll_register_callbacks cb 0x7fe90c005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:20.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.474+0000 7f8e185f8700 1 -- 172.21.15.2:0/1662722334 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f8dfc0a1460 msgr2=0x7f8dfc0a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.474+0000 7f8e185f8700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f8dfc0a1460 0x7f8dfc0a3910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8e10197bf0 tx=0x7f8e00009310 comp rx=0 tx=0).stop 2023-11-16T18:26:20.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.474+0000 7f8e185f8700 1 -- 172.21.15.2:0/1662722334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100fea80 msgr2=0x7f8e101965e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.474+0000 7f8e185f8700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100fea80 0x7f8e101965e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00e9a0 tx=0x7f8e0c00ecb0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92176c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a5576b0 0x555e0a710f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92176c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a5576b0 0x555e0a710f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51192/0 (socket says 172.21.15.2:51192) 2023-11-16T18:26:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92176c700 1 -- 172.21.15.2:0/3659671337 learned_addr learned my addr 172.21.15.2:0/3659671337 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe920f6b700 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a701720 0x555e0a711490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.474+0000 7f8e185f8700 1 -- 172.21.15.2:0/1662722334 shutdown_connections 2023-11-16T18:26:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.474+0000 7f8e185f8700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f8dfc0a1460 0x7f8dfc0a3910 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.474+0000 7f8e185f8700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e101055d0 0x7f8e1019af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.474+0000 7f8e185f8700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100fea80 0x7f8e101965e0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00e9a0 tx=0x7f8e0c00ecb0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.474+0000 7f8e185f8700 1 --2- 172.21.15.2:0/1662722334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e100ff420 0x7f8e10196b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.474+0000 7f8e185f8700 1 -- 172.21.15.2:0/1662722334 >> 172.21.15.2:0/1662722334 conn(0x7f8e10075810 msgr2=0x7f8e10103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe921f6d700 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a711a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92176c700 1 -- 172.21.15.2:0/3659671337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a701720 msgr2=0x555e0a711490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92176c700 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a701720 0x555e0a711490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92176c700 1 -- 172.21.15.2:0/3659671337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 msgr2=0x555e0a711a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92176c700 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a711a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.474+0000 7f8e185f8700 1 -- 172.21.15.2:0/1662722334 shutdown_connections 2023-11-16T18:26:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe92176c700 1 -- 172.21.15.2:0/3659671337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e0a613130 con 0x555e0a5576b0 2023-11-16T18:26:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe921f6d700 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a711a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:20.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:20.474+0000 7f8e185f8700 1 -- 172.21.15.2:0/1662722334 wait complete. 2023-11-16T18:26:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.474+0000 7fe920f6b700 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a701720 0x555e0a711490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.475+0000 7fe92176c700 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a5576b0 0x555e0a710f50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x555e0a720080 tx=0x7fe9180047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63cdb6181da6998e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.475+0000 7fe904ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.475+0000 7fe904ff9700 1 -- 172.21.15.2:0/3659671337 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe918004040 con 0x555e0a5576b0 2023-11-16T18:26:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.475+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/3659671337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe90c0a23b0 con 0x555e0a5576b0 2023-11-16T18:26:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.475+0000 7fe904ff9700 1 -- 172.21.15.2:0/3659671337 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe918030070 con 0x555e0a5576b0 2023-11-16T18:26:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.475+0000 7fe904ff9700 1 -- 172.21.15.2:0/3659671337 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9180041a0 con 0x555e0a5576b0 2023-11-16T18:26:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.475+0000 7fe92fb5cb80 10 client.16023 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.475+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/3659671337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x555e0a47ad80 con 0x555e0a5576b0 2023-11-16T18:26:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.475+0000 7fe904ff9700 1 -- 172.21.15.2:0/3659671337 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe90c0a23b0 con 0x555e0a5576b0 2023-11-16T18:26:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.476+0000 7fe904ff9700 1 -- 172.21.15.2:0/3659671337 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe91801d410 con 0x555e0a5576b0 2023-11-16T18:26:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.476+0000 7fe92fb5cb80 10 client.16023 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.476+0000 7fe92fb5cb80 10 client.16023 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.476+0000 7fe92fb5cb80 20 client.16023 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.476+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555e0a771ed0 0x555e0a7742e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.476+0000 7fe92fb5cb80 4 client.16023 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:26:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.476+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/3659671337 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555e0a61b600 con 0x555e0a771ed0 2023-11-16T18:26:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.476+0000 7fe920f6b700 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555e0a771ed0 0x555e0a7742e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.476+0000 7fe920f6b700 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555e0a771ed0 0x555e0a7742e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef75636649d2042 server_cookie=8f6ce40a2e2d678d in_seq=0 out_seq=0 2023-11-16T18:26:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.476+0000 7fe904ff9700 10 client.16023 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe904ff9700 1 -- 172.21.15.2:0/3659671337 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555e0a61b600 con 0x555e0a771ed0 2023-11-16T18:26:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe904ff9700 10 client.16023 handle_command_reply: tid=0 2023-11-16T18:26:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe92fb5cb80 1 client.16023 shutdown 2023-11-16T18:26:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/3659671337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a5576b0 msgr2=0x555e0a710f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a5576b0 0x555e0a710f50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x555e0a720080 tx=0x7fe9180047c0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/3659671337 shutdown_connections 2023-11-16T18:26:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555e0a771ed0 0x555e0a7742e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e0a701b90 0x555e0a711a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe904ff9700 0 client.16023 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555e0a5576b0 0x555e0a710f50 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe92fb5cb80 1 --2- 172.21.15.2:0/3659671337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555e0a701720 0x555e0a711490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/3659671337 >> 172.21.15.2:0/3659671337 conn(0x555e0a6f0f50 msgr2=0x555e0a703510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/3659671337 shutdown_connections 2023-11-16T18:26:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe92fb5cb80 1 -- 172.21.15.2:0/3659671337 wait complete. 2023-11-16T18:26:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.511+0000 7fe92fb5cb80 20 client.16023 trim_cache size 0 max 16384 2023-11-16T18:26:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:26:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:26:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:26:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:26:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.512+0000 7fe9239d0700 1 -- 172.21.15.2:0/3767241010 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe9080a13e0 msgr2=0x7fe9080a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.512+0000 7fe9239d0700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe9080a13e0 0x7fe9080a3890 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe918009ef0 tx=0x7fe918009260 comp rx=0 tx=0).stop 2023-11-16T18:26:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.512+0000 7fe9239d0700 1 -- 172.21.15.2:0/3767241010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe91c101d50 msgr2=0x7fe91c196960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.512+0000 7fe9239d0700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe91c101d50 0x7fe91c196960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00e9f0 tx=0x7fe90c00ed00 comp rx=0 tx=0).stop 2023-11-16T18:26:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.512+0000 7fe9239d0700 1 -- 172.21.15.2:0/3767241010 shutdown_connections 2023-11-16T18:26:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.512+0000 7fe9239d0700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe9080a13e0 0x7fe9080a3890 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.512+0000 7fe9239d0700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe91c10a810 0x7fe91c19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.512+0000 7fe9239d0700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe91c101440 0x7fe91c196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.512+0000 7fe9239d0700 1 --2- 172.21.15.2:0/3767241010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe91c101d50 0x7fe91c196960 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.512+0000 7fe9239d0700 1 -- 172.21.15.2:0/3767241010 >> 172.21.15.2:0/3767241010 conn(0x7fe91c0763a0 msgr2=0x7fe91c0fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.512+0000 7fe9239d0700 1 -- 172.21.15.2:0/3767241010 shutdown_connections 2023-11-16T18:26:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:20.512+0000 7fe9239d0700 1 -- 172.21.15.2:0/3767241010 wait complete. 2023-11-16T18:26:20.605 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:20.605 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:20.606 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-11-16T18:26:20.637 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:20.637 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:20.656 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:20.656 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:20.656 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 54 -lt 100 ']' 2023-11-16T18:26:20.658 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 54 + 1 2023-11-16T18:26:20.682 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=55 2023-11-16T18:26:20.682 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:20.706 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:20.706 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:20.706 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:20.707 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:20.707 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:21.169 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:21.169 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:21.275 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:21.275 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:21.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:21.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:21.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 55 -lt 100 ']' 2023-11-16T18:26:21.298 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 55 + 1 2023-11-16T18:26:21.340 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=56 2023-11-16T18:26:21.340 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:21.398 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:21.399 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:21.399 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:21.399 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:21.399 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:21.919 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:21.919 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:22.073 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:22.074 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:22.096 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:22.097 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:22.097 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 56 -lt 100 ']' 2023-11-16T18:26:22.097 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 56 + 1 2023-11-16T18:26:22.140 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=57 2023-11-16T18:26:22.140 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:22.189 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:22.189 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:22.190 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:22.190 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:22.190 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:22.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:22 smithi002 ceph-mon[98807]: pgmap v48: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-11-16T18:26:22.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:22 smithi040 ceph-mon[100251]: pgmap v48: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-11-16T18:26:22.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:22 smithi188 ceph-mon[100639]: pgmap v48: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-11-16T18:26:22.760 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:22.760 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:23.007 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:23.007 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:23.057 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:23.057 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:23.057 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 57 -lt 100 ']' 2023-11-16T18:26:23.057 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 57 + 1 2023-11-16T18:26:23.090 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=58 2023-11-16T18:26:23.090 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:23.119 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:23.119 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:23.120 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:23.120 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:23.120 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:23.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:23 smithi188 ceph-mon[100639]: mgrmap e25: x(active, since 91s), standbys: z, y 2023-11-16T18:26:23.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:23 smithi188 ceph-mon[100639]: pgmap v49: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 17 op/s 2023-11-16T18:26:23.556 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.553+0000 7fdd67c90700 1 -- 172.21.15.2:0/2421479354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd601002a0 msgr2=0x7fdd6010bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:23.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.553+0000 7fdd67c90700 1 --2- 172.21.15.2:0/2421479354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd601002a0 0x7fdd6010bfd0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800b0d0 tx=0x7fdd5800b3e0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.553+0000 7fdd67c90700 1 -- 172.21.15.2:0/2421479354 shutdown_connections 2023-11-16T18:26:23.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.553+0000 7fdd67c90700 1 --2- 172.21.15.2:0/2421479354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd60104140 0x7fdd60104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.553+0000 7fdd67c90700 1 --2- 172.21.15.2:0/2421479354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd601002a0 0x7fdd6010bfd0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.553+0000 7fdd67c90700 1 --2- 172.21.15.2:0/2421479354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd600ff8f0 0x7fdd600ffd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.553+0000 7fdd67c90700 1 -- 172.21.15.2:0/2421479354 >> 172.21.15.2:0/2421479354 conn(0x7fdd60075880 msgr2=0x7fdd60075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:23.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.553+0000 7fdd67c90700 1 -- 172.21.15.2:0/2421479354 shutdown_connections 2023-11-16T18:26:23.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.553+0000 7fdd67c90700 1 -- 172.21.15.2:0/2421479354 wait complete. 2023-11-16T18:26:23.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd67c90700 1 Processor -- start 2023-11-16T18:26:23.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd67c90700 1 -- start start 2023-11-16T18:26:23.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd67c90700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd600ff8f0 0x7fdd60196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd67c90700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd601002a0 0x7fdd60196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd67c90700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd60104140 0x7fdd6019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd67c90700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6019b420 con 0x7fdd601002a0 2023-11-16T18:26:23.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd67c90700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6019b590 con 0x7fdd60104140 2023-11-16T18:26:23.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd67c90700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6019b750 con 0x7fdd600ff8f0 2023-11-16T18:26:23.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd6522b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd601002a0 0x7fdd60196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd6522b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd601002a0 0x7fdd60196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51208/0 (socket says 172.21.15.2:51208) 2023-11-16T18:26:23.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd6522b700 1 -- 172.21.15.2:0/2579074089 learned_addr learned my addr 172.21.15.2:0/2579074089 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:23.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd65a2c700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd600ff8f0 0x7fdd60196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd6522b700 1 -- 172.21.15.2:0/2579074089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd600ff8f0 msgr2=0x7fdd60196440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:23.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd6522b700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd600ff8f0 0x7fdd60196440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd6622d700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd60104140 0x7fdd6019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd6522b700 1 -- 172.21.15.2:0/2579074089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd60104140 msgr2=0x7fdd6019ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:23.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd6522b700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd60104140 0x7fdd6019ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd6522b700 1 -- 172.21.15.2:0/2579074089 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd58009d00 con 0x7fdd601002a0 2023-11-16T18:26:23.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd65a2c700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd600ff8f0 0x7fdd60196440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:23.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.554+0000 7fdd6622d700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd60104140 0x7fdd6019ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:26:23.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.555+0000 7fdd6522b700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd601002a0 0x7fdd60196980 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00c920 tx=0x7fdd5c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:23.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.555+0000 7fdd52ffd700 1 -- 172.21.15.2:0/2579074089 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c007b00 con 0x7fdd601002a0 2023-11-16T18:26:23.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.555+0000 7fdd52ffd700 1 -- 172.21.15.2:0/2579074089 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd5c004cf0 con 0x7fdd601002a0 2023-11-16T18:26:23.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.555+0000 7fdd67c90700 1 -- 172.21.15.2:0/2579074089 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6010f1d0 con 0x7fdd601002a0 2023-11-16T18:26:23.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.555+0000 7fdd52ffd700 1 -- 172.21.15.2:0/2579074089 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c005610 con 0x7fdd601002a0 2023-11-16T18:26:23.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.555+0000 7fdd67c90700 1 -- 172.21.15.2:0/2579074089 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6010f040 con 0x7fdd601002a0 2023-11-16T18:26:23.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.557+0000 7fdd52ffd700 1 -- 172.21.15.2:0/2579074089 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fdd5c004750 con 0x7fdd601002a0 2023-11-16T18:26:23.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.557+0000 7fdd52ffd700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdd4c0a1340 0x7fdd4c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.558+0000 7fdd65a2c700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdd4c0a1340 0x7fdd4c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.558+0000 7fdd52ffd700 1 -- 172.21.15.2:0/2579074089 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fdd5c0ccfc0 con 0x7fdd601002a0 2023-11-16T18:26:23.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.558+0000 7fdd65a2c700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdd4c0a1340 0x7fdd4c0a37f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdd54005e10 tx=0x7fdd54005d60 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:23.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:23 smithi002 ceph-mon[98807]: mgrmap e25: x(active, since 91s), standbys: z, y 2023-11-16T18:26:23.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:23 smithi002 ceph-mon[98807]: pgmap v49: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 17 op/s 2023-11-16T18:26:23.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.615+0000 7fdd73e1cb80 1 Processor -- start 2023-11-16T18:26:23.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.615+0000 7fdd73e1cb80 1 -- start start 2023-11-16T18:26:23.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.615+0000 7fdd73e1cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa2bb60 0x556f5aa2bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.615+0000 7fdd73e1cb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2c470 0x556f5aa30370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.615+0000 7fdd73e1cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa308b0 0x556f5aa30d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.615+0000 7fdd73e1cb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5a931060 con 0x556f5aa2c470 2023-11-16T18:26:23.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.615+0000 7fdd73e1cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5a9838a0 con 0x556f5aa2bb60 2023-11-16T18:26:23.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.615+0000 7fdd73e1cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5a97cd30 con 0x556f5aa308b0 2023-11-16T18:26:23.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.615+0000 7fdd6522b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2c470 0x556f5aa30370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.615+0000 7fdd6522b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2c470 0x556f5aa30370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51216/0 (socket says 172.21.15.2:51216) 2023-11-16T18:26:23.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.615+0000 7fdd6522b700 1 -- 172.21.15.2:0/3027711674 learned_addr learned my addr 172.21.15.2:0/3027711674 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:23.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.615+0000 7fdd65a2c700 1 --2- 172.21.15.2:0/3027711674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa2bb60 0x556f5aa2bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.616+0000 7fdd6622d700 1 --2- 172.21.15.2:0/3027711674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa308b0 0x556f5aa30d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.616+0000 7fdd6522b700 1 -- 172.21.15.2:0/3027711674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa308b0 msgr2=0x556f5aa30d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:23.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.616+0000 7fdd6522b700 1 --2- 172.21.15.2:0/3027711674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa308b0 0x556f5aa30d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.616+0000 7fdd6522b700 1 -- 172.21.15.2:0/3027711674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa2bb60 msgr2=0x556f5aa2bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:23.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.616+0000 7fdd6522b700 1 --2- 172.21.15.2:0/3027711674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa2bb60 0x556f5aa2bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.616+0000 7fdd6522b700 1 -- 172.21.15.2:0/3027711674 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f5a7fe630 con 0x556f5aa2c470 2023-11-16T18:26:23.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.616+0000 7fdd6522b700 1 --2- 172.21.15.2:0/3027711674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2c470 0x556f5aa30370 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c096790 tx=0x7fdd5c099680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b544a44085ea157 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:23.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.616+0000 7fdd4bfff700 1 -- 172.21.15.2:0/3027711674 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c0a2170 con 0x556f5aa2c470 2023-11-16T18:26:23.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.616+0000 7fdd4bfff700 1 -- 172.21.15.2:0/3027711674 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd5c097e60 con 0x556f5aa2c470 2023-11-16T18:26:23.619 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.616+0000 7fdd4bfff700 1 -- 172.21.15.2:0/3027711674 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c004b60 con 0x556f5aa2c470 2023-11-16T18:26:23.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.617+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/3027711674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2c470 msgr2=0x556f5aa30370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:23.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.617+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/3027711674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2c470 0x556f5aa30370 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c096790 tx=0x7fdd5c099680 comp rx=0 tx=0).stop 2023-11-16T18:26:23.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.617+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/3027711674 shutdown_connections 2023-11-16T18:26:23.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.617+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/3027711674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa308b0 0x556f5aa30d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.617+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/3027711674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa2bb60 0x556f5aa2bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.617+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/3027711674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2c470 0x556f5aa30370 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.617+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/3027711674 >> 172.21.15.2:0/3027711674 conn(0x556f5aa21360 msgr2=0x556f5aa26b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:23.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.617+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/3027711674 shutdown_connections 2023-11-16T18:26:23.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.617+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/3027711674 wait complete. 2023-11-16T18:26:23.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.617+0000 7fdd73e1cb80 1 -- start start 2023-11-16T18:26:23.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd73e1cb80 1 Processor -- start 2023-11-16T18:26:23.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd73e1cb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 0x556f5aa35380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd73e1cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 0x556f5aa358c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd73e1cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa308b0 0x556f5aa35e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd73e1cb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5a91e8e0 con 0x556f5aa2bb60 2023-11-16T18:26:23.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd73e1cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5a91e580 con 0x556f5aa308b0 2023-11-16T18:26:23.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd73e1cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5a800920 con 0x556f5aa2c470 2023-11-16T18:26:23.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd73e1cb80 10 client.0 _ll_register_callbacks cb 0x556f5a93c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:23.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd65a2c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 0x556f5aa35380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd65a2c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 0x556f5aa35380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51220/0 (socket says 172.21.15.2:51220) 2023-11-16T18:26:23.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd65a2c700 1 -- 172.21.15.2:0/1315019299 learned_addr learned my addr 172.21.15.2:0/1315019299 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:23.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd6522b700 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 0x556f5aa358c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd6622d700 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa308b0 0x556f5aa35e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd65a2c700 1 -- 172.21.15.2:0/1315019299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 msgr2=0x556f5aa358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:23.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd65a2c700 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 0x556f5aa358c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd65a2c700 1 -- 172.21.15.2:0/1315019299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa308b0 msgr2=0x556f5aa35e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:23.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd65a2c700 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa308b0 0x556f5aa35e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd65a2c700 1 -- 172.21.15.2:0/1315019299 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6010f040 con 0x556f5aa2bb60 2023-11-16T18:26:23.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.618+0000 7fdd6622d700 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa308b0 0x556f5aa35e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:26:23.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.619+0000 7fdd65a2c700 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 0x556f5aa35380 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400fb80 tx=0x7fdd54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=981615c9438ad04b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:23.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.619+0000 7fdd497fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:23.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.619+0000 7fdd497fa700 1 -- 172.21.15.2:0/1315019299 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54017040 con 0x556f5aa2bb60 2023-11-16T18:26:23.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.619+0000 7fdd497fa700 1 -- 172.21.15.2:0/1315019299 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fdd54003a90 con 0x556f5aa2bb60 2023-11-16T18:26:23.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.619+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/1315019299 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556f5aa1eb20 con 0x556f5aa2bb60 2023-11-16T18:26:23.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.619+0000 7fdd497fa700 1 -- 172.21.15.2:0/1315019299 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd54003c90 con 0x556f5aa2bb60 2023-11-16T18:26:23.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.619+0000 7fdd73e1cb80 10 client.16035 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:23.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.619+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/1315019299 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556f5a943190 con 0x556f5aa2bb60 2023-11-16T18:26:23.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.619+0000 7fdd497fa700 1 -- 172.21.15.2:0/1315019299 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556f5aa1eb20 con 0x556f5aa2bb60 2023-11-16T18:26:23.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.619+0000 7fdd497fa700 1 -- 172.21.15.2:0/1315019299 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdd540204a0 con 0x556f5aa2bb60 2023-11-16T18:26:23.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.619+0000 7fdd73e1cb80 10 client.16035 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:23.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.619+0000 7fdd73e1cb80 10 client.16035 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-11-16T18:26:23.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.620+0000 7fdd73e1cb80 20 client.16035 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:23.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.620+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x556f5aa45940 0x556f5aa47d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.620+0000 7fdd73e1cb80 4 client.16035 mds_command: new command op to 24460 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:23.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.620+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/1315019299 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556f5a93f0a0 con 0x556f5aa45940 2023-11-16T18:26:23.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.620+0000 7fdd6522b700 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x556f5aa45940 0x556f5aa47d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.620+0000 7fdd6522b700 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x556f5aa45940 0x556f5aa47d50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1990932f5f1c361b server_cookie=55f179fd693b5a88 in_seq=0 out_seq=0 2023-11-16T18:26:23.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.620+0000 7fdd497fa700 10 client.16035 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:26:23.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd497fa700 1 -- 172.21.15.2:0/1315019299 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x556f5a93f0a0 con 0x556f5aa45940 2023-11-16T18:26:23.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd497fa700 10 client.16035 handle_command_reply: tid=0 2023-11-16T18:26:23.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd73e1cb80 1 client.16035 shutdown 2023-11-16T18:26:23.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/1315019299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 msgr2=0x556f5aa35380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:23.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 0x556f5aa35380 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400fb80 tx=0x7fdd54004750 comp rx=0 tx=0).stop 2023-11-16T18:26:23.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/1315019299 shutdown_connections 2023-11-16T18:26:23.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x556f5aa45940 0x556f5aa47d50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa308b0 0x556f5aa35e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 0x556f5aa35380 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd497fa700 0 client.16035 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:26:23.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/1315019299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 0x556f5aa358c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/1315019299 >> 172.21.15.2:0/1315019299 conn(0x556f5aa21360 msgr2=0x556f5aa25de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:23.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/1315019299 shutdown_connections 2023-11-16T18:26:23.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/1315019299 wait complete. 2023-11-16T18:26:23.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.621+0000 7fdd73e1cb80 20 client.16035 trim_cache size 0 max 16384 2023-11-16T18:26:23.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.635+0000 7fdd73e1cb80 1 Processor -- start 2023-11-16T18:26:23.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.635+0000 7fdd73e1cb80 1 -- start start 2023-11-16T18:26:23.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.635+0000 7fdd73e1cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa2bb60 0x556f5aa3c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.635+0000 7fdd73e1cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 0x556f5aa35c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.635+0000 7fdd73e1cb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa308b0 0x556f5aa32dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.635+0000 7fdd73e1cb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5aa281c0 con 0x556f5aa308b0 2023-11-16T18:26:23.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.635+0000 7fdd73e1cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5aa33900 con 0x556f5aa2bb60 2023-11-16T18:26:23.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.635+0000 7fdd73e1cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5aa2f4b0 con 0x556f5aa2c470 2023-11-16T18:26:23.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.635+0000 7fdd6622d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa308b0 0x556f5aa32dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.635+0000 7fdd6622d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa308b0 0x556f5aa32dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51232/0 (socket says 172.21.15.2:51232) 2023-11-16T18:26:23.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.635+0000 7fdd6622d700 1 -- 172.21.15.2:0/2771784103 learned_addr learned my addr 172.21.15.2:0/2771784103 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:23.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.635+0000 7fdd6522b700 1 --2- 172.21.15.2:0/2771784103 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 0x556f5aa35c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.636+0000 7fdd65a2c700 1 --2- 172.21.15.2:0/2771784103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa2bb60 0x556f5aa3c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.636+0000 7fdd6622d700 1 -- 172.21.15.2:0/2771784103 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 msgr2=0x556f5aa35c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:23.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.636+0000 7fdd6622d700 1 --2- 172.21.15.2:0/2771784103 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 0x556f5aa35c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.636+0000 7fdd6622d700 1 -- 172.21.15.2:0/2771784103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa2bb60 msgr2=0x556f5aa3c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:23.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.636+0000 7fdd6622d700 1 --2- 172.21.15.2:0/2771784103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa2bb60 0x556f5aa3c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.636+0000 7fdd6622d700 1 -- 172.21.15.2:0/2771784103 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f5aa22b00 con 0x556f5aa308b0 2023-11-16T18:26:23.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.636+0000 7fdd6622d700 1 --2- 172.21.15.2:0/2771784103 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa308b0 0x556f5aa32dc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fdd58009c50 tx=0x7fdd58009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa767581e462399f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:23.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.636+0000 7fdd50dac700 1 -- 172.21.15.2:0/2771784103 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5800e070 con 0x556f5aa308b0 2023-11-16T18:26:23.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.636+0000 7fdd50dac700 1 -- 172.21.15.2:0/2771784103 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd58009400 con 0x556f5aa308b0 2023-11-16T18:26:23.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.636+0000 7fdd50dac700 1 -- 172.21.15.2:0/2771784103 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd58025980 con 0x556f5aa308b0 2023-11-16T18:26:23.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.637+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/2771784103 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa308b0 msgr2=0x556f5aa32dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:23.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.637+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/2771784103 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa308b0 0x556f5aa32dc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fdd58009c50 tx=0x7fdd58009c80 comp rx=0 tx=0).stop 2023-11-16T18:26:23.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.637+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/2771784103 shutdown_connections 2023-11-16T18:26:23.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.637+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/2771784103 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa308b0 0x556f5aa32dc0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.637+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/2771784103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa2bb60 0x556f5aa3c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.637+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/2771784103 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 0x556f5aa35c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.637+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/2771784103 >> 172.21.15.2:0/2771784103 conn(0x556f5aa21360 msgr2=0x556f5aa2e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:23.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.637+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/2771784103 shutdown_connections 2023-11-16T18:26:23.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.637+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/2771784103 wait complete. 2023-11-16T18:26:23.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.639+0000 7fdd73e1cb80 1 -- start start 2023-11-16T18:26:23.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.639+0000 7fdd73e1cb80 1 Processor -- start 2023-11-16T18:26:23.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.640+0000 7fdd73e1cb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 0x556f5aa38980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.640+0000 7fdd73e1cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 0x556f5aa38ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.640+0000 7fdd73e1cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa308b0 0x556f5aa987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.640+0000 7fdd73e1cb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5aa397d0 con 0x556f5aa2bb60 2023-11-16T18:26:23.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.640+0000 7fdd73e1cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5aa98cf0 con 0x556f5aa308b0 2023-11-16T18:26:23.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.640+0000 7fdd73e1cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5aa98e30 con 0x556f5aa2c470 2023-11-16T18:26:23.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.640+0000 7fdd73e1cb80 10 client.0 _ll_register_callbacks cb 0x556f5a93c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:23.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.640+0000 7fdd65a2c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 0x556f5aa38980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.640+0000 7fdd65a2c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 0x556f5aa38980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51234/0 (socket says 172.21.15.2:51234) 2023-11-16T18:26:23.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.640+0000 7fdd65a2c700 1 -- 172.21.15.2:0/3542015384 learned_addr learned my addr 172.21.15.2:0/3542015384 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:23.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.641+0000 7fdd6622d700 1 --2- 172.21.15.2:0/3542015384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa308b0 0x556f5aa987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.641+0000 7fdd65a2c700 1 -- 172.21.15.2:0/3542015384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 msgr2=0x556f5aa38ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T18:26:23.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.641+0000 7fdd65a2c700 1 --2- 172.21.15.2:0/3542015384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 0x556f5aa38ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.641+0000 7fdd65a2c700 1 -- 172.21.15.2:0/3542015384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa308b0 msgr2=0x556f5aa987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:23.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.641+0000 7fdd65a2c700 1 --2- 172.21.15.2:0/3542015384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa308b0 0x556f5aa987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.641+0000 7fdd65a2c700 1 -- 172.21.15.2:0/3542015384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6010c860 con 0x556f5aa2bb60 2023-11-16T18:26:23.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.642+0000 7fdd65a2c700 1 --2- 172.21.15.2:0/3542015384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 0x556f5aa38980 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400fb50 tx=0x7fdd54021fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=724dd06b5c26ff11 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:23.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.642+0000 7fdd497fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:23.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.642+0000 7fdd497fa700 1 -- 172.21.15.2:0/3542015384 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd540105d0 con 0x556f5aa2bb60 2023-11-16T18:26:23.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.642+0000 7fdd497fa700 1 -- 172.21.15.2:0/3542015384 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fdd54020620 con 0x556f5aa2bb60 2023-11-16T18:26:23.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.642+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/3542015384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd5c097e60 con 0x556f5aa2bb60 2023-11-16T18:26:23.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.643+0000 7fdd497fa700 1 -- 172.21.15.2:0/3542015384 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd54020a00 con 0x556f5aa2bb60 2023-11-16T18:26:23.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.643+0000 7fdd73e1cb80 10 client.16041 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:23.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.643+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/3542015384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556f5a7aade0 con 0x556f5aa2bb60 2023-11-16T18:26:23.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.643+0000 7fdd497fa700 1 -- 172.21.15.2:0/3542015384 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd5c097e60 con 0x556f5aa2bb60 2023-11-16T18:26:23.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.644+0000 7fdd497fa700 1 -- 172.21.15.2:0/3542015384 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdd540054b0 con 0x556f5aa2bb60 2023-11-16T18:26:23.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.644+0000 7fdd73e1cb80 10 client.16041 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:23.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.644+0000 7fdd73e1cb80 10 client.16041 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-11-16T18:26:23.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.644+0000 7fdd73e1cb80 20 client.16041 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:23.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.644+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/3542015384 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x556f5aa9a3f0 0x556f5aa9c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:23.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.644+0000 7fdd73e1cb80 4 client.16041 mds_command: new command op to 24460 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:26:23.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.644+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/3542015384 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556f5a94b660 con 0x556f5aa9a3f0 2023-11-16T18:26:23.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.645+0000 7fdd6522b700 1 --2- 172.21.15.2:0/3542015384 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x556f5aa9a3f0 0x556f5aa9c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:23.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.645+0000 7fdd6522b700 1 --2- 172.21.15.2:0/3542015384 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x556f5aa9a3f0 0x556f5aa9c7d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4f02dc28cd7df67f server_cookie=9e3d025975d8e83a in_seq=0 out_seq=0 2023-11-16T18:26:23.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.645+0000 7fdd497fa700 10 client.16041 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:26:23.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.646+0000 7fdd497fa700 1 -- 172.21.15.2:0/3542015384 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556f5a94b660 con 0x556f5aa9a3f0 2023-11-16T18:26:23.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.646+0000 7fdd497fa700 10 client.16041 handle_command_reply: tid=0 2023-11-16T18:26:23.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.646+0000 7fdd73e1cb80 1 client.16041 shutdown 2023-11-16T18:26:23.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.646+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/3542015384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 msgr2=0x556f5aa38980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:23.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.646+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/3542015384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 0x556f5aa38980 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400fb50 tx=0x7fdd54021fa0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.646+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/3542015384 shutdown_connections 2023-11-16T18:26:23.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.646+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/3542015384 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x556f5aa9a3f0 0x556f5aa9c7d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.646+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/3542015384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f5aa308b0 0x556f5aa987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.646+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/3542015384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556f5aa2bb60 0x556f5aa38980 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.646+0000 7fdd73e1cb80 1 --2- 172.21.15.2:0/3542015384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556f5aa2c470 0x556f5aa38ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.646+0000 7fdd497fa700 0 client.16041 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:26:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.646+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/3542015384 >> 172.21.15.2:0/3542015384 conn(0x556f5aa21360 msgr2=0x556f5aa35b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.646+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/3542015384 shutdown_connections 2023-11-16T18:26:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.647+0000 7fdd73e1cb80 1 -- 172.21.15.2:0/3542015384 wait complete. 2023-11-16T18:26:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.647+0000 7fdd73e1cb80 20 client.16041 trim_cache size 0 max 16384 2023-11-16T18:26:23.662 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:26:23.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.648+0000 7fdd67c90700 1 -- 172.21.15.2:0/2579074089 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdd4c0a1340 msgr2=0x7fdd4c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:23.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.648+0000 7fdd67c90700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdd4c0a1340 0x7fdd4c0a37f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdd54005e10 tx=0x7fdd54005d60 comp rx=0 tx=0).stop 2023-11-16T18:26:23.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.648+0000 7fdd67c90700 1 -- 172.21.15.2:0/2579074089 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd601002a0 msgr2=0x7fdd60196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:23.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.648+0000 7fdd67c90700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd601002a0 0x7fdd60196980 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00c920 tx=0x7fdd5c00cce0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.648+0000 7fdd67c90700 1 -- 172.21.15.2:0/2579074089 shutdown_connections 2023-11-16T18:26:23.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.648+0000 7fdd67c90700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdd4c0a1340 0x7fdd4c0a37f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.648+0000 7fdd67c90700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd600ff8f0 0x7fdd60196440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.648+0000 7fdd67c90700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd60104140 0x7fdd6019ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.648+0000 7fdd67c90700 1 --2- 172.21.15.2:0/2579074089 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd601002a0 0x7fdd60196980 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:23.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.648+0000 7fdd67c90700 1 -- 172.21.15.2:0/2579074089 >> 172.21.15.2:0/2579074089 conn(0x7fdd60075880 msgr2=0x7fdd60102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:23.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.649+0000 7fdd67c90700 1 -- 172.21.15.2:0/2579074089 shutdown_connections 2023-11-16T18:26:23.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:23.649+0000 7fdd67c90700 1 -- 172.21.15.2:0/2579074089 wait complete. 2023-11-16T18:26:23.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:23 smithi040 ceph-mon[100251]: mgrmap e25: x(active, since 91s), standbys: z, y 2023-11-16T18:26:23.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:23 smithi040 ceph-mon[100251]: pgmap v49: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 17 op/s 2023-11-16T18:26:23.752 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:23.752 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:23.754 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T18:26:23.949 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:23.949 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:23.972 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:23.972 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:23.972 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 58 -lt 100 ']' 2023-11-16T18:26:23.972 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 58 + 1 2023-11-16T18:26:24.006 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=59 2023-11-16T18:26:24.007 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:24.016 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:24.016 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:24.017 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:24.017 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:24.017 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:24.552 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:24.552 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:24.625 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:24.625 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:24.642 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:24.642 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:24.642 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 59 -lt 100 ']' 2023-11-16T18:26:24.642 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 59 + 1 2023-11-16T18:26:24.690 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=60 2023-11-16T18:26:24.690 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:24.715 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:24.715 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:24.716 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:24.716 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:24.717 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:25.185 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:25.185 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:25.365 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:25.365 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:25.372 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:25.372 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:25.372 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 60 -lt 100 ']' 2023-11-16T18:26:25.372 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 60 + 1 2023-11-16T18:26:25.415 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=61 2023-11-16T18:26:25.415 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:25.452 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:25.452 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:25.453 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:25.453 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:25.453 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:26.243 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.674 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.676 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:26 smithi002 ceph-mon[98807]: pgmap v50: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 12 op/s 2023-11-16T18:26:33.676 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:26 smithi188 ceph-mon[100639]: pgmap v50: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 12 op/s 2023-11-16T18:26:33.677 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:27 smithi188 ceph-mon[100639]: pgmap v51: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 16 op/s 2023-11-16T18:26:33.677 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:30 smithi188 ceph-mon[100639]: pgmap v52: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 12 op/s 2023-11-16T18:26:33.677 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:32 smithi188 ceph-mon[100639]: pgmap v53: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 12 op/s 2023-11-16T18:26:33.716 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:26 smithi040 ceph-mon[100251]: pgmap v50: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 12 op/s 2023-11-16T18:26:33.716 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:27 smithi040 ceph-mon[100251]: pgmap v51: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 16 op/s 2023-11-16T18:26:33.716 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:30 smithi040 ceph-mon[100251]: pgmap v52: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 12 op/s 2023-11-16T18:26:33.716 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:32 smithi040 ceph-mon[100251]: pgmap v53: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 12 op/s 2023-11-16T18:26:33.717 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:33.717 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:33.717 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.717 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.717 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 61 -lt 100 ']' 2023-11-16T18:26:33.717 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 61 + 1 2023-11-16T18:26:33.717 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=62 2023-11-16T18:26:33.718 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:33.718 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.718 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.718 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:33.718 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:33.718 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:33.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.687+0000 7f39fe2aa700 1 -- 172.21.15.2:0/1933485137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39f8105250 msgr2=0x7f39f8105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:33.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.687+0000 7f39fe2aa700 1 --2- 172.21.15.2:0/1933485137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39f8105250 0x7f39f8105620 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f39e8009990 tx=0x7f39e8009ca0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.687+0000 7f39fe2aa700 1 -- 172.21.15.2:0/1933485137 shutdown_connections 2023-11-16T18:26:33.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.687+0000 7f39fe2aa700 1 --2- 172.21.15.2:0/1933485137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39f8068770 0x7f39f810d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.687+0000 7f39fe2aa700 1 --2- 172.21.15.2:0/1933485137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39f8105250 0x7f39f8105620 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.687+0000 7f39fe2aa700 1 --2- 172.21.15.2:0/1933485137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f8067dc0 0x7f39f8068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.687+0000 7f39fe2aa700 1 -- 172.21.15.2:0/1933485137 >> 172.21.15.2:0/1933485137 conn(0x7f39f80763a0 msgr2=0x7f39f80767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:33.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.687+0000 7f39fe2aa700 1 -- 172.21.15.2:0/1933485137 shutdown_connections 2023-11-16T18:26:33.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.687+0000 7f39fe2aa700 1 -- 172.21.15.2:0/1933485137 wait complete. 2023-11-16T18:26:33.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.688+0000 7f39fe2aa700 1 Processor -- start 2023-11-16T18:26:33.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.688+0000 7f39fe2aa700 1 -- start start 2023-11-16T18:26:33.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.688+0000 7f39fe2aa700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39f8067dc0 0x7f39f8196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.688+0000 7f39fe2aa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39f8068770 0x7f39f81969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.688+0000 7f39fe2aa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f8105250 0x7f39f819ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.688+0000 7f39fe2aa700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f819b440 con 0x7f39f8068770 2023-11-16T18:26:33.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.688+0000 7f39fe2aa700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f819b5b0 con 0x7f39f8105250 2023-11-16T18:26:33.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.688+0000 7f39fe2aa700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f819b770 con 0x7f39f8067dc0 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.688+0000 7f39effff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39f8068770 0x7f39f81969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.688+0000 7f39effff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39f8068770 0x7f39f81969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51250/0 (socket says 172.21.15.2:51250) 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.688+0000 7f39effff700 1 -- 172.21.15.2:0/3367497312 learned_addr learned my addr 172.21.15.2:0/3367497312 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.688+0000 7f39f7fff700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39f8067dc0 0x7f39f8196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.689+0000 7f39fc847700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f8105250 0x7f39f819ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.689+0000 7f39effff700 1 -- 172.21.15.2:0/3367497312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39f8067dc0 msgr2=0x7f39f8196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.689+0000 7f39effff700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39f8067dc0 0x7f39f8196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.689+0000 7f39effff700 1 -- 172.21.15.2:0/3367497312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f8105250 msgr2=0x7f39f819ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.689+0000 7f39effff700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f8105250 0x7f39f819ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.689+0000 7f39effff700 1 -- 172.21.15.2:0/3367497312 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e80095f0 con 0x7f39f8068770 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.689+0000 7f39effff700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39f8068770 0x7f39f81969a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f39e000d950 tx=0x7f39e000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.689+0000 7f39f5ffb700 1 -- 172.21.15.2:0/3367497312 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e0009940 con 0x7f39f8068770 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.689+0000 7f39f5ffb700 1 -- 172.21.15.2:0/3367497312 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39e0010460 con 0x7f39f8068770 2023-11-16T18:26:33.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.689+0000 7f39fe2aa700 1 -- 172.21.15.2:0/3367497312 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f81104a0 con 0x7f39f8068770 2023-11-16T18:26:33.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.689+0000 7f39f5ffb700 1 -- 172.21.15.2:0/3367497312 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e000f620 con 0x7f39f8068770 2023-11-16T18:26:33.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.689+0000 7f39fe2aa700 1 -- 172.21.15.2:0/3367497312 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f8110310 con 0x7f39f8068770 2023-11-16T18:26:33.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.692+0000 7f39f5ffb700 1 -- 172.21.15.2:0/3367497312 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f39e00108d0 con 0x7f39f8068770 2023-11-16T18:26:33.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.692+0000 7f39f5ffb700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f39e40a12c0 0x7f39e40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.692+0000 7f39f5ffb700 1 -- 172.21.15.2:0/3367497312 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f39e00ccf70 con 0x7f39f8068770 2023-11-16T18:26:33.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.692+0000 7f39f7fff700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f39e40a12c0 0x7f39e40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.693+0000 7f39f7fff700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f39e40a12c0 0x7f39e40a3770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f39e800b5c0 tx=0x7f39e8005fb0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:33.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.773+0000 7f3a0a436b80 1 Processor -- start 2023-11-16T18:26:33.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.773+0000 7f3a0a436b80 1 -- start start 2023-11-16T18:26:33.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.773+0000 7f3a0a436b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 0x555df2b8df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.773+0000 7f3a0a436b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 0x555df2b92370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.773+0000 7f3a0a436b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b928b0 0x555df2b92d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.773+0000 7f3a0a436b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555df2a93060 con 0x555df2b8db60 2023-11-16T18:26:33.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.773+0000 7f3a0a436b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555df2ae58a0 con 0x555df2b8e470 2023-11-16T18:26:33.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.773+0000 7f3a0a436b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555df2aded30 con 0x555df2b928b0 2023-11-16T18:26:33.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.774+0000 7f39f7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 0x555df2b8df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.774+0000 7f39f7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 0x555df2b8df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51256/0 (socket says 172.21.15.2:51256) 2023-11-16T18:26:33.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.774+0000 7f39f7fff700 1 -- 172.21.15.2:0/2460357870 learned_addr learned my addr 172.21.15.2:0/2460357870 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:33.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.774+0000 7f39fc847700 1 --2- 172.21.15.2:0/2460357870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b928b0 0x555df2b92d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.774+0000 7f39effff700 1 --2- 172.21.15.2:0/2460357870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 0x555df2b92370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.774+0000 7f39f7fff700 1 -- 172.21.15.2:0/2460357870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b928b0 msgr2=0x555df2b92d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:33.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.774+0000 7f39f7fff700 1 --2- 172.21.15.2:0/2460357870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b928b0 0x555df2b92d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.774+0000 7f39f7fff700 1 -- 172.21.15.2:0/2460357870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 msgr2=0x555df2b92370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:33.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.774+0000 7f39f7fff700 1 --2- 172.21.15.2:0/2460357870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 0x555df2b92370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.774+0000 7f39f7fff700 1 -- 172.21.15.2:0/2460357870 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555df2960630 con 0x555df2b8db60 2023-11-16T18:26:33.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.774+0000 7f39f7fff700 1 --2- 172.21.15.2:0/2460357870 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 0x555df2b8df30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f39e800b8d0 tx=0x7f39e8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5893b0952e0f2ee server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:33.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.775+0000 7f39ee7fc700 1 -- 172.21.15.2:0/2460357870 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e802b070 con 0x555df2b8db60 2023-11-16T18:26:33.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.775+0000 7f39ee7fc700 1 -- 172.21.15.2:0/2460357870 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39e801e040 con 0x555df2b8db60 2023-11-16T18:26:33.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.775+0000 7f39ee7fc700 1 -- 172.21.15.2:0/2460357870 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e800f750 con 0x555df2b8db60 2023-11-16T18:26:33.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.776+0000 7f3a0a436b80 1 -- 172.21.15.2:0/2460357870 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 msgr2=0x555df2b8df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:33.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.776+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/2460357870 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 0x555df2b8df30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f39e800b8d0 tx=0x7f39e8020000 comp rx=0 tx=0).stop 2023-11-16T18:26:33.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.776+0000 7f3a0a436b80 1 -- 172.21.15.2:0/2460357870 shutdown_connections 2023-11-16T18:26:33.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.776+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/2460357870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b928b0 0x555df2b92d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.776+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/2460357870 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 0x555df2b8df30 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.776+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/2460357870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 0x555df2b92370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.776+0000 7f3a0a436b80 1 -- 172.21.15.2:0/2460357870 >> 172.21.15.2:0/2460357870 conn(0x555df2b83360 msgr2=0x555df2b88b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:33.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.776+0000 7f3a0a436b80 1 -- 172.21.15.2:0/2460357870 shutdown_connections 2023-11-16T18:26:33.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.776+0000 7f3a0a436b80 1 -- 172.21.15.2:0/2460357870 wait complete. 2023-11-16T18:26:33.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.776+0000 7f3a0a436b80 1 -- start start 2023-11-16T18:26:33.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.777+0000 7f3a0a436b80 1 Processor -- start 2023-11-16T18:26:33.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.777+0000 7f3a0a436b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 0x555df2b9a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.777+0000 7f3a0a436b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 0x555df2b97310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.777+0000 7f3a0a436b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b928b0 0x555df2b97a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.777+0000 7f3a0a436b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555df2a808e0 con 0x555df2b8db60 2023-11-16T18:26:33.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.777+0000 7f3a0a436b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555df2a80580 con 0x555df2b8e470 2023-11-16T18:26:33.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.777+0000 7f3a0a436b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555df2962920 con 0x555df2b928b0 2023-11-16T18:26:33.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.777+0000 7f3a0a436b80 10 client.0 _ll_register_callbacks cb 0x555df2a9e780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:33.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.777+0000 7f39f7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 0x555df2b9a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.777+0000 7f39f7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 0x555df2b9a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51262/0 (socket says 172.21.15.2:51262) 2023-11-16T18:26:33.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.777+0000 7f39f7fff700 1 -- 172.21.15.2:0/3886128739 learned_addr learned my addr 172.21.15.2:0/3886128739 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:33.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.777+0000 7f39effff700 1 --2- 172.21.15.2:0/3886128739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 0x555df2b97310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.777+0000 7f39fc847700 1 --2- 172.21.15.2:0/3886128739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b928b0 0x555df2b97a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.778+0000 7f39f7fff700 1 -- 172.21.15.2:0/3886128739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b928b0 msgr2=0x555df2b97a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:33.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.778+0000 7f39f7fff700 1 --2- 172.21.15.2:0/3886128739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b928b0 0x555df2b97a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.778+0000 7f39f7fff700 1 -- 172.21.15.2:0/3886128739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 msgr2=0x555df2b97310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:33.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.778+0000 7f39f7fff700 1 --2- 172.21.15.2:0/3886128739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 0x555df2b97310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.778+0000 7f39f7fff700 1 -- 172.21.15.2:0/3886128739 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f810da60 con 0x555df2b8db60 2023-11-16T18:26:33.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.778+0000 7f39f7fff700 1 --2- 172.21.15.2:0/3886128739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 0x555df2b9a090 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f39e8009990 tx=0x7f39e800fcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57d8482f11ac258 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:33.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.778+0000 7f39d77fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:33.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.778+0000 7f39d77fe700 1 -- 172.21.15.2:0/3886128739 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e802b040 con 0x555df2b8db60 2023-11-16T18:26:33.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.778+0000 7f39d77fe700 1 -- 172.21.15.2:0/3886128739 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f39e8005450 con 0x555df2b8db60 2023-11-16T18:26:33.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.778+0000 7f3a0a436b80 1 -- 172.21.15.2:0/3886128739 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555df2b80b20 con 0x555df2b8db60 2023-11-16T18:26:33.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.778+0000 7f39d77fe700 1 -- 172.21.15.2:0/3886128739 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39e8005700 con 0x555df2b8db60 2023-11-16T18:26:33.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.778+0000 7f3a0a436b80 10 client.16053 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:33.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.778+0000 7f3a0a436b80 1 -- 172.21.15.2:0/3886128739 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x555df2aa5190 con 0x555df2b8db60 2023-11-16T18:26:33.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.779+0000 7f39d77fe700 1 -- 172.21.15.2:0/3886128739 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555df2b80b20 con 0x555df2b8db60 2023-11-16T18:26:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.779+0000 7f39d77fe700 1 -- 172.21.15.2:0/3886128739 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f39e8024b40 con 0x555df2b8db60 2023-11-16T18:26:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.779+0000 7f3a0a436b80 10 client.16053 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.779+0000 7f3a0a436b80 10 client.16053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.779+0000 7f3a0a436b80 20 client.16053 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.779+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/3886128739 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555df2bab750 0x555df2badb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.779+0000 7f3a0a436b80 4 client.16053 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.779+0000 7f3a0a436b80 1 -- 172.21.15.2:0/3886128739 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555df2aa10a0 con 0x555df2bab750 2023-11-16T18:26:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.779+0000 7f39effff700 1 --2- 172.21.15.2:0/3886128739 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555df2bab750 0x555df2badb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.780+0000 7f39effff700 1 --2- 172.21.15.2:0/3886128739 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555df2bab750 0x555df2badb30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a32a0f7a30169bd2 server_cookie=f55083ca1cba4428 in_seq=0 out_seq=0 2023-11-16T18:26:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.780+0000 7f39d77fe700 10 client.16053 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.780+0000 7f39d77fe700 1 -- 172.21.15.2:0/3886128739 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555df2aa10a0 con 0x555df2bab750 2023-11-16T18:26:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.780+0000 7f39d77fe700 10 client.16053 handle_command_reply: tid=0 2023-11-16T18:26:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.780+0000 7f3a0a436b80 1 client.16053 shutdown 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.781+0000 7f3a0a436b80 1 -- 172.21.15.2:0/3886128739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 msgr2=0x555df2b9a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.781+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/3886128739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 0x555df2b9a090 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f39e8009990 tx=0x7f39e800fcc0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.781+0000 7f3a0a436b80 1 -- 172.21.15.2:0/3886128739 shutdown_connections 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.781+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/3886128739 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555df2bab750 0x555df2badb30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.781+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/3886128739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b928b0 0x555df2b97a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.781+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/3886128739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b8db60 0x555df2b9a090 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.781+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/3886128739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 0x555df2b97310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.781+0000 7f39d77fe700 0 client.16053 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.781+0000 7f3a0a436b80 1 -- 172.21.15.2:0/3886128739 >> 172.21.15.2:0/3886128739 conn(0x555df2b83360 msgr2=0x555df2b87de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.781+0000 7f3a0a436b80 1 -- 172.21.15.2:0/3886128739 shutdown_connections 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.781+0000 7f3a0a436b80 1 -- 172.21.15.2:0/3886128739 wait complete. 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.781+0000 7f3a0a436b80 20 client.16053 trim_cache size 0 max 16384 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.796+0000 7f3a0a436b80 1 Processor -- start 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.796+0000 7f3a0a436b80 1 -- start start 2023-11-16T18:26:33.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.796+0000 7f3a0a436b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8db60 0x555df2b911a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.796+0000 7f3a0a436b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b8e470 0x555df2b88560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.796+0000 7f3a0a436b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 0x555df2b88aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.796+0000 7f3a0a436b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555df2b82ee0 con 0x555df2b928b0 2023-11-16T18:26:33.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.796+0000 7f3a0a436b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555df2b988c0 con 0x555df2b8db60 2023-11-16T18:26:33.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.796+0000 7f3a0a436b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555df2b8a1c0 con 0x555df2b8e470 2023-11-16T18:26:33.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39fc847700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 0x555df2b88aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39f7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8db60 0x555df2b911a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39fc847700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 0x555df2b88aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51274/0 (socket says 172.21.15.2:51274) 2023-11-16T18:26:33.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39fc847700 1 -- 172.21.15.2:0/3509494951 learned_addr learned my addr 172.21.15.2:0/3509494951 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:33.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39effff700 1 --2- 172.21.15.2:0/3509494951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b8e470 0x555df2b88560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39fc847700 1 -- 172.21.15.2:0/3509494951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b8e470 msgr2=0x555df2b88560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:33.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39fc847700 1 --2- 172.21.15.2:0/3509494951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b8e470 0x555df2b88560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39fc847700 1 -- 172.21.15.2:0/3509494951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8db60 msgr2=0x555df2b911a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39fc847700 1 --2- 172.21.15.2:0/3509494951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8db60 0x555df2b911a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39fc847700 1 -- 172.21.15.2:0/3509494951 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555df2aa4ee0 con 0x555df2b928b0 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39f7fff700 1 --2- 172.21.15.2:0/3509494951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8db60 0x555df2b911a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39fc847700 1 --2- 172.21.15.2:0/3509494951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 0x555df2b88aa0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f39f0009fb0 tx=0x7f39f000e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774847fcf7bd08bb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39eeffd700 1 -- 172.21.15.2:0/3509494951 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f0011050 con 0x555df2b928b0 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39eeffd700 1 -- 172.21.15.2:0/3509494951 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39f0012040 con 0x555df2b928b0 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.797+0000 7f39eeffd700 1 -- 172.21.15.2:0/3509494951 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f0016620 con 0x555df2b928b0 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.798+0000 7f3a0a436b80 1 -- 172.21.15.2:0/3509494951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 msgr2=0x555df2b88aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.798+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/3509494951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 0x555df2b88aa0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f39f0009fb0 tx=0x7f39f000e040 comp rx=0 tx=0).stop 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.798+0000 7f3a0a436b80 1 -- 172.21.15.2:0/3509494951 shutdown_connections 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.798+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/3509494951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 0x555df2b88aa0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.798+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/3509494951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8db60 0x555df2b911a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.798+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/3509494951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b8e470 0x555df2b88560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.798+0000 7f3a0a436b80 1 -- 172.21.15.2:0/3509494951 >> 172.21.15.2:0/3509494951 conn(0x555df2b83360 msgr2=0x555df2ba7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.798+0000 7f3a0a436b80 1 -- 172.21.15.2:0/3509494951 shutdown_connections 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.798+0000 7f3a0a436b80 1 -- 172.21.15.2:0/3509494951 wait complete. 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.799+0000 7f3a0a436b80 1 -- start start 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.799+0000 7f3a0a436b80 1 Processor -- start 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.799+0000 7f3a0a436b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b8db60 0x555df2bf7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.799+0000 7f3a0a436b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 0x555df2bf7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.799+0000 7f3a0a436b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 0x555df2b9ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.799+0000 7f3a0a436b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555df2b9f3e0 con 0x555df2b928b0 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.799+0000 7f3a0a436b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555df2b9f550 con 0x555df2b8e470 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.799+0000 7f3a0a436b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555df2b9f710 con 0x555df2b8db60 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.799+0000 7f3a0a436b80 10 client.0 _ll_register_callbacks cb 0x7f39e0096030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39fc847700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 0x555df2b9ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39fc847700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 0x555df2b9ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51280/0 (socket says 172.21.15.2:51280) 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39effff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 0x555df2bf7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39fc847700 1 -- 172.21.15.2:0/268292611 learned_addr learned my addr 172.21.15.2:0/268292611 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39f7fff700 1 --2- 172.21.15.2:0/268292611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b8db60 0x555df2bf7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39fc847700 1 -- 172.21.15.2:0/268292611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b8db60 msgr2=0x555df2bf7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39fc847700 1 --2- 172.21.15.2:0/268292611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b8db60 0x555df2bf7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39fc847700 1 -- 172.21.15.2:0/268292611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 msgr2=0x555df2bf7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39fc847700 1 --2- 172.21.15.2:0/268292611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 0x555df2bf7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39fc847700 1 -- 172.21.15.2:0/268292611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555df2aa5190 con 0x555df2b928b0 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39effff700 1 --2- 172.21.15.2:0/268292611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 0x555df2bf7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39fc847700 1 --2- 172.21.15.2:0/268292611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 0x555df2b9ed50 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f39f0017710 tx=0x7f39f0017740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e3059b6780fd50 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39d77fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.800+0000 7f39d77fe700 1 -- 172.21.15.2:0/268292611 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f0021850 con 0x555df2b928b0 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.801+0000 7f3a0a436b80 1 -- 172.21.15.2:0/268292611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39e801e040 con 0x555df2b928b0 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.801+0000 7f39d77fe700 1 -- 172.21.15.2:0/268292611 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f39f0006e40 con 0x555df2b928b0 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.801+0000 7f39d77fe700 1 -- 172.21.15.2:0/268292611 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39f0011020 con 0x555df2b928b0 2023-11-16T18:26:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.801+0000 7f3a0a436b80 10 client.16059 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.801+0000 7f3a0a436b80 1 -- 172.21.15.2:0/268292611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f39f0009be0 con 0x555df2b928b0 2023-11-16T18:26:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.801+0000 7f39d77fe700 1 -- 172.21.15.2:0/268292611 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39e801e040 con 0x555df2b928b0 2023-11-16T18:26:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.801+0000 7f39d77fe700 1 -- 172.21.15.2:0/268292611 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f39f001e050 con 0x555df2b928b0 2023-11-16T18:26:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.801+0000 7f3a0a436b80 10 client.16059 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.801+0000 7f3a0a436b80 10 client.16059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.801+0000 7f3a0a436b80 20 client.16059 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.801+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/268292611 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555df2c00610 0x555df2c02a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.801+0000 7f3a0a436b80 4 client.16059 mds_command: new command op to 24473 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:26:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.801+0000 7f3a0a436b80 1 -- 172.21.15.2:0/268292611 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555df2aad660 con 0x555df2c00610 2023-11-16T18:26:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.802+0000 7f39f7fff700 1 --2- 172.21.15.2:0/268292611 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555df2c00610 0x555df2c02a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.802+0000 7f39f7fff700 1 --2- 172.21.15.2:0/268292611 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555df2c00610 0x555df2c02a20 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=394c97c041cc1d38 server_cookie=fc85d263795b906c in_seq=0 out_seq=0 2023-11-16T18:26:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.802+0000 7f39d77fe700 10 client.16059 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.802+0000 7f39d77fe700 1 -- 172.21.15.2:0/268292611 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555df2aad660 con 0x555df2c00610 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.802+0000 7f39d77fe700 10 client.16059 handle_command_reply: tid=0 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.802+0000 7f3a0a436b80 1 client.16059 shutdown 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.802+0000 7f3a0a436b80 1 -- 172.21.15.2:0/268292611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 msgr2=0x555df2b9ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.802+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/268292611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 0x555df2b9ed50 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f39f0017710 tx=0x7f39f0017740 comp rx=0 tx=0).stop 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f3a0a436b80 1 -- 172.21.15.2:0/268292611 shutdown_connections 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/268292611 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555df2c00610 0x555df2c02a20 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/268292611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555df2b8db60 0x555df2bf7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/268292611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555df2b928b0 0x555df2b9ed50 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f39d77fe700 0 client.16059 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f3a0a436b80 1 --2- 172.21.15.2:0/268292611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555df2b8e470 0x555df2bf7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f3a0a436b80 1 -- 172.21.15.2:0/268292611 >> 172.21.15.2:0/268292611 conn(0x555df2b83360 msgr2=0x555df2b8c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f3a0a436b80 1 -- 172.21.15.2:0/268292611 shutdown_connections 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f3a0a436b80 1 -- 172.21.15.2:0/268292611 wait complete. 2023-11-16T18:26:33.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f3a0a436b80 20 client.16059 trim_cache size 0 max 16384 2023-11-16T18:26:33.748 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:26:33.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f39fe2aa700 1 -- 172.21.15.2:0/3367497312 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f39e40a12c0 msgr2=0x7f39e40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:33.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f39fe2aa700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f39e40a12c0 0x7f39e40a3770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f39e800b5c0 tx=0x7f39e8005fb0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f39fe2aa700 1 -- 172.21.15.2:0/3367497312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39f8068770 msgr2=0x7f39f81969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:33.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f39fe2aa700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39f8068770 0x7f39f81969a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f39e000d950 tx=0x7f39e000dc60 comp rx=0 tx=0).stop 2023-11-16T18:26:33.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f39fe2aa700 1 -- 172.21.15.2:0/3367497312 shutdown_connections 2023-11-16T18:26:33.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f39fe2aa700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f39e40a12c0 0x7f39e40a3770 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f39fe2aa700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39f8067dc0 0x7f39f8196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f39fe2aa700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f8105250 0x7f39f819ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f39fe2aa700 1 --2- 172.21.15.2:0/3367497312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39f8068770 0x7f39f81969a0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:33.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.803+0000 7f39fe2aa700 1 -- 172.21.15.2:0/3367497312 >> 172.21.15.2:0/3367497312 conn(0x7f39f80763a0 msgr2=0x7f39f80fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:33.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.804+0000 7f39fe2aa700 1 -- 172.21.15.2:0/3367497312 shutdown_connections 2023-11-16T18:26:33.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:26.804+0000 7f39fe2aa700 1 -- 172.21.15.2:0/3367497312 wait complete. 2023-11-16T18:26:33.751 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.751 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.752 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:33.752 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:33.752 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.752 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.752 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 62 -lt 100 ']' 2023-11-16T18:26:33.752 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 62 + 1 2023-11-16T18:26:33.752 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=63 2023-11-16T18:26:33.752 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:33.752 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.752 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.752 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:33.752 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:33.752 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:27 smithi002 ceph-mon[98807]: pgmap v51: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 16 op/s 2023-11-16T18:26:33.757 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.757 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.757 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:33.758 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:33.758 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.758 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.758 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 63 -lt 100 ']' 2023-11-16T18:26:33.758 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 63 + 1 2023-11-16T18:26:33.758 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=64 2023-11-16T18:26:33.758 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:33.758 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.758 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.758 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:33.758 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:33.758 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:33.758 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.759 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.759 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:33.759 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:33.759 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.759 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.759 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 64 -lt 100 ']' 2023-11-16T18:26:33.759 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 64 + 1 2023-11-16T18:26:33.759 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=65 2023-11-16T18:26:33.759 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:33.759 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.759 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.759 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 65 -lt 100 ']' 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 65 + 1 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=66 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.760 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:33.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:33.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:33.761 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.761 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:33.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:33.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 66 -lt 100 ']' 2023-11-16T18:26:33.761 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 66 + 1 2023-11-16T18:26:33.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=67 2023-11-16T18:26:33.762 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:33.762 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.762 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.762 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:33.762 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:33.762 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:33.762 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.762 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:30 smithi002 ceph-mon[98807]: pgmap v52: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 12 op/s 2023-11-16T18:26:33.768 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:33.781 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:33.782 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.782 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.782 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 67 -lt 100 ']' 2023-11-16T18:26:33.782 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 67 + 1 2023-11-16T18:26:33.782 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=68 2023-11-16T18:26:33.782 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:33.782 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.782 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.782 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:33.782 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:33.782 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:33.782 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 68 -lt 100 ']' 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 68 + 1 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=69 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:33.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:33.784 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.784 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.784 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:33.784 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:33.784 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.784 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.784 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 69 -lt 100 ']' 2023-11-16T18:26:33.784 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 69 + 1 2023-11-16T18:26:33.784 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=70 2023-11-16T18:26:33.784 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:33.784 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.784 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 70 -lt 100 ']' 2023-11-16T18:26:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 70 + 1 2023-11-16T18:26:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=71 2023-11-16T18:26:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:33.786 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.786 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.786 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:33.787 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:33.787 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:33.787 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:33.787 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:33.983 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:33.983 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:33.983 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.983 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:33.984 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 71 -lt 100 ']' 2023-11-16T18:26:33.984 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 71 + 1 2023-11-16T18:26:33.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:32 smithi002 ceph-mon[98807]: pgmap v53: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 12 op/s 2023-11-16T18:26:33.988 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=72 2023-11-16T18:26:34.008 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:34.008 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:34.008 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:34.008 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:34.008 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:34.008 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:34.009 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:34.009 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:34.009 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:34.009 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:34.009 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:34.009 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:34.009 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 72 -lt 100 ']' 2023-11-16T18:26:34.009 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 72 + 1 2023-11-16T18:26:34.009 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=73 2023-11-16T18:26:34.009 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:34.009 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:34.009 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:34.010 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:34.010 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:34.010 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:34.010 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:34.010 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:34.010 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:34.010 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:34.010 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:34.010 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:34.010 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 73 -lt 100 ']' 2023-11-16T18:26:34.010 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 73 + 1 2023-11-16T18:26:34.010 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=74 2023-11-16T18:26:34.010 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:34.011 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:34.011 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:34.011 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:34.011 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:34.011 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:34.011 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:34.011 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:34.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:34 smithi002 ceph-mon[98807]: pgmap v54: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 33 KiB/s wr, 16 op/s 2023-11-16T18:26:34.207 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:34.207 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:34.212 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:34.212 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:34.212 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 74 -lt 100 ']' 2023-11-16T18:26:34.212 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 74 + 1 2023-11-16T18:26:34.255 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=75 2023-11-16T18:26:34.255 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:34.292 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:34.293 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:34.293 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:34.294 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:34.294 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:34.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:34 smithi040 ceph-mon[100251]: pgmap v54: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 33 KiB/s wr, 16 op/s 2023-11-16T18:26:34.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:34 smithi188 ceph-mon[100639]: pgmap v54: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 33 KiB/s wr, 16 op/s 2023-11-16T18:26:34.751 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T18:26:34.751 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T18:26:35.217 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:35.217 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:35.356 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:35.356 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:35.380 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:35.380 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:35.380 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 75 -lt 100 ']' 2023-11-16T18:26:35.380 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 75 + 1 2023-11-16T18:26:35.413 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=76 2023-11-16T18:26:35.413 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:35.422 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:35.422 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:35.423 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:35.424 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:35.424 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:35.992 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:35.992 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:36.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:36 smithi002 ceph-mon[98807]: pgmap v55: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-11-16T18:26:36.223 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:36.223 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.235+0000 7f6aba1e2700 1 -- 172.21.15.2:0/1543934709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab4105560 msgr2=0x7f6ab410d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.235+0000 7f6aba1e2700 1 --2- 172.21.15.2:0/1543934709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab4105560 0x7f6ab410d170 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00b0d0 tx=0x7f6aac00b3e0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.235+0000 7f6aba1e2700 1 -- 172.21.15.2:0/1543934709 shutdown_connections 2023-11-16T18:26:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.235+0000 7f6aba1e2700 1 --2- 172.21.15.2:0/1543934709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab4105560 0x7f6ab410d170 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.235+0000 7f6aba1e2700 1 --2- 172.21.15.2:0/1543934709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab4100c50 0x7f6ab4101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.235+0000 7f6aba1e2700 1 --2- 172.21.15.2:0/1543934709 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ab41015f0 0x7f6ab4104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.235+0000 7f6aba1e2700 1 -- 172.21.15.2:0/1543934709 >> 172.21.15.2:0/1543934709 conn(0x7f6ab4077550 msgr2=0x7f6ab4077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.235+0000 7f6aba1e2700 1 -- 172.21.15.2:0/1543934709 shutdown_connections 2023-11-16T18:26:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.236+0000 7f6aba1e2700 1 -- 172.21.15.2:0/1543934709 wait complete. 2023-11-16T18:26:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.236+0000 7f6aba1e2700 1 Processor -- start 2023-11-16T18:26:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.237+0000 7f6aba1e2700 1 -- start start 2023-11-16T18:26:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.237+0000 7f6aba1e2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab4100c50 0x7f6ab4072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.237+0000 7f6aba1e2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ab41015f0 0x7f6ab4073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.237+0000 7f6aba1e2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab4105560 0x7f6ab41a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.237+0000 7f6aba1e2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab4073b60 con 0x7f6ab4100c50 2023-11-16T18:26:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.237+0000 7f6aba1e2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab41a4b10 con 0x7f6ab4105560 2023-11-16T18:26:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.237+0000 7f6aba1e2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab41a4c50 con 0x7f6ab41015f0 2023-11-16T18:26:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.237+0000 7f6ab37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab4100c50 0x7f6ab4072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.237+0000 7f6ab37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab4100c50 0x7f6ab4072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51290/0 (socket says 172.21.15.2:51290) 2023-11-16T18:26:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.237+0000 7f6ab37fe700 1 -- 172.21.15.2:0/3572479410 learned_addr learned my addr 172.21.15.2:0/3572479410 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.237+0000 7f6ab2ffd700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ab41015f0 0x7f6ab4073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.237+0000 7f6ab3fff700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab4105560 0x7f6ab41a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.238+0000 7f6ab37fe700 1 -- 172.21.15.2:0/3572479410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ab41015f0 msgr2=0x7f6ab4073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.238+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ab41015f0 0x7f6ab4073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.238+0000 7f6ab37fe700 1 -- 172.21.15.2:0/3572479410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab4105560 msgr2=0x7f6ab41a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.238+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab4105560 0x7f6ab41a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.238+0000 7f6ab37fe700 1 -- 172.21.15.2:0/3572479410 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac009d00 con 0x7f6ab4100c50 2023-11-16T18:26:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.238+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab4100c50 0x7f6ab4072df0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400e9a0 tx=0x7f6aa400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.238+0000 7f6ab0ff9700 1 -- 172.21.15.2:0/3572479410 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4009790 con 0x7f6ab4100c50 2023-11-16T18:26:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.238+0000 7f6ab0ff9700 1 -- 172.21.15.2:0/3572479410 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6aa4010430 con 0x7f6ab4100c50 2023-11-16T18:26:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.238+0000 7f6aba1e2700 1 -- 172.21.15.2:0/3572479410 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab4110410 con 0x7f6ab4100c50 2023-11-16T18:26:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.238+0000 7f6ab0ff9700 1 -- 172.21.15.2:0/3572479410 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4018680 con 0x7f6ab4100c50 2023-11-16T18:26:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.238+0000 7f6aba1e2700 1 -- 172.21.15.2:0/3572479410 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab4110280 con 0x7f6ab4100c50 2023-11-16T18:26:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.241+0000 7f6ab0ff9700 1 -- 172.21.15.2:0/3572479410 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f6aa40105a0 con 0x7f6ab4100c50 2023-11-16T18:26:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.242+0000 7f6ab0ff9700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6a9c0a1340 0x7f6a9c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.242+0000 7f6ab2ffd700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6a9c0a1340 0x7f6a9c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.242+0000 7f6ab0ff9700 1 -- 172.21.15.2:0/3572479410 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6aa4014070 con 0x7f6ab4100c50 2023-11-16T18:26:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.242+0000 7f6ab2ffd700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6a9c0a1340 0x7f6a9c0a37f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40745e0 tx=0x7f6aa8009190 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:36.261 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:36.261 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:36.261 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 76 -lt 100 ']' 2023-11-16T18:26:36.261 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 76 + 1 2023-11-16T18:26:36.297 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=77 2023-11-16T18:26:36.297 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:36.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.341+0000 7f6ac636eb80 1 Processor -- start 2023-11-16T18:26:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.341+0000 7f6ac636eb80 1 -- start start 2023-11-16T18:26:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.341+0000 7f6ac636eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06e11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.341+0000 7f6ac636eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 0x5626b06db560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.341+0000 7f6ac636eb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1b90 0x5626b06dbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.341+0000 7f6ac636eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b05e1000 con 0x5626b05376b0 2023-11-16T18:26:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.341+0000 7f6ac636eb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b0633840 con 0x5626b06e1b90 2023-11-16T18:26:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.341+0000 7f6ac636eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b062ccd0 con 0x5626b06e1720 2023-11-16T18:26:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.341+0000 7f6ab37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06e11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6ab37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06e11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51298/0 (socket says 172.21.15.2:51298) 2023-11-16T18:26:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6ab3fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1b90 0x5626b06dbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6ab37fe700 1 -- 172.21.15.2:0/3042335183 learned_addr learned my addr 172.21.15.2:0/3042335183 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6ab2ffd700 1 --2- 172.21.15.2:0/3042335183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 0x5626b06db560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6ab37fe700 1 -- 172.21.15.2:0/3042335183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 msgr2=0x5626b06db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/3042335183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 0x5626b06db560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6ab37fe700 1 -- 172.21.15.2:0/3042335183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1b90 msgr2=0x5626b06dbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/3042335183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1b90 0x5626b06dbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6ab37fe700 1 -- 172.21.15.2:0/3042335183 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626b04ae5d0 con 0x5626b05376b0 2023-11-16T18:26:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6ab3fff700 1 --2- 172.21.15.2:0/3042335183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1b90 0x5626b06dbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/3042335183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06e11e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4096440 tx=0x7f6aa4099b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36639d4fb24c2437 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6a99ffb700 1 -- 172.21.15.2:0/3042335183 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa409df70 con 0x5626b05376b0 2023-11-16T18:26:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6a99ffb700 1 -- 172.21.15.2:0/3042335183 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6aa40a2390 con 0x5626b05376b0 2023-11-16T18:26:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.342+0000 7f6a99ffb700 1 -- 172.21.15.2:0/3042335183 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4099690 con 0x5626b05376b0 2023-11-16T18:26:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.343+0000 7f6ac636eb80 1 -- 172.21.15.2:0/3042335183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 msgr2=0x5626b06e11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.343+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/3042335183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06e11e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4096440 tx=0x7f6aa4099b70 comp rx=0 tx=0).stop 2023-11-16T18:26:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.343+0000 7f6ac636eb80 1 -- 172.21.15.2:0/3042335183 shutdown_connections 2023-11-16T18:26:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.343+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/3042335183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1b90 0x5626b06dbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.343+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/3042335183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06e11e0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.343+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/3042335183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 0x5626b06db560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.343+0000 7f6ac636eb80 1 -- 172.21.15.2:0/3042335183 >> 172.21.15.2:0/3042335183 conn(0x5626b06d0f50 msgr2=0x5626b06d6270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.343+0000 7f6ac636eb80 1 -- 172.21.15.2:0/3042335183 shutdown_connections 2023-11-16T18:26:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.343+0000 7f6ac636eb80 1 -- 172.21.15.2:0/3042335183 wait complete. 2023-11-16T18:26:36.347 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:36.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.344+0000 7f6ac636eb80 1 -- start start 2023-11-16T18:26:36.348 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.344+0000 7f6ac636eb80 1 Processor -- start 2023-11-16T18:26:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.344+0000 7f6ac636eb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b05376b0 0x5626b06e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.344+0000 7f6ac636eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 0x5626b06e52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.344+0000 7f6ac636eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b06e1b90 0x5626b06e5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ac636eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b05ce880 con 0x5626b06e1b90 2023-11-16T18:26:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ac636eb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b05ce520 con 0x5626b05376b0 2023-11-16T18:26:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ac636eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b04b08c0 con 0x5626b06e1720 2023-11-16T18:26:36.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ac636eb80 10 client.0 _ll_register_callbacks cb 0x5626b05ec720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:36.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ab37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b05376b0 0x5626b06e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ab37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b05376b0 0x5626b06e4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:41742/0 (socket says 172.21.15.2:41742) 2023-11-16T18:26:36.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ab37fe700 1 -- 172.21.15.2:0/1808010607 learned_addr learned my addr 172.21.15.2:0/1808010607 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:36.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ab3fff700 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b06e1b90 0x5626b06e5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ab2ffd700 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 0x5626b06e52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.350 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:36.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ab37fe700 1 -- 172.21.15.2:0/1808010607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 msgr2=0x5626b06e52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:36.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 0x5626b06e52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ab37fe700 1 -- 172.21.15.2:0/1808010607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b06e1b90 msgr2=0x5626b06e5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:36.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b06e1b90 0x5626b06e5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ab37fe700 1 -- 172.21.15.2:0/1808010607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab4110280 con 0x5626b05376b0 2023-11-16T18:26:36.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ab3fff700 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b06e1b90 0x5626b06e5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:36.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ab2ffd700 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 0x5626b06e52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:36.352 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:36.353 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:36.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b05376b0 0x5626b06e4d90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6aa40cab40 tx=0x7f6aa4010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=699895509a850f54 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.345+0000 7f6a92ffd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:26:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6a92ffd700 1 -- 172.21.15.2:0/1808010607 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa40a1f00 con 0x5626b05376b0 2023-11-16T18:26:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6a92ffd700 1 -- 172.21.15.2:0/1808010607 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6aa40a2060 con 0x5626b05376b0 2023-11-16T18:26:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6ac636eb80 1 -- 172.21.15.2:0/1808010607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626b06ce710 con 0x5626b05376b0 2023-11-16T18:26:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6a92ffd700 1 -- 172.21.15.2:0/1808010607 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6aa40b82a0 con 0x5626b05376b0 2023-11-16T18:26:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6ac636eb80 10 client.25631 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6ac636eb80 1 -- 172.21.15.2:0/1808010607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5626b05f3130 con 0x5626b05376b0 2023-11-16T18:26:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6a92ffd700 1 -- 172.21.15.2:0/1808010607 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5626b06ce710 con 0x5626b05376b0 2023-11-16T18:26:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6a92ffd700 1 -- 172.21.15.2:0/1808010607 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6aa4099040 con 0x5626b05376b0 2023-11-16T18:26:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6ac636eb80 10 client.25631 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6ac636eb80 10 client.25631 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6ac636eb80 20 client.25631 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626b06f4b60 0x5626b06f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6ac636eb80 4 client.25631 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.346+0000 7f6ac636eb80 1 -- 172.21.15.2:0/1808010607 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626b05ef040 con 0x5626b06f4b60 2023-11-16T18:26:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.347+0000 7f6ab2ffd700 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626b06f4b60 0x5626b06f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.347+0000 7f6a92ffd700 10 client.25631 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.347+0000 7f6ab2ffd700 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626b06f4b60 0x5626b06f6f40 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32c1db0f5f22f858 server_cookie=beaba11e7111ff16 in_seq=0 out_seq=0 2023-11-16T18:26:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6a92ffd700 1 -- 172.21.15.2:0/1808010607 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5626b05ef040 con 0x5626b06f4b60 2023-11-16T18:26:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6a92ffd700 10 client.25631 handle_command_reply: tid=0 2023-11-16T18:26:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6ac636eb80 1 client.25631 shutdown 2023-11-16T18:26:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6ac636eb80 1 -- 172.21.15.2:0/1808010607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b05376b0 msgr2=0x5626b06e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b05376b0 0x5626b06e4d90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6aa40cab40 tx=0x7f6aa4010c80 comp rx=0 tx=0).stop 2023-11-16T18:26:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6ac636eb80 1 -- 172.21.15.2:0/1808010607 shutdown_connections 2023-11-16T18:26:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626b06f4b60 0x5626b06f6f40 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b06e1b90 0x5626b06e5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b05376b0 0x5626b06e4d90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/1808010607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 0x5626b06e52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6a92ffd700 0 client.25631 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6ac636eb80 1 -- 172.21.15.2:0/1808010607 >> 172.21.15.2:0/1808010607 conn(0x5626b06d0f50 msgr2=0x5626b06d57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6ac636eb80 1 -- 172.21.15.2:0/1808010607 shutdown_connections 2023-11-16T18:26:36.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6ac636eb80 1 -- 172.21.15.2:0/1808010607 wait complete. 2023-11-16T18:26:36.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.348+0000 7f6ac636eb80 20 client.25631 trim_cache size 0 max 16384 2023-11-16T18:26:36.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.364+0000 7f6ac636eb80 1 Processor -- start 2023-11-16T18:26:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.364+0000 7f6ac636eb80 1 -- start start 2023-11-16T18:26:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.364+0000 7f6ac636eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06ec090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.364+0000 7f6ac636eb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1720 0x5626b06d5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ac636eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1b90 0x5626b06e5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ac636eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b06d2fd0 con 0x5626b05376b0 2023-11-16T18:26:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ac636eb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b06d8050 con 0x5626b06e1720 2023-11-16T18:26:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ac636eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b06d6fb0 con 0x5626b06e1b90 2023-11-16T18:26:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ab37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ab37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06ec090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51312/0 (socket says 172.21.15.2:51312) 2023-11-16T18:26:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ab37fe700 1 -- 172.21.15.2:0/1981298593 learned_addr learned my addr 172.21.15.2:0/1981298593 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ab2ffd700 1 --2- 172.21.15.2:0/1981298593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1720 0x5626b06d5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ab3fff700 1 --2- 172.21.15.2:0/1981298593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1b90 0x5626b06e5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ab37fe700 1 -- 172.21.15.2:0/1981298593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1b90 msgr2=0x5626b06e5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/1981298593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1b90 0x5626b06e5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ab37fe700 1 -- 172.21.15.2:0/1981298593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1720 msgr2=0x5626b06d5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/1981298593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1720 0x5626b06d5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ab37fe700 1 -- 172.21.15.2:0/1981298593 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626b06d26f0 con 0x5626b05376b0 2023-11-16T18:26:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/1981298593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06ec090 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6aa40969b0 tx=0x7f6aa4099a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196e5f2554980d14 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6a9a7fc700 1 -- 172.21.15.2:0/1981298593 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa409dff0 con 0x5626b05376b0 2023-11-16T18:26:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6a9a7fc700 1 -- 172.21.15.2:0/1981298593 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6aa40998a0 con 0x5626b05376b0 2023-11-16T18:26:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.365+0000 7f6a9a7fc700 1 -- 172.21.15.2:0/1981298593 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4094b50 con 0x5626b05376b0 2023-11-16T18:26:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.366+0000 7f6ac636eb80 1 -- 172.21.15.2:0/1981298593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 msgr2=0x5626b06ec090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.366+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/1981298593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06ec090 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6aa40969b0 tx=0x7f6aa4099a80 comp rx=0 tx=0).stop 2023-11-16T18:26:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.366+0000 7f6ac636eb80 1 -- 172.21.15.2:0/1981298593 shutdown_connections 2023-11-16T18:26:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.366+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/1981298593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1b90 0x5626b06e5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.366+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/1981298593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06ec090 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.366+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/1981298593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1720 0x5626b06d5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.366+0000 7f6ac636eb80 1 -- 172.21.15.2:0/1981298593 >> 172.21.15.2:0/1981298593 conn(0x5626b06d0f50 msgr2=0x5626b06defc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.366+0000 7f6ac636eb80 1 -- 172.21.15.2:0/1981298593 shutdown_connections 2023-11-16T18:26:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.366+0000 7f6ac636eb80 1 -- 172.21.15.2:0/1981298593 wait complete. 2023-11-16T18:26:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.367+0000 7f6ac636eb80 1 -- start start 2023-11-16T18:26:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.367+0000 7f6ac636eb80 1 Processor -- start 2023-11-16T18:26:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.367+0000 7f6ac636eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06ec0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.367+0000 7f6ac636eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 0x5626b06e9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.367+0000 7f6ac636eb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1b90 0x5626b06e9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.367+0000 7f6ac636eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b06ea070 con 0x5626b05376b0 2023-11-16T18:26:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.367+0000 7f6ac636eb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b06ea1e0 con 0x5626b06e1b90 2023-11-16T18:26:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.367+0000 7f6ac636eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b07479f0 con 0x5626b06e1720 2023-11-16T18:26:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6ac636eb80 10 client.0 _ll_register_callbacks cb 0x7f6aa4005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6ab37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06ec0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6ab37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06ec0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51318/0 (socket says 172.21.15.2:51318) 2023-11-16T18:26:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6ab37fe700 1 -- 172.21.15.2:0/4236396151 learned_addr learned my addr 172.21.15.2:0/4236396151 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6ab2ffd700 1 --2- 172.21.15.2:0/4236396151 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 0x5626b06e9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6ab3fff700 1 --2- 172.21.15.2:0/4236396151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1b90 0x5626b06e9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6ab37fe700 1 -- 172.21.15.2:0/4236396151 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 msgr2=0x5626b06e9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/4236396151 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 0x5626b06e9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6ab37fe700 1 -- 172.21.15.2:0/4236396151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1b90 msgr2=0x5626b06e9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/4236396151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1b90 0x5626b06e9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6ab37fe700 1 -- 172.21.15.2:0/4236396151 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626b05f3130 con 0x5626b05376b0 2023-11-16T18:26:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6a92ffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6ab37fe700 1 --2- 172.21.15.2:0/4236396151 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06ec0a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4000c00 tx=0x7f6aa400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b6331e4ed679e9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6a92ffd700 1 -- 172.21.15.2:0/4236396151 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa400cdd0 con 0x5626b05376b0 2023-11-16T18:26:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6a92ffd700 1 -- 172.21.15.2:0/4236396151 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6aa409df70 con 0x5626b05376b0 2023-11-16T18:26:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.368+0000 7f6ac636eb80 1 -- 172.21.15.2:0/4236396151 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6aa40a2390 con 0x5626b05376b0 2023-11-16T18:26:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.369+0000 7f6a92ffd700 1 -- 172.21.15.2:0/4236396151 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6aa40a1f00 con 0x5626b05376b0 2023-11-16T18:26:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.369+0000 7f6ac636eb80 10 client.16077 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.369+0000 7f6ac636eb80 1 -- 172.21.15.2:0/4236396151 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5626b045ad80 con 0x5626b05376b0 2023-11-16T18:26:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.369+0000 7f6a92ffd700 1 -- 172.21.15.2:0/4236396151 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6aa40a2390 con 0x5626b05376b0 2023-11-16T18:26:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.369+0000 7f6a92ffd700 1 -- 172.21.15.2:0/4236396151 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6aa4094c50 con 0x5626b05376b0 2023-11-16T18:26:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.369+0000 7f6ac636eb80 10 client.16077 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.369+0000 7f6ac636eb80 10 client.16077 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.369+0000 7f6ac636eb80 20 client.16077 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.369+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/4236396151 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626b0748b90 0x5626b074af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.369+0000 7f6ac636eb80 4 client.16077 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:26:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.369+0000 7f6ac636eb80 1 -- 172.21.15.2:0/4236396151 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5626b05fb600 con 0x5626b0748b90 2023-11-16T18:26:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.369+0000 7f6ab2ffd700 1 --2- 172.21.15.2:0/4236396151 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626b0748b90 0x5626b074af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.370+0000 7f6ab2ffd700 1 --2- 172.21.15.2:0/4236396151 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626b0748b90 0x5626b074af70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e188ac1021aa860 server_cookie=5f7a97f0d531a733 in_seq=0 out_seq=0 2023-11-16T18:26:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.370+0000 7f6a92ffd700 10 client.16077 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6a92ffd700 1 -- 172.21.15.2:0/4236396151 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5626b05fb600 con 0x5626b0748b90 2023-11-16T18:26:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6a92ffd700 10 client.16077 handle_command_reply: tid=0 2023-11-16T18:26:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6ac636eb80 1 client.16077 shutdown 2023-11-16T18:26:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6ac636eb80 1 -- 172.21.15.2:0/4236396151 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 msgr2=0x5626b06ec0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/4236396151 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06ec0a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4000c00 tx=0x7f6aa400cb10 comp rx=0 tx=0).stop 2023-11-16T18:26:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6ac636eb80 1 -- 172.21.15.2:0/4236396151 shutdown_connections 2023-11-16T18:26:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/4236396151 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5626b0748b90 0x5626b074af70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/4236396151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626b06e1b90 0x5626b06e9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/4236396151 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626b05376b0 0x5626b06ec0a0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6ac636eb80 1 --2- 172.21.15.2:0/4236396151 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5626b06e1720 0x5626b06e9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6a92ffd700 0 client.16077 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6ac636eb80 1 -- 172.21.15.2:0/4236396151 >> 172.21.15.2:0/4236396151 conn(0x5626b06d0f50 msgr2=0x5626b06e5340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6ac636eb80 1 -- 172.21.15.2:0/4236396151 shutdown_connections 2023-11-16T18:26:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6ac636eb80 1 -- 172.21.15.2:0/4236396151 wait complete. 2023-11-16T18:26:36.403 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.398+0000 7f6ac636eb80 20 client.16077 trim_cache size 0 max 16384 2023-11-16T18:26:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:26:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:26:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:26:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:26:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.399+0000 7f6aba1e2700 1 -- 172.21.15.2:0/3572479410 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6a9c0a1340 msgr2=0x7f6a9c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.399+0000 7f6aba1e2700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6a9c0a1340 0x7f6a9c0a37f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40745e0 tx=0x7f6aa8009190 comp rx=0 tx=0).stop 2023-11-16T18:26:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.399+0000 7f6aba1e2700 1 -- 172.21.15.2:0/3572479410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab4100c50 msgr2=0x7f6ab4072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.399+0000 7f6aba1e2700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab4100c50 0x7f6ab4072df0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400e9a0 tx=0x7f6aa400ecb0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.399+0000 7f6aba1e2700 1 -- 172.21.15.2:0/3572479410 shutdown_connections 2023-11-16T18:26:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.399+0000 7f6aba1e2700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6a9c0a1340 0x7f6a9c0a37f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.399+0000 7f6aba1e2700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab4105560 0x7f6ab41a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.399+0000 7f6aba1e2700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab4100c50 0x7f6ab4072df0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.399+0000 7f6aba1e2700 1 --2- 172.21.15.2:0/3572479410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ab41015f0 0x7f6ab4073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.399+0000 7f6aba1e2700 1 -- 172.21.15.2:0/3572479410 >> 172.21.15.2:0/3572479410 conn(0x7f6ab4077550 msgr2=0x7f6ab40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.399+0000 7f6aba1e2700 1 -- 172.21.15.2:0/3572479410 shutdown_connections 2023-11-16T18:26:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:36.399+0000 7f6aba1e2700 1 -- 172.21.15.2:0/3572479410 wait complete. 2023-11-16T18:26:36.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:36 smithi040 ceph-mon[100251]: pgmap v55: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-11-16T18:26:36.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:36 smithi188 ceph-mon[100639]: pgmap v55: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-11-16T18:26:36.858 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:36.858 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:37.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:37.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:37.112 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:37.112 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:37.112 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 77 -lt 100 ']' 2023-11-16T18:26:37.112 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 77 + 1 2023-11-16T18:26:37.213 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=78 2023-11-16T18:26:37.213 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:37.242 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:37.242 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:37.243 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:37.243 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:37.243 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:37.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:37 smithi188 ceph-mon[100639]: pgmap v56: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 35 KiB/s wr, 19 op/s 2023-11-16T18:26:37.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:37 smithi002 ceph-mon[98807]: pgmap v56: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 35 KiB/s wr, 19 op/s 2023-11-16T18:26:37.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:37 smithi040 ceph-mon[100251]: pgmap v56: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 35 KiB/s wr, 19 op/s 2023-11-16T18:26:37.933 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:37.933 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:38.163 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:38.163 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:38.179 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:38.179 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:38.179 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 78 -lt 100 ']' 2023-11-16T18:26:38.179 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 78 + 1 2023-11-16T18:26:38.213 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=79 2023-11-16T18:26:38.213 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:38.404 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:38.404 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:38.405 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:38.405 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:38.405 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:38.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.493+0000 7ff9685b7700 1 -- 172.21.15.2:0/3373691035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9601055c0 msgr2=0x7ff96010d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:38.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.493+0000 7ff9685b7700 1 --2- 172.21.15.2:0/3373691035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9601055c0 0x7ff96010d1d0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff95800b0d0 tx=0x7ff95800b3e0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.493+0000 7ff9685b7700 1 -- 172.21.15.2:0/3373691035 shutdown_connections 2023-11-16T18:26:38.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.493+0000 7ff9685b7700 1 --2- 172.21.15.2:0/3373691035 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff960100c50 0x7ff960101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.493+0000 7ff9685b7700 1 --2- 172.21.15.2:0/3373691035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9601055c0 0x7ff96010d1d0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.493+0000 7ff9685b7700 1 --2- 172.21.15.2:0/3373691035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9601015f0 0x7ff960104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.493+0000 7ff9685b7700 1 -- 172.21.15.2:0/3373691035 >> 172.21.15.2:0/3373691035 conn(0x7ff960077550 msgr2=0x7ff960077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:38.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.493+0000 7ff9685b7700 1 -- 172.21.15.2:0/3373691035 shutdown_connections 2023-11-16T18:26:38.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.493+0000 7ff9685b7700 1 -- 172.21.15.2:0/3373691035 wait complete. 2023-11-16T18:26:38.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.494+0000 7ff9685b7700 1 Processor -- start 2023-11-16T18:26:38.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.494+0000 7ff9685b7700 1 -- start start 2023-11-16T18:26:38.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff9685b7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff960100c50 0x7ff960196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff9685b7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9601015f0 0x7ff960196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff9685b7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9601055c0 0x7ff96019aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff9685b7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96019b4f0 con 0x7ff9601055c0 2023-11-16T18:26:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff9685b7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96019b660 con 0x7ff9601015f0 2023-11-16T18:26:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff9685b7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96019b820 con 0x7ff960100c50 2023-11-16T18:26:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff966b54700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9601055c0 0x7ff96019aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff966b54700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9601055c0 0x7ff96019aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51336/0 (socket says 172.21.15.2:51336) 2023-11-16T18:26:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff966b54700 1 -- 172.21.15.2:0/329832635 learned_addr learned my addr 172.21.15.2:0/329832635 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff965b52700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9601015f0 0x7ff960196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff966353700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff960100c50 0x7ff960196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff966b54700 1 -- 172.21.15.2:0/329832635 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff960100c50 msgr2=0x7ff960196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff966b54700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff960100c50 0x7ff960196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff966b54700 1 -- 172.21.15.2:0/329832635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9601015f0 msgr2=0x7ff960196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff966b54700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9601015f0 0x7ff960196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff966b54700 1 -- 172.21.15.2:0/329832635 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff958009d00 con 0x7ff9601055c0 2023-11-16T18:26:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.495+0000 7ff965b52700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9601015f0 0x7ff960196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.496+0000 7ff966b54700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9601055c0 0x7ff96019aeb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff958007870 tx=0x7ff9580078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.496+0000 7ff9577fe700 1 -- 172.21.15.2:0/329832635 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95800e040 con 0x7ff9601055c0 2023-11-16T18:26:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.496+0000 7ff9577fe700 1 -- 172.21.15.2:0/329832635 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff958005070 con 0x7ff9601055c0 2023-11-16T18:26:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.496+0000 7ff9685b7700 1 -- 172.21.15.2:0/329832635 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9601103e0 con 0x7ff9601055c0 2023-11-16T18:26:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.496+0000 7ff9577fe700 1 -- 172.21.15.2:0/329832635 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95801b6c0 con 0x7ff9601055c0 2023-11-16T18:26:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.496+0000 7ff9685b7700 1 -- 172.21.15.2:0/329832635 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff960110250 con 0x7ff9601055c0 2023-11-16T18:26:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.498+0000 7ff9577fe700 1 -- 172.21.15.2:0/329832635 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7ff9580051e0 con 0x7ff9601055c0 2023-11-16T18:26:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.499+0000 7ff9577fe700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff94c0a12c0 0x7ff94c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.499+0000 7ff966353700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff94c0a12c0 0x7ff94c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.499+0000 7ff9577fe700 1 -- 172.21.15.2:0/329832635 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff9580ce230 con 0x7ff9601055c0 2023-11-16T18:26:38.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.500+0000 7ff966353700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff94c0a12c0 0x7ff94c0a3770 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff95c009c60 tx=0x7ff95c009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:38.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.579+0000 7ff974743b80 1 Processor -- start 2023-11-16T18:26:38.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.580+0000 7ff974743b80 1 -- start start 2023-11-16T18:26:38.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.580+0000 7ff974743b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c946be0 0x55c73c946fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.580+0000 7ff974743b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 0x55c73c94b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.580+0000 7ff974743b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c94bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.580+0000 7ff974743b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c84c0e0 con 0x55c73c9474f0 2023-11-16T18:26:38.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.580+0000 7ff974743b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c89e920 con 0x55c73c946be0 2023-11-16T18:26:38.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.580+0000 7ff974743b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c897db0 con 0x55c73c94b930 2023-11-16T18:26:38.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.580+0000 7ff965b52700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 0x55c73c94b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.580+0000 7ff965b52700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 0x55c73c94b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51342/0 (socket says 172.21.15.2:51342) 2023-11-16T18:26:38.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.580+0000 7ff965b52700 1 -- 172.21.15.2:0/4121476317 learned_addr learned my addr 172.21.15.2:0/4121476317 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:38.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.580+0000 7ff966353700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c946be0 0x55c73c946fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.580+0000 7ff966b54700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c94bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.581+0000 7ff965b52700 1 -- 172.21.15.2:0/4121476317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 msgr2=0x55c73c94bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:38.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.581+0000 7ff965b52700 1 --2- 172.21.15.2:0/4121476317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c94bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.581+0000 7ff965b52700 1 -- 172.21.15.2:0/4121476317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c946be0 msgr2=0x55c73c946fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:38.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.581+0000 7ff965b52700 1 --2- 172.21.15.2:0/4121476317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c946be0 0x55c73c946fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.581+0000 7ff965b52700 1 -- 172.21.15.2:0/4121476317 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c73c7196b0 con 0x55c73c9474f0 2023-11-16T18:26:38.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.581+0000 7ff966b54700 1 --2- 172.21.15.2:0/4121476317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c94bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:38.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.581+0000 7ff966353700 1 --2- 172.21.15.2:0/4121476317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c946be0 0x55c73c946fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:38.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.581+0000 7ff965b52700 1 --2- 172.21.15.2:0/4121476317 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 0x55c73c94b3f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff950007fa0 tx=0x7ff9500106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4db3fe739175a3f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:38.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.581+0000 7ff954ff9700 1 -- 172.21.15.2:0/4121476317 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950004830 con 0x55c73c9474f0 2023-11-16T18:26:38.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.581+0000 7ff954ff9700 1 -- 172.21.15.2:0/4121476317 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff950015070 con 0x55c73c9474f0 2023-11-16T18:26:38.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.581+0000 7ff954ff9700 1 -- 172.21.15.2:0/4121476317 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950011640 con 0x55c73c9474f0 2023-11-16T18:26:38.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.582+0000 7ff974743b80 1 -- 172.21.15.2:0/4121476317 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 msgr2=0x55c73c94b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:38.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.582+0000 7ff974743b80 1 --2- 172.21.15.2:0/4121476317 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 0x55c73c94b3f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff950007fa0 tx=0x7ff9500106b0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.582+0000 7ff974743b80 1 -- 172.21.15.2:0/4121476317 shutdown_connections 2023-11-16T18:26:38.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.582+0000 7ff974743b80 1 --2- 172.21.15.2:0/4121476317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c94bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.582+0000 7ff974743b80 1 --2- 172.21.15.2:0/4121476317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c946be0 0x55c73c946fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.582+0000 7ff974743b80 1 --2- 172.21.15.2:0/4121476317 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 0x55c73c94b3f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.582+0000 7ff974743b80 1 -- 172.21.15.2:0/4121476317 >> 172.21.15.2:0/4121476317 conn(0x55c73c93c3e0 msgr2=0x55c73c941bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:38.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.582+0000 7ff974743b80 1 -- 172.21.15.2:0/4121476317 shutdown_connections 2023-11-16T18:26:38.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.582+0000 7ff974743b80 1 -- 172.21.15.2:0/4121476317 wait complete. 2023-11-16T18:26:38.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.583+0000 7ff974743b80 1 -- start start 2023-11-16T18:26:38.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.583+0000 7ff974743b80 1 Processor -- start 2023-11-16T18:26:38.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.583+0000 7ff974743b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c946be0 0x55c73c953100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.583+0000 7ff974743b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c9474f0 0x55c73c950310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.583+0000 7ff974743b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c950a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.583+0000 7ff974743b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c839960 con 0x55c73c946be0 2023-11-16T18:26:38.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.583+0000 7ff974743b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c839600 con 0x55c73c9474f0 2023-11-16T18:26:38.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.583+0000 7ff974743b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c71b9a0 con 0x55c73c94b930 2023-11-16T18:26:38.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.583+0000 7ff974743b80 10 client.0 _ll_register_callbacks cb 0x55c73c857800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:38.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.583+0000 7ff966353700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c946be0 0x55c73c953100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.583+0000 7ff966353700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c946be0 0x55c73c953100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51346/0 (socket says 172.21.15.2:51346) 2023-11-16T18:26:38.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.583+0000 7ff966353700 1 -- 172.21.15.2:0/2344846141 learned_addr learned my addr 172.21.15.2:0/2344846141 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:38.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.584+0000 7ff965b52700 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c9474f0 0x55c73c950310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.584+0000 7ff966b54700 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c950a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.584+0000 7ff966353700 1 -- 172.21.15.2:0/2344846141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 msgr2=0x55c73c950a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:38.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.584+0000 7ff966353700 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c950a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.584+0000 7ff966353700 1 -- 172.21.15.2:0/2344846141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c9474f0 msgr2=0x55c73c950310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:38.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.584+0000 7ff966353700 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c9474f0 0x55c73c950310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.584+0000 7ff966353700 1 -- 172.21.15.2:0/2344846141 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff950007bd0 con 0x55c73c946be0 2023-11-16T18:26:38.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.584+0000 7ff966b54700 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c950a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:38.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.584+0000 7ff966353700 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c946be0 0x55c73c953100 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff95c009b20 tx=0x7ff95c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6458c586eebedd33 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:38.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.584+0000 7ff949ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:38.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.584+0000 7ff949ffb700 1 -- 172.21.15.2:0/2344846141 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c022070 con 0x55c73c946be0 2023-11-16T18:26:38.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.584+0000 7ff949ffb700 1 -- 172.21.15.2:0/2344846141 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff95c01d720 con 0x55c73c946be0 2023-11-16T18:26:38.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.584+0000 7ff974743b80 1 -- 172.21.15.2:0/2344846141 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c73c939ba0 con 0x55c73c946be0 2023-11-16T18:26:38.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.585+0000 7ff949ffb700 1 -- 172.21.15.2:0/2344846141 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff95c01d8e0 con 0x55c73c946be0 2023-11-16T18:26:38.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.585+0000 7ff974743b80 10 client.16089 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:38.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.585+0000 7ff974743b80 1 -- 172.21.15.2:0/2344846141 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c73c85e210 con 0x55c73c946be0 2023-11-16T18:26:38.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.585+0000 7ff949ffb700 1 -- 172.21.15.2:0/2344846141 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c73c939ba0 con 0x55c73c946be0 2023-11-16T18:26:38.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.585+0000 7ff949ffb700 1 -- 172.21.15.2:0/2344846141 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff95c027c00 con 0x55c73c946be0 2023-11-16T18:26:38.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.585+0000 7ff974743b80 10 client.16089 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:38.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.585+0000 7ff974743b80 10 client.16089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:38.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.585+0000 7ff974743b80 20 client.16089 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:38.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.585+0000 7ff974743b80 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c73c964da0 0x55c73c9671b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.585+0000 7ff974743b80 4 client.16089 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:38.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.585+0000 7ff974743b80 1 -- 172.21.15.2:0/2344846141 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c73c85a120 con 0x55c73c964da0 2023-11-16T18:26:38.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.585+0000 7ff965b52700 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c73c964da0 0x55c73c9671b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.586+0000 7ff965b52700 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c73c964da0 0x55c73c9671b0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bdfb1d666ab7fee server_cookie=d3dc051ed43bc1c4 in_seq=0 out_seq=0 2023-11-16T18:26:38.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.586+0000 7ff949ffb700 10 client.16089 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:38.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff949ffb700 1 -- 172.21.15.2:0/2344846141 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c73c85a120 con 0x55c73c964da0 2023-11-16T18:26:38.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff949ffb700 10 client.16089 handle_command_reply: tid=0 2023-11-16T18:26:38.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff974743b80 1 client.16089 shutdown 2023-11-16T18:26:38.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff974743b80 1 -- 172.21.15.2:0/2344846141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c946be0 msgr2=0x55c73c953100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:38.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff974743b80 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c946be0 0x55c73c953100 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff95c009b20 tx=0x7ff95c0047c0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff974743b80 1 -- 172.21.15.2:0/2344846141 shutdown_connections 2023-11-16T18:26:38.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff974743b80 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c73c964da0 0x55c73c9671b0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff974743b80 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c950a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff949ffb700 0 client.16089 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:38.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff974743b80 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c946be0 0x55c73c953100 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff974743b80 1 --2- 172.21.15.2:0/2344846141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c9474f0 0x55c73c950310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff974743b80 1 -- 172.21.15.2:0/2344846141 >> 172.21.15.2:0/2344846141 conn(0x55c73c93c3e0 msgr2=0x55c73c940e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:38.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff974743b80 1 -- 172.21.15.2:0/2344846141 shutdown_connections 2023-11-16T18:26:38.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff974743b80 1 -- 172.21.15.2:0/2344846141 wait complete. 2023-11-16T18:26:38.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.587+0000 7ff974743b80 20 client.16089 trim_cache size 0 max 16384 2023-11-16T18:26:38.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.605+0000 7ff974743b80 1 Processor -- start 2023-11-16T18:26:38.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.605+0000 7ff974743b80 1 -- start start 2023-11-16T18:26:38.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.605+0000 7ff974743b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c946be0 0x55c73c952c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.605+0000 7ff974743b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c9474f0 0x55c73c95be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.605+0000 7ff974743b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c94b930 0x55c73c9415e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.605+0000 7ff974743b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c947c80 con 0x55c73c94b930 2023-11-16T18:26:38.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.605+0000 7ff974743b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c93bf60 con 0x55c73c9474f0 2023-11-16T18:26:38.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.605+0000 7ff974743b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c94a440 con 0x55c73c946be0 2023-11-16T18:26:38.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff966b54700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c94b930 0x55c73c9415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff966b54700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c94b930 0x55c73c9415e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51358/0 (socket says 172.21.15.2:51358) 2023-11-16T18:26:38.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff966b54700 1 -- 172.21.15.2:0/1194917395 learned_addr learned my addr 172.21.15.2:0/1194917395 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:38.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff965b52700 1 --2- 172.21.15.2:0/1194917395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c9474f0 0x55c73c95be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff966353700 1 --2- 172.21.15.2:0/1194917395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c946be0 0x55c73c952c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff966b54700 1 -- 172.21.15.2:0/1194917395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c946be0 msgr2=0x55c73c952c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:38.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff966b54700 1 --2- 172.21.15.2:0/1194917395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c946be0 0x55c73c952c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff966b54700 1 -- 172.21.15.2:0/1194917395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c9474f0 msgr2=0x55c73c95be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:38.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff966b54700 1 --2- 172.21.15.2:0/1194917395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c9474f0 0x55c73c95be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff966b54700 1 -- 172.21.15.2:0/1194917395 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c73c93db80 con 0x55c73c94b930 2023-11-16T18:26:38.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff966b54700 1 --2- 172.21.15.2:0/1194917395 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c94b930 0x55c73c9415e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff9580cb610 tx=0x7ff958009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d2d9304816f39c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:38.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff9557fa700 1 -- 172.21.15.2:0/1194917395 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95809afc0 con 0x55c73c94b930 2023-11-16T18:26:38.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff9557fa700 1 -- 172.21.15.2:0/1194917395 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff95809eff0 con 0x55c73c94b930 2023-11-16T18:26:38.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.606+0000 7ff9557fa700 1 -- 172.21.15.2:0/1194917395 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95809b480 con 0x55c73c94b930 2023-11-16T18:26:38.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.607+0000 7ff974743b80 1 -- 172.21.15.2:0/1194917395 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c94b930 msgr2=0x55c73c9415e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:38.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.607+0000 7ff974743b80 1 --2- 172.21.15.2:0/1194917395 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c94b930 0x55c73c9415e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff9580cb610 tx=0x7ff958009b80 comp rx=0 tx=0).stop 2023-11-16T18:26:38.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.607+0000 7ff974743b80 1 -- 172.21.15.2:0/1194917395 shutdown_connections 2023-11-16T18:26:38.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.607+0000 7ff974743b80 1 --2- 172.21.15.2:0/1194917395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c946be0 0x55c73c952c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.607+0000 7ff974743b80 1 --2- 172.21.15.2:0/1194917395 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c94b930 0x55c73c9415e0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.607+0000 7ff974743b80 1 --2- 172.21.15.2:0/1194917395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c9474f0 0x55c73c95be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.607+0000 7ff974743b80 1 -- 172.21.15.2:0/1194917395 >> 172.21.15.2:0/1194917395 conn(0x55c73c93c3e0 msgr2=0x55c73c94dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:38.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.607+0000 7ff974743b80 1 -- 172.21.15.2:0/1194917395 shutdown_connections 2023-11-16T18:26:38.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.607+0000 7ff974743b80 1 -- 172.21.15.2:0/1194917395 wait complete. 2023-11-16T18:26:38.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.607+0000 7ff974743b80 1 -- start start 2023-11-16T18:26:38.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.608+0000 7ff974743b80 1 Processor -- start 2023-11-16T18:26:38.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.608+0000 7ff974743b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c946be0 0x55c73c95c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.608+0000 7ff974743b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 0x55c73c958450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.608+0000 7ff974743b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c958b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.608+0000 7ff974743b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c9592a0 con 0x55c73c9474f0 2023-11-16T18:26:38.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.608+0000 7ff974743b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c9b8c60 con 0x55c73c946be0 2023-11-16T18:26:38.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.608+0000 7ff974743b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c9b8da0 con 0x55c73c94b930 2023-11-16T18:26:38.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.608+0000 7ff974743b80 10 client.0 _ll_register_callbacks cb 0x7ff95c0044c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:38.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.608+0000 7ff965b52700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 0x55c73c958450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.608+0000 7ff965b52700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 0x55c73c958450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51362/0 (socket says 172.21.15.2:51362) 2023-11-16T18:26:38.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.608+0000 7ff965b52700 1 -- 172.21.15.2:0/1480414161 learned_addr learned my addr 172.21.15.2:0/1480414161 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:38.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.608+0000 7ff966b54700 1 --2- 172.21.15.2:0/1480414161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c958b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.608+0000 7ff966353700 1 --2- 172.21.15.2:0/1480414161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c946be0 0x55c73c95c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.609+0000 7ff965b52700 1 -- 172.21.15.2:0/1480414161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 msgr2=0x55c73c958b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.609+0000 7ff965b52700 1 --2- 172.21.15.2:0/1480414161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c958b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.609+0000 7ff965b52700 1 -- 172.21.15.2:0/1480414161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c946be0 msgr2=0x55c73c95c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.609+0000 7ff965b52700 1 --2- 172.21.15.2:0/1480414161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c946be0 0x55c73c95c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.609+0000 7ff965b52700 1 -- 172.21.15.2:0/1480414161 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9601103e0 con 0x55c73c9474f0 2023-11-16T18:26:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.609+0000 7ff965b52700 1 --2- 172.21.15.2:0/1480414161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 0x55c73c958450 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff950007fa0 tx=0x7ff950007460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3e3d4fa3612bf1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.609+0000 7ff949ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.609+0000 7ff949ffb700 1 -- 172.21.15.2:0/1480414161 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950020040 con 0x55c73c9474f0 2023-11-16T18:26:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.609+0000 7ff949ffb700 1 -- 172.21.15.2:0/1480414161 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff9500050c0 con 0x55c73c9474f0 2023-11-16T18:26:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.609+0000 7ff974743b80 1 -- 172.21.15.2:0/1480414161 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff950015070 con 0x55c73c9474f0 2023-11-16T18:26:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.609+0000 7ff949ffb700 1 -- 172.21.15.2:0/1480414161 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff950027620 con 0x55c73c9474f0 2023-11-16T18:26:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.609+0000 7ff974743b80 10 client.16095 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.609+0000 7ff974743b80 1 -- 172.21.15.2:0/1480414161 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c73c6c5e60 con 0x55c73c9474f0 2023-11-16T18:26:38.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.610+0000 7ff949ffb700 1 -- 172.21.15.2:0/1480414161 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff950015070 con 0x55c73c9474f0 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.610+0000 7ff949ffb700 1 -- 172.21.15.2:0/1480414161 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff95001a4f0 con 0x55c73c9474f0 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.610+0000 7ff974743b80 10 client.16095 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.610+0000 7ff974743b80 10 client.16095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.610+0000 7ff974743b80 20 client.16095 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.610+0000 7ff974743b80 1 --2- 172.21.15.2:0/1480414161 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c73c9ba440 0x55c73c9bc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.610+0000 7ff974743b80 4 client.16095 mds_command: new command op to 24473 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.610+0000 7ff974743b80 1 -- 172.21.15.2:0/1480414161 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c73c8666e0 con 0x55c73c9ba440 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.610+0000 7ff966353700 1 --2- 172.21.15.2:0/1480414161 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c73c9ba440 0x55c73c9bc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.611+0000 7ff949ffb700 10 client.16095 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.611+0000 7ff966353700 1 --2- 172.21.15.2:0/1480414161 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c73c9ba440 0x55c73c9bc850 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbcba01ffbcf7712 server_cookie=8829d19f425d4c2c in_seq=0 out_seq=0 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.612+0000 7ff949ffb700 1 -- 172.21.15.2:0/1480414161 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c73c8666e0 con 0x55c73c9ba440 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.612+0000 7ff949ffb700 10 client.16095 handle_command_reply: tid=0 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.612+0000 7ff974743b80 1 client.16095 shutdown 2023-11-16T18:26:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.612+0000 7ff974743b80 1 -- 172.21.15.2:0/1480414161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 msgr2=0x55c73c958450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:38.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.612+0000 7ff974743b80 1 --2- 172.21.15.2:0/1480414161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 0x55c73c958450 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff950007fa0 tx=0x7ff950007460 comp rx=0 tx=0).stop 2023-11-16T18:26:38.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff974743b80 1 -- 172.21.15.2:0/1480414161 shutdown_connections 2023-11-16T18:26:38.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff974743b80 1 --2- 172.21.15.2:0/1480414161 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c73c9ba440 0x55c73c9bc850 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff974743b80 1 --2- 172.21.15.2:0/1480414161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c73c94b930 0x55c73c958b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff974743b80 1 --2- 172.21.15.2:0/1480414161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73c946be0 0x55c73c95c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff949ffb700 0 client.16095 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:38.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff974743b80 1 --2- 172.21.15.2:0/1480414161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c73c9474f0 0x55c73c958450 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff974743b80 1 -- 172.21.15.2:0/1480414161 >> 172.21.15.2:0/1480414161 conn(0x55c73c93c3e0 msgr2=0x55c73c94dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:38.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff974743b80 1 -- 172.21.15.2:0/1480414161 shutdown_connections 2023-11-16T18:26:38.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff974743b80 1 -- 172.21.15.2:0/1480414161 wait complete. 2023-11-16T18:26:38.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff974743b80 20 client.16095 trim_cache size 0 max 16384 2023-11-16T18:26:38.659 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:26:38.659 INFO:teuthology.orchestra.run.smithi002.stdout: "return_code": 0, 2023-11-16T18:26:38.659 INFO:teuthology.orchestra.run.smithi002.stdout: "scrub_tag": "e734baa5-bac3-42c1-b8cc-06ca75eb2e6b", 2023-11-16T18:26:38.659 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": "asynchronous" 2023-11-16T18:26:38.659 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:26:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff9685b7700 1 -- 172.21.15.2:0/329832635 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff94c0a12c0 msgr2=0x7ff94c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff9685b7700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff94c0a12c0 0x7ff94c0a3770 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff95c009c60 tx=0x7ff95c009260 comp rx=0 tx=0).stop 2023-11-16T18:26:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff9685b7700 1 -- 172.21.15.2:0/329832635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9601055c0 msgr2=0x7ff96019aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.613+0000 7ff9685b7700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9601055c0 0x7ff96019aeb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff958007870 tx=0x7ff9580078a0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.614+0000 7ff9685b7700 1 -- 172.21.15.2:0/329832635 shutdown_connections 2023-11-16T18:26:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.614+0000 7ff9685b7700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff94c0a12c0 0x7ff94c0a3770 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.614+0000 7ff9685b7700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff960100c50 0x7ff960196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.614+0000 7ff9685b7700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9601055c0 0x7ff96019aeb0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.614+0000 7ff9685b7700 1 --2- 172.21.15.2:0/329832635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9601015f0 0x7ff960196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.614+0000 7ff9685b7700 1 -- 172.21.15.2:0/329832635 >> 172.21.15.2:0/329832635 conn(0x7ff960077550 msgr2=0x7ff9600fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.614+0000 7ff9685b7700 1 -- 172.21.15.2:0/329832635 shutdown_connections 2023-11-16T18:26:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:38.614+0000 7ff9685b7700 1 -- 172.21.15.2:0/329832635 wait complete. 2023-11-16T18:26:38.681 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:26:38.933 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:38.933 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:39.037 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:39.038 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:39.049 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:39.049 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:39.050 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 79 -lt 100 ']' 2023-11-16T18:26:39.050 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 79 + 1 2023-11-16T18:26:39.071 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=80 2023-11-16T18:26:39.071 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:39.080 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:39.080 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:39.081 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:39.081 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:39.081 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:39.466 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:39.466 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:39.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:39.688 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:39.713 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:39.714 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:39.714 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 80 -lt 100 ']' 2023-11-16T18:26:39.714 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 80 + 1 2023-11-16T18:26:39.762 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=81 2023-11-16T18:26:39.763 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:39.820 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:39.820 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:39.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:39.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:39.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:40.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:40 smithi002 ceph-mon[98807]: scrub queued for path: / 2023-11-16T18:26:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:40 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:26:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:40 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:26:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:40 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:26:40.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:40 smithi002 ceph-mon[98807]: pgmap v57: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 26 KiB/s wr, 15 op/s 2023-11-16T18:26:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:40 smithi040 ceph-mon[100251]: scrub queued for path: / 2023-11-16T18:26:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:40 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:26:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:40 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:26:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:40 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:26:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:40 smithi040 ceph-mon[100251]: pgmap v57: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 26 KiB/s wr, 15 op/s 2023-11-16T18:26:40.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:40 smithi188 ceph-mon[100639]: scrub queued for path: / 2023-11-16T18:26:40.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:40 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:26:40.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:40 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:26:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:40 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:26:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:40 smithi188 ceph-mon[100639]: pgmap v57: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 26 KiB/s wr, 15 op/s 2023-11-16T18:26:40.682 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:40.683 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:40.946 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:40.946 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:40.971 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:40.971 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:40.971 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 81 -lt 100 ']' 2023-11-16T18:26:40.971 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 81 + 1 2023-11-16T18:26:41.046 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=82 2023-11-16T18:26:41.046 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:41.100 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:41.100 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:41.101 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:41.101 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:41.101 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.808+0000 7f65b9b73700 1 -- 172.21.15.2:0/1112888583 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65b410c8c0 msgr2=0x7f65b410cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.808+0000 7f65b9b73700 1 --2- 172.21.15.2:0/1112888583 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65b410c8c0 0x7f65b410cc90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f65a40099a0 tx=0x7f65a4009cb0 comp rx=0 tx=0).stop 2023-11-16T18:26:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.808+0000 7f65b9b73700 1 -- 172.21.15.2:0/1112888583 shutdown_connections 2023-11-16T18:26:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.808+0000 7f65b9b73700 1 --2- 172.21.15.2:0/1112888583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65b4107050 0x7f65b4113710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.808+0000 7f65b9b73700 1 --2- 172.21.15.2:0/1112888583 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65b410c8c0 0x7f65b410cc90 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.808+0000 7f65b9b73700 1 --2- 172.21.15.2:0/1112888583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65b410d260 0x7f65b4106920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.808+0000 7f65b9b73700 1 -- 172.21.15.2:0/1112888583 >> 172.21.15.2:0/1112888583 conn(0x7f65b4073840 msgr2=0x7f65b4075c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.808+0000 7f65b9b73700 1 -- 172.21.15.2:0/1112888583 shutdown_connections 2023-11-16T18:26:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.808+0000 7f65b9b73700 1 -- 172.21.15.2:0/1112888583 wait complete. 2023-11-16T18:26:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.830+0000 7f65b9b73700 1 Processor -- start 2023-11-16T18:26:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.831+0000 7f65b9b73700 1 -- start start 2023-11-16T18:26:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.831+0000 7f65b9b73700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65b4107050 0x7f65b41a2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.831+0000 7f65b9b73700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65b410c8c0 0x7f65b41a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.831+0000 7f65b9b73700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65b410d260 0x7f65b419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.831+0000 7f65b9b73700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b419ce70 con 0x7f65b410c8c0 2023-11-16T18:26:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.831+0000 7f65b9b73700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b419cfe0 con 0x7f65b4107050 2023-11-16T18:26:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.831+0000 7f65b9b73700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b419d1a0 con 0x7f65b410d260 2023-11-16T18:26:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.831+0000 7f65b2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65b410c8c0 0x7f65b41a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.831+0000 7f65b2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65b410c8c0 0x7f65b41a2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51376/0 (socket says 172.21.15.2:51376) 2023-11-16T18:26:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.831+0000 7f65b2ffd700 1 -- 172.21.15.2:0/2224045144 learned_addr learned my addr 172.21.15.2:0/2224045144 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.831+0000 7f65b3fff700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65b410d260 0x7f65b419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.831+0000 7f65b37fe700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65b4107050 0x7f65b41a2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.832+0000 7f65b2ffd700 1 -- 172.21.15.2:0/2224045144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65b410d260 msgr2=0x7f65b419c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.832+0000 7f65b2ffd700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65b410d260 0x7f65b419c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.832+0000 7f65b2ffd700 1 -- 172.21.15.2:0/2224045144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65b4107050 msgr2=0x7f65b41a2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.832+0000 7f65b2ffd700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65b4107050 0x7f65b41a2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.832+0000 7f65b2ffd700 1 -- 172.21.15.2:0/2224045144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a4009580 con 0x7f65b410c8c0 2023-11-16T18:26:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.832+0000 7f65b2ffd700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65b410c8c0 0x7f65b41a2cd0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f65a800e970 tx=0x7f65a800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.832+0000 7f65b0ff9700 1 -- 172.21.15.2:0/2224045144 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8009790 con 0x7f65b410c8c0 2023-11-16T18:26:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.832+0000 7f65b0ff9700 1 -- 172.21.15.2:0/2224045144 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65a800c7e0 con 0x7f65b410c8c0 2023-11-16T18:26:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.832+0000 7f65b9b73700 1 -- 172.21.15.2:0/2224045144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b41169d0 con 0x7f65b410c8c0 2023-11-16T18:26:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.832+0000 7f65b0ff9700 1 -- 172.21.15.2:0/2224045144 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a80054e0 con 0x7f65b410c8c0 2023-11-16T18:26:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.832+0000 7f65b9b73700 1 -- 172.21.15.2:0/2224045144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b4116840 con 0x7f65b410c8c0 2023-11-16T18:26:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.835+0000 7f65b0ff9700 1 -- 172.21.15.2:0/2224045144 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f65a800c950 con 0x7f65b410c8c0 2023-11-16T18:26:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.835+0000 7f65b0ff9700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f659c0a12c0 0x7f659c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.835+0000 7f65b0ff9700 1 -- 172.21.15.2:0/2224045144 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f65a8014070 con 0x7f65b410c8c0 2023-11-16T18:26:41.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.835+0000 7f65b37fe700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f659c0a12c0 0x7f659c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:41.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.836+0000 7f65b37fe700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f659c0a12c0 0x7f659c0a3770 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f65a400b2e0 tx=0x7f65a400b610 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:41.866 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:41.866 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:41.989 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.986+0000 7f65c5cffb80 1 Processor -- start 2023-11-16T18:26:41.989 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.986+0000 7f65c5cffb80 1 -- start start 2023-11-16T18:26:41.989 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.986+0000 7f65c5cffb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 0x560ef60aef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.986+0000 7f65c5cffb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60af470 0x560ef60b3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.986+0000 7f65c5cffb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65c5cffb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef5fb4060 con 0x560ef60aeb60 2023-11-16T18:26:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65c5cffb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef60068a0 con 0x560ef60af470 2023-11-16T18:26:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65c5cffb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef5fffd30 con 0x560ef60b38b0 2023-11-16T18:26:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65b37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 0x560ef60aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65b37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 0x560ef60aef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51382/0 (socket says 172.21.15.2:51382) 2023-11-16T18:26:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65b2ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60af470 0x560ef60b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65b37fe700 1 -- 172.21.15.2:0/2939552312 learned_addr learned my addr 172.21.15.2:0/2939552312 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65b3fff700 1 --2- 172.21.15.2:0/2939552312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65b37fe700 1 -- 172.21.15.2:0/2939552312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 msgr2=0x560ef60b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65b37fe700 1 --2- 172.21.15.2:0/2939552312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65b37fe700 1 -- 172.21.15.2:0/2939552312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60af470 msgr2=0x560ef60b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65b37fe700 1 --2- 172.21.15.2:0/2939552312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60af470 0x560ef60b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65b37fe700 1 -- 172.21.15.2:0/2939552312 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ef5e81630 con 0x560ef60aeb60 2023-11-16T18:26:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.987+0000 7f65b2ffd700 1 --2- 172.21.15.2:0/2939552312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60af470 0x560ef60b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:41.994 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.988+0000 7f65b37fe700 1 --2- 172.21.15.2:0/2939552312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 0x560ef60aef30 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f65a4015000 tx=0x7f65a4021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238e9256bc193846 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:41.994 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.988+0000 7f6599ffb700 1 -- 172.21.15.2:0/2939552312 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a402b070 con 0x560ef60aeb60 2023-11-16T18:26:41.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.988+0000 7f6599ffb700 1 -- 172.21.15.2:0/2939552312 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65a401f040 con 0x560ef60aeb60 2023-11-16T18:26:41.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.988+0000 7f6599ffb700 1 -- 172.21.15.2:0/2939552312 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a4018af0 con 0x560ef60aeb60 2023-11-16T18:26:41.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.989+0000 7f65c5cffb80 1 -- 172.21.15.2:0/2939552312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 msgr2=0x560ef60aef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:41.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.989+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/2939552312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 0x560ef60aef30 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f65a4015000 tx=0x7f65a4021000 comp rx=0 tx=0).stop 2023-11-16T18:26:41.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.989+0000 7f65c5cffb80 1 -- 172.21.15.2:0/2939552312 shutdown_connections 2023-11-16T18:26:41.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.989+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/2939552312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:41.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.989+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/2939552312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 0x560ef60aef30 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:41.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.989+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/2939552312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60af470 0x560ef60b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:41.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.989+0000 7f65c5cffb80 1 -- 172.21.15.2:0/2939552312 >> 172.21.15.2:0/2939552312 conn(0x560ef60a4360 msgr2=0x560ef60a9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:41.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.989+0000 7f65c5cffb80 1 -- 172.21.15.2:0/2939552312 shutdown_connections 2023-11-16T18:26:42.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.989+0000 7f65c5cffb80 1 -- 172.21.15.2:0/2939552312 wait complete. 2023-11-16T18:26:42.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.989+0000 7f65c5cffb80 1 -- start start 2023-11-16T18:26:42.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.990+0000 7f65c5cffb80 1 Processor -- start 2023-11-16T18:26:42.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.990+0000 7f65c5cffb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 0x560ef60bb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:42.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.990+0000 7f65c5cffb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60af470 0x560ef60b8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:42.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.990+0000 7f65c5cffb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:42.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.990+0000 7f65c5cffb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef5fa18e0 con 0x560ef60aeb60 2023-11-16T18:26:42.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65c5cffb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef5fa1580 con 0x560ef60af470 2023-11-16T18:26:42.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65c5cffb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef5e83920 con 0x560ef60b38b0 2023-11-16T18:26:42.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65c5cffb80 10 client.0 _ll_register_callbacks cb 0x560ef5fbf780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:42.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65b37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 0x560ef60bb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:42.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65b37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 0x560ef60bb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51388/0 (socket says 172.21.15.2:51388) 2023-11-16T18:26:42.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65b37fe700 1 -- 172.21.15.2:0/3234660773 learned_addr learned my addr 172.21.15.2:0/3234660773 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:42.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65b2ffd700 1 --2- 172.21.15.2:0/3234660773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60af470 0x560ef60b8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:42.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65b3fff700 1 --2- 172.21.15.2:0/3234660773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:42.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65b37fe700 1 -- 172.21.15.2:0/3234660773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 msgr2=0x560ef60b8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:42.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65b37fe700 1 --2- 172.21.15.2:0/3234660773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60b8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65b37fe700 1 -- 172.21.15.2:0/3234660773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60af470 msgr2=0x560ef60b8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65b37fe700 1 --2- 172.21.15.2:0/3234660773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60af470 0x560ef60b8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65b37fe700 1 -- 172.21.15.2:0/3234660773 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b4113f80 con 0x560ef60aeb60 2023-11-16T18:26:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f65b37fe700 1 --2- 172.21.15.2:0/3234660773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 0x560ef60bb080 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f65a40098c0 tx=0x7f65a4018450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d6eed32057be91 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.991+0000 7f6592ffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f6592ffd700 1 -- 172.21.15.2:0/3234660773 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a402b040 con 0x560ef60aeb60 2023-11-16T18:26:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f6592ffd700 1 -- 172.21.15.2:0/3234660773 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f65a400fc80 con 0x560ef60aeb60 2023-11-16T18:26:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f65c5cffb80 1 -- 172.21.15.2:0/3234660773 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560ef60a1b20 con 0x560ef60aeb60 2023-11-16T18:26:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f6592ffd700 1 -- 172.21.15.2:0/3234660773 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65a400fe40 con 0x560ef60aeb60 2023-11-16T18:26:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f65c5cffb80 10 client.16107 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f65c5cffb80 1 -- 172.21.15.2:0/3234660773 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560ef5fc6190 con 0x560ef60aeb60 2023-11-16T18:26:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f6592ffd700 1 -- 172.21.15.2:0/3234660773 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560ef60a1b20 con 0x560ef60aeb60 2023-11-16T18:26:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f6592ffd700 1 -- 172.21.15.2:0/3234660773 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f65a402f510 con 0x560ef60aeb60 2023-11-16T18:26:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f65c5cffb80 10 client.16107 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f65c5cffb80 10 client.16107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f65c5cffb80 20 client.16107 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/3234660773 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560ef60c8320 0x560ef60ca700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f65c5cffb80 4 client.16107 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.992+0000 7f65c5cffb80 1 -- 172.21.15.2:0/3234660773 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ef5fc20a0 con 0x560ef60c8320 2023-11-16T18:26:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.993+0000 7f65b2ffd700 1 --2- 172.21.15.2:0/3234660773 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560ef60c8320 0x560ef60ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:42.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.993+0000 7f65b2ffd700 1 --2- 172.21.15.2:0/3234660773 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560ef60c8320 0x560ef60ca700 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2238567ab8b4a55 server_cookie=b658219c4eaf94f8 in_seq=0 out_seq=0 2023-11-16T18:26:42.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.993+0000 7f6592ffd700 10 client.16107 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:42.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f6592ffd700 1 -- 172.21.15.2:0/3234660773 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560ef5fc20a0 con 0x560ef60c8320 2023-11-16T18:26:42.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f6592ffd700 10 client.16107 handle_command_reply: tid=0 2023-11-16T18:26:42.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f65c5cffb80 1 client.16107 shutdown 2023-11-16T18:26:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f65c5cffb80 1 -- 172.21.15.2:0/3234660773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 msgr2=0x560ef60bb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/3234660773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 0x560ef60bb080 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f65a40098c0 tx=0x7f65a4018450 comp rx=0 tx=0).stop 2023-11-16T18:26:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f65c5cffb80 1 -- 172.21.15.2:0/3234660773 shutdown_connections 2023-11-16T18:26:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/3234660773 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560ef60c8320 0x560ef60ca700 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/3234660773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60b8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f6592ffd700 0 client.16107 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/3234660773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60aeb60 0x560ef60bb080 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/3234660773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60af470 0x560ef60b8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f65c5cffb80 1 -- 172.21.15.2:0/3234660773 >> 172.21.15.2:0/3234660773 conn(0x560ef60a4360 msgr2=0x560ef60a8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f65c5cffb80 1 -- 172.21.15.2:0/3234660773 shutdown_connections 2023-11-16T18:26:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.994+0000 7f65c5cffb80 1 -- 172.21.15.2:0/3234660773 wait complete. 2023-11-16T18:26:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:41.995+0000 7f65c5cffb80 20 client.16107 trim_cache size 0 max 16384 2023-11-16T18:26:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.012+0000 7f65c5cffb80 1 Processor -- start 2023-11-16T18:26:42.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.012+0000 7f65c5cffb80 1 -- start start 2023-11-16T18:26:42.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.012+0000 7f65c5cffb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60aeb60 0x560ef60bf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:42.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.012+0000 7f65c5cffb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 0x560ef60c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:42.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.012+0000 7f65c5cffb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60babb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:42.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.012+0000 7f65c5cffb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef60b2550 con 0x560ef60af470 2023-11-16T18:26:42.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.012+0000 7f65c5cffb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef60a3ee0 con 0x560ef60aeb60 2023-11-16T18:26:42.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.012+0000 7f65c5cffb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef60b9020 con 0x560ef60b38b0 2023-11-16T18:26:42.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.012+0000 7f65b2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 0x560ef60c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:42.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.012+0000 7f65b37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60aeb60 0x560ef60bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:42.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.012+0000 7f65b2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 0x560ef60c4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51398/0 (socket says 172.21.15.2:51398) 2023-11-16T18:26:42.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.013+0000 7f65b2ffd700 1 -- 172.21.15.2:0/4286003113 learned_addr learned my addr 172.21.15.2:0/4286003113 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:42.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.013+0000 7f65b3fff700 1 --2- 172.21.15.2:0/4286003113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60babb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:42.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.013+0000 7f65b37fe700 1 -- 172.21.15.2:0/4286003113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 msgr2=0x560ef60babb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:42.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.013+0000 7f65b37fe700 1 --2- 172.21.15.2:0/4286003113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60babb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.013+0000 7f65b37fe700 1 -- 172.21.15.2:0/4286003113 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 msgr2=0x560ef60c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:42.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.013+0000 7f65b37fe700 1 --2- 172.21.15.2:0/4286003113 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 0x560ef60c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.013+0000 7f65b37fe700 1 -- 172.21.15.2:0/4286003113 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ef5fc5ee0 con 0x560ef60aeb60 2023-11-16T18:26:42.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.013+0000 7f65b2ffd700 1 --2- 172.21.15.2:0/4286003113 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 0x560ef60c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:42.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.013+0000 7f65b37fe700 1 --2- 172.21.15.2:0/4286003113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60aeb60 0x560ef60bf940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f65a40057f0 tx=0x7f65a400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc7d1f5d833ab80f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:42.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.013+0000 7f659a7fc700 1 -- 172.21.15.2:0/4286003113 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a402f610 con 0x560ef60aeb60 2023-11-16T18:26:42.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.013+0000 7f659a7fc700 1 -- 172.21.15.2:0/4286003113 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65a402fb40 con 0x560ef60aeb60 2023-11-16T18:26:42.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.013+0000 7f659a7fc700 1 -- 172.21.15.2:0/4286003113 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a40235f0 con 0x560ef60aeb60 2023-11-16T18:26:42.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.014+0000 7f65c5cffb80 1 -- 172.21.15.2:0/4286003113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60aeb60 msgr2=0x560ef60bf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:42.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.014+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/4286003113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60aeb60 0x560ef60bf940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f65a40057f0 tx=0x7f65a400bfd0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.014+0000 7f65c5cffb80 1 -- 172.21.15.2:0/4286003113 shutdown_connections 2023-11-16T18:26:42.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.014+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/4286003113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60babb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.014+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/4286003113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60aeb60 0x560ef60bf940 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.014+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/4286003113 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 0x560ef60c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.014+0000 7f65c5cffb80 1 -- 172.21.15.2:0/4286003113 >> 172.21.15.2:0/4286003113 conn(0x560ef60a4360 msgr2=0x560ef60b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:42.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.014+0000 7f65c5cffb80 1 -- 172.21.15.2:0/4286003113 shutdown_connections 2023-11-16T18:26:42.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.014+0000 7f65c5cffb80 1 -- 172.21.15.2:0/4286003113 wait complete. 2023-11-16T18:26:42.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.014+0000 7f65c5cffb80 1 -- start start 2023-11-16T18:26:42.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.015+0000 7f65c5cffb80 1 Processor -- start 2023-11-16T18:26:42.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.015+0000 7f65c5cffb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60aeb60 0x560ef60b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:42.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.015+0000 7f65c5cffb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 0x560ef60b7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:42.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.015+0000 7f65c5cffb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60b68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:42.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.015+0000 7f65c5cffb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef60b8590 con 0x560ef60af470 2023-11-16T18:26:42.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.015+0000 7f65c5cffb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef60b86d0 con 0x560ef60aeb60 2023-11-16T18:26:42.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.015+0000 7f65c5cffb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef60b6e40 con 0x560ef60b38b0 2023-11-16T18:26:42.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.015+0000 7f65c5cffb80 10 client.0 _ll_register_callbacks cb 0x7f65a8096340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:42.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.015+0000 7f65b2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 0x560ef60b7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:42.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.015+0000 7f65b2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 0x560ef60b7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51404/0 (socket says 172.21.15.2:51404) 2023-11-16T18:26:42.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.015+0000 7f65b2ffd700 1 -- 172.21.15.2:0/694918588 learned_addr learned my addr 172.21.15.2:0/694918588 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.015+0000 7f65b3fff700 1 --2- 172.21.15.2:0/694918588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60b68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.015+0000 7f65b37fe700 1 --2- 172.21.15.2:0/694918588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60aeb60 0x560ef60b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f65b2ffd700 1 -- 172.21.15.2:0/694918588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 msgr2=0x560ef60b68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f65b2ffd700 1 --2- 172.21.15.2:0/694918588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60b68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f65b2ffd700 1 -- 172.21.15.2:0/694918588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60aeb60 msgr2=0x560ef60b7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f65b2ffd700 1 --2- 172.21.15.2:0/694918588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60aeb60 0x560ef60b7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f65b2ffd700 1 -- 172.21.15.2:0/694918588 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b41169d0 con 0x560ef60af470 2023-11-16T18:26:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f65b2ffd700 1 --2- 172.21.15.2:0/694918588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 0x560ef60b7fc0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f65a8099d00 tx=0x7f65a8096470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd7767675ea57d9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f6592ffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f6592ffd700 1 -- 172.21.15.2:0/694918588 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a80a2450 con 0x560ef60af470 2023-11-16T18:26:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f6592ffd700 1 -- 172.21.15.2:0/694918588 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f65a80ca900 con 0x560ef60af470 2023-11-16T18:26:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f65c5cffb80 1 -- 172.21.15.2:0/694918588 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f65a401f040 con 0x560ef60af470 2023-11-16T18:26:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f6592ffd700 1 -- 172.21.15.2:0/694918588 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65a80cab00 con 0x560ef60af470 2023-11-16T18:26:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f65c5cffb80 10 client.16113 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f65c5cffb80 1 -- 172.21.15.2:0/694918588 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560ef5fc6190 con 0x560ef60af470 2023-11-16T18:26:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.016+0000 7f6592ffd700 1 -- 172.21.15.2:0/694918588 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65a401f040 con 0x560ef60af470 2023-11-16T18:26:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.017+0000 7f6592ffd700 1 -- 172.21.15.2:0/694918588 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f65a80994f0 con 0x560ef60af470 2023-11-16T18:26:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.017+0000 7f65c5cffb80 10 client.16113 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.017+0000 7f65c5cffb80 10 client.16113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.017+0000 7f65c5cffb80 20 client.16113 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.017+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/694918588 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560ef611cdb0 0x560ef611f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.017+0000 7f65c5cffb80 4 client.16113 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:26:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.017+0000 7f65c5cffb80 1 -- 172.21.15.2:0/694918588 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560ef5fce660 con 0x560ef611cdb0 2023-11-16T18:26:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.017+0000 7f65b37fe700 1 --2- 172.21.15.2:0/694918588 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560ef611cdb0 0x560ef611f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f65b37fe700 1 --2- 172.21.15.2:0/694918588 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560ef611cdb0 0x560ef611f1c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f50dd0c867d8c0f5 server_cookie=8acf987cdb0c6510 in_seq=0 out_seq=0 2023-11-16T18:26:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f6592ffd700 10 client.16113 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f6592ffd700 1 -- 172.21.15.2:0/694918588 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560ef5fce660 con 0x560ef611cdb0 2023-11-16T18:26:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f6592ffd700 10 client.16113 handle_command_reply: tid=0 2023-11-16T18:26:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f65c5cffb80 1 client.16113 shutdown 2023-11-16T18:26:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f65c5cffb80 1 -- 172.21.15.2:0/694918588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 msgr2=0x560ef60b7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/694918588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 0x560ef60b7fc0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f65a8099d00 tx=0x7f65a8096470 comp rx=0 tx=0).stop 2023-11-16T18:26:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f65c5cffb80 1 -- 172.21.15.2:0/694918588 shutdown_connections 2023-11-16T18:26:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/694918588 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x560ef611cdb0 0x560ef611f1c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/694918588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560ef60b38b0 0x560ef60b68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f6592ffd700 0 client.16113 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/694918588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ef60aeb60 0x560ef60b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f65c5cffb80 1 --2- 172.21.15.2:0/694918588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560ef60af470 0x560ef60b7fc0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f65c5cffb80 1 -- 172.21.15.2:0/694918588 >> 172.21.15.2:0/694918588 conn(0x560ef60a4360 msgr2=0x560ef60b2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:42.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f65c5cffb80 1 -- 172.21.15.2:0/694918588 shutdown_connections 2023-11-16T18:26:42.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f65c5cffb80 1 -- 172.21.15.2:0/694918588 wait complete. 2023-11-16T18:26:42.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.018+0000 7f65c5cffb80 20 client.16113 trim_cache size 0 max 16384 2023-11-16T18:26:42.028 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:26:42.028 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "no active scrubs running", 2023-11-16T18:26:42.028 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": {} 2023-11-16T18:26:42.028 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:26:42.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.019+0000 7f65b9b73700 1 -- 172.21.15.2:0/2224045144 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f659c0a12c0 msgr2=0x7f659c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:42.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.019+0000 7f65b9b73700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f659c0a12c0 0x7f659c0a3770 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f65a400b2e0 tx=0x7f65a400b610 comp rx=0 tx=0).stop 2023-11-16T18:26:42.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.019+0000 7f65b9b73700 1 -- 172.21.15.2:0/2224045144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65b410c8c0 msgr2=0x7f65b41a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:42.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.019+0000 7f65b9b73700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65b410c8c0 0x7f65b41a2cd0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f65a800e970 tx=0x7f65a800ec80 comp rx=0 tx=0).stop 2023-11-16T18:26:42.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.019+0000 7f65b9b73700 1 -- 172.21.15.2:0/2224045144 shutdown_connections 2023-11-16T18:26:42.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.019+0000 7f65b9b73700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f659c0a12c0 0x7f659c0a3770 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.019+0000 7f65b9b73700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65b410d260 0x7f65b419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.019+0000 7f65b9b73700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65b4107050 0x7f65b41a2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.019+0000 7f65b9b73700 1 --2- 172.21.15.2:0/2224045144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65b410c8c0 0x7f65b41a2cd0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:42.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.019+0000 7f65b9b73700 1 -- 172.21.15.2:0/2224045144 >> 172.21.15.2:0/2224045144 conn(0x7f65b4073840 msgr2=0x7f65b4074dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:42.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.019+0000 7f65b9b73700 1 -- 172.21.15.2:0/2224045144 shutdown_connections 2023-11-16T18:26:42.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:42.019+0000 7f65b9b73700 1 -- 172.21.15.2:0/2224045144 wait complete. 2023-11-16T18:26:42.072 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T18:26:42.072 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:26:42.103 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:42.104 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:42.128 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:42.128 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:42.129 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 82 -lt 100 ']' 2023-11-16T18:26:42.137 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 82 + 1 2023-11-16T18:26:42.165 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=83 2023-11-16T18:26:42.166 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:42.181 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:42.181 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:42.182 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:42.182 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:42.182 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:42.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:42 smithi002 ceph-mon[98807]: pgmap v58: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 26 KiB/s wr, 15 op/s 2023-11-16T18:26:42.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:42 smithi002 ceph-mon[98807]: scrub summary: idle 2023-11-16T18:26:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:42 smithi040 ceph-mon[100251]: pgmap v58: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 26 KiB/s wr, 15 op/s 2023-11-16T18:26:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:42 smithi040 ceph-mon[100251]: scrub summary: idle 2023-11-16T18:26:42.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:42 smithi188 ceph-mon[100639]: pgmap v58: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 26 KiB/s wr, 15 op/s 2023-11-16T18:26:42.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:42 smithi188 ceph-mon[100639]: scrub summary: idle 2023-11-16T18:26:42.557 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:42.557 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:42.613 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:42.613 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:42.660 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:42.660 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:42.660 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 83 -lt 100 ']' 2023-11-16T18:26:42.660 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 83 + 1 2023-11-16T18:26:42.716 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=84 2023-11-16T18:26:42.716 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:42.770 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:42.770 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:42.771 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:42.771 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:42.771 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:43.307 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:43.307 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:43.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:43.479 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:43.503 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:43.503 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:43.503 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 84 -lt 100 ']' 2023-11-16T18:26:43.504 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 84 + 1 2023-11-16T18:26:43.554 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=85 2023-11-16T18:26:43.554 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:43.616 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:43.617 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:43.617 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:43.618 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:43.618 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:44.225 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:44.226 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:44.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:44 smithi002 ceph-mon[98807]: pgmap v59: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 35 KiB/s wr, 20 op/s 2023-11-16T18:26:44.428 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:44.429 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:44.436 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:44.437 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:44.437 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 85 -lt 100 ']' 2023-11-16T18:26:44.437 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 85 + 1 2023-11-16T18:26:44.470 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=86 2023-11-16T18:26:44.470 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:44.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:44 smithi040 ceph-mon[100251]: pgmap v59: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 35 KiB/s wr, 20 op/s 2023-11-16T18:26:44.510 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:44.510 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:44.511 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:44.511 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:44.511 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:44.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:44 smithi188 ceph-mon[100639]: pgmap v59: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 35 KiB/s wr, 20 op/s 2023-11-16T18:26:44.993 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.991+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2515257173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52e80ffe30 msgr2=0x7f52e8103df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:44.994 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.991+0000 7f52ecb9b700 1 --2- 172.21.15.2:0/2515257173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52e80ffe30 0x7f52e8103df0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f52dc0099c0 tx=0x7f52dc009cd0 comp rx=0 tx=0).stop 2023-11-16T18:26:44.994 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.991+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2515257173 shutdown_connections 2023-11-16T18:26:44.994 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.991+0000 7f52ecb9b700 1 --2- 172.21.15.2:0/2515257173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52e81044d0 0x7f52e810c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:44.994 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.991+0000 7f52ecb9b700 1 --2- 172.21.15.2:0/2515257173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52e80ff490 0x7f52e80ff860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:44.994 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.991+0000 7f52ecb9b700 1 --2- 172.21.15.2:0/2515257173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52e80ffe30 0x7f52e8103df0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:44.994 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.991+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2515257173 >> 172.21.15.2:0/2515257173 conn(0x7f52e8076100 msgr2=0x7f52e8076500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:44.994 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.991+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2515257173 shutdown_connections 2023-11-16T18:26:44.994 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.991+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2515257173 wait complete. 2023-11-16T18:26:44.996 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.992+0000 7f52ecb9b700 1 Processor -- start 2023-11-16T18:26:44.996 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.992+0000 7f52ecb9b700 1 -- start start 2023-11-16T18:26:44.996 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52ecb9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52e80ff490 0x7f52e8100e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:44.996 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52ecb9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52e80ffe30 0x7f52e81013d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:44.996 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52ecb9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52e81044d0 0x7f52e81a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:44.996 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52ecb9b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e8101c00 con 0x7f52e80ffe30 2023-11-16T18:26:44.996 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52ecb9b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e81a4ba0 con 0x7f52e81044d0 2023-11-16T18:26:44.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52ecb9b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e81a4ce0 con 0x7f52e80ff490 2023-11-16T18:26:44.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52e5d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52e80ffe30 0x7f52e81013d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:44.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52e5d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52e80ffe30 0x7f52e81013d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51418/0 (socket says 172.21.15.2:51418) 2023-11-16T18:26:44.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52e5d9b700 1 -- 172.21.15.2:0/2259110304 learned_addr learned my addr 172.21.15.2:0/2259110304 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:44.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52e6d9d700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52e81044d0 0x7f52e81a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:44.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52e659c700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52e80ff490 0x7f52e8100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:44.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52e5d9b700 1 -- 172.21.15.2:0/2259110304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52e80ff490 msgr2=0x7f52e8100e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:44.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52e5d9b700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52e80ff490 0x7f52e8100e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:44.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52e5d9b700 1 -- 172.21.15.2:0/2259110304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52e81044d0 msgr2=0x7f52e81a4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:44.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52e5d9b700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52e81044d0 0x7f52e81a4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:44.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52e5d9b700 1 -- 172.21.15.2:0/2259110304 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc0095f0 con 0x7f52e80ffe30 2023-11-16T18:26:44.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.993+0000 7f52e5d9b700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52e80ffe30 0x7f52e81013d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009fd0 tx=0x7f52dc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:44.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.994+0000 7f52d77fe700 1 -- 172.21.15.2:0/2259110304 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc01d070 con 0x7f52e80ffe30 2023-11-16T18:26:44.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.994+0000 7f52d77fe700 1 -- 172.21.15.2:0/2259110304 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52dc005020 con 0x7f52e80ffe30 2023-11-16T18:26:44.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.994+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2259110304 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e810f250 con 0x7f52e80ffe30 2023-11-16T18:26:44.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.994+0000 7f52d77fe700 1 -- 172.21.15.2:0/2259110304 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc004a90 con 0x7f52e80ffe30 2023-11-16T18:26:44.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.994+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2259110304 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e810f0c0 con 0x7f52e80ffe30 2023-11-16T18:26:44.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.995+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2259110304 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e810f3e0 con 0x7f52e80ffe30 2023-11-16T18:26:44.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.996+0000 7f52d77fe700 1 -- 172.21.15.2:0/2259110304 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f52dc0054b0 con 0x7f52e80ffe30 2023-11-16T18:26:44.999 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.997+0000 7f52d77fe700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f52d00a12c0 0x7f52d00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:44.999 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.997+0000 7f52d77fe700 1 -- 172.21.15.2:0/2259110304 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f52dc0cdcd0 con 0x7f52e80ffe30 2023-11-16T18:26:44.999 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.997+0000 7f52e659c700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f52d00a12c0 0x7f52d00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:45.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:44.998+0000 7f52e659c700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f52d00a12c0 0x7f52d00a3770 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009c60 tx=0x7f52d8009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:45.003 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.001+0000 7f52d77fe700 1 -- 172.21.15.2:0/2259110304 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f52dc096e00 con 0x7f52e80ffe30 2023-11-16T18:26:45.090 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:45.090 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:45.264 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.262+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2259110304 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52e8067260 con 0x7f52e80ffe30 2023-11-16T18:26:45.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.263+0000 7f52d77fe700 1 -- 172.21.15.2:0/2259110304 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10774 (secure 0 0 0) 0x7f52dc09a420 con 0x7f52e80ffe30 2023-11-16T18:26:45.265 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:26:45.266 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:23:07.909835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:26:45.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.265+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2259110304 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f52d00a12c0 msgr2=0x7f52d00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:45.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.265+0000 7f52ecb9b700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f52d00a12c0 0x7f52d00a3770 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009c60 tx=0x7f52d8009260 comp rx=0 tx=0).stop 2023-11-16T18:26:45.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.265+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2259110304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52e80ffe30 msgr2=0x7f52e81013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:45.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.265+0000 7f52ecb9b700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52e80ffe30 0x7f52e81013d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009fd0 tx=0x7f52dc009580 comp rx=0 tx=0).stop 2023-11-16T18:26:45.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.265+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2259110304 shutdown_connections 2023-11-16T18:26:45.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.265+0000 7f52ecb9b700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f52d00a12c0 0x7f52d00a3770 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:45.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.265+0000 7f52ecb9b700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52e80ff490 0x7f52e8100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:45.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.265+0000 7f52ecb9b700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52e81044d0 0x7f52e81a4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:45.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.265+0000 7f52ecb9b700 1 --2- 172.21.15.2:0/2259110304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52e80ffe30 0x7f52e81013d0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:45.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.265+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2259110304 >> 172.21.15.2:0/2259110304 conn(0x7f52e8076100 msgr2=0x7f52e8068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:45.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.265+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2259110304 shutdown_connections 2023-11-16T18:26:45.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:45.265+0000 7f52ecb9b700 1 -- 172.21.15.2:0/2259110304 wait complete. 2023-11-16T18:26:45.269 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T18:26:45.328 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:45.328 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:45.346 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-11-16T18:26:45.352 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:45.352 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:45.352 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 86 -lt 100 ']' 2023-11-16T18:26:45.352 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 86 + 1 2023-11-16T18:26:45.387 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=87 2023-11-16T18:26:45.387 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:45.422 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:45.422 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:45.422 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:45.422 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:45.423 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:45.940 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:45.940 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:46.008 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:46.008 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:46.027 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:46.028 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:46.028 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 87 -lt 100 ']' 2023-11-16T18:26:46.028 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 87 + 1 2023-11-16T18:26:46.070 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=88 2023-11-16T18:26:46.070 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:46.134 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:46.134 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:46.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:46.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:46.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:46.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:46 smithi002 ceph-mon[98807]: pgmap v60: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 17 op/s 2023-11-16T18:26:46.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:46 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2259110304' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:26:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:46 smithi040 ceph-mon[100251]: pgmap v60: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 17 op/s 2023-11-16T18:26:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:46 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2259110304' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:26:46.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:46 smithi188 ceph-mon[100639]: pgmap v60: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 17 op/s 2023-11-16T18:26:46.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:46 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2259110304' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:26:46.598 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:46.598 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:46.791 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:46.792 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:46.795 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:46.795 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:46.795 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 88 -lt 100 ']' 2023-11-16T18:26:46.796 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 88 + 1 2023-11-16T18:26:46.828 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=89 2023-11-16T18:26:46.828 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:46.893 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:46.893 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:46.894 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:46.894 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:46.894 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:47.540 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:47.540 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:47.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:47 smithi188 ceph-mon[100639]: pgmap v61: 129 pgs: 129 active+clean; 2.7 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 21 op/s 2023-11-16T18:26:47.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:47 smithi002 ceph-mon[98807]: pgmap v61: 129 pgs: 129 active+clean; 2.7 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 21 op/s 2023-11-16T18:26:47.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:47 smithi040 ceph-mon[100251]: pgmap v61: 129 pgs: 129 active+clean; 2.7 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 21 op/s 2023-11-16T18:26:47.736 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:47.736 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:47.745 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:47.746 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:47.746 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 89 -lt 100 ']' 2023-11-16T18:26:47.746 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 89 + 1 2023-11-16T18:26:47.811 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=90 2023-11-16T18:26:47.811 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:47.920 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:47.920 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:47.921 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:47.921 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:47.921 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:48.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.007+0000 7ff208787700 1 -- 172.21.15.2:0/3767894300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2000ffa70 msgr2=0x7ff2000ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:48.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.007+0000 7ff208787700 1 --2- 172.21.15.2:0/3767894300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2000ffa70 0x7ff2000ffee0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0009990 tx=0x7ff1f0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.007+0000 7ff208787700 1 -- 172.21.15.2:0/3767894300 shutdown_connections 2023-11-16T18:26:48.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.007+0000 7ff208787700 1 --2- 172.21.15.2:0/3767894300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff200100420 0x7ff20010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.007+0000 7ff208787700 1 --2- 172.21.15.2:0/3767894300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff200105250 0x7ff200105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.007+0000 7ff208787700 1 --2- 172.21.15.2:0/3767894300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2000ffa70 0x7ff2000ffee0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.007+0000 7ff208787700 1 -- 172.21.15.2:0/3767894300 >> 172.21.15.2:0/3767894300 conn(0x7ff2000763a0 msgr2=0x7ff2000767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:48.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.007+0000 7ff208787700 1 -- 172.21.15.2:0/3767894300 shutdown_connections 2023-11-16T18:26:48.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.007+0000 7ff208787700 1 -- 172.21.15.2:0/3767894300 wait complete. 2023-11-16T18:26:48.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.007+0000 7ff208787700 1 Processor -- start 2023-11-16T18:26:48.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.007+0000 7ff208787700 1 -- start start 2023-11-16T18:26:48.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff208787700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2000ffa70 0x7ff200197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff208787700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff200100420 0x7ff200197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff208787700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff200105250 0x7ff2001a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff208787700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff200198070 con 0x7ff2000ffa70 2023-11-16T18:26:48.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff208787700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2001a27e0 con 0x7ff200105250 2023-11-16T18:26:48.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff208787700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2001a2950 con 0x7ff200100420 2023-11-16T18:26:48.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff206523700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2000ffa70 0x7ff200197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff206523700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2000ffa70 0x7ff200197220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51430/0 (socket says 172.21.15.2:51430) 2023-11-16T18:26:48.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff206523700 1 -- 172.21.15.2:0/3132414203 learned_addr learned my addr 172.21.15.2:0/3132414203 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:48.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff205d22700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff200100420 0x7ff200197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff206d24700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff200105250 0x7ff2001a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff206523700 1 -- 172.21.15.2:0/3132414203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff200100420 msgr2=0x7ff200197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:48.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff206523700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff200100420 0x7ff200197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff206523700 1 -- 172.21.15.2:0/3132414203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff200105250 msgr2=0x7ff2001a22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:48.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff206523700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff200105250 0x7ff2001a22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff206523700 1 -- 172.21.15.2:0/3132414203 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f00095f0 con 0x7ff2000ffa70 2023-11-16T18:26:48.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff205d22700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff200100420 0x7ff200197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:48.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff206523700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2000ffa70 0x7ff200197220 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00e9a0 tx=0x7ff1fc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:48.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.008+0000 7ff1f77fe700 1 -- 172.21.15.2:0/3132414203 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc009790 con 0x7ff2000ffa70 2023-11-16T18:26:48.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.009+0000 7ff1f77fe700 1 -- 172.21.15.2:0/3132414203 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1fc00cbd0 con 0x7ff2000ffa70 2023-11-16T18:26:48.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.009+0000 7ff208787700 1 -- 172.21.15.2:0/3132414203 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20010f1e0 con 0x7ff2000ffa70 2023-11-16T18:26:48.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.009+0000 7ff1f77fe700 1 -- 172.21.15.2:0/3132414203 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc018680 con 0x7ff2000ffa70 2023-11-16T18:26:48.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.009+0000 7ff208787700 1 -- 172.21.15.2:0/3132414203 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20010f050 con 0x7ff2000ffa70 2023-11-16T18:26:48.014 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.011+0000 7ff1f77fe700 1 -- 172.21.15.2:0/3132414203 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7ff1fc00cd40 con 0x7ff2000ffa70 2023-11-16T18:26:48.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.011+0000 7ff1f77fe700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff1ec0a1340 0x7ff1ec0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.012+0000 7ff1f77fe700 1 -- 172.21.15.2:0/3132414203 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff1fc014070 con 0x7ff2000ffa70 2023-11-16T18:26:48.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.012+0000 7ff205d22700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff1ec0a1340 0x7ff1ec0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.012+0000 7ff205d22700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff1ec0a1340 0x7ff1ec0a37f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000b2e0 tx=0x7ff1f0005fb0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:48.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.089+0000 7ff214913b80 1 Processor -- start 2023-11-16T18:26:48.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff214913b80 1 -- start start 2023-11-16T18:26:48.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff214913b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 0x565227474f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff214913b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x565227479370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff214913b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652274798b0 0x565227479d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff214913b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56522737a060 con 0x565227474b60 2023-11-16T18:26:48.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff214913b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652273cc8a0 con 0x565227475470 2023-11-16T18:26:48.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff214913b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652273c5d30 con 0x5652274798b0 2023-11-16T18:26:48.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff206523700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 0x565227474f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff206523700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 0x565227474f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51438/0 (socket says 172.21.15.2:51438) 2023-11-16T18:26:48.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff206523700 1 -- 172.21.15.2:0/3720804455 learned_addr learned my addr 172.21.15.2:0/3720804455 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:48.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff205d22700 1 --2- 172.21.15.2:0/3720804455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x565227479370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff206d24700 1 --2- 172.21.15.2:0/3720804455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652274798b0 0x565227479d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff206523700 1 -- 172.21.15.2:0/3720804455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652274798b0 msgr2=0x565227479d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:48.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff206523700 1 --2- 172.21.15.2:0/3720804455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652274798b0 0x565227479d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.090+0000 7ff206523700 1 -- 172.21.15.2:0/3720804455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 msgr2=0x565227479370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:48.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.091+0000 7ff206523700 1 --2- 172.21.15.2:0/3720804455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x565227479370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.091+0000 7ff206523700 1 -- 172.21.15.2:0/3720804455 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565227247630 con 0x565227474b60 2023-11-16T18:26:48.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.091+0000 7ff205d22700 1 --2- 172.21.15.2:0/3720804455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x565227479370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:26:48.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.091+0000 7ff206523700 1 --2- 172.21.15.2:0/3720804455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 0x565227474f30 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0a1fc0 tx=0x7ff1fc09a120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc3c02851ee092e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:48.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.091+0000 7ff1f4ff9700 1 -- 172.21.15.2:0/3720804455 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc0cd170 con 0x565227474b60 2023-11-16T18:26:48.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.091+0000 7ff1f4ff9700 1 -- 172.21.15.2:0/3720804455 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1fc09a300 con 0x565227474b60 2023-11-16T18:26:48.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.091+0000 7ff1f4ff9700 1 -- 172.21.15.2:0/3720804455 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc0a22c0 con 0x565227474b60 2023-11-16T18:26:48.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.092+0000 7ff214913b80 1 -- 172.21.15.2:0/3720804455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 msgr2=0x565227474f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:48.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.092+0000 7ff214913b80 1 --2- 172.21.15.2:0/3720804455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 0x565227474f30 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0a1fc0 tx=0x7ff1fc09a120 comp rx=0 tx=0).stop 2023-11-16T18:26:48.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.092+0000 7ff214913b80 1 -- 172.21.15.2:0/3720804455 shutdown_connections 2023-11-16T18:26:48.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.092+0000 7ff214913b80 1 --2- 172.21.15.2:0/3720804455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652274798b0 0x565227479d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.092+0000 7ff214913b80 1 --2- 172.21.15.2:0/3720804455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 0x565227474f30 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.092+0000 7ff214913b80 1 --2- 172.21.15.2:0/3720804455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x565227479370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.092+0000 7ff214913b80 1 -- 172.21.15.2:0/3720804455 >> 172.21.15.2:0/3720804455 conn(0x56522746a360 msgr2=0x56522746fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:48.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.092+0000 7ff214913b80 1 -- 172.21.15.2:0/3720804455 shutdown_connections 2023-11-16T18:26:48.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.092+0000 7ff214913b80 1 -- 172.21.15.2:0/3720804455 wait complete. 2023-11-16T18:26:48.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.092+0000 7ff214913b80 1 -- start start 2023-11-16T18:26:48.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.092+0000 7ff214913b80 1 Processor -- start 2023-11-16T18:26:48.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff214913b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227474b60 0x5652274810c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff214913b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x56522747e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff214913b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 0x56522747ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff214913b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652273678e0 con 0x5652274798b0 2023-11-16T18:26:48.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff214913b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x565227367580 con 0x565227475470 2023-11-16T18:26:48.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff214913b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x565227249920 con 0x565227474b60 2023-11-16T18:26:48.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff214913b80 10 client.0 _ll_register_callbacks cb 0x565227385780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:48.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff206d24700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 0x56522747ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff206d24700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 0x56522747ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51448/0 (socket says 172.21.15.2:51448) 2023-11-16T18:26:48.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff206d24700 1 -- 172.21.15.2:0/2051501566 learned_addr learned my addr 172.21.15.2:0/2051501566 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:48.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff205d22700 1 --2- 172.21.15.2:0/2051501566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x56522747e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff206523700 1 --2- 172.21.15.2:0/2051501566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227474b60 0x5652274810c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff206d24700 1 -- 172.21.15.2:0/2051501566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227474b60 msgr2=0x5652274810c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:48.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff206d24700 1 --2- 172.21.15.2:0/2051501566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227474b60 0x5652274810c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff206d24700 1 -- 172.21.15.2:0/2051501566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 msgr2=0x56522747e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:48.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff206d24700 1 --2- 172.21.15.2:0/2051501566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x56522747e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.093+0000 7ff206d24700 1 -- 172.21.15.2:0/2051501566 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20010f050 con 0x5652274798b0 2023-11-16T18:26:48.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff206d24700 1 --2- 172.21.15.2:0/2051501566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 0x56522747ea30 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8007940 tx=0x7ff1f8007d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a14a68d8332b8912 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:48.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff1e9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:48.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff1e9ffb700 1 -- 172.21.15.2:0/2051501566 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8010860 con 0x5652274798b0 2023-11-16T18:26:48.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff1e9ffb700 1 -- 172.21.15.2:0/2051501566 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff1f80109c0 con 0x5652274798b0 2023-11-16T18:26:48.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff214913b80 1 -- 172.21.15.2:0/2051501566 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565227467b20 con 0x5652274798b0 2023-11-16T18:26:48.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff1e9ffb700 1 -- 172.21.15.2:0/2051501566 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1f8010b80 con 0x5652274798b0 2023-11-16T18:26:48.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff214913b80 10 client.16131 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:48.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff214913b80 1 -- 172.21.15.2:0/2051501566 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56522738c190 con 0x5652274798b0 2023-11-16T18:26:48.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff1e9ffb700 1 -- 172.21.15.2:0/2051501566 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x565227467b20 con 0x5652274798b0 2023-11-16T18:26:48.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff1e9ffb700 1 -- 172.21.15.2:0/2051501566 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff1f801d3a0 con 0x5652274798b0 2023-11-16T18:26:48.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff214913b80 10 client.16131 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:48.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff214913b80 10 client.16131 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2023-11-16T18:26:48.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff214913b80 20 client.16131 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:48.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff214913b80 1 --2- 172.21.15.2:0/2051501566 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x56522748e970 0x565227490d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff214913b80 4 client.16131 mds_command: new command op to 24446 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:48.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.094+0000 7ff214913b80 1 -- 172.21.15.2:0/2051501566 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652273880a0 con 0x56522748e970 2023-11-16T18:26:48.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.095+0000 7ff206523700 1 --2- 172.21.15.2:0/2051501566 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x56522748e970 0x565227490d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.095+0000 7ff1e9ffb700 10 client.16131 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:26:48.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.095+0000 7ff206523700 1 --2- 172.21.15.2:0/2051501566 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x56522748e970 0x565227490d80 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f113ef55e33ffa48 server_cookie=adf2f2ca9a7a8da7 in_seq=0 out_seq=0 2023-11-16T18:26:48.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff1e9ffb700 1 -- 172.21.15.2:0/2051501566 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5652273880a0 con 0x56522748e970 2023-11-16T18:26:48.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff1e9ffb700 10 client.16131 handle_command_reply: tid=0 2023-11-16T18:26:48.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff214913b80 1 client.16131 shutdown 2023-11-16T18:26:48.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff214913b80 1 -- 172.21.15.2:0/2051501566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 msgr2=0x56522747ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:48.106 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff214913b80 1 --2- 172.21.15.2:0/2051501566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 0x56522747ea30 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8007940 tx=0x7ff1f8007d00 comp rx=0 tx=0).stop 2023-11-16T18:26:48.106 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff214913b80 1 -- 172.21.15.2:0/2051501566 shutdown_connections 2023-11-16T18:26:48.106 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff214913b80 1 --2- 172.21.15.2:0/2051501566 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x56522748e970 0x565227490d80 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.106 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff214913b80 1 --2- 172.21.15.2:0/2051501566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227474b60 0x5652274810c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.106 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff214913b80 1 --2- 172.21.15.2:0/2051501566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 0x56522747ea30 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.106 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff1e9ffb700 0 client.16131 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:26:48.106 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff214913b80 1 --2- 172.21.15.2:0/2051501566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x56522747e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.106 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff214913b80 1 -- 172.21.15.2:0/2051501566 >> 172.21.15.2:0/2051501566 conn(0x56522746a360 msgr2=0x56522747ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:48.106 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff214913b80 1 -- 172.21.15.2:0/2051501566 shutdown_connections 2023-11-16T18:26:48.106 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff214913b80 1 -- 172.21.15.2:0/2051501566 wait complete. 2023-11-16T18:26:48.106 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.096+0000 7ff214913b80 20 client.16131 trim_cache size 0 max 16384 2023-11-16T18:26:48.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.110+0000 7ff214913b80 1 Processor -- start 2023-11-16T18:26:48.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.110+0000 7ff214913b80 1 -- start start 2023-11-16T18:26:48.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.110+0000 7ff214913b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227474b60 0x565227477930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.110+0000 7ff214913b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x565227477e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.110+0000 7ff214913b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 0x56522747dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.110+0000 7ff214913b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x565227475bc0 con 0x5652274798b0 2023-11-16T18:26:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.110+0000 7ff214913b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x565227469ee0 con 0x565227475470 2023-11-16T18:26:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.110+0000 7ff214913b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56522747f010 con 0x565227474b60 2023-11-16T18:26:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.110+0000 7ff206d24700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 0x56522747dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.110+0000 7ff206d24700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 0x56522747dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51456/0 (socket says 172.21.15.2:51456) 2023-11-16T18:26:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.110+0000 7ff206d24700 1 -- 172.21.15.2:0/3112534193 learned_addr learned my addr 172.21.15.2:0/3112534193 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.110+0000 7ff206523700 1 --2- 172.21.15.2:0/3112534193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227474b60 0x565227477930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.110+0000 7ff205d22700 1 --2- 172.21.15.2:0/3112534193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x565227477e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.111+0000 7ff206d24700 1 -- 172.21.15.2:0/3112534193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227474b60 msgr2=0x565227477930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.111+0000 7ff206d24700 1 --2- 172.21.15.2:0/3112534193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227474b60 0x565227477930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.111+0000 7ff206d24700 1 -- 172.21.15.2:0/3112534193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 msgr2=0x565227477e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.111+0000 7ff206d24700 1 --2- 172.21.15.2:0/3112534193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x565227477e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.111+0000 7ff206d24700 1 -- 172.21.15.2:0/3112534193 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565227479f50 con 0x5652274798b0 2023-11-16T18:26:48.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.111+0000 7ff205d22700 1 --2- 172.21.15.2:0/3112534193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x565227477e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:48.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.111+0000 7ff206d24700 1 --2- 172.21.15.2:0/3112534193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 0x56522747dc30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff1f801d800 tx=0x7ff1f80093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e62df592114700 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:48.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.111+0000 7ff1f57fa700 1 -- 172.21.15.2:0/3112534193 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8010440 con 0x5652274798b0 2023-11-16T18:26:48.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.111+0000 7ff1f57fa700 1 -- 172.21.15.2:0/3112534193 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1f801d3a0 con 0x5652274798b0 2023-11-16T18:26:48.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.111+0000 7ff1f57fa700 1 -- 172.21.15.2:0/3112534193 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f801dcb0 con 0x5652274798b0 2023-11-16T18:26:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.112+0000 7ff214913b80 1 -- 172.21.15.2:0/3112534193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 msgr2=0x56522747dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.112+0000 7ff214913b80 1 --2- 172.21.15.2:0/3112534193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 0x56522747dc30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff1f801d800 tx=0x7ff1f80093f0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.112+0000 7ff214913b80 1 -- 172.21.15.2:0/3112534193 shutdown_connections 2023-11-16T18:26:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.112+0000 7ff214913b80 1 --2- 172.21.15.2:0/3112534193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227474b60 0x565227477930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.112+0000 7ff214913b80 1 --2- 172.21.15.2:0/3112534193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652274798b0 0x56522747dc30 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.112+0000 7ff214913b80 1 --2- 172.21.15.2:0/3112534193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565227475470 0x565227477e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.112+0000 7ff214913b80 1 -- 172.21.15.2:0/3112534193 >> 172.21.15.2:0/3112534193 conn(0x56522746a360 msgr2=0x56522748a150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.112+0000 7ff214913b80 1 -- 172.21.15.2:0/3112534193 shutdown_connections 2023-11-16T18:26:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.112+0000 7ff214913b80 1 -- 172.21.15.2:0/3112534193 wait complete. 2023-11-16T18:26:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.112+0000 7ff214913b80 1 -- start start 2023-11-16T18:26:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.112+0000 7ff214913b80 1 Processor -- start 2023-11-16T18:26:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.112+0000 7ff214913b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 0x565227483990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff214913b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227475470 0x565227483ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff214913b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652274798b0 0x565227484410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff214913b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652274e1890 con 0x565227474b60 2023-11-16T18:26:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff214913b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652274e19d0 con 0x5652274798b0 2023-11-16T18:26:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff214913b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652274e1b90 con 0x565227475470 2023-11-16T18:26:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff214913b80 10 client.0 _ll_register_callbacks cb 0x7ff1f8007500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff206523700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 0x565227483990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff206523700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 0x565227483990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51458/0 (socket says 172.21.15.2:51458) 2023-11-16T18:26:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff206523700 1 -- 172.21.15.2:0/501678156 learned_addr learned my addr 172.21.15.2:0/501678156 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff206d24700 1 --2- 172.21.15.2:0/501678156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652274798b0 0x565227484410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff205d22700 1 --2- 172.21.15.2:0/501678156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227475470 0x565227483ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff206523700 1 -- 172.21.15.2:0/501678156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227475470 msgr2=0x565227483ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff206523700 1 --2- 172.21.15.2:0/501678156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227475470 0x565227483ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff206523700 1 -- 172.21.15.2:0/501678156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652274798b0 msgr2=0x565227484410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff206523700 1 --2- 172.21.15.2:0/501678156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652274798b0 0x565227484410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff206523700 1 -- 172.21.15.2:0/501678156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56522738c190 con 0x565227474b60 2023-11-16T18:26:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff206523700 1 --2- 172.21.15.2:0/501678156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 0x565227483990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc096b10 tx=0x7ff1fc00efa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95c75b1c4f4250eb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff1e9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff1e9ffb700 1 -- 172.21.15.2:0/501678156 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc099790 con 0x565227474b60 2023-11-16T18:26:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff1e9ffb700 1 -- 172.21.15.2:0/501678156 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff1fc0cd1f0 con 0x565227474b60 2023-11-16T18:26:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.113+0000 7ff214913b80 1 -- 172.21.15.2:0/501678156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff1fc09a300 con 0x565227474b60 2023-11-16T18:26:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.114+0000 7ff1e9ffb700 1 -- 172.21.15.2:0/501678156 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1fc0b82b0 con 0x565227474b60 2023-11-16T18:26:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.114+0000 7ff214913b80 10 client.16137 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.114+0000 7ff214913b80 1 -- 172.21.15.2:0/501678156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5652271f3de0 con 0x565227474b60 2023-11-16T18:26:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.114+0000 7ff1e9ffb700 1 -- 172.21.15.2:0/501678156 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1fc0ca080 con 0x565227474b60 2023-11-16T18:26:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.114+0000 7ff1e9ffb700 1 -- 172.21.15.2:0/501678156 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff1fc0ca380 con 0x565227474b60 2023-11-16T18:26:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.114+0000 7ff214913b80 10 client.16137 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.114+0000 7ff214913b80 10 client.16137 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2023-11-16T18:26:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.114+0000 7ff214913b80 20 client.16137 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.114+0000 7ff214913b80 1 --2- 172.21.15.2:0/501678156 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x5652274e2e60 0x5652274e5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.114+0000 7ff214913b80 4 client.16137 mds_command: new command op to 24446 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:26:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.114+0000 7ff214913b80 1 -- 172.21.15.2:0/501678156 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x565227394660 con 0x5652274e2e60 2023-11-16T18:26:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.114+0000 7ff205d22700 1 --2- 172.21.15.2:0/501678156 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x5652274e2e60 0x5652274e5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff1e9ffb700 10 client.16137 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:26:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff205d22700 1 --2- 172.21.15.2:0/501678156 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x5652274e2e60 0x5652274e5240 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=41f6c1eed69264b3 server_cookie=331fe161978cc26c in_seq=0 out_seq=0 2023-11-16T18:26:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff1e9ffb700 1 -- 172.21.15.2:0/501678156 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x565227394660 con 0x5652274e2e60 2023-11-16T18:26:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff1e9ffb700 10 client.16137 handle_command_reply: tid=0 2023-11-16T18:26:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff214913b80 1 client.16137 shutdown 2023-11-16T18:26:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff214913b80 1 -- 172.21.15.2:0/501678156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 msgr2=0x565227483990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff214913b80 1 --2- 172.21.15.2:0/501678156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 0x565227483990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc096b10 tx=0x7ff1fc00efa0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff214913b80 1 -- 172.21.15.2:0/501678156 shutdown_connections 2023-11-16T18:26:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff214913b80 1 --2- 172.21.15.2:0/501678156 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x5652274e2e60 0x5652274e5240 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff214913b80 1 --2- 172.21.15.2:0/501678156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652274798b0 0x565227484410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff1e9ffb700 0 client.16137 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:26:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff214913b80 1 --2- 172.21.15.2:0/501678156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565227474b60 0x565227483990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff214913b80 1 --2- 172.21.15.2:0/501678156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565227475470 0x565227483ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff214913b80 1 -- 172.21.15.2:0/501678156 >> 172.21.15.2:0/501678156 conn(0x56522746a360 msgr2=0x565227477930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff214913b80 1 -- 172.21.15.2:0/501678156 shutdown_connections 2023-11-16T18:26:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.115+0000 7ff214913b80 1 -- 172.21.15.2:0/501678156 wait complete. 2023-11-16T18:26:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.116+0000 7ff214913b80 20 client.16137 trim_cache size 0 max 16384 2023-11-16T18:26:48.126 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:26:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.116+0000 7ff208787700 1 -- 172.21.15.2:0/3132414203 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff1ec0a1340 msgr2=0x7ff1ec0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.116+0000 7ff208787700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff1ec0a1340 0x7ff1ec0a37f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000b2e0 tx=0x7ff1f0005fb0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.116+0000 7ff208787700 1 -- 172.21.15.2:0/3132414203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2000ffa70 msgr2=0x7ff200197220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.116+0000 7ff208787700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2000ffa70 0x7ff200197220 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00e9a0 tx=0x7ff1fc00ed60 comp rx=0 tx=0).stop 2023-11-16T18:26:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.116+0000 7ff208787700 1 -- 172.21.15.2:0/3132414203 shutdown_connections 2023-11-16T18:26:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.116+0000 7ff208787700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff1ec0a1340 0x7ff1ec0a37f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.116+0000 7ff208787700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff200105250 0x7ff2001a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.116+0000 7ff208787700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2000ffa70 0x7ff200197220 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.116+0000 7ff208787700 1 --2- 172.21.15.2:0/3132414203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff200100420 0x7ff200197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.116+0000 7ff208787700 1 -- 172.21.15.2:0/3132414203 >> 172.21.15.2:0/3132414203 conn(0x7ff2000763a0 msgr2=0x7ff2000fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.116+0000 7ff208787700 1 -- 172.21.15.2:0/3132414203 shutdown_connections 2023-11-16T18:26:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:48.116+0000 7ff208787700 1 -- 172.21.15.2:0/3132414203 wait complete. 2023-11-16T18:26:48.221 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-11-16T18:26:48.431 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:48.431 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:48.487 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:48.487 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:48.509 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:48.509 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:48.509 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 90 -lt 100 ']' 2023-11-16T18:26:48.509 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 90 + 1 2023-11-16T18:26:48.536 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=91 2023-11-16T18:26:48.536 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:48.553 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:48.553 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:48.553 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:48.554 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:48.554 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:49.073 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:49.073 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:49.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:49.128 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:49.169 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:49.169 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:49.169 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 91 -lt 100 ']' 2023-11-16T18:26:49.169 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 91 + 1 2023-11-16T18:26:49.219 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=92 2023-11-16T18:26:49.220 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:49.267 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:49.268 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:49.268 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:49.268 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:49.269 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:49.848 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:49.848 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:50.028 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:50.028 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:50.034 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:50.034 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:50.034 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 92 -lt 100 ']' 2023-11-16T18:26:50.034 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 92 + 1 2023-11-16T18:26:50.094 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=93 2023-11-16T18:26:50.094 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:50.128 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:50.128 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:50.129 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:50.129 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:50.129 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:50.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:50 smithi002 ceph-mon[98807]: pgmap v62: 129 pgs: 129 active+clean; 2.7 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-11-16T18:26:50.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:50 smithi040 ceph-mon[100251]: pgmap v62: 129 pgs: 129 active+clean; 2.7 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-11-16T18:26:50.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:50 smithi188 ceph-mon[100639]: pgmap v62: 129 pgs: 129 active+clean; 2.7 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-11-16T18:26:50.723 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:50.723 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:50.969 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:50.969 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:50.994 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:50.994 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:50.994 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 93 -lt 100 ']' 2023-11-16T18:26:50.994 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 93 + 1 2023-11-16T18:26:51.019 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=94 2023-11-16T18:26:51.019 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:51.087 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:51.087 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:51.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:51.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:51.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:51.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.365+0000 7fa73bf5c700 1 -- 172.21.15.2:0/575383933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa734104140 msgr2=0x7fa734104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.365+0000 7fa73bf5c700 1 --2- 172.21.15.2:0/575383933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa734104140 0x7fa734104510 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fa7300099c0 tx=0x7fa730009cd0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.365+0000 7fa73bf5c700 1 -- 172.21.15.2:0/575383933 shutdown_connections 2023-11-16T18:26:51.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.365+0000 7fa73bf5c700 1 --2- 172.21.15.2:0/575383933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7341002a0 0x7fa73410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.365+0000 7fa73bf5c700 1 --2- 172.21.15.2:0/575383933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa734104140 0x7fa734104510 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.365+0000 7fa73bf5c700 1 --2- 172.21.15.2:0/575383933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7340ff8f0 0x7fa7340ffd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.365+0000 7fa73bf5c700 1 -- 172.21.15.2:0/575383933 >> 172.21.15.2:0/575383933 conn(0x7fa734075880 msgr2=0x7fa734075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:51.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.365+0000 7f03c0418700 1 -- 172.21.15.2:0/3070485777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03b8104400 msgr2=0x7f03b810bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.365+0000 7f03c0418700 1 --2- 172.21.15.2:0/3070485777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03b8104400 0x7f03b810bfd0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f03b00099c0 tx=0x7f03b0009cd0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.365+0000 7fa73bf5c700 1 -- 172.21.15.2:0/575383933 shutdown_connections 2023-11-16T18:26:51.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.365+0000 7fa73bf5c700 1 -- 172.21.15.2:0/575383933 wait complete. 2023-11-16T18:26:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.365+0000 7f03c0418700 1 -- 172.21.15.2:0/3070485777 shutdown_connections 2023-11-16T18:26:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.365+0000 7f03c0418700 1 --2- 172.21.15.2:0/3070485777 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b80ff3c0 0x7f03b80ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.365+0000 7f03c0418700 1 --2- 172.21.15.2:0/3070485777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03b8104400 0x7f03b810bfd0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.365+0000 7f03c0418700 1 --2- 172.21.15.2:0/3070485777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b80ffd60 0x7f03b8103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.365+0000 7f03c0418700 1 -- 172.21.15.2:0/3070485777 >> 172.21.15.2:0/3070485777 conn(0x7f03b80763a0 msgr2=0x7f03b80767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.365+0000 7f03c0418700 1 -- 172.21.15.2:0/3070485777 shutdown_connections 2023-11-16T18:26:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.365+0000 7f03c0418700 1 -- 172.21.15.2:0/3070485777 wait complete. 2023-11-16T18:26:51.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa73bf5c700 1 Processor -- start 2023-11-16T18:26:51.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa73bf5c700 1 -- start start 2023-11-16T18:26:51.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03c0418700 1 Processor -- start 2023-11-16T18:26:51.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03c0418700 1 -- start start 2023-11-16T18:26:51.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa73bf5c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7340ff8f0 0x7fa734195180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa73bf5c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7341002a0 0x7fa7341956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa73bf5c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa734104140 0x7fa734199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa73bf5c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73419a160 con 0x7fa7341002a0 2023-11-16T18:26:51.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa73bf5c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73419a2d0 con 0x7fa734104140 2023-11-16T18:26:51.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa73bf5c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73419a490 con 0x7fa7340ff8f0 2023-11-16T18:26:51.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa7394f7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7341002a0 0x7fa7341956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa7394f7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7341002a0 0x7fa7341956c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51478/0 (socket says 172.21.15.2:51478) 2023-11-16T18:26:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03c0418700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b80ff3c0 0x7f03b8194150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03c0418700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03b80ffd60 0x7f03b8194690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa7394f7700 1 -- 172.21.15.2:0/697298872 learned_addr learned my addr 172.21.15.2:0/697298872 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03c0418700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b8104400 0x7f03b8198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03c0418700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b8199130 con 0x7f03b80ffd60 2023-11-16T18:26:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03c0418700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b81992a0 con 0x7f03b8104400 2023-11-16T18:26:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03c0418700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b8199460 con 0x7f03b80ff3c0 2023-11-16T18:26:51.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa73a4f9700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa734104140 0x7fa734199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa739cf8700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7340ff8f0 0x7fa734195180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03bd9b3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03b80ffd60 0x7f03b8194690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03be1b4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b80ff3c0 0x7f03b8194150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03be1b4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b80ff3c0 0x7f03b8194150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43012/0 (socket says 172.21.15.2:43012) 2023-11-16T18:26:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03be9b5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b8104400 0x7f03b8198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03be1b4700 1 -- 172.21.15.2:0/2725798252 learned_addr learned my addr 172.21.15.2:0/2725798252 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.366+0000 7f03bd9b3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03b80ffd60 0x7f03b8194690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51484/0 (socket says 172.21.15.2:51484) 2023-11-16T18:26:51.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa7394f7700 1 -- 172.21.15.2:0/697298872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7340ff8f0 msgr2=0x7fa734195180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa7394f7700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7340ff8f0 0x7fa734195180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa7394f7700 1 -- 172.21.15.2:0/697298872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa734104140 msgr2=0x7fa734199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.366+0000 7fa7394f7700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa734104140 0x7fa734199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.367+0000 7fa7394f7700 1 -- 172.21.15.2:0/697298872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7300095f0 con 0x7fa7341002a0 2023-11-16T18:26:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.367+0000 7f03be1b4700 1 -- 172.21.15.2:0/2725798252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b8104400 msgr2=0x7f03b8198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.367+0000 7f03be1b4700 1 --2- 172.21.15.2:0/2725798252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b8104400 0x7f03b8198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.367+0000 7f03be1b4700 1 -- 172.21.15.2:0/2725798252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03b80ffd60 msgr2=0x7f03b8194690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.367+0000 7f03be1b4700 1 --2- 172.21.15.2:0/2725798252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03b80ffd60 0x7f03b8194690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.367+0000 7f03be1b4700 1 -- 172.21.15.2:0/2725798252 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b00095f0 con 0x7f03b80ff3c0 2023-11-16T18:26:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.367+0000 7f03bd9b3700 1 --2- 172.21.15.2:0/2725798252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03b80ffd60 0x7f03b8194690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:51.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.367+0000 7fa7394f7700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7341002a0 0x7fa7341956c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa72400eab0 tx=0x7fa72400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:51.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.367+0000 7fa72affd700 1 -- 172.21.15.2:0/697298872 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa724009790 con 0x7fa7341002a0 2023-11-16T18:26:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.367+0000 7f03be1b4700 1 --2- 172.21.15.2:0/2725798252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b80ff3c0 0x7f03b8194150 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f03b4009b00 tx=0x7f03b4009e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:51.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.367+0000 7fa72affd700 1 -- 172.21.15.2:0/697298872 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa72400ee90 con 0x7fa7341002a0 2023-11-16T18:26:51.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.367+0000 7fa72affd700 1 -- 172.21.15.2:0/697298872 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72401d660 con 0x7fa7341002a0 2023-11-16T18:26:51.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.367+0000 7fa73bf5c700 1 -- 172.21.15.2:0/697298872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa73410f1b0 con 0x7fa7341002a0 2023-11-16T18:26:51.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.367+0000 7fa73bf5c700 1 -- 172.21.15.2:0/697298872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa73410f020 con 0x7fa7341002a0 2023-11-16T18:26:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.367+0000 7f03af7fe700 1 -- 172.21.15.2:0/2725798252 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b4010070 con 0x7f03b80ff3c0 2023-11-16T18:26:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.367+0000 7f03af7fe700 1 -- 172.21.15.2:0/2725798252 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03b40090d0 con 0x7f03b80ff3c0 2023-11-16T18:26:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.367+0000 7f03c0418700 1 -- 172.21.15.2:0/2725798252 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b810e180 con 0x7f03b80ff3c0 2023-11-16T18:26:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.367+0000 7f03af7fe700 1 -- 172.21.15.2:0/2725798252 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b4004280 con 0x7f03b80ff3c0 2023-11-16T18:26:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.367+0000 7f03c0418700 1 -- 172.21.15.2:0/2725798252 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b810dff0 con 0x7f03b80ff3c0 2023-11-16T18:26:51.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.370+0000 7fa72affd700 1 -- 172.21.15.2:0/697298872 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fa72400bbc0 con 0x7fa7341002a0 2023-11-16T18:26:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.370+0000 7f03af7fe700 1 -- 172.21.15.2:0/2725798252 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f03b4014070 con 0x7f03b80ff3c0 2023-11-16T18:26:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.370+0000 7f03af7fe700 1 --2- 172.21.15.2:0/2725798252 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f03a40a12c0 0x7f03a40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.370+0000 7fa72affd700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa7200a1340 0x7fa7200a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.370+0000 7f03af7fe700 1 -- 172.21.15.2:0/2725798252 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f03b40cc300 con 0x7f03b80ff3c0 2023-11-16T18:26:51.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.370+0000 7fa739cf8700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa7200a1340 0x7fa7200a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.370+0000 7fa72affd700 1 -- 172.21.15.2:0/697298872 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa7240d10e0 con 0x7fa7341002a0 2023-11-16T18:26:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.370+0000 7f03bd9b3700 1 --2- 172.21.15.2:0/2725798252 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f03a40a12c0 0x7f03a40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.371+0000 7fa739cf8700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa7200a1340 0x7fa7200a37f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa730009990 tx=0x7fa730005fb0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.371+0000 7f03bd9b3700 1 --2- 172.21.15.2:0/2725798252 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f03a40a12c0 0x7f03a40a3770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03a800ffd0 tx=0x7f03a8011040 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:51.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.426+0000 7fa7480e8b80 1 Processor -- start 2023-11-16T18:26:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.426+0000 7f03cc5a4b80 1 Processor -- start 2023-11-16T18:26:51.428 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.426+0000 7f03cc5a4b80 1 -- start start 2023-11-16T18:26:51.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.426+0000 7fa7480e8b80 1 -- start start 2023-11-16T18:26:51.429 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03cc5a4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 0x556b6a70e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.429 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03cc5a4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70e720 0x556b6a708560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7480e8b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916126f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.430 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03cc5a4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70eb90 0x556b6a708b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.430 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03cc5a4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6a60e000 con 0x556b6a70e720 2023-11-16T18:26:51.430 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03cc5a4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6a660840 con 0x556b6a5646b0 2023-11-16T18:26:51.430 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03cc5a4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6a659cd0 con 0x556b6a70eb90 2023-11-16T18:26:51.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7480e8b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557916127470 0x55791612b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7480e8b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55791612b8b0 0x55791612bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7480e8b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55791602c060 con 0x557916127470 2023-11-16T18:26:51.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7480e8b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55791607e8a0 con 0x557916126b60 2023-11-16T18:26:51.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7480e8b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557916077d30 con 0x55791612b8b0 2023-11-16T18:26:51.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03bd9b3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70e720 0x556b6a708560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7394f7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557916127470 0x55791612b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7394f7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557916127470 0x55791612b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51506/0 (socket says 172.21.15.2:51506) 2023-11-16T18:26:51.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7394f7700 1 -- 172.21.15.2:0/4115629888 learned_addr learned my addr 172.21.15.2:0/4115629888 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:51.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa739cf8700 1 --2- 172.21.15.2:0/4115629888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916126f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03bd9b3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70e720 0x556b6a708560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51500/0 (socket says 172.21.15.2:51500) 2023-11-16T18:26:51.433 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03bd9b3700 1 -- 172.21.15.2:0/3433689901 learned_addr learned my addr 172.21.15.2:0/3433689901 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:51.433 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03be9b5700 1 --2- 172.21.15.2:0/3433689901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70eb90 0x556b6a708b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.433 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03be1b4700 1 --2- 172.21.15.2:0/3433689901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 0x556b6a70e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa73a4f9700 1 --2- 172.21.15.2:0/4115629888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55791612b8b0 0x55791612bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.433 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03be9b5700 1 -- 172.21.15.2:0/3433689901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 msgr2=0x556b6a70e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.433 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03be9b5700 1 --2- 172.21.15.2:0/3433689901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 0x556b6a70e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7394f7700 1 -- 172.21.15.2:0/4115629888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55791612b8b0 msgr2=0x55791612bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7394f7700 1 --2- 172.21.15.2:0/4115629888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55791612b8b0 0x55791612bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7394f7700 1 -- 172.21.15.2:0/4115629888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 msgr2=0x557916126f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7394f7700 1 --2- 172.21.15.2:0/4115629888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916126f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.427+0000 7fa7394f7700 1 -- 172.21.15.2:0/4115629888 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557915ef9630 con 0x557916127470 2023-11-16T18:26:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03be9b5700 1 -- 172.21.15.2:0/3433689901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70e720 msgr2=0x556b6a708560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03be9b5700 1 --2- 172.21.15.2:0/3433689901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70e720 0x556b6a708560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03be9b5700 1 -- 172.21.15.2:0/3433689901 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b6a4db5d0 con 0x556b6a70eb90 2023-11-16T18:26:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.427+0000 7f03bd9b3700 1 --2- 172.21.15.2:0/3433689901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70e720 0x556b6a708560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:51.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.428+0000 7fa7394f7700 1 --2- 172.21.15.2:0/4115629888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557916127470 0x55791612b370 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fa72400be70 tx=0x7fa72409d7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11394a9b4be7852 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.428+0000 7f03be9b5700 1 --2- 172.21.15.2:0/3433689901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70eb90 0x556b6a708b00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f03b00098e0 tx=0x7f03b0005c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2de236ba0565b221 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:51.436 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.428+0000 7fa71ffff700 1 -- 172.21.15.2:0/4115629888 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7240a6290 con 0x557916127470 2023-11-16T18:26:51.436 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.428+0000 7fa71ffff700 1 -- 172.21.15.2:0/4115629888 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa72409a550 con 0x557916127470 2023-11-16T18:26:51.436 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.428+0000 7fa71ffff700 1 -- 172.21.15.2:0/4115629888 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72409d590 con 0x557916127470 2023-11-16T18:26:51.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.428+0000 7f03acdac700 1 -- 172.21.15.2:0/3433689901 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b000b990 con 0x556b6a70eb90 2023-11-16T18:26:51.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.428+0000 7f03acdac700 1 -- 172.21.15.2:0/3433689901 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03b001e040 con 0x556b6a70eb90 2023-11-16T18:26:51.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.428+0000 7f03acdac700 1 -- 172.21.15.2:0/3433689901 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b000fdc0 con 0x556b6a70eb90 2023-11-16T18:26:51.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.429+0000 7fa7480e8b80 1 -- 172.21.15.2:0/4115629888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557916127470 msgr2=0x55791612b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.429+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/4115629888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557916127470 0x55791612b370 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fa72400be70 tx=0x7fa72409d7a0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.429+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3433689901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70eb90 msgr2=0x556b6a708b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.429+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3433689901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70eb90 0x556b6a708b00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f03b00098e0 tx=0x7f03b0005c70 comp rx=0 tx=0).stop 2023-11-16T18:26:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.429+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3433689901 shutdown_connections 2023-11-16T18:26:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.429+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3433689901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70eb90 0x556b6a708b00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.429+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3433689901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 0x556b6a70e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.429+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3433689901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70e720 0x556b6a708560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.429+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3433689901 >> 172.21.15.2:0/3433689901 conn(0x556b6a6fdf50 msgr2=0x556b6a703270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:51.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.429+0000 7fa7480e8b80 1 -- 172.21.15.2:0/4115629888 shutdown_connections 2023-11-16T18:26:51.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.429+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/4115629888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55791612b8b0 0x55791612bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.429+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/4115629888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916126f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.429+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/4115629888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557916127470 0x55791612b370 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.429+0000 7fa7480e8b80 1 -- 172.21.15.2:0/4115629888 >> 172.21.15.2:0/4115629888 conn(0x55791611c360 msgr2=0x557916121b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.429+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3433689901 shutdown_connections 2023-11-16T18:26:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.429+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3433689901 wait complete. 2023-11-16T18:26:51.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.429+0000 7fa7480e8b80 1 -- 172.21.15.2:0/4115629888 shutdown_connections 2023-11-16T18:26:51.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.429+0000 7fa7480e8b80 1 -- 172.21.15.2:0/4115629888 wait complete. 2023-11-16T18:26:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03cc5a4b80 1 -- start start 2023-11-16T18:26:51.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.430+0000 7fa7480e8b80 1 -- start start 2023-11-16T18:26:51.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.430+0000 7fa7480e8b80 1 Processor -- start 2023-11-16T18:26:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03cc5a4b80 1 Processor -- start 2023-11-16T18:26:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03cc5a4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 0x556b6a714a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03cc5a4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70e720 0x556b6a711c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03cc5a4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70eb90 0x556b6a712250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03cc5a4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6a5fb880 con 0x556b6a70e720 2023-11-16T18:26:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03cc5a4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6a5fb520 con 0x556b6a5646b0 2023-11-16T18:26:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03cc5a4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6a4dd8c0 con 0x556b6a70eb90 2023-11-16T18:26:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03cc5a4b80 10 client.0 _ll_register_callbacks cb 0x556b6a619720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:51.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.430+0000 7fa7480e8b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916130340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.430+0000 7fa7480e8b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557916127470 0x557916130880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.430+0000 7fa7480e8b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55791612b8b0 0x557916130dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.430+0000 7fa7480e8b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579160198e0 con 0x557916127470 2023-11-16T18:26:51.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.430+0000 7fa7480e8b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557916019580 con 0x557916126b60 2023-11-16T18:26:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03bd9b3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70e720 0x556b6a711c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.430+0000 7fa7480e8b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557915efb920 con 0x55791612b8b0 2023-11-16T18:26:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03be1b4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 0x556b6a714a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03bd9b3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70e720 0x556b6a711c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51510/0 (socket says 172.21.15.2:51510) 2023-11-16T18:26:51.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.430+0000 7fa7480e8b80 10 client.0 _ll_register_callbacks cb 0x557916037780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03bd9b3700 1 -- 172.21.15.2:0/3069408716 learned_addr learned my addr 172.21.15.2:0/3069408716 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.430+0000 7f03be9b5700 1 --2- 172.21.15.2:0/3069408716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70eb90 0x556b6a712250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.430+0000 7fa7394f7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557916127470 0x557916130880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.430+0000 7fa739cf8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916130340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa739cf8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916130340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:41952/0 (socket says 172.21.15.2:41952) 2023-11-16T18:26:51.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa739cf8700 1 -- 172.21.15.2:0/3152642073 learned_addr learned my addr 172.21.15.2:0/3152642073 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.431+0000 7f03be1b4700 1 -- 172.21.15.2:0/3069408716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70eb90 msgr2=0x556b6a712250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.431+0000 7f03be1b4700 1 --2- 172.21.15.2:0/3069408716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70eb90 0x556b6a712250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.431+0000 7f03be1b4700 1 -- 172.21.15.2:0/3069408716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70e720 msgr2=0x556b6a711c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.431+0000 7f03be1b4700 1 --2- 172.21.15.2:0/3069408716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70e720 0x556b6a711c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.431+0000 7f03be1b4700 1 -- 172.21.15.2:0/3069408716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b810c810 con 0x556b6a5646b0 2023-11-16T18:26:51.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa73a4f9700 1 --2- 172.21.15.2:0/3152642073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55791612b8b0 0x557916130dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa739cf8700 1 -- 172.21.15.2:0/3152642073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55791612b8b0 msgr2=0x557916130dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa739cf8700 1 --2- 172.21.15.2:0/3152642073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55791612b8b0 0x557916130dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa739cf8700 1 -- 172.21.15.2:0/3152642073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557916127470 msgr2=0x557916130880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa739cf8700 1 --2- 172.21.15.2:0/3152642073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557916127470 0x557916130880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa739cf8700 1 -- 172.21.15.2:0/3152642073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa73410f020 con 0x557916126b60 2023-11-16T18:26:51.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa7394f7700 1 --2- 172.21.15.2:0/3152642073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557916127470 0x557916130880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.431+0000 7f03be1b4700 1 --2- 172.21.15.2:0/3069408716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 0x556b6a714a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f03b40c96e0 tx=0x7f03b4008f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e53e957d4a99ea6a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.431+0000 7f03a1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:26:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.431+0000 7f03a1ffb700 1 -- 172.21.15.2:0/3069408716 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b40b0070 con 0x556b6a5646b0 2023-11-16T18:26:51.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa739cf8700 1 --2- 172.21.15.2:0/3152642073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916130340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa7300099c0 tx=0x7fa730009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80365a81e008bcfd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:51.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa71d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:26:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.431+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3069408716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556b6a6fb710 con 0x556b6a5646b0 2023-11-16T18:26:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.431+0000 7f03a1ffb700 1 -- 172.21.15.2:0/3069408716 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f03b409d090 con 0x556b6a5646b0 2023-11-16T18:26:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.431+0000 7f03a1ffb700 1 -- 172.21.15.2:0/3069408716 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03b40a14b0 con 0x556b6a5646b0 2023-11-16T18:26:51.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa71d7fa700 1 -- 172.21.15.2:0/3152642073 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73002c040 con 0x557916126b60 2023-11-16T18:26:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.431+0000 7f03cc5a4b80 10 client.25715 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.431+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3069408716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556b6a620130 con 0x556b6a5646b0 2023-11-16T18:26:51.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa71d7fa700 1 -- 172.21.15.2:0/3152642073 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa7300044e0 con 0x557916126b60 2023-11-16T18:26:51.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.431+0000 7fa7480e8b80 1 -- 172.21.15.2:0/3152642073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557916119b20 con 0x557916126b60 2023-11-16T18:26:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.432+0000 7f03a1ffb700 1 -- 172.21.15.2:0/3069408716 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03b40c7080 con 0x556b6a5646b0 2023-11-16T18:26:51.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.432+0000 7fa71d7fa700 1 -- 172.21.15.2:0/3152642073 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa730003ad0 con 0x557916126b60 2023-11-16T18:26:51.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.432+0000 7fa7480e8b80 10 client.25718 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:51.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.432+0000 7fa7480e8b80 1 -- 172.21.15.2:0/3152642073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55791603e190 con 0x557916126b60 2023-11-16T18:26:51.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.432+0000 7f03a1ffb700 1 -- 172.21.15.2:0/3069408716 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f03b40a1020 con 0x556b6a5646b0 2023-11-16T18:26:51.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.432+0000 7f03cc5a4b80 10 client.25715 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:51.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.432+0000 7f03cc5a4b80 10 client.25715 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:51.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.432+0000 7f03cc5a4b80 20 client.25715 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:51.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.432+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3069408716 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x556b6a721f60 0x556b6a724370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.432+0000 7f03cc5a4b80 4 client.25715 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:51.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.432+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3069408716 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b6a61c040 con 0x556b6a721f60 2023-11-16T18:26:51.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.432+0000 7fa71d7fa700 1 -- 172.21.15.2:0/3152642073 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557916119b20 con 0x557916126b60 2023-11-16T18:26:51.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.432+0000 7f03bd9b3700 1 --2- 172.21.15.2:0/3069408716 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x556b6a721f60 0x556b6a724370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.432+0000 7fa71d7fa700 1 -- 172.21.15.2:0/3152642073 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa730026430 con 0x557916126b60 2023-11-16T18:26:51.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.432+0000 7fa7480e8b80 10 client.25718 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:51.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.432+0000 7fa7480e8b80 10 client.25718 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-11-16T18:26:51.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.432+0000 7fa7480e8b80 20 client.25718 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:51.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.432+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/3152642073 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x557916140820 0x557916142c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.432+0000 7fa7480e8b80 4 client.25718 mds_command: new command op to 24460 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:51.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.432+0000 7fa7480e8b80 1 -- 172.21.15.2:0/3152642073 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55791603a0a0 con 0x557916140820 2023-11-16T18:26:51.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.433+0000 7fa7394f7700 1 --2- 172.21.15.2:0/3152642073 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x557916140820 0x557916142c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.433+0000 7f03a1ffb700 10 client.25715 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:51.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.433+0000 7f03bd9b3700 1 --2- 172.21.15.2:0/3069408716 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x556b6a721f60 0x556b6a724370 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61451f3d7cf92c4c server_cookie=12cfd8bdcbfefae in_seq=0 out_seq=0 2023-11-16T18:26:51.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.433+0000 7fa71d7fa700 10 client.25718 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:26:51.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.433+0000 7fa7394f7700 1 --2- 172.21.15.2:0/3152642073 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x557916140820 0x557916142c30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=60beaaa40870eae3 server_cookie=e7594f7c13812d49 in_seq=0 out_seq=0 2023-11-16T18:26:51.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.433+0000 7f03a1ffb700 1 -- 172.21.15.2:0/3069408716 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x556b6a61c040 con 0x556b6a721f60 2023-11-16T18:26:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.433+0000 7f03a1ffb700 10 client.25715 handle_command_reply: tid=0 2023-11-16T18:26:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.433+0000 7f03cc5a4b80 1 client.25715 shutdown 2023-11-16T18:26:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.434+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3069408716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 msgr2=0x556b6a714a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.434+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3069408716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 0x556b6a714a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f03b40c96e0 tx=0x7f03b4008f20 comp rx=0 tx=0).stop 2023-11-16T18:26:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.434+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3069408716 shutdown_connections 2023-11-16T18:26:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.434+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3069408716 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x556b6a721f60 0x556b6a724370 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.434+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3069408716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70eb90 0x556b6a712250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.434+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3069408716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 0x556b6a714a50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.434+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3069408716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70e720 0x556b6a711c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.434+0000 7f03a1ffb700 0 client.25715 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.434+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3069408716 >> 172.21.15.2:0/3069408716 conn(0x556b6a6fdf50 msgr2=0x556b6a7027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.434+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3069408716 shutdown_connections 2023-11-16T18:26:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.434+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3069408716 wait complete. 2023-11-16T18:26:51.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.434+0000 7f03cc5a4b80 20 client.25715 trim_cache size 0 max 16384 2023-11-16T18:26:51.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa71d7fa700 1 -- 172.21.15.2:0/3152642073 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55791603a0a0 con 0x557916140820 2023-11-16T18:26:51.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa71d7fa700 10 client.25718 handle_command_reply: tid=0 2023-11-16T18:26:51.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa7480e8b80 1 client.25718 shutdown 2023-11-16T18:26:51.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa7480e8b80 1 -- 172.21.15.2:0/3152642073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 msgr2=0x557916130340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/3152642073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916130340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa7300099c0 tx=0x7fa730009cd0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa7480e8b80 1 -- 172.21.15.2:0/3152642073 shutdown_connections 2023-11-16T18:26:51.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/3152642073 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x557916140820 0x557916142c30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/3152642073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55791612b8b0 0x557916130dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/3152642073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916130340 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa71d7fa700 0 client.25718 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:26:51.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/3152642073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557916127470 0x557916130880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa7480e8b80 1 -- 172.21.15.2:0/3152642073 >> 172.21.15.2:0/3152642073 conn(0x55791611c360 msgr2=0x557916120de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:51.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa7480e8b80 1 -- 172.21.15.2:0/3152642073 shutdown_connections 2023-11-16T18:26:51.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa7480e8b80 1 -- 172.21.15.2:0/3152642073 wait complete. 2023-11-16T18:26:51.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.434+0000 7fa7480e8b80 20 client.25718 trim_cache size 0 max 16384 2023-11-16T18:26:51.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.448+0000 7fa7480e8b80 1 Processor -- start 2023-11-16T18:26:51.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.448+0000 7fa7480e8b80 1 -- start start 2023-11-16T18:26:51.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.448+0000 7fa7480e8b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916137880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.448+0000 7fa7480e8b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557916127470 0x557916130bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.448+0000 7fa7480e8b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55791612b8b0 0x55791612dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.448+0000 7fa7480e8b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579161228b0 con 0x55791612b8b0 2023-11-16T18:26:51.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.448+0000 7fa7480e8b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55791612c120 con 0x557916126b60 2023-11-16T18:26:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.448+0000 7fa7480e8b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55791612a470 con 0x557916127470 2023-11-16T18:26:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.448+0000 7fa73a4f9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55791612b8b0 0x55791612dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.448+0000 7fa73a4f9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55791612b8b0 0x55791612dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51528/0 (socket says 172.21.15.2:51528) 2023-11-16T18:26:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.448+0000 7fa73a4f9700 1 -- 172.21.15.2:0/2148992542 learned_addr learned my addr 172.21.15.2:0/2148992542 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.448+0000 7fa7394f7700 1 --2- 172.21.15.2:0/2148992542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557916127470 0x557916130bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.448+0000 7fa739cf8700 1 --2- 172.21.15.2:0/2148992542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.449+0000 7fa73a4f9700 1 -- 172.21.15.2:0/2148992542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557916127470 msgr2=0x557916130bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.449+0000 7fa73a4f9700 1 --2- 172.21.15.2:0/2148992542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557916127470 0x557916130bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.449+0000 7fa73a4f9700 1 -- 172.21.15.2:0/2148992542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 msgr2=0x557916137880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.449+0000 7fa73a4f9700 1 --2- 172.21.15.2:0/2148992542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916137880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.449+0000 7fa73a4f9700 1 -- 172.21.15.2:0/2148992542 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55791603dee0 con 0x55791612b8b0 2023-11-16T18:26:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.449+0000 7fa7394f7700 1 --2- 172.21.15.2:0/2148992542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557916127470 0x557916130bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.449+0000 7fa739cf8700 1 --2- 172.21.15.2:0/2148992542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916137880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.449+0000 7fa73a4f9700 1 --2- 172.21.15.2:0/2148992542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55791612b8b0 0x55791612dd60 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009e90 tx=0x7fa72c009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d69c44a60050369c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.449+0000 7fa728ff9700 1 -- 172.21.15.2:0/2148992542 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c007e40 con 0x55791612b8b0 2023-11-16T18:26:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.449+0000 7fa728ff9700 1 -- 172.21.15.2:0/2148992542 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa72c00f040 con 0x55791612b8b0 2023-11-16T18:26:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.449+0000 7fa728ff9700 1 -- 172.21.15.2:0/2148992542 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c01f7e0 con 0x55791612b8b0 2023-11-16T18:26:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03cc5a4b80 1 Processor -- start 2023-11-16T18:26:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03cc5a4b80 1 -- start start 2023-11-16T18:26:51.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.450+0000 7fa7480e8b80 1 -- 172.21.15.2:0/2148992542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55791612b8b0 msgr2=0x55791612dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.450+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/2148992542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55791612b8b0 0x55791612dd60 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009e90 tx=0x7fa72c009ec0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.450+0000 7fa7480e8b80 1 -- 172.21.15.2:0/2148992542 shutdown_connections 2023-11-16T18:26:51.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.450+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/2148992542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55791612b8b0 0x55791612dd60 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.450+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/2148992542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916137880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.450+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/2148992542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557916127470 0x557916130bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.450+0000 7fa7480e8b80 1 -- 172.21.15.2:0/2148992542 >> 172.21.15.2:0/2148992542 conn(0x55791611c360 msgr2=0x557916129970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:51.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.450+0000 7fa7480e8b80 1 -- 172.21.15.2:0/2148992542 shutdown_connections 2023-11-16T18:26:51.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.450+0000 7fa7480e8b80 1 -- 172.21.15.2:0/2148992542 wait complete. 2023-11-16T18:26:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03cc5a4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 0x556b6a70f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03cc5a4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70e720 0x556b6a719090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03cc5a4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70eb90 0x556b6a702f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03cc5a4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6a712830 con 0x556b6a70eb90 2023-11-16T18:26:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03cc5a4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6a703fb0 con 0x556b6a5646b0 2023-11-16T18:26:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03cc5a4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6a70e1e0 con 0x556b6a70e720 2023-11-16T18:26:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03be9b5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70eb90 0x556b6a702f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03be9b5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70eb90 0x556b6a702f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51534/0 (socket says 172.21.15.2:51534) 2023-11-16T18:26:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03be9b5700 1 -- 172.21.15.2:0/3247394688 learned_addr learned my addr 172.21.15.2:0/3247394688 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:51.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.450+0000 7fa7480e8b80 1 -- start start 2023-11-16T18:26:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03bd9b3700 1 --2- 172.21.15.2:0/3247394688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70e720 0x556b6a719090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03be1b4700 1 --2- 172.21.15.2:0/3247394688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 0x556b6a70f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.450+0000 7fa7480e8b80 1 Processor -- start 2023-11-16T18:26:51.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03be9b5700 1 -- 172.21.15.2:0/3247394688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70e720 msgr2=0x556b6a719090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03be9b5700 1 --2- 172.21.15.2:0/3247394688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70e720 0x556b6a719090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03be9b5700 1 -- 172.21.15.2:0/3247394688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 msgr2=0x556b6a70f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03be9b5700 1 --2- 172.21.15.2:0/3247394688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 0x556b6a70f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.450+0000 7f03be9b5700 1 -- 172.21.15.2:0/3247394688 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b6a61fe80 con 0x556b6a70eb90 2023-11-16T18:26:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.451+0000 7f03be9b5700 1 --2- 172.21.15.2:0/3247394688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70eb90 0x556b6a702f60 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f03b00058e0 tx=0x7f03b0005980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeecd116bf75b49f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.451+0000 7f03ad5ad700 1 -- 172.21.15.2:0/3247394688 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b000b790 con 0x556b6a70eb90 2023-11-16T18:26:51.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7480e8b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916133860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.451+0000 7f03ad5ad700 1 -- 172.21.15.2:0/3247394688 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03b000bdb0 con 0x556b6a70eb90 2023-11-16T18:26:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.451+0000 7f03ad5ad700 1 -- 172.21.15.2:0/3247394688 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b000fdc0 con 0x556b6a70eb90 2023-11-16T18:26:51.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7480e8b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557916127470 0x557916133da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7480e8b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55791612b8b0 0x5579161342e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7480e8b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557916193740 con 0x55791612b8b0 2023-11-16T18:26:51.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7480e8b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579161938b0 con 0x557916126b60 2023-11-16T18:26:51.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7480e8b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557916193a70 con 0x557916127470 2023-11-16T18:26:51.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7480e8b80 10 client.0 _ll_register_callbacks cb 0x557916037780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:51.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7394f7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557916127470 0x557916133da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7394f7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557916127470 0x557916133da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43068/0 (socket says 172.21.15.2:43068) 2023-11-16T18:26:51.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7394f7700 1 -- 172.21.15.2:0/1725280245 learned_addr learned my addr 172.21.15.2:0/1725280245 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:51.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa739cf8700 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916133860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa73a4f9700 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55791612b8b0 0x5579161342e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7394f7700 1 -- 172.21.15.2:0/1725280245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 msgr2=0x557916133860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7394f7700 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916133860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7394f7700 1 -- 172.21.15.2:0/1725280245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55791612b8b0 msgr2=0x5579161342e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7394f7700 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55791612b8b0 0x5579161342e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.451+0000 7fa7394f7700 1 -- 172.21.15.2:0/1725280245 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55791603e190 con 0x557916127470 2023-11-16T18:26:51.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.452+0000 7fa73a4f9700 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55791612b8b0 0x5579161342e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.452+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3247394688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70eb90 msgr2=0x556b6a702f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.452+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3247394688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70eb90 0x556b6a702f60 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f03b00058e0 tx=0x7f03b0005980 comp rx=0 tx=0).stop 2023-11-16T18:26:51.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.452+0000 7fa7394f7700 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557916127470 0x557916133da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa72400ea80 tx=0x7fa72409d8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ffea36acd3bc2df server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:51.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.452+0000 7fa71d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:26:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.452+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3247394688 shutdown_connections 2023-11-16T18:26:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.452+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3247394688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a70eb90 0x556b6a702f60 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.452+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3247394688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a5646b0 0x556b6a70f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.452+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/3247394688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70e720 0x556b6a719090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.452+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3247394688 >> 172.21.15.2:0/3247394688 conn(0x556b6a6fdf50 msgr2=0x556b6a70ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.452+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3247394688 shutdown_connections 2023-11-16T18:26:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.452+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/3247394688 wait complete. 2023-11-16T18:26:51.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.452+0000 7fa71d7fa700 1 -- 172.21.15.2:0/1725280245 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7240a6290 con 0x557916127470 2023-11-16T18:26:51.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.452+0000 7fa71d7fa700 1 -- 172.21.15.2:0/1725280245 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa7240ce1e0 con 0x557916127470 2023-11-16T18:26:51.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.452+0000 7fa7480e8b80 1 -- 172.21.15.2:0/1725280245 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa72409a550 con 0x557916127470 2023-11-16T18:26:51.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.452+0000 7fa71d7fa700 1 -- 172.21.15.2:0/1725280245 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7240ce3a0 con 0x557916127470 2023-11-16T18:26:51.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.452+0000 7f03cc5a4b80 1 -- start start 2023-11-16T18:26:51.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.452+0000 7fa7480e8b80 10 client.25678 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:51.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.452+0000 7fa7480e8b80 1 -- 172.21.15.2:0/1725280245 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7fa7240ba040 con 0x557916127470 2023-11-16T18:26:51.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.452+0000 7f03cc5a4b80 1 Processor -- start 2023-11-16T18:26:51.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.452+0000 7fa71d7fa700 1 -- 172.21.15.2:0/1725280245 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa72409a550 con 0x557916127470 2023-11-16T18:26:51.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.453+0000 7fa71d7fa700 1 -- 172.21.15.2:0/1725280245 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa7240a5e00 con 0x557916127470 2023-11-16T18:26:51.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.453+0000 7fa7480e8b80 10 client.25678 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:51.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.453+0000 7fa7480e8b80 10 client.25678 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-11-16T18:26:51.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.453+0000 7fa7480e8b80 20 client.25678 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:51.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.453+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x557916194d70 0x557916197180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.453+0000 7fa7480e8b80 4 client.25678 mds_command: new command op to 24460 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:26:51.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.453+0000 7fa7480e8b80 1 -- 172.21.15.2:0/1725280245 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557916046660 con 0x557916194d70 2023-11-16T18:26:51.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03cc5a4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a5646b0 0x556b6a770fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03cc5a4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70e720 0x556b6a771520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03cc5a4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a70eb90 0x556b6a718060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03cc5a4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6a771cb0 con 0x556b6a5646b0 2023-11-16T18:26:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03cc5a4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6a7185d0 con 0x556b6a70eb90 2023-11-16T18:26:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03cc5a4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6a718790 con 0x556b6a70e720 2023-11-16T18:26:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03cc5a4b80 10 client.0 _ll_register_callbacks cb 0x7f03b4098c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03be1b4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a5646b0 0x556b6a770fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03bd9b3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70e720 0x556b6a771520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03be1b4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a5646b0 0x556b6a770fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51544/0 (socket says 172.21.15.2:51544) 2023-11-16T18:26:51.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03be1b4700 1 -- 172.21.15.2:0/925553099 learned_addr learned my addr 172.21.15.2:0/925553099 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:51.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.453+0000 7fa739cf8700 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x557916194d70 0x557916197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03be9b5700 1 --2- 172.21.15.2:0/925553099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a70eb90 0x556b6a718060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03be1b4700 1 -- 172.21.15.2:0/925553099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70e720 msgr2=0x556b6a771520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03be1b4700 1 --2- 172.21.15.2:0/925553099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70e720 0x556b6a771520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03be1b4700 1 -- 172.21.15.2:0/925553099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a70eb90 msgr2=0x556b6a718060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03be1b4700 1 --2- 172.21.15.2:0/925553099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a70eb90 0x556b6a718060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03be1b4700 1 -- 172.21.15.2:0/925553099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b6a4db5d0 con 0x556b6a5646b0 2023-11-16T18:26:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03be1b4700 1 --2- 172.21.15.2:0/925553099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a5646b0 0x556b6a770fe0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f03b4098ff0 tx=0x7f03b4004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca3f4dcadb34482 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.453+0000 7f03a1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:51.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.453+0000 7fa71d7fa700 10 client.25678 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:26:51.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.453+0000 7fa739cf8700 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x557916194d70 0x557916197180 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ba2f37227adf63a4 server_cookie=ac3ac058596ba0f0 in_seq=0 out_seq=0 2023-11-16T18:26:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03a1ffb700 1 -- 172.21.15.2:0/925553099 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b40b0040 con 0x556b6a5646b0 2023-11-16T18:26:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03a1ffb700 1 -- 172.21.15.2:0/925553099 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f03b4008840 con 0x556b6a5646b0 2023-11-16T18:26:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/925553099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f03b001e040 con 0x556b6a5646b0 2023-11-16T18:26:51.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa71d7fa700 1 -- 172.21.15.2:0/1725280245 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557916046660 con 0x557916194d70 2023-11-16T18:26:51.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa71d7fa700 10 client.25678 handle_command_reply: tid=0 2023-11-16T18:26:51.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa7480e8b80 1 client.25678 shutdown 2023-11-16T18:26:51.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa7480e8b80 1 -- 172.21.15.2:0/1725280245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557916127470 msgr2=0x557916133da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557916127470 0x557916133da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa72400ea80 tx=0x7fa72409d8d0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03a1ffb700 1 -- 172.21.15.2:0/925553099 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03b40953a0 con 0x556b6a5646b0 2023-11-16T18:26:51.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03cc5a4b80 10 client.16173 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:51.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/925553099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556b6a470c00 con 0x556b6a5646b0 2023-11-16T18:26:51.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa7480e8b80 1 -- 172.21.15.2:0/1725280245 shutdown_connections 2023-11-16T18:26:51.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x557916194d70 0x557916197180 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55791612b8b0 0x5579161342e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557916126b60 0x557916133860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa7480e8b80 1 --2- 172.21.15.2:0/1725280245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557916127470 0x557916133da0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa71d7fa700 0 client.25678 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:26:51.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa7480e8b80 1 -- 172.21.15.2:0/1725280245 >> 172.21.15.2:0/1725280245 conn(0x55791611c360 msgr2=0x5579161308b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:51.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa7480e8b80 1 -- 172.21.15.2:0/1725280245 shutdown_connections 2023-11-16T18:26:51.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03a1ffb700 1 -- 172.21.15.2:0/925553099 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03b001e040 con 0x556b6a5646b0 2023-11-16T18:26:51.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa7480e8b80 1 -- 172.21.15.2:0/1725280245 wait complete. 2023-11-16T18:26:51.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.454+0000 7fa7480e8b80 20 client.25678 trim_cache size 0 max 16384 2023-11-16T18:26:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03a1ffb700 1 -- 172.21.15.2:0/925553099 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f03b4098150 con 0x556b6a5646b0 2023-11-16T18:26:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03cc5a4b80 10 client.16173 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03cc5a4b80 10 client.16173 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03cc5a4b80 20 client.16173 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/925553099 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x556b6a70a040 0x556b6a70a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03cc5a4b80 4 client.16173 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:26:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.454+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/925553099 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556b6a628600 con 0x556b6a70a040 2023-11-16T18:26:51.494 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:26:51.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.455+0000 7f03bd9b3700 1 --2- 172.21.15.2:0/925553099 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x556b6a70a040 0x556b6a70a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:51.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.455+0000 7fa73bf5c700 1 -- 172.21.15.2:0/697298872 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa7200a1340 msgr2=0x7fa7200a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.455+0000 7fa73bf5c700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa7200a1340 0x7fa7200a37f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa730009990 tx=0x7fa730005fb0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.455+0000 7fa73bf5c700 1 -- 172.21.15.2:0/697298872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7341002a0 msgr2=0x7fa7341956c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.455+0000 7fa73bf5c700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7341002a0 0x7fa7341956c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa72400eab0 tx=0x7fa72400edc0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.455+0000 7fa73bf5c700 1 -- 172.21.15.2:0/697298872 shutdown_connections 2023-11-16T18:26:51.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.455+0000 7fa73bf5c700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa7200a1340 0x7fa7200a37f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.455+0000 7fa73bf5c700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7340ff8f0 0x7fa734195180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.455+0000 7fa73bf5c700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa734104140 0x7fa734199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.455+0000 7fa73bf5c700 1 --2- 172.21.15.2:0/697298872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7341002a0 0x7fa7341956c0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.455+0000 7fa73bf5c700 1 -- 172.21.15.2:0/697298872 >> 172.21.15.2:0/697298872 conn(0x7fa734075880 msgr2=0x7fa7340fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:51.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.455+0000 7fa73bf5c700 1 -- 172.21.15.2:0/697298872 shutdown_connections 2023-11-16T18:26:51.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:51.455+0000 7fa73bf5c700 1 -- 172.21.15.2:0/697298872 wait complete. 2023-11-16T18:26:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.455+0000 7f03a1ffb700 10 client.16173 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.455+0000 7f03bd9b3700 1 --2- 172.21.15.2:0/925553099 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x556b6a70a040 0x556b6a70a440 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79138f2f01ee3506 server_cookie=3a8430b18f21a721 in_seq=0 out_seq=0 2023-11-16T18:26:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.475+0000 7f03a1ffb700 1 -- 172.21.15.2:0/925553099 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556b6a628600 con 0x556b6a70a040 2023-11-16T18:26:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.475+0000 7f03a1ffb700 10 client.16173 handle_command_reply: tid=0 2023-11-16T18:26:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.475+0000 7f03cc5a4b80 1 client.16173 shutdown 2023-11-16T18:26:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.475+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/925553099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a5646b0 msgr2=0x556b6a770fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.475+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/925553099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a5646b0 0x556b6a770fe0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f03b4098ff0 tx=0x7f03b4004d00 comp rx=0 tx=0).stop 2023-11-16T18:26:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.475+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/925553099 shutdown_connections 2023-11-16T18:26:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.475+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/925553099 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x556b6a70a040 0x556b6a70a440 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.475+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/925553099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6a70eb90 0x556b6a718060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.475+0000 7f03a1ffb700 0 client.16173 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.475+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/925553099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556b6a5646b0 0x556b6a770fe0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.475+0000 7f03cc5a4b80 1 --2- 172.21.15.2:0/925553099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556b6a70e720 0x556b6a771520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.475+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/925553099 >> 172.21.15.2:0/925553099 conn(0x556b6a6fdf50 msgr2=0x556b6a710410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/925553099 shutdown_connections 2023-11-16T18:26:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03cc5a4b80 1 -- 172.21.15.2:0/925553099 wait complete. 2023-11-16T18:26:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03cc5a4b80 20 client.16173 trim_cache size 0 max 16384 2023-11-16T18:26:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:26:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:26:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:26:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:26:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03c0418700 1 -- 172.21.15.2:0/2725798252 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f03a40a12c0 msgr2=0x7f03a40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03c0418700 1 --2- 172.21.15.2:0/2725798252 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f03a40a12c0 0x7f03a40a3770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03a800ffd0 tx=0x7f03a8011040 comp rx=0 tx=0).stop 2023-11-16T18:26:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03c0418700 1 -- 172.21.15.2:0/2725798252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b80ff3c0 msgr2=0x7f03b8194150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03c0418700 1 --2- 172.21.15.2:0/2725798252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b80ff3c0 0x7f03b8194150 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f03b4009b00 tx=0x7f03b4009e10 comp rx=0 tx=0).stop 2023-11-16T18:26:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03c0418700 1 -- 172.21.15.2:0/2725798252 shutdown_connections 2023-11-16T18:26:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03c0418700 1 --2- 172.21.15.2:0/2725798252 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f03a40a12c0 0x7f03a40a3770 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03c0418700 1 --2- 172.21.15.2:0/2725798252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b80ff3c0 0x7f03b8194150 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03c0418700 1 --2- 172.21.15.2:0/2725798252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b8104400 0x7f03b8198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03c0418700 1 --2- 172.21.15.2:0/2725798252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03b80ffd60 0x7f03b8194690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03c0418700 1 -- 172.21.15.2:0/2725798252 >> 172.21.15.2:0/2725798252 conn(0x7f03b80763a0 msgr2=0x7f03b80fc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03c0418700 1 -- 172.21.15.2:0/2725798252 shutdown_connections 2023-11-16T18:26:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:26:51.476+0000 7f03c0418700 1 -- 172.21.15.2:0/2725798252 wait complete. 2023-11-16T18:26:51.501 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T18:26:51.689 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:51.689 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:51.837 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:51.838 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:51.867 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:51.867 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:51.867 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 94 -lt 100 ']' 2023-11-16T18:26:51.867 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 94 + 1 2023-11-16T18:26:51.894 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=95 2023-11-16T18:26:51.894 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:51.912 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:52.006 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:52.007 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:52.007 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:52.007 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:52.297 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:52.297 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:52.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:52 smithi002 ceph-mon[98807]: pgmap v63: 129 pgs: 129 active+clean; 2.7 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-11-16T18:26:52.388 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:52.388 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:52.426 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:52.427 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:52.427 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 95 -lt 100 ']' 2023-11-16T18:26:52.427 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 95 + 1 2023-11-16T18:26:52.461 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=96 2023-11-16T18:26:52.461 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:52.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:52 smithi040 ceph-mon[100251]: pgmap v63: 129 pgs: 129 active+clean; 2.7 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-11-16T18:26:52.525 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:52.526 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:52.526 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:52.526 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:52.527 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:52.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:52 smithi188 ceph-mon[100639]: pgmap v63: 129 pgs: 129 active+clean; 2.7 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-11-16T18:26:53.014 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:53.014 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:53.193 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:53.194 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:53.209 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:53.209 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:53.210 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 96 -lt 100 ']' 2023-11-16T18:26:53.210 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 96 + 1 2023-11-16T18:26:53.252 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=97 2023-11-16T18:26:53.252 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:53.309 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:53.309 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:53.310 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:53.310 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:53.310 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:54.031 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:57.564 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:57.571 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:54 smithi040 ceph-mon[100251]: pgmap v64: 129 pgs: 129 active+clean; 2.8 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 17 op/s 2023-11-16T18:26:57.571 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:56 smithi040 ceph-mon[100251]: pgmap v65: 129 pgs: 129 active+clean; 2.8 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 12 op/s 2023-11-16T18:26:57.571 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:54 smithi188 ceph-mon[100639]: pgmap v64: 129 pgs: 129 active+clean; 2.8 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 17 op/s 2023-11-16T18:26:57.604 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:56 smithi188 ceph-mon[100639]: pgmap v65: 129 pgs: 129 active+clean; 2.8 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 12 op/s 2023-11-16T18:26:57.604 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:26:57 smithi188 ceph-mon[100639]: pgmap v66: 129 pgs: 129 active+clean; 2.9 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 16 op/s 2023-11-16T18:26:57.605 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:57.605 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:57.605 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:57.605 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:57.605 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 97 -lt 100 ']' 2023-11-16T18:26:57.605 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 97 + 1 2023-11-16T18:26:57.606 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=98 2023-11-16T18:26:57.606 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:57.606 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:57.606 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:57.606 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:57.608 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:57.608 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:57.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:54 smithi002 ceph-mon[98807]: pgmap v64: 129 pgs: 129 active+clean; 2.8 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 17 op/s 2023-11-16T18:26:57.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.589+0000 7f05d2789700 1 -- 172.21.15.2:0/3347017643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05cc0689d0 msgr2=0x7f05cc104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:57.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.589+0000 7f05d2789700 1 --2- 172.21.15.2:0/3347017643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05cc0689d0 0x7f05cc104ea0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f05b40099c0 tx=0x7f05b4009cd0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.589+0000 7f05d2789700 1 -- 172.21.15.2:0/3347017643 shutdown_connections 2023-11-16T18:26:57.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.589+0000 7f05d2789700 1 --2- 172.21.15.2:0/3347017643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05cc1055d0 0x7f05cc10d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.589+0000 7f05d2789700 1 --2- 172.21.15.2:0/3347017643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05cc068030 0x7f05cc068400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.589+0000 7f05d2789700 1 --2- 172.21.15.2:0/3347017643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05cc0689d0 0x7f05cc104ea0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.589+0000 7f05d2789700 1 -- 172.21.15.2:0/3347017643 >> 172.21.15.2:0/3347017643 conn(0x7f05cc0763a0 msgr2=0x7f05cc0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:57.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.589+0000 7f05d2789700 1 -- 172.21.15.2:0/3347017643 shutdown_connections 2023-11-16T18:26:57.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.589+0000 7f05d2789700 1 -- 172.21.15.2:0/3347017643 wait complete. 2023-11-16T18:26:57.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.590+0000 7f05d2789700 1 Processor -- start 2023-11-16T18:26:57.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.590+0000 7f05d2789700 1 -- start start 2023-11-16T18:26:57.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.590+0000 7f05d2789700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05cc068030 0x7f05cc196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.590+0000 7f05d2789700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05cc0689d0 0x7f05cc196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.590+0000 7f05d2789700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05cc1055d0 0x7f05cc19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.590+0000 7f05d2789700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc19b3e0 con 0x7f05cc1055d0 2023-11-16T18:26:57.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.590+0000 7f05d2789700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc19b550 con 0x7f05cc068030 2023-11-16T18:26:57.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.590+0000 7f05d2789700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc19b710 con 0x7f05cc0689d0 2023-11-16T18:26:57.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.590+0000 7f05d0d26700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05cc1055d0 0x7f05cc19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.590+0000 7f05d0d26700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05cc1055d0 0x7f05cc19ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51560/0 (socket says 172.21.15.2:51560) 2023-11-16T18:26:57.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.590+0000 7f05d0d26700 1 -- 172.21.15.2:0/1093186029 learned_addr learned my addr 172.21.15.2:0/1093186029 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:57.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05cbfff700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05cc068030 0x7f05cc196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05c3fff700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05cc0689d0 0x7f05cc196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05d0d26700 1 -- 172.21.15.2:0/1093186029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05cc0689d0 msgr2=0x7f05cc196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:57.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05d0d26700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05cc0689d0 0x7f05cc196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05d0d26700 1 -- 172.21.15.2:0/1093186029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05cc068030 msgr2=0x7f05cc196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:57.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05d0d26700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05cc068030 0x7f05cc196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05d0d26700 1 -- 172.21.15.2:0/1093186029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b40095f0 con 0x7f05cc1055d0 2023-11-16T18:26:57.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05c3fff700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05cc0689d0 0x7f05cc196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:57.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05d0d26700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05cc1055d0 0x7f05cc19ada0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f05c4007ee0 tx=0x7f05c400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:57.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05c9ffb700 1 -- 172.21.15.2:0/1093186029 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c4017070 con 0x7f05cc1055d0 2023-11-16T18:26:57.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05c9ffb700 1 -- 172.21.15.2:0/1093186029 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05c400f040 con 0x7f05cc1055d0 2023-11-16T18:26:57.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05d2789700 1 -- 172.21.15.2:0/1093186029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05cc110300 con 0x7f05cc1055d0 2023-11-16T18:26:57.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05c9ffb700 1 -- 172.21.15.2:0/1093186029 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c4005740 con 0x7f05cc1055d0 2023-11-16T18:26:57.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.591+0000 7f05d2789700 1 -- 172.21.15.2:0/1093186029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05cc110170 con 0x7f05cc1055d0 2023-11-16T18:26:57.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.594+0000 7f05c9ffb700 1 -- 172.21.15.2:0/1093186029 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f05c400dc60 con 0x7f05cc1055d0 2023-11-16T18:26:57.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.594+0000 7f05c9ffb700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f05b80a1340 0x7f05b80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.594+0000 7f05cbfff700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f05b80a1340 0x7f05b80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.594+0000 7f05c9ffb700 1 -- 172.21.15.2:0/1093186029 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f05c40cc7c0 con 0x7f05cc1055d0 2023-11-16T18:26:57.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.595+0000 7f05cbfff700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f05b80a1340 0x7f05b80a37f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f05bc005950 tx=0x7f05bc00b410 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:57.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.659+0000 7f05de915b80 1 Processor -- start 2023-11-16T18:26:57.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.659+0000 7f05de915b80 1 -- start start 2023-11-16T18:26:57.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.659+0000 7f05de915b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a6eb60 0x559934a6ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.659+0000 7f05de915b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a73370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.659+0000 7f05de915b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a738b0 0x559934a73d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.659+0000 7f05de915b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559934974060 con 0x559934a6f470 2023-11-16T18:26:57.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.659+0000 7f05de915b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599349c68a0 con 0x559934a6eb60 2023-11-16T18:26:57.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.659+0000 7f05de915b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599349bfd30 con 0x559934a738b0 2023-11-16T18:26:57.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.659+0000 7f05c3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a73370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.659+0000 7f05c3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a73370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51568/0 (socket says 172.21.15.2:51568) 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.659+0000 7f05c3fff700 1 -- 172.21.15.2:0/3642425589 learned_addr learned my addr 172.21.15.2:0/3642425589 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.660+0000 7f05cbfff700 1 --2- 172.21.15.2:0/3642425589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a6eb60 0x559934a6ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.660+0000 7f05c3fff700 1 -- 172.21.15.2:0/3642425589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a738b0 msgr2=0x559934a73d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.660+0000 7f05c3fff700 1 --2- 172.21.15.2:0/3642425589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a738b0 0x559934a73d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.660+0000 7f05c3fff700 1 -- 172.21.15.2:0/3642425589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a6eb60 msgr2=0x559934a6ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.660+0000 7f05c3fff700 1 --2- 172.21.15.2:0/3642425589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a6eb60 0x559934a6ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.660+0000 7f05c3fff700 1 -- 172.21.15.2:0/3642425589 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559934841630 con 0x559934a6f470 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.660+0000 7f05c3fff700 1 --2- 172.21.15.2:0/3642425589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a73370 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f05b400b580 tx=0x7f05b4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1820cf1943764446 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.660+0000 7f05c27fc700 1 -- 172.21.15.2:0/3642425589 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4005230 con 0x559934a6f470 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.660+0000 7f05c27fc700 1 -- 172.21.15.2:0/3642425589 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05b401e040 con 0x559934a6f470 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.660+0000 7f05c27fc700 1 -- 172.21.15.2:0/3642425589 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4003ae0 con 0x559934a6f470 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.661+0000 7f05de915b80 1 -- 172.21.15.2:0/3642425589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 msgr2=0x559934a73370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:57.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.661+0000 7f05de915b80 1 --2- 172.21.15.2:0/3642425589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a73370 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f05b400b580 tx=0x7f05b4005e80 comp rx=0 tx=0).stop 2023-11-16T18:26:57.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.661+0000 7f05de915b80 1 -- 172.21.15.2:0/3642425589 shutdown_connections 2023-11-16T18:26:57.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.661+0000 7f05de915b80 1 --2- 172.21.15.2:0/3642425589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a738b0 0x559934a73d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.661+0000 7f05de915b80 1 --2- 172.21.15.2:0/3642425589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a6eb60 0x559934a6ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.661+0000 7f05de915b80 1 --2- 172.21.15.2:0/3642425589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a73370 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.661+0000 7f05de915b80 1 -- 172.21.15.2:0/3642425589 >> 172.21.15.2:0/3642425589 conn(0x559934a64360 msgr2=0x559934a69b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:57.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.661+0000 7f05de915b80 1 -- 172.21.15.2:0/3642425589 shutdown_connections 2023-11-16T18:26:57.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.661+0000 7f05de915b80 1 -- 172.21.15.2:0/3642425589 wait complete. 2023-11-16T18:26:57.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.661+0000 7f05de915b80 1 -- start start 2023-11-16T18:26:57.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.662+0000 7f05de915b80 1 Processor -- start 2023-11-16T18:26:57.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.662+0000 7f05de915b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a6eb60 0x559934a78380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.662+0000 7f05de915b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a788c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.662+0000 7f05de915b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a738b0 0x559934a78e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.662+0000 7f05de915b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599349618e0 con 0x559934a6f470 2023-11-16T18:26:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.662+0000 7f05de915b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559934961580 con 0x559934a6eb60 2023-11-16T18:26:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.662+0000 7f05de915b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x559934843920 con 0x559934a738b0 2023-11-16T18:26:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.662+0000 7f05de915b80 10 client.0 _ll_register_callbacks cb 0x55993497f780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.662+0000 7f05c3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a788c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.662+0000 7f05c3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a788c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51574/0 (socket says 172.21.15.2:51574) 2023-11-16T18:26:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.662+0000 7f05c3fff700 1 -- 172.21.15.2:0/888411512 learned_addr learned my addr 172.21.15.2:0/888411512 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05cbfff700 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a6eb60 0x559934a78380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05d0d26700 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a738b0 0x559934a78e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05c3fff700 1 -- 172.21.15.2:0/888411512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a738b0 msgr2=0x559934a78e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05c3fff700 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a738b0 0x559934a78e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05c3fff700 1 -- 172.21.15.2:0/888411512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a6eb60 msgr2=0x559934a78380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05c3fff700 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a6eb60 0x559934a78380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05c3fff700 1 -- 172.21.15.2:0/888411512 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05cc10d990 con 0x559934a6f470 2023-11-16T18:26:57.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05d0d26700 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a738b0 0x559934a78e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:26:57.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05cbfff700 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a6eb60 0x559934a78380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:57.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05c3fff700 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a788c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009d10 tx=0x7f05b40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e26862e6ddde26 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:57.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05ab7fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:57.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05ab7fe700 1 -- 172.21.15.2:0/888411512 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b400bdd0 con 0x559934a6f470 2023-11-16T18:26:57.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05ab7fe700 1 -- 172.21.15.2:0/888411512 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f05b40038c0 con 0x559934a6f470 2023-11-16T18:26:57.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05de915b80 1 -- 172.21.15.2:0/888411512 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559934a61b20 con 0x559934a6f470 2023-11-16T18:26:57.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05ab7fe700 1 -- 172.21.15.2:0/888411512 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05b4003a80 con 0x559934a6f470 2023-11-16T18:26:57.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05de915b80 10 client.16185 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:57.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.663+0000 7f05de915b80 1 -- 172.21.15.2:0/888411512 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x559934986190 con 0x559934a6f470 2023-11-16T18:26:57.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.664+0000 7f05ab7fe700 1 -- 172.21.15.2:0/888411512 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559934a61b20 con 0x559934a6f470 2023-11-16T18:26:57.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.664+0000 7f05ab7fe700 1 -- 172.21.15.2:0/888411512 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f05b4021070 con 0x559934a6f470 2023-11-16T18:26:57.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.664+0000 7f05de915b80 10 client.16185 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:57.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.664+0000 7f05de915b80 10 client.16185 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:57.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.664+0000 7f05de915b80 20 client.16185 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:57.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.664+0000 7f05de915b80 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559934a88430 0x559934a8a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.664+0000 7f05de915b80 4 client.16185 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:57.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.664+0000 7f05de915b80 1 -- 172.21.15.2:0/888411512 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599349820a0 con 0x559934a88430 2023-11-16T18:26:57.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.664+0000 7f05cbfff700 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559934a88430 0x559934a8a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.665+0000 7f05ab7fe700 10 client.16185 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:57.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.665+0000 7f05cbfff700 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559934a88430 0x559934a8a840 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=391ac05ab22ae92e server_cookie=d695d1d031afc45f in_seq=0 out_seq=0 2023-11-16T18:26:57.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.665+0000 7f05ab7fe700 1 -- 172.21.15.2:0/888411512 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5599349820a0 con 0x559934a88430 2023-11-16T18:26:57.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.665+0000 7f05ab7fe700 10 client.16185 handle_command_reply: tid=0 2023-11-16T18:26:57.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.665+0000 7f05de915b80 1 client.16185 shutdown 2023-11-16T18:26:57.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.666+0000 7f05de915b80 1 -- 172.21.15.2:0/888411512 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 msgr2=0x559934a788c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.666+0000 7f05de915b80 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a788c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009d10 tx=0x7f05b40057b0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.666+0000 7f05de915b80 1 -- 172.21.15.2:0/888411512 shutdown_connections 2023-11-16T18:26:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.666+0000 7f05de915b80 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559934a88430 0x559934a8a840 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.666+0000 7f05de915b80 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a738b0 0x559934a78e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.666+0000 7f05de915b80 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a6eb60 0x559934a78380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.666+0000 7f05ab7fe700 0 client.16185 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.666+0000 7f05de915b80 1 --2- 172.21.15.2:0/888411512 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a788c0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.666+0000 7f05de915b80 1 -- 172.21.15.2:0/888411512 >> 172.21.15.2:0/888411512 conn(0x559934a64360 msgr2=0x559934a68de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.666+0000 7f05de915b80 1 -- 172.21.15.2:0/888411512 shutdown_connections 2023-11-16T18:26:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.666+0000 7f05de915b80 1 -- 172.21.15.2:0/888411512 wait complete. 2023-11-16T18:26:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.666+0000 7f05de915b80 20 client.16185 trim_cache size 0 max 16384 2023-11-16T18:26:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.679+0000 7f05de915b80 1 Processor -- start 2023-11-16T18:26:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.679+0000 7f05de915b80 1 -- start start 2023-11-16T18:26:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05de915b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a6eb60 0x559934a78c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05de915b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a69560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05de915b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a738b0 0x559934a69aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05de915b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559934a8aea0 con 0x559934a6f470 2023-11-16T18:26:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05de915b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559934a76900 con 0x559934a738b0 2023-11-16T18:26:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05de915b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x559934a724e0 con 0x559934a6eb60 2023-11-16T18:26:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05c3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a69560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05c3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a69560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51582/0 (socket says 172.21.15.2:51582) 2023-11-16T18:26:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05c3fff700 1 -- 172.21.15.2:0/4173994046 learned_addr learned my addr 172.21.15.2:0/4173994046 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05cbfff700 1 --2- 172.21.15.2:0/4173994046 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a6eb60 0x559934a78c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05c3fff700 1 -- 172.21.15.2:0/4173994046 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a6eb60 msgr2=0x559934a78c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05c3fff700 1 --2- 172.21.15.2:0/4173994046 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a6eb60 0x559934a78c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05c3fff700 1 -- 172.21.15.2:0/4173994046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a738b0 msgr2=0x559934a69aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:57.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05c3fff700 1 --2- 172.21.15.2:0/4173994046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a738b0 0x559934a69aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05c3fff700 1 -- 172.21.15.2:0/4173994046 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559934985ee0 con 0x559934a6f470 2023-11-16T18:26:57.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05cbfff700 1 --2- 172.21.15.2:0/4173994046 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a6eb60 0x559934a78c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:26:57.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05c3fff700 1 --2- 172.21.15.2:0/4173994046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a69560 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f05b4004ac0 tx=0x7f05b4009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848f318f17c2d3da server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:57.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05c2ffd700 1 -- 172.21.15.2:0/4173994046 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b400bdd0 con 0x559934a6f470 2023-11-16T18:26:57.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05c2ffd700 1 -- 172.21.15.2:0/4173994046 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05b400b840 con 0x559934a6f470 2023-11-16T18:26:57.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.680+0000 7f05c2ffd700 1 -- 172.21.15.2:0/4173994046 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4022820 con 0x559934a6f470 2023-11-16T18:26:57.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.681+0000 7f05de915b80 1 -- 172.21.15.2:0/4173994046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 msgr2=0x559934a69560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:57.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.681+0000 7f05de915b80 1 --2- 172.21.15.2:0/4173994046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a69560 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f05b4004ac0 tx=0x7f05b4009290 comp rx=0 tx=0).stop 2023-11-16T18:26:57.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.681+0000 7f05de915b80 1 -- 172.21.15.2:0/4173994046 shutdown_connections 2023-11-16T18:26:57.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.681+0000 7f05de915b80 1 --2- 172.21.15.2:0/4173994046 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a6eb60 0x559934a78c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.681+0000 7f05de915b80 1 --2- 172.21.15.2:0/4173994046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a738b0 0x559934a69aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.681+0000 7f05de915b80 1 --2- 172.21.15.2:0/4173994046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a69560 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.681+0000 7f05de915b80 1 -- 172.21.15.2:0/4173994046 >> 172.21.15.2:0/4173994046 conn(0x559934a64360 msgr2=0x559934a75cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.681+0000 7f05de915b80 1 -- 172.21.15.2:0/4173994046 shutdown_connections 2023-11-16T18:26:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.681+0000 7f05de915b80 1 -- 172.21.15.2:0/4173994046 wait complete. 2023-11-16T18:26:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.681+0000 7f05de915b80 1 -- start start 2023-11-16T18:26:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05de915b80 1 Processor -- start 2023-11-16T18:26:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05de915b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a6eb60 0x559934a7f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05de915b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a7b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05de915b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a738b0 0x559934a7be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05de915b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559934a7c5a0 con 0x559934a6f470 2023-11-16T18:26:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05de915b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559934a7c710 con 0x559934a738b0 2023-11-16T18:26:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05de915b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x559934adb2a0 con 0x559934a6eb60 2023-11-16T18:26:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05de915b80 10 client.0 _ll_register_callbacks cb 0x7f05bc00b990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05c3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a7b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05c3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a7b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51588/0 (socket says 172.21.15.2:51588) 2023-11-16T18:26:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05c3fff700 1 -- 172.21.15.2:0/4178056257 learned_addr learned my addr 172.21.15.2:0/4178056257 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05d0d26700 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a738b0 0x559934a7be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05cbfff700 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a6eb60 0x559934a7f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05c3fff700 1 -- 172.21.15.2:0/4178056257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a6eb60 msgr2=0x559934a7f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05c3fff700 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a6eb60 0x559934a7f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05c3fff700 1 -- 172.21.15.2:0/4178056257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a738b0 msgr2=0x559934a7be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05c3fff700 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a738b0 0x559934a7be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05c3fff700 1 -- 172.21.15.2:0/4178056257 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559934986190 con 0x559934a6f470 2023-11-16T18:26:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05cbfff700 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a6eb60 0x559934a7f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:26:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.682+0000 7f05d0d26700 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a738b0 0x559934a7be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05c3fff700 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a7b950 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f05b4004cb0 tx=0x7f05b400bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8ae79134a87e9e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05ab7fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05ab7fe700 1 -- 172.21.15.2:0/4178056257 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4004d30 con 0x559934a6f470 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05ab7fe700 1 -- 172.21.15.2:0/4178056257 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f05b4000e40 con 0x559934a6f470 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05de915b80 1 -- 172.21.15.2:0/4178056257 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55993498e660 con 0x559934a6f470 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05ab7fe700 1 -- 172.21.15.2:0/4178056257 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05b400b840 con 0x559934a6f470 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05de915b80 10 client.16191 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05de915b80 1 -- 172.21.15.2:0/4178056257 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5599347d6c60 con 0x559934a6f470 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05ab7fe700 1 -- 172.21.15.2:0/4178056257 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55993498e660 con 0x559934a6f470 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05ab7fe700 1 -- 172.21.15.2:0/4178056257 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f05b4046070 con 0x559934a6f470 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05de915b80 10 client.16191 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05de915b80 10 client.16191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05de915b80 20 client.16191 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05de915b80 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559934adc3a0 0x559934ade780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05de915b80 4 client.16191 mds_command: new command op to 24473 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:26:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.683+0000 7f05de915b80 1 -- 172.21.15.2:0/4178056257 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55993480ef70 con 0x559934adc3a0 2023-11-16T18:26:57.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.684+0000 7f05cbfff700 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559934adc3a0 0x559934ade780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:57.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.684+0000 7f05ab7fe700 10 client.16191 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:57.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.684+0000 7f05cbfff700 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559934adc3a0 0x559934ade780 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dd4df986df6089c server_cookie=7fbd194e309e4777 in_seq=0 out_seq=0 2023-11-16T18:26:57.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.684+0000 7f05ab7fe700 1 -- 172.21.15.2:0/4178056257 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55993480ef70 con 0x559934adc3a0 2023-11-16T18:26:57.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.684+0000 7f05ab7fe700 10 client.16191 handle_command_reply: tid=0 2023-11-16T18:26:57.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.684+0000 7f05de915b80 1 client.16191 shutdown 2023-11-16T18:26:57.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05de915b80 1 -- 172.21.15.2:0/4178056257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 msgr2=0x559934a7b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:57.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05de915b80 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a7b950 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f05b4004cb0 tx=0x7f05b400bf30 comp rx=0 tx=0).stop 2023-11-16T18:26:57.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05de915b80 1 -- 172.21.15.2:0/4178056257 shutdown_connections 2023-11-16T18:26:57.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05de915b80 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559934adc3a0 0x559934ade780 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05de915b80 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559934a6eb60 0x559934a7f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05de915b80 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559934a738b0 0x559934a7be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05de915b80 1 --2- 172.21.15.2:0/4178056257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559934a6f470 0x559934a7b950 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05ab7fe700 0 client.16191 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:57.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05de915b80 1 -- 172.21.15.2:0/4178056257 >> 172.21.15.2:0/4178056257 conn(0x559934a64360 msgr2=0x559934a73180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:57.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05de915b80 1 -- 172.21.15.2:0/4178056257 shutdown_connections 2023-11-16T18:26:57.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05de915b80 1 -- 172.21.15.2:0/4178056257 wait complete. 2023-11-16T18:26:57.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05de915b80 20 client.16191 trim_cache size 0 max 16384 2023-11-16T18:26:57.640 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:26:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05d2789700 1 -- 172.21.15.2:0/1093186029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f05b80a1340 msgr2=0x7f05b80a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05d2789700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f05b80a1340 0x7f05b80a37f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f05bc005950 tx=0x7f05bc00b410 comp rx=0 tx=0).stop 2023-11-16T18:26:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05d2789700 1 -- 172.21.15.2:0/1093186029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05cc1055d0 msgr2=0x7f05cc19ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05d2789700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05cc1055d0 0x7f05cc19ada0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f05c4007ee0 tx=0x7f05c400d3b0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05d2789700 1 -- 172.21.15.2:0/1093186029 shutdown_connections 2023-11-16T18:26:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05d2789700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f05b80a1340 0x7f05b80a37f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05d2789700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05cc1055d0 0x7f05cc19ada0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05d2789700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05cc068030 0x7f05cc196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05d2789700 1 --2- 172.21.15.2:0/1093186029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05cc0689d0 0x7f05cc196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.685+0000 7f05d2789700 1 -- 172.21.15.2:0/1093186029 >> 172.21.15.2:0/1093186029 conn(0x7f05cc0763a0 msgr2=0x7f05cc0fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.686+0000 7f05d2789700 1 -- 172.21.15.2:0/1093186029 shutdown_connections 2023-11-16T18:26:57.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:54.686+0000 7f05d2789700 1 -- 172.21.15.2:0/1093186029 wait complete. 2023-11-16T18:26:57.650 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:57.650 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:57.650 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:57.650 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:57.651 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:57.686 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:57.686 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 98 -lt 100 ']' 2023-11-16T18:26:57.686 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 98 + 1 2023-11-16T18:26:57.686 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=99 2023-11-16T18:26:57.686 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:57.686 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:57.686 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:57.686 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:57.686 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:57.686 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:57.686 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 99 -lt 100 ']' 2023-11-16T18:26:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 99 + 1 2023-11-16T18:26:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=100 2023-11-16T18:26:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T18:26:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T18:26:57.688 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T18:26:57.688 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T18:26:57.688 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T18:26:57.688 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T18:26:57.688 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:57.688 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:57.688 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T18:26:57.688 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 100 -lt 100 ']' 2023-11-16T18:26:57.688 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T18:26:57.688 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n system.posix_acl_access -v 0x02000000 d1 2023-11-16T18:26:57.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n system.posix_acl_default -v 0x02000000 . 2023-11-16T18:26:57.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ expect_failure getfattr -n system.posix_acl_access d1 2023-11-16T18:26:57.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ getfattr -n system.posix_acl_access d1 2023-11-16T18:26:57.700 INFO:tasks.workunit.client.0.smithi002.stderr:d1: system.posix_acl_access: No such attribute 2023-11-16T18:26:57.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ return 0 2023-11-16T18:26:57.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ expect_failure getfattr -n system.posix_acl_default . 2023-11-16T18:26:57.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ getfattr -n system.posix_acl_default . 2023-11-16T18:26:57.700 INFO:tasks.workunit.client.0.smithi002.stderr:.: system.posix_acl_default: No such attribute 2023-11-16T18:26:57.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ return 0 2023-11-16T18:26:57.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T18:26:57.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ cd .. 2023-11-16T18:26:57.701 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir testdir 2023-11-16T18:26:57.701 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo OK 2023-11-16T18:26:57.702 INFO:tasks.workunit.client.0.smithi002.stdout:OK 2023-11-16T18:26:57.703 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T18:26:57.703 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T18:26:57.704 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:56 smithi002 ceph-mon[98807]: pgmap v65: 129 pgs: 129 active+clean; 2.8 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 12 op/s 2023-11-16T18:26:57.704 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:26:57 smithi002 ceph-mon[98807]: pgmap v66: 129 pgs: 129 active+clean; 2.9 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 16 op/s 2023-11-16T18:26:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:26:57 smithi040 ceph-mon[100251]: pgmap v66: 129 pgs: 129 active+clean; 2.9 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 16 op/s 2023-11-16T18:26:58.017 INFO:tasks.workunit:Running workunit fs/misc/chmod.sh... 2023-11-16T18:26:58.017 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/chmod.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh 2023-11-16T18:26:58.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2023-11-16T18:26:58.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ file=test_chmod.155351 2023-11-16T18:26:58.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo foo 2023-11-16T18:26:58.092 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T18:26:58.092 INFO:tasks.workunit.client.0.smithi002.stderr:+ chmod 400 test_chmod.155351 2023-11-16T18:26:58.094 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T18:26:58.094 INFO:tasks.workunit.client.0.smithi002.stderr:+ check_perms test_chmod.155351 -r-------- 2023-11-16T18:26:58.094 INFO:tasks.workunit.client.0.smithi002.stderr:+ file=test_chmod.155351 2023-11-16T18:26:58.094 INFO:tasks.workunit.client.0.smithi002.stderr:++ ls -la test_chmod.155351 2023-11-16T18:26:58.096 INFO:tasks.workunit.client.0.smithi002.stderr:+ r='-r--------. 1 ubuntu ubuntu 4 Nov 16 18:26 test_chmod.155351' 2023-11-16T18:26:58.097 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T18:26:58.097 INFO:tasks.workunit.client.0.smithi002.stderr:+ perms=-r-------- 2023-11-16T18:26:58.097 INFO:tasks.workunit.client.0.smithi002.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Nov 16 18:26 test_chmod.155351 2023-11-16T18:26:58.097 INFO:tasks.workunit.client.0.smithi002.stderr:++ awk '{print $1}' 2023-11-16T18:26:58.153 INFO:tasks.workunit.client.0.smithi002.stderr:+ test -r-------- '!=' -r--------. 2023-11-16T18:26:58.153 INFO:tasks.workunit.client.0.smithi002.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Nov 16 18:26 test_chmod.155351 2023-11-16T18:26:58.153 INFO:tasks.workunit.client.0.smithi002.stderr:++ awk '{print $1}' 2023-11-16T18:26:58.155 INFO:tasks.workunit.client.0.smithi002.stderr:+ test -r--------. '!=' -r--------. 2023-11-16T18:26:58.155 INFO:tasks.workunit.client.0.smithi002.stderr:+ set +e 2023-11-16T18:26:58.155 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo bar 2023-11-16T18:26:58.155 INFO:tasks.workunit.client.0.smithi002.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh: line 40: test_chmod.155351: Permission denied 2023-11-16T18:26:58.156 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 1 = 0 2023-11-16T18:26:58.156 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2023-11-16T18:26:58.156 INFO:tasks.workunit.client.0.smithi002.stderr:+ chmod 600 test_chmod.155351 2023-11-16T18:26:58.156 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo bar 2023-11-16T18:26:58.157 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T18:26:58.157 INFO:tasks.workunit.client.0.smithi002.stderr:+ check_perms test_chmod.155351 -rw------- 2023-11-16T18:26:58.157 INFO:tasks.workunit.client.0.smithi002.stderr:+ file=test_chmod.155351 2023-11-16T18:26:58.157 INFO:tasks.workunit.client.0.smithi002.stderr:++ ls -la test_chmod.155351 2023-11-16T18:26:58.158 INFO:tasks.workunit.client.0.smithi002.stderr:+ r='-rw-------. 1 ubuntu ubuntu 8 Nov 16 18:26 test_chmod.155351' 2023-11-16T18:26:58.158 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T18:26:58.158 INFO:tasks.workunit.client.0.smithi002.stderr:+ perms=-rw------- 2023-11-16T18:26:58.158 INFO:tasks.workunit.client.0.smithi002.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Nov 16 18:26 test_chmod.155351 2023-11-16T18:26:58.195 INFO:tasks.workunit.client.0.smithi002.stderr:++ awk '{print $1}' 2023-11-16T18:26:58.197 INFO:tasks.workunit.client.0.smithi002.stderr:+ test -rw------- '!=' -rw-------. 2023-11-16T18:26:58.197 INFO:tasks.workunit.client.0.smithi002.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Nov 16 18:26 test_chmod.155351 2023-11-16T18:26:58.197 INFO:tasks.workunit.client.0.smithi002.stderr:++ awk '{print $1}' 2023-11-16T18:26:58.197 INFO:tasks.workunit.client.0.smithi002.stderr:+ test -rw-------. '!=' -rw-------. 2023-11-16T18:26:58.197 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo foo 2023-11-16T18:26:58.197 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T18:26:58.197 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T18:26:58.197 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T18:26:58.251 INFO:tasks.workunit:Running workunit fs/misc/dac_override.sh... 2023-11-16T18:26:58.252 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/dac_override.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dac_override.sh 2023-11-16T18:26:58.279 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2023-11-16T18:26:58.279 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir -p testdir 2023-11-16T18:26:58.281 INFO:tasks.workunit.client.0.smithi002.stderr:+ file=test_chmod.155410 2023-11-16T18:26:58.282 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo foo 2023-11-16T18:26:58.283 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo chmod 600 testdir 2023-11-16T18:26:58.302 INFO:tasks.workunit.client.0.smithi002.stderr:+ expect_failure cat testdir/test_chmod.155410 2023-11-16T18:26:58.303 INFO:tasks.workunit.client.0.smithi002.stderr:+ cat testdir/test_chmod.155410 2023-11-16T18:26:58.304 INFO:tasks.workunit.client.0.smithi002.stderr:cat: testdir/test_chmod.155410: Permission denied 2023-11-16T18:26:58.304 INFO:tasks.workunit.client.0.smithi002.stderr:+ return 0 2023-11-16T18:26:58.304 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo cat testdir/test_chmod.155410 2023-11-16T18:26:58.315 INFO:tasks.workunit.client.0.smithi002.stdout:foo 2023-11-16T18:26:58.317 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T18:26:58.317 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T18:26:58.414 INFO:tasks.workunit:Running workunit fs/misc/direct_io.py... 2023-11-16T18:26:58.414 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/direct_io.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/direct_io.py 2023-11-16T18:26:58.644 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T18:26:58.644 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T18:26:59.390 INFO:tasks.workunit.client.0.smithi002.stdout:ok 2023-11-16T18:26:59.395 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T18:26:59.395 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T18:26:59.444 INFO:tasks.workunit:Running workunit fs/misc/dirfrag.sh... 2023-11-16T18:26:59.444 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/dirfrag.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh 2023-11-16T18:26:59.497 INFO:tasks.workunit.client.0.smithi002.stdout:creating folder hierarchy 2023-11-16T18:26:59.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.507+0000 7fe5f3323700 1 -- 172.21.15.2:0/2836612251 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5ec100420 msgr2=0x7fe5ec10bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.507+0000 7fe5f3323700 1 --2- 172.21.15.2:0/2836612251 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5ec100420 0x7fe5ec10bfd0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009990 tx=0x7fe5e4009ca0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.507+0000 7fe5f3323700 1 -- 172.21.15.2:0/2836612251 shutdown_connections 2023-11-16T18:26:59.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.507+0000 7fe5f3323700 1 --2- 172.21.15.2:0/2836612251 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5ec100420 0x7fe5ec10bfd0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.507+0000 7fe5f3323700 1 --2- 172.21.15.2:0/2836612251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5ec105250 0x7fe5ec105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.507+0000 7fe5f3323700 1 --2- 172.21.15.2:0/2836612251 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5ec0ffa70 0x7fe5ec0ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.507+0000 7fe5f3323700 1 -- 172.21.15.2:0/2836612251 >> 172.21.15.2:0/2836612251 conn(0x7fe5ec0763a0 msgr2=0x7fe5ec0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:59.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.508+0000 7fe5f3323700 1 -- 172.21.15.2:0/2836612251 shutdown_connections 2023-11-16T18:26:59.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.508+0000 7fe5f3323700 1 -- 172.21.15.2:0/2836612251 wait complete. 2023-11-16T18:26:59.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.508+0000 7fe5f3323700 1 Processor -- start 2023-11-16T18:26:59.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.508+0000 7fe5f3323700 1 -- start start 2023-11-16T18:26:59.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.508+0000 7fe5f3323700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5ec0ffa70 0x7fe5ec100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.508+0000 7fe5f3323700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5ec100420 0x7fe5ec1013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.508+0000 7fe5f3323700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5ec105250 0x7fe5ec1a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.508+0000 7fe5f3323700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec101c50 con 0x7fe5ec100420 2023-11-16T18:26:59.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.508+0000 7fe5f3323700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec1a49b0 con 0x7fe5ec105250 2023-11-16T18:26:59.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f3323700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec1a4af0 con 0x7fe5ec0ffa70 2023-11-16T18:26:59.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f08be700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5ec100420 0x7fe5ec1013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f08be700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5ec100420 0x7fe5ec1013b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51648/0 (socket says 172.21.15.2:51648) 2023-11-16T18:26:59.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f08be700 1 -- 172.21.15.2:0/2846707612 learned_addr learned my addr 172.21.15.2:0/2846707612 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:59.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f18c0700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5ec105250 0x7fe5ec1a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5ec0ffa70 0x7fe5ec100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f08be700 1 -- 172.21.15.2:0/2846707612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5ec0ffa70 msgr2=0x7fe5ec100e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f08be700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5ec0ffa70 0x7fe5ec100e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f08be700 1 -- 172.21.15.2:0/2846707612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5ec105250 msgr2=0x7fe5ec1a4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f08be700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5ec105250 0x7fe5ec1a4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f08be700 1 -- 172.21.15.2:0/2846707612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e40095f0 con 0x7fe5ec100420 2023-11-16T18:26:59.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5ec0ffa70 0x7fe5ec100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:26:59.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f08be700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5ec100420 0x7fe5ec1013b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000c950 tx=0x7fe5e000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:59.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5ea7fc700 1 -- 172.21.15.2:0/2846707612 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e000cea0 con 0x7fe5ec100420 2023-11-16T18:26:59.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5f3323700 1 -- 172.21.15.2:0/2846707612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ec10f1e0 con 0x7fe5ec100420 2023-11-16T18:26:59.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5ea7fc700 1 -- 172.21.15.2:0/2846707612 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5e0004830 con 0x7fe5ec100420 2023-11-16T18:26:59.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.509+0000 7fe5ea7fc700 1 -- 172.21.15.2:0/2846707612 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0005460 con 0x7fe5ec100420 2023-11-16T18:26:59.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.510+0000 7fe5f3323700 1 -- 172.21.15.2:0/2846707612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ec10f050 con 0x7fe5ec100420 2023-11-16T18:26:59.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.513+0000 7fe5ea7fc700 1 -- 172.21.15.2:0/2846707612 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fe5e0007ac0 con 0x7fe5ec100420 2023-11-16T18:26:59.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.513+0000 7fe5ea7fc700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe5d40a12c0 0x7fe5d40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.513+0000 7fe5ea7fc700 1 -- 172.21.15.2:0/2846707612 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe5e00cccb0 con 0x7fe5ec100420 2023-11-16T18:26:59.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.513+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe5d40a12c0 0x7fe5d40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.514+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe5d40a12c0 0x7fe5d40a3770 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009fd0 tx=0x7fe5dc00b040 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:59.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.559+0000 7fe5ff4afb80 1 Processor -- start 2023-11-16T18:26:59.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.559+0000 7fe5ff4afb80 1 -- start start 2023-11-16T18:26:59.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5ff4afb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f188afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5ff4afb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 0x5615f188f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5ff4afb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f188fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5ff4afb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f17900e0 con 0x5615f188f930 2023-11-16T18:26:59.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5ff4afb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f17e2920 con 0x5615f188abe0 2023-11-16T18:26:59.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5ff4afb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f17dbdb0 con 0x5615f188b4f0 2023-11-16T18:26:59.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5f18c0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f188fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5f18c0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f188fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51658/0 (socket says 172.21.15.2:51658) 2023-11-16T18:26:59.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5f10bf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f188afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5f18c0700 1 -- 172.21.15.2:0/3086480985 learned_addr learned my addr 172.21.15.2:0/3086480985 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:59.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5f08be700 1 --2- 172.21.15.2:0/3086480985 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 0x5615f188f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5f18c0700 1 -- 172.21.15.2:0/3086480985 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 msgr2=0x5615f188f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5f18c0700 1 --2- 172.21.15.2:0/3086480985 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 0x5615f188f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5f18c0700 1 -- 172.21.15.2:0/3086480985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 msgr2=0x5615f188afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5f18c0700 1 --2- 172.21.15.2:0/3086480985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f188afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5f18c0700 1 -- 172.21.15.2:0/3086480985 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615f165d6b0 con 0x5615f188f930 2023-11-16T18:26:59.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/3086480985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f188afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:59.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.560+0000 7fe5f18c0700 1 --2- 172.21.15.2:0/3086480985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f188fd80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4015000 tx=0x7fe5e4005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61377eb3ffe267ff server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:59.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.561+0000 7fe5d37fe700 1 -- 172.21.15.2:0/3086480985 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e401d070 con 0x5615f188f930 2023-11-16T18:26:59.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.561+0000 7fe5d37fe700 1 -- 172.21.15.2:0/3086480985 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5e401e040 con 0x5615f188f930 2023-11-16T18:26:59.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.561+0000 7fe5d37fe700 1 -- 172.21.15.2:0/3086480985 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e40049c0 con 0x5615f188f930 2023-11-16T18:26:59.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.562+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3086480985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 msgr2=0x5615f188fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.562+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3086480985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f188fd80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4015000 tx=0x7fe5e4005be0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.562+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3086480985 shutdown_connections 2023-11-16T18:26:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.562+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3086480985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f188fd80 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.562+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3086480985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f188afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.562+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3086480985 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 0x5615f188f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.562+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3086480985 >> 172.21.15.2:0/3086480985 conn(0x5615f18803e0 msgr2=0x5615f1885bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.562+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3086480985 shutdown_connections 2023-11-16T18:26:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.562+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3086480985 wait complete. 2023-11-16T18:26:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.562+0000 7fe5ff4afb80 1 -- start start 2023-11-16T18:26:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.562+0000 7fe5ff4afb80 1 Processor -- start 2023-11-16T18:26:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5ff4afb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f1897030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5ff4afb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 0x5615f1894240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5ff4afb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f18947c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5ff4afb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f177d960 con 0x5615f188f930 2023-11-16T18:26:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5ff4afb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f177d600 con 0x5615f188abe0 2023-11-16T18:26:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5ff4afb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f165f9a0 con 0x5615f188b4f0 2023-11-16T18:26:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5ff4afb80 10 client.0 _ll_register_callbacks cb 0x5615f179b800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5f18c0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f18947c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5f18c0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f18947c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51664/0 (socket says 172.21.15.2:51664) 2023-11-16T18:26:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5f18c0700 1 -- 172.21.15.2:0/3882886610 learned_addr learned my addr 172.21.15.2:0/3882886610 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5f10bf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f1897030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5f08be700 1 --2- 172.21.15.2:0/3882886610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 0x5615f1894240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5f18c0700 1 -- 172.21.15.2:0/3882886610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 msgr2=0x5615f1894240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5f18c0700 1 --2- 172.21.15.2:0/3882886610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 0x5615f1894240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5f18c0700 1 -- 172.21.15.2:0/3882886610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 msgr2=0x5615f1897030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5f18c0700 1 --2- 172.21.15.2:0/3882886610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f1897030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.563+0000 7fe5f18c0700 1 -- 172.21.15.2:0/3882886610 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ec10c8d0 con 0x5615f188f930 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.564+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/3882886610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f1897030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.564+0000 7fe5f18c0700 1 --2- 172.21.15.2:0/3882886610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f18947c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009ce0 tx=0x7fe5e4022b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c3e193290ec8fc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.564+0000 7fe5d0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.564+0000 7fe5d0ff9700 1 -- 172.21.15.2:0/3882886610 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e401d030 con 0x5615f188f930 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.564+0000 7fe5d0ff9700 1 -- 172.21.15.2:0/3882886610 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe5e400f660 con 0x5615f188f930 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.564+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3882886610 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5615f187dba0 con 0x5615f188f930 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.564+0000 7fe5d0ff9700 1 -- 172.21.15.2:0/3882886610 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5e400f820 con 0x5615f188f930 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.564+0000 7fe5ff4afb80 10 client.16221 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.564+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3882886610 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5615f17a2210 con 0x5615f188f930 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.564+0000 7fe5d0ff9700 1 -- 172.21.15.2:0/3882886610 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5615f187dba0 con 0x5615f188f930 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.565+0000 7fe5d0ff9700 1 -- 172.21.15.2:0/3882886610 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe5e4038070 con 0x5615f188f930 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.565+0000 7fe5ff4afb80 10 client.16221 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.565+0000 7fe5ff4afb80 10 client.16221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.565+0000 7fe5ff4afb80 20 client.16221 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.565+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3882886610 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5615f18a4330 0x5615f18a6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.565+0000 7fe5ff4afb80 4 client.16221 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.565+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3882886610 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5615f179e120 con 0x5615f18a4330 2023-11-16T18:26:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.565+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/3882886610 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5615f18a4330 0x5615f18a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.565+0000 7fe5d0ff9700 10 client.16221 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.565+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/3882886610 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5615f18a4330 0x5615f18a6710 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27ac533e266975ea server_cookie=adb90179e33e9aa7 in_seq=0 out_seq=0 2023-11-16T18:26:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5d0ff9700 1 -- 172.21.15.2:0/3882886610 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5615f179e120 con 0x5615f18a4330 2023-11-16T18:26:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5d0ff9700 10 client.16221 handle_command_reply: tid=0 2023-11-16T18:26:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5ff4afb80 1 client.16221 shutdown 2023-11-16T18:26:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3882886610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 msgr2=0x5615f18947c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3882886610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f18947c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009ce0 tx=0x7fe5e4022b90 comp rx=0 tx=0).stop 2023-11-16T18:26:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3882886610 shutdown_connections 2023-11-16T18:26:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3882886610 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5615f18a4330 0x5615f18a6710 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3882886610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f18947c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3882886610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f1897030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3882886610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 0x5615f1894240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3882886610 >> 172.21.15.2:0/3882886610 conn(0x5615f18803e0 msgr2=0x5615f1890de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5d0ff9700 0 client.16221 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3882886610 shutdown_connections 2023-11-16T18:26:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3882886610 wait complete. 2023-11-16T18:26:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.568+0000 7fe5ff4afb80 20 client.16221 trim_cache size 0 max 16384 2023-11-16T18:26:59.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.586+0000 7fe5ff4afb80 1 Processor -- start 2023-11-16T18:26:59.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.586+0000 7fe5ff4afb80 1 -- start start 2023-11-16T18:26:59.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.586+0000 7fe5ff4afb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188abe0 0x5615f189ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.586+0000 7fe5ff4afb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188b4f0 0x5615f1896b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.586+0000 7fe5ff4afb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188f930 0x5615f1885600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.586+0000 7fe5ff4afb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f18977b0 con 0x5615f188abe0 2023-11-16T18:26:59.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.586+0000 7fe5ff4afb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f1894da0 con 0x5615f188b4f0 2023-11-16T18:26:59.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.586+0000 7fe5ff4afb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f188e3b0 con 0x5615f188f930 2023-11-16T18:26:59.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5f10bf700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188abe0 0x5615f189ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5f10bf700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188abe0 0x5615f189ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51668/0 (socket says 172.21.15.2:51668) 2023-11-16T18:26:59.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5f10bf700 1 -- 172.21.15.2:0/3616731373 learned_addr learned my addr 172.21.15.2:0/3616731373 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:59.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5f08be700 1 --2- 172.21.15.2:0/3616731373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188b4f0 0x5615f1896b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5f18c0700 1 --2- 172.21.15.2:0/3616731373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188f930 0x5615f1885600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5f10bf700 1 -- 172.21.15.2:0/3616731373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188f930 msgr2=0x5615f1885600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/3616731373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188f930 0x5615f1885600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5f10bf700 1 -- 172.21.15.2:0/3616731373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188b4f0 msgr2=0x5615f1896b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/3616731373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188b4f0 0x5615f1896b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5f10bf700 1 -- 172.21.15.2:0/3616731373 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615f1891fe0 con 0x5615f188abe0 2023-11-16T18:26:59.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/3616731373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188abe0 0x5615f189ba30 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009550 tx=0x7fe5dc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dae814e609df2e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:59.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5d3fff700 1 -- 172.21.15.2:0/3616731373 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc020040 con 0x5615f188abe0 2023-11-16T18:26:59.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5d3fff700 1 -- 172.21.15.2:0/3616731373 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5dc011040 con 0x5615f188abe0 2023-11-16T18:26:59.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.587+0000 7fe5d3fff700 1 -- 172.21.15.2:0/3616731373 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc02a310 con 0x5615f188abe0 2023-11-16T18:26:59.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.588+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3616731373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188abe0 msgr2=0x5615f189ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.588+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3616731373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188abe0 0x5615f189ba30 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009550 tx=0x7fe5dc012000 comp rx=0 tx=0).stop 2023-11-16T18:26:59.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.588+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3616731373 shutdown_connections 2023-11-16T18:26:59.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.588+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3616731373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188f930 0x5615f1885600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.588+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3616731373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188abe0 0x5615f189ba30 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.588+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/3616731373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188b4f0 0x5615f1896b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.588+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3616731373 >> 172.21.15.2:0/3616731373 conn(0x5615f18803e0 msgr2=0x5615f1894520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:59.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.588+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3616731373 shutdown_connections 2023-11-16T18:26:59.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.588+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/3616731373 wait complete. 2023-11-16T18:26:59.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.588+0000 7fe5ff4afb80 1 -- start start 2023-11-16T18:26:59.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.589+0000 7fe5ff4afb80 1 Processor -- start 2023-11-16T18:26:59.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.589+0000 7fe5ff4afb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f189a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.589+0000 7fe5ff4afb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 0x5615f189a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.589+0000 7fe5ff4afb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f189ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.589+0000 7fe5ff4afb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f18f8190 con 0x5615f188f930 2023-11-16T18:26:59.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.589+0000 7fe5ff4afb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f18f82d0 con 0x5615f188abe0 2023-11-16T18:26:59.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.589+0000 7fe5ff4afb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f18f8410 con 0x5615f188b4f0 2023-11-16T18:26:59.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.589+0000 7fe5ff4afb80 10 client.0 _ll_register_callbacks cb 0x7fe5dc009050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:59.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.589+0000 7fe5f18c0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f189ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.589+0000 7fe5f10bf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f189a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.589+0000 7fe5f18c0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f189ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51678/0 (socket says 172.21.15.2:51678) 2023-11-16T18:26:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.589+0000 7fe5f18c0700 1 -- 172.21.15.2:0/2837440159 learned_addr learned my addr 172.21.15.2:0/2837440159 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.589+0000 7fe5f08be700 1 --2- 172.21.15.2:0/2837440159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 0x5615f189a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5f18c0700 1 -- 172.21.15.2:0/2837440159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 msgr2=0x5615f189a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5f18c0700 1 --2- 172.21.15.2:0/2837440159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 0x5615f189a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5f18c0700 1 -- 172.21.15.2:0/2837440159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 msgr2=0x5615f189a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5f18c0700 1 --2- 172.21.15.2:0/2837440159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f189a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5f18c0700 1 -- 172.21.15.2:0/2837440159 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5dc007d30 con 0x5615f188f930 2023-11-16T18:26:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/2837440159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f189a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5f18c0700 1 --2- 172.21.15.2:0/2837440159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f189ab50 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4000c00 tx=0x7fe5e402cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91c0c34945a970cc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5d0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:26:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5d0ff9700 1 -- 172.21.15.2:0/2837440159 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e401d040 con 0x5615f188f930 2023-11-16T18:26:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5d0ff9700 1 -- 172.21.15.2:0/2837440159 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe5e4000e40 con 0x5615f188f930 2023-11-16T18:26:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/2837440159 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5e401e040 con 0x5615f188f930 2023-11-16T18:26:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5d0ff9700 1 -- 172.21.15.2:0/2837440159 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5e4005bc0 con 0x5615f188f930 2023-11-16T18:26:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5ff4afb80 10 client.16227 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/2837440159 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5615f1609e60 con 0x5615f188f930 2023-11-16T18:26:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.590+0000 7fe5d0ff9700 1 -- 172.21.15.2:0/2837440159 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5e401e040 con 0x5615f188f930 2023-11-16T18:26:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.591+0000 7fe5d0ff9700 1 -- 172.21.15.2:0/2837440159 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe5e4038070 con 0x5615f188f930 2023-11-16T18:26:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.591+0000 7fe5ff4afb80 10 client.16227 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.591+0000 7fe5ff4afb80 10 client.16227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:59.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.591+0000 7fe5ff4afb80 20 client.16227 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:59.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.591+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/2837440159 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5615f18f96a0 0x5615f18fba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.591+0000 7fe5ff4afb80 4 client.16227 mds_command: new command op to 24473 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T18:26:59.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.591+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/2837440159 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5615f17aa6e0 con 0x5615f18f96a0 2023-11-16T18:26:59.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.591+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/2837440159 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5615f18f96a0 0x5615f18fba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.591+0000 7fe5d0ff9700 10 client.16227 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:59.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.591+0000 7fe5f10bf700 1 --2- 172.21.15.2:0/2837440159 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5615f18f96a0 0x5615f18fba80 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4d9c475b67df2cf server_cookie=e328f5fd9fbb8eec in_seq=0 out_seq=0 2023-11-16T18:26:59.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.595+0000 7fe5d0ff9700 1 -- 172.21.15.2:0/2837440159 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5615f17aa6e0 con 0x5615f18f96a0 2023-11-16T18:26:59.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.595+0000 7fe5d0ff9700 10 client.16227 handle_command_reply: tid=0 2023-11-16T18:26:59.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.595+0000 7fe5ff4afb80 1 client.16227 shutdown 2023-11-16T18:26:59.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.595+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/2837440159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 msgr2=0x5615f189ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.595+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/2837440159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f189ab50 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4000c00 tx=0x7fe5e402cf70 comp rx=0 tx=0).stop 2023-11-16T18:26:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/2837440159 shutdown_connections 2023-11-16T18:26:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/2837440159 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5615f18f96a0 0x5615f18fba80 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/2837440159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5615f188f930 0x5615f189ab50 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/2837440159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615f188abe0 0x5615f189a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5ff4afb80 1 --2- 172.21.15.2:0/2837440159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f188b4f0 0x5615f189a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5d0ff9700 0 client.16227 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/2837440159 >> 172.21.15.2:0/2837440159 conn(0x5615f18803e0 msgr2=0x5615f1894520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/2837440159 shutdown_connections 2023-11-16T18:26:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5ff4afb80 1 -- 172.21.15.2:0/2837440159 wait complete. 2023-11-16T18:26:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5ff4afb80 20 client.16227 trim_cache size 0 max 16384 2023-11-16T18:26:59.601 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:26:59.601 INFO:teuthology.orchestra.run.smithi002.stdout: "return_code": 0, 2023-11-16T18:26:59.601 INFO:teuthology.orchestra.run.smithi002.stdout: "scrub_tag": "578db078-ceba-435a-b0d9-6f2bac227202", 2023-11-16T18:26:59.601 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": "asynchronous" 2023-11-16T18:26:59.601 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:26:59.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5f3323700 1 -- 172.21.15.2:0/2846707612 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe5d40a12c0 msgr2=0x7fe5d40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5f3323700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe5d40a12c0 0x7fe5d40a3770 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009fd0 tx=0x7fe5dc00b040 comp rx=0 tx=0).stop 2023-11-16T18:26:59.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5f3323700 1 -- 172.21.15.2:0/2846707612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5ec100420 msgr2=0x7fe5ec1013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5f3323700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5ec100420 0x7fe5ec1013b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000c950 tx=0x7fe5e000cc60 comp rx=0 tx=0).stop 2023-11-16T18:26:59.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5f3323700 1 -- 172.21.15.2:0/2846707612 shutdown_connections 2023-11-16T18:26:59.602 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5f3323700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe5d40a12c0 0x7fe5d40a3770 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5f3323700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5ec0ffa70 0x7fe5ec100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5f3323700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5ec105250 0x7fe5ec1a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5f3323700 1 --2- 172.21.15.2:0/2846707612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5ec100420 0x7fe5ec1013b0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5f3323700 1 -- 172.21.15.2:0/2846707612 >> 172.21.15.2:0/2846707612 conn(0x7fe5ec0763a0 msgr2=0x7fe5ec0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:59.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.596+0000 7fe5f3323700 1 -- 172.21.15.2:0/2846707612 shutdown_connections 2023-11-16T18:26:59.603 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.597+0000 7fe5f3323700 1 -- 172.21.15.2:0/2846707612 wait complete. 2023-11-16T18:26:59.612 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:26:59.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.745+0000 7f6bf5cd3700 1 -- 172.21.15.2:0/1629210195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf0105250 msgr2=0x7f6bf0105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.745+0000 7f6bf5cd3700 1 --2- 172.21.15.2:0/1629210195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf0105250 0x7f6bf0105620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6be00099c0 tx=0x7f6be0009cd0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.745+0000 7f6bf5cd3700 1 -- 172.21.15.2:0/1629210195 shutdown_connections 2023-11-16T18:26:59.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.745+0000 7f6bf5cd3700 1 --2- 172.21.15.2:0/1629210195 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bf0100420 0x7f6bf010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.745+0000 7f6bf5cd3700 1 --2- 172.21.15.2:0/1629210195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf0105250 0x7f6bf0105620 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.745+0000 7f6bf5cd3700 1 --2- 172.21.15.2:0/1629210195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bf00ffa70 0x7f6bf00ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.745+0000 7f6bf5cd3700 1 -- 172.21.15.2:0/1629210195 >> 172.21.15.2:0/1629210195 conn(0x7f6bf00763a0 msgr2=0x7f6bf00767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:59.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.745+0000 7f6bf5cd3700 1 -- 172.21.15.2:0/1629210195 shutdown_connections 2023-11-16T18:26:59.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.745+0000 7f6bf5cd3700 1 -- 172.21.15.2:0/1629210195 wait complete. 2023-11-16T18:26:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.746+0000 7f6bf5cd3700 1 Processor -- start 2023-11-16T18:26:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.746+0000 7f6bf5cd3700 1 -- start start 2023-11-16T18:26:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.746+0000 7f6bf5cd3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bf00ffa70 0x7f6bf0197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.746+0000 7f6bf5cd3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf0100420 0x7f6bf0197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.746+0000 7f6bf5cd3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bf0105250 0x7f6bf01a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.746+0000 7f6bf5cd3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf01a27e0 con 0x7f6bf0105250 2023-11-16T18:26:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.746+0000 7f6bf5cd3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf01a2920 con 0x7f6bf0100420 2023-11-16T18:26:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.746+0000 7f6bf5cd3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf01a2a60 con 0x7f6bf00ffa70 2023-11-16T18:26:59.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.746+0000 7f6beeffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf0100420 0x7f6bf0197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.746+0000 7f6beffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bf0105250 0x7f6bf01a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.746+0000 7f6beeffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf0100420 0x7f6bf0197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42128/0 (socket says 172.21.15.2:42128) 2023-11-16T18:26:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6beeffd700 1 -- 172.21.15.2:0/3303648701 learned_addr learned my addr 172.21.15.2:0/3303648701 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.746+0000 7f6beffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bf0105250 0x7f6bf01a22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51692/0 (socket says 172.21.15.2:51692) 2023-11-16T18:26:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bf00ffa70 0x7f6bf0197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6beeffd700 1 -- 172.21.15.2:0/3303648701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bf00ffa70 msgr2=0x7f6bf0197220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6beeffd700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bf00ffa70 0x7f6bf0197220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6beeffd700 1 -- 172.21.15.2:0/3303648701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bf0105250 msgr2=0x7f6bf01a22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6beeffd700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bf0105250 0x7f6bf01a22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6beeffd700 1 -- 172.21.15.2:0/3303648701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be00095f0 con 0x7f6bf0100420 2023-11-16T18:26:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bf00ffa70 0x7f6bf0197220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:59.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6beeffd700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf0100420 0x7f6bf0197760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6be400e9a0 tx=0x7f6be400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6becff9700 1 -- 172.21.15.2:0/3303648701 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4009790 con 0x7f6bf0100420 2023-11-16T18:26:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6becff9700 1 -- 172.21.15.2:0/3303648701 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6be400cbd0 con 0x7f6bf0100420 2023-11-16T18:26:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6bf5cd3700 1 -- 172.21.15.2:0/3303648701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf010f1b0 con 0x7f6bf0100420 2023-11-16T18:26:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6becff9700 1 -- 172.21.15.2:0/3303648701 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4018680 con 0x7f6bf0100420 2023-11-16T18:26:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.747+0000 7f6bf5cd3700 1 -- 172.21.15.2:0/3303648701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf010f020 con 0x7f6bf0100420 2023-11-16T18:26:59.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.750+0000 7f6becff9700 1 -- 172.21.15.2:0/3303648701 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f6be400cd40 con 0x7f6bf0100420 2023-11-16T18:26:59.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.751+0000 7f6becff9700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6bd80a1340 0x7f6bd80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.751+0000 7f6becff9700 1 -- 172.21.15.2:0/3303648701 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6be4014070 con 0x7f6bf0100420 2023-11-16T18:26:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.751+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6bd80a1340 0x7f6bd80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.751+0000 7f6c01e5fb80 1 Processor -- start 2023-11-16T18:26:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.751+0000 7f6c01e5fb80 1 -- start start 2023-11-16T18:26:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6c01e5fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56406358eb60 0x56406358ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6bd80a1340 0x7f6bd80a37f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009990 tx=0x7f6be0005fb0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6c01e5fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x564063593370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6c01e5fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640635938b0 0x564063593d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6c01e5fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x564063494060 con 0x56406358f470 2023-11-16T18:26:59.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6c01e5fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640634e68a0 con 0x5640635938b0 2023-11-16T18:26:59.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6c01e5fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640634dfd30 con 0x56406358eb60 2023-11-16T18:26:59.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6beeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x564063593370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6beeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x564063593370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51698/0 (socket says 172.21.15.2:51698) 2023-11-16T18:26:59.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6beeffd700 1 -- 172.21.15.2:0/68984563 learned_addr learned my addr 172.21.15.2:0/68984563 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:59.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6beffff700 1 --2- 172.21.15.2:0/68984563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640635938b0 0x564063593d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/68984563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56406358eb60 0x56406358ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6beeffd700 1 -- 172.21.15.2:0/68984563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56406358eb60 msgr2=0x56406358ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6beeffd700 1 --2- 172.21.15.2:0/68984563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56406358eb60 0x56406358ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6beeffd700 1 -- 172.21.15.2:0/68984563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640635938b0 msgr2=0x564063593d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6beeffd700 1 --2- 172.21.15.2:0/68984563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640635938b0 0x564063593d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6beeffd700 1 -- 172.21.15.2:0/68984563 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564063361630 con 0x56406358f470 2023-11-16T18:26:59.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/68984563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56406358eb60 0x56406358ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:26:59.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6beffff700 1 --2- 172.21.15.2:0/68984563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640635938b0 0x564063593d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:59.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.752+0000 7f6beeffd700 1 --2- 172.21.15.2:0/68984563 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x564063593370 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6be4096a80 tx=0x7f6be40999f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66efccae771027c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:59.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.753+0000 7f6bd5ffb700 1 -- 172.21.15.2:0/68984563 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be409e040 con 0x56406358f470 2023-11-16T18:26:59.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.753+0000 7f6bd5ffb700 1 -- 172.21.15.2:0/68984563 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6be40a2460 con 0x56406358f470 2023-11-16T18:26:59.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.753+0000 7f6bd5ffb700 1 -- 172.21.15.2:0/68984563 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be40a2230 con 0x56406358f470 2023-11-16T18:26:59.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.753+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/68984563 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 msgr2=0x564063593370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.753+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/68984563 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x564063593370 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6be4096a80 tx=0x7f6be40999f0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.754+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/68984563 shutdown_connections 2023-11-16T18:26:59.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.754+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/68984563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56406358eb60 0x56406358ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.754+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/68984563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640635938b0 0x564063593d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.754+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/68984563 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x564063593370 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.754+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/68984563 >> 172.21.15.2:0/68984563 conn(0x564063584360 msgr2=0x564063589b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.754+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/68984563 shutdown_connections 2023-11-16T18:26:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.754+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/68984563 wait complete. 2023-11-16T18:26:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.754+0000 7f6c01e5fb80 1 -- start start 2023-11-16T18:26:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.754+0000 7f6c01e5fb80 1 Processor -- start 2023-11-16T18:26:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6c01e5fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6c01e5fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x5640635982e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6c01e5fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640635938b0 0x5640635989d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6c01e5fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640634818e0 con 0x56406358f470 2023-11-16T18:26:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6c01e5fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564063481580 con 0x56406358eb60 2023-11-16T18:26:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6c01e5fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x564063363920 con 0x5640635938b0 2023-11-16T18:26:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6c01e5fb80 10 client.0 _ll_register_callbacks cb 0x56406349f780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6beeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x5640635982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6beeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x5640635982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51704/0 (socket says 172.21.15.2:51704) 2023-11-16T18:26:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6beeffd700 1 -- 172.21.15.2:0/2997984565 learned_addr learned my addr 172.21.15.2:0/2997984565 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/2997984565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6bef7fe700 1 -- 172.21.15.2:0/2997984565 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640635938b0 msgr2=0x5640635989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/2997984565 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640635938b0 0x5640635989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6bef7fe700 1 -- 172.21.15.2:0/2997984565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 msgr2=0x5640635982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/2997984565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x5640635982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6bef7fe700 1 -- 172.21.15.2:0/2997984565 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf010f020 con 0x56406358eb60 2023-11-16T18:26:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6beeffd700 1 --2- 172.21.15.2:0/2997984565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x5640635982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/2997984565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359b0d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6be00098e0 tx=0x7f6be0009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2908e53169db4233 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.755+0000 7f6bceffd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:26:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6bceffd700 1 -- 172.21.15.2:0/2997984565 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be002e040 con 0x56406358eb60 2023-11-16T18:26:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6bceffd700 1 -- 172.21.15.2:0/2997984565 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6be0033480 con 0x56406358eb60 2023-11-16T18:26:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/2997984565 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564063581b20 con 0x56406358eb60 2023-11-16T18:26:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6bceffd700 1 -- 172.21.15.2:0/2997984565 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6be0022070 con 0x56406358eb60 2023-11-16T18:26:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6c01e5fb80 10 client.25790 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/2997984565 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5640634a6190 con 0x56406358eb60 2023-11-16T18:26:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6bceffd700 1 -- 172.21.15.2:0/2997984565 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564063581b20 con 0x56406358eb60 2023-11-16T18:26:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6bceffd700 1 -- 172.21.15.2:0/2997984565 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6be00287a0 con 0x56406358eb60 2023-11-16T18:26:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6c01e5fb80 10 client.25790 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6c01e5fb80 10 client.25790 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6c01e5fb80 20 client.25790 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/2997984565 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5640635a8800 0x5640635aac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6c01e5fb80 4 client.25790 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:26:59.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/2997984565 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5640634a20a0 con 0x5640635a8800 2023-11-16T18:26:59.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.756+0000 7f6beeffd700 1 --2- 172.21.15.2:0/2997984565 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5640635a8800 0x5640635aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.757+0000 7f6bceffd700 10 client.25790 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:59.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.757+0000 7f6beeffd700 1 --2- 172.21.15.2:0/2997984565 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5640635a8800 0x5640635aac10 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26559f31e537036d server_cookie=7acc3c5b01edc982 in_seq=0 out_seq=0 2023-11-16T18:26:59.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6bceffd700 1 -- 172.21.15.2:0/2997984565 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5640634a20a0 con 0x5640635a8800 2023-11-16T18:26:59.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6bceffd700 10 client.25790 handle_command_reply: tid=0 2023-11-16T18:26:59.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6c01e5fb80 1 client.25790 shutdown 2023-11-16T18:26:59.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/2997984565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 msgr2=0x56406359b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/2997984565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359b0d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6be00098e0 tx=0x7f6be0009cd0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/2997984565 shutdown_connections 2023-11-16T18:26:59.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/2997984565 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5640635a8800 0x5640635aac10 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/2997984565 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640635938b0 0x5640635989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/2997984565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359b0d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/2997984565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x5640635982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6bceffd700 0 client.25790 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:59.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/2997984565 >> 172.21.15.2:0/2997984565 conn(0x564063584360 msgr2=0x564063594d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:59.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/2997984565 shutdown_connections 2023-11-16T18:26:59.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/2997984565 wait complete. 2023-11-16T18:26:59.768 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.759+0000 7f6c01e5fb80 20 client.25790 trim_cache size 0 max 16384 2023-11-16T18:26:59.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.777+0000 7f6c01e5fb80 1 Processor -- start 2023-11-16T18:26:59.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.777+0000 7f6c01e5fb80 1 -- start start 2023-11-16T18:26:59.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.777+0000 7f6c01e5fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.777+0000 7f6c01e5fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56406358f470 0x56406359ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.777+0000 7f6c01e5fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5640635938b0 0x564063597b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.777+0000 7f6c01e5fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640635819a0 con 0x5640635938b0 2023-11-16T18:26:59.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.777+0000 7f6c01e5fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640635a4800 con 0x56406358eb60 2023-11-16T18:26:59.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.777+0000 7f6c01e5fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x564063592410 con 0x56406358f470 2023-11-16T18:26:59.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.777+0000 7f6beffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5640635938b0 0x564063597b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6bef7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6beffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5640635938b0 0x564063597b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51714/0 (socket says 172.21.15.2:51714) 2023-11-16T18:26:59.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6beffff700 1 -- 172.21.15.2:0/3423275828 learned_addr learned my addr 172.21.15.2:0/3423275828 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:59.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6beeffd700 1 --2- 172.21.15.2:0/3423275828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56406358f470 0x56406359ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6beffff700 1 -- 172.21.15.2:0/3423275828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56406358f470 msgr2=0x56406359ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6beffff700 1 --2- 172.21.15.2:0/3423275828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56406358f470 0x56406359ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6beffff700 1 -- 172.21.15.2:0/3423275828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 msgr2=0x56406359f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6beffff700 1 --2- 172.21.15.2:0/3423275828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6beffff700 1 -- 172.21.15.2:0/3423275828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640634a5ee0 con 0x5640635938b0 2023-11-16T18:26:59.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/3423275828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:59.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6beffff700 1 --2- 172.21.15.2:0/3423275828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5640635938b0 0x564063597b80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6be8011830 tx=0x7f6be8011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1d36d469845a3b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:59.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6bd67fc700 1 -- 172.21.15.2:0/3423275828 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be800e040 con 0x5640635938b0 2023-11-16T18:26:59.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6bd67fc700 1 -- 172.21.15.2:0/3423275828 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6be800f040 con 0x5640635938b0 2023-11-16T18:26:59.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.778+0000 7f6bd67fc700 1 -- 172.21.15.2:0/3423275828 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8016a00 con 0x5640635938b0 2023-11-16T18:26:59.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.779+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/3423275828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5640635938b0 msgr2=0x564063597b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.779+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/3423275828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5640635938b0 0x564063597b80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6be8011830 tx=0x7f6be8011b40 comp rx=0 tx=0).stop 2023-11-16T18:26:59.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.779+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/3423275828 shutdown_connections 2023-11-16T18:26:59.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.779+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/3423275828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5640635938b0 0x564063597b80 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.779+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/3423275828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.779+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/3423275828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56406358f470 0x56406359ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.779+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/3423275828 >> 172.21.15.2:0/3423275828 conn(0x564063584360 msgr2=0x5640635a40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:59.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.779+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/3423275828 shutdown_connections 2023-11-16T18:26:59.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.779+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/3423275828 wait complete. 2023-11-16T18:26:59.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.779+0000 7f6c01e5fb80 1 -- start start 2023-11-16T18:26:59.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.780+0000 7f6c01e5fb80 1 Processor -- start 2023-11-16T18:26:59.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.780+0000 7f6c01e5fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.780+0000 7f6c01e5fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x56406359f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.780+0000 7f6c01e5fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640635938b0 0x56406359d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.780+0000 7f6c01e5fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56406359df50 con 0x56406358f470 2023-11-16T18:26:59.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.780+0000 7f6c01e5fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56406359e0c0 con 0x56406358eb60 2023-11-16T18:26:59.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.780+0000 7f6c01e5fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56406359e280 con 0x5640635938b0 2023-11-16T18:26:59.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.780+0000 7f6c01e5fb80 10 client.0 _ll_register_callbacks cb 0x7f6be000be60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:26:59.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.780+0000 7f6beeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x56406359f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.780+0000 7f6bef7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.780+0000 7f6beeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x56406359f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51718/0 (socket says 172.21.15.2:51718) 2023-11-16T18:26:59.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.780+0000 7f6bef7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42154/0 (socket says 172.21.15.2:42154) 2023-11-16T18:26:59.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.780+0000 7f6beeffd700 1 -- 172.21.15.2:0/4104153243 learned_addr learned my addr 172.21.15.2:0/4104153243 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:26:59.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6bef7fe700 1 -- 172.21.15.2:0/4104153243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640635938b0 msgr2=0x56406359d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/4104153243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640635938b0 0x56406359d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6bef7fe700 1 -- 172.21.15.2:0/4104153243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 msgr2=0x56406359f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:26:59.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/4104153243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x56406359f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6bef7fe700 1 -- 172.21.15.2:0/4104153243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640634a6190 con 0x56406358eb60 2023-11-16T18:26:59.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6beeffd700 1 --2- 172.21.15.2:0/4104153243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x56406359f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:26:59.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6bef7fe700 1 --2- 172.21.15.2:0/4104153243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359f240 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6be0004820 tx=0x7f6be0028620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8137b1bb5a0e6124 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:26:59.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6bceffd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:26:59.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6bceffd700 1 -- 172.21.15.2:0/4104153243 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be0022070 con 0x56406358eb60 2023-11-16T18:26:59.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6bceffd700 1 -- 172.21.15.2:0/4104153243 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6be000b970 con 0x56406358eb60 2023-11-16T18:26:59.787 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/4104153243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6be40a2460 con 0x56406358eb60 2023-11-16T18:26:59.788 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6bceffd700 1 -- 172.21.15.2:0/4104153243 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6be002e040 con 0x56406358eb60 2023-11-16T18:26:59.788 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6c01e5fb80 10 client.25796 fetch_fsmap learned FSMap version 24 2023-11-16T18:26:59.788 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/4104153243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f6be80114e0 con 0x56406358eb60 2023-11-16T18:26:59.788 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.781+0000 7f6bceffd700 1 -- 172.21.15.2:0/4104153243 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6be40a2460 con 0x56406358eb60 2023-11-16T18:26:59.788 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.782+0000 7f6bceffd700 1 -- 172.21.15.2:0/4104153243 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6be003b070 con 0x56406358eb60 2023-11-16T18:26:59.788 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.782+0000 7f6c01e5fb80 10 client.25796 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:26:59.788 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.782+0000 7f6c01e5fb80 10 client.25796 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:26:59.788 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.782+0000 7f6c01e5fb80 20 client.25796 populate_metadata read hostname 'smithi002' 2023-11-16T18:26:59.788 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.782+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/4104153243 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5640635fd300 0x5640635ff6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:26:59.788 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.782+0000 7f6c01e5fb80 4 client.25796 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:26:59.788 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.782+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/4104153243 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5640634ae660 con 0x5640635fd300 2023-11-16T18:26:59.788 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.782+0000 7f6beeffd700 1 --2- 172.21.15.2:0/4104153243 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5640635fd300 0x5640635ff6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:26:59.789 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.782+0000 7f6bceffd700 10 client.25796 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:59.789 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.782+0000 7f6beeffd700 1 --2- 172.21.15.2:0/4104153243 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5640635fd300 0x5640635ff6e0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84bf59b9ca3c2fd4 server_cookie=402f3046b3595050 in_seq=0 out_seq=0 2023-11-16T18:26:59.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6bceffd700 1 -- 172.21.15.2:0/4104153243 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5640634ae660 con 0x5640635fd300 2023-11-16T18:26:59.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6bceffd700 10 client.25796 handle_command_reply: tid=0 2023-11-16T18:26:59.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6c01e5fb80 1 client.25796 shutdown 2023-11-16T18:26:59.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/4104153243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 msgr2=0x56406359f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/4104153243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359f240 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6be0004820 tx=0x7f6be0028620 comp rx=0 tx=0).stop 2023-11-16T18:26:59.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/4104153243 shutdown_connections 2023-11-16T18:26:59.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/4104153243 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5640635fd300 0x5640635ff6e0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/4104153243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640635938b0 0x56406359d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/4104153243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56406358eb60 0x56406359f240 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6c01e5fb80 1 --2- 172.21.15.2:0/4104153243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56406358f470 0x56406359f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/4104153243 >> 172.21.15.2:0/4104153243 conn(0x564063584360 msgr2=0x56406358d960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:59.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6bceffd700 0 client.25796 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:26:59.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/4104153243 shutdown_connections 2023-11-16T18:26:59.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6c01e5fb80 1 -- 172.21.15.2:0/4104153243 wait complete. 2023-11-16T18:26:59.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.785+0000 7f6c01e5fb80 20 client.25796 trim_cache size 0 max 16384 2023-11-16T18:26:59.791 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:26:59.792 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-16T18:26:59.792 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": { 2023-11-16T18:26:59.792 INFO:teuthology.orchestra.run.smithi002.stdout: "578db078-ceba-435a-b0d9-6f2bac227202": { 2023-11-16T18:26:59.792 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/", 2023-11-16T18:26:59.792 INFO:teuthology.orchestra.run.smithi002.stdout: "tag": "578db078-ceba-435a-b0d9-6f2bac227202", 2023-11-16T18:26:59.792 INFO:teuthology.orchestra.run.smithi002.stdout: "options": "recursive,force" 2023-11-16T18:26:59.792 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:26:59.792 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:26:59.792 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:26:59.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.786+0000 7f6bf5cd3700 1 -- 172.21.15.2:0/3303648701 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6bd80a1340 msgr2=0x7f6bd80a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.786+0000 7f6bf5cd3700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6bd80a1340 0x7f6bd80a37f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009990 tx=0x7f6be0005fb0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.786+0000 7f6bf5cd3700 1 -- 172.21.15.2:0/3303648701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf0100420 msgr2=0x7f6bf0197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:26:59.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.786+0000 7f6bf5cd3700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf0100420 0x7f6bf0197760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6be400e9a0 tx=0x7f6be400ecb0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.786+0000 7f6bf5cd3700 1 -- 172.21.15.2:0/3303648701 shutdown_connections 2023-11-16T18:26:59.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.786+0000 7f6bf5cd3700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6bd80a1340 0x7f6bd80a37f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.786+0000 7f6bf5cd3700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bf00ffa70 0x7f6bf0197220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.786+0000 7f6bf5cd3700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bf0105250 0x7f6bf01a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.786+0000 7f6bf5cd3700 1 --2- 172.21.15.2:0/3303648701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf0100420 0x7f6bf0197760 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:26:59.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.786+0000 7f6bf5cd3700 1 -- 172.21.15.2:0/3303648701 >> 172.21.15.2:0/3303648701 conn(0x7f6bf00763a0 msgr2=0x7f6bf00fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:26:59.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.786+0000 7f6bf5cd3700 1 -- 172.21.15.2:0/3303648701 shutdown_connections 2023-11-16T18:26:59.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:26:59.786+0000 7f6bf5cd3700 1 -- 172.21.15.2:0/3303648701 wait complete. 2023-11-16T18:26:59.802 INFO:tasks.cephfs.filesystem:scrub status for tag:578db078-ceba-435a-b0d9-6f2bac227202 - {'path': '/', 'tag': '578db078-ceba-435a-b0d9-6f2bac227202', 'options': 'recursive,force'} 2023-11-16T18:27:00.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:00 smithi002 ceph-mon[98807]: pgmap v67: 129 pgs: 129 active+clean; 2.9 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 12 op/s 2023-11-16T18:27:00.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:00 smithi040 ceph-mon[100251]: pgmap v67: 129 pgs: 129 active+clean; 2.9 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 12 op/s 2023-11-16T18:27:00.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:00 smithi188 ceph-mon[100639]: pgmap v67: 129 pgs: 129 active+clean; 2.9 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 12 op/s 2023-11-16T18:27:01.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:01 smithi002 ceph-mon[98807]: scrub queued for path: / 2023-11-16T18:27:01.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:01 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:01.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:01 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:27:01.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:01 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:01.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:01 smithi040 ceph-mon[100251]: scrub queued for path: / 2023-11-16T18:27:01.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:01 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:01.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:01 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:27:01.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:01 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:01.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:01 smithi188 ceph-mon[100639]: scrub queued for path: / 2023-11-16T18:27:01.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:01 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:01.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:01 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:27:01.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:01 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:02.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:02 smithi002 ceph-mon[98807]: pgmap v68: 129 pgs: 129 active+clean; 2.9 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 12 op/s 2023-11-16T18:27:02.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:02 smithi002 ceph-mon[98807]: scrub summary: idle 2023-11-16T18:27:02.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:02 smithi040 ceph-mon[100251]: pgmap v68: 129 pgs: 129 active+clean; 2.9 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 12 op/s 2023-11-16T18:27:02.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:02 smithi040 ceph-mon[100251]: scrub summary: idle 2023-11-16T18:27:02.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:02 smithi188 ceph-mon[100639]: pgmap v68: 129 pgs: 129 active+clean; 2.9 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 12 op/s 2023-11-16T18:27:02.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:02 smithi188 ceph-mon[100639]: scrub summary: idle 2023-11-16T18:27:04.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:04 smithi002 ceph-mon[98807]: pgmap v69: 129 pgs: 129 active+clean; 6.8 MiB data, 94 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 445 KiB/s wr, 39 op/s 2023-11-16T18:27:04.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:04 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:27:04.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:04 smithi040 ceph-mon[100251]: pgmap v69: 129 pgs: 129 active+clean; 6.8 MiB data, 94 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 445 KiB/s wr, 39 op/s 2023-11-16T18:27:04.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:04 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:27:04.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:04 smithi188 ceph-mon[100639]: pgmap v69: 129 pgs: 129 active+clean; 6.8 MiB data, 94 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 445 KiB/s wr, 39 op/s 2023-11-16T18:27:04.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:04 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:27:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.631+0000 7f58bde47700 1 -- 172.21.15.2:0/13506692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b80ff3c0 msgr2=0x7f58b80ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.631+0000 7f58bde47700 1 --2- 172.21.15.2:0/13506692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b80ff3c0 0x7f58b80ff790 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f58a80099c0 tx=0x7f58a8009cd0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.631+0000 7f58bde47700 1 -- 172.21.15.2:0/13506692 shutdown_connections 2023-11-16T18:27:04.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.631+0000 7f58bde47700 1 --2- 172.21.15.2:0/13506692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58b8104400 0x7f58b810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.631+0000 7f58bde47700 1 --2- 172.21.15.2:0/13506692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b80ff3c0 0x7f58b80ff790 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.631+0000 7f58bde47700 1 --2- 172.21.15.2:0/13506692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b80ffd60 0x7f58b8103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.631+0000 7f58bde47700 1 -- 172.21.15.2:0/13506692 >> 172.21.15.2:0/13506692 conn(0x7f58b80763a0 msgr2=0x7f58b80767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:04.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.631+0000 7f58bde47700 1 -- 172.21.15.2:0/13506692 shutdown_connections 2023-11-16T18:27:04.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.631+0000 7f58bde47700 1 -- 172.21.15.2:0/13506692 wait complete. 2023-11-16T18:27:04.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.632+0000 7f58bde47700 1 Processor -- start 2023-11-16T18:27:04.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.632+0000 7f58bde47700 1 -- start start 2023-11-16T18:27:04.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.632+0000 7f58bde47700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58b80ff3c0 0x7f58b81963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.632+0000 7f58bde47700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b80ffd60 0x7f58b8196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.632+0000 7f58bde47700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b8104400 0x7f58b819ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.632+0000 7f58bde47700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b819b3c0 con 0x7f58b8104400 2023-11-16T18:27:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.632+0000 7f58bde47700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b819b530 con 0x7f58b80ff3c0 2023-11-16T18:27:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.632+0000 7f58bde47700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b819b6f0 con 0x7f58b80ffd60 2023-11-16T18:27:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.632+0000 7f58b7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b8104400 0x7f58b819ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.632+0000 7f58b7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b8104400 0x7f58b819ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51732/0 (socket says 172.21.15.2:51732) 2023-11-16T18:27:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.632+0000 7f58b7fff700 1 -- 172.21.15.2:0/1298892081 learned_addr learned my addr 172.21.15.2:0/1298892081 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.632+0000 7f58b77fe700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58b80ff3c0 0x7f58b81963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.632+0000 7f58b6ffd700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b80ffd60 0x7f58b8196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.633+0000 7f58b77fe700 1 -- 172.21.15.2:0/1298892081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b80ffd60 msgr2=0x7f58b8196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.633+0000 7f58b77fe700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b80ffd60 0x7f58b8196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.633+0000 7f58b77fe700 1 -- 172.21.15.2:0/1298892081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b8104400 msgr2=0x7f58b819ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.633+0000 7f58b77fe700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b8104400 0x7f58b819ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.633+0000 7f58b77fe700 1 -- 172.21.15.2:0/1298892081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a80095f0 con 0x7f58b80ff3c0 2023-11-16T18:27:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.633+0000 7f58b7fff700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b8104400 0x7f58b819ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.633+0000 7f58b6ffd700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b80ffd60 0x7f58b8196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.633+0000 7f58b77fe700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58b80ff3c0 0x7f58b81963e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f58a80098e0 tx=0x7f58a800f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.633+0000 7f58b4ff9700 1 -- 172.21.15.2:0/1298892081 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a801d070 con 0x7f58b80ff3c0 2023-11-16T18:27:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.633+0000 7f58bde47700 1 -- 172.21.15.2:0/1298892081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b810f1c0 con 0x7f58b80ff3c0 2023-11-16T18:27:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.633+0000 7f58b4ff9700 1 -- 172.21.15.2:0/1298892081 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58a8005050 con 0x7f58b80ff3c0 2023-11-16T18:27:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.633+0000 7f58b4ff9700 1 -- 172.21.15.2:0/1298892081 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8004970 con 0x7f58b80ff3c0 2023-11-16T18:27:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.633+0000 7f58bde47700 1 -- 172.21.15.2:0/1298892081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b810f030 con 0x7f58b80ff3c0 2023-11-16T18:27:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.635+0000 7f58b4ff9700 1 -- 172.21.15.2:0/1298892081 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f58a8005470 con 0x7f58b80ff3c0 2023-11-16T18:27:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.636+0000 7f58b4ff9700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f58a00a12c0 0x7f58a00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.636+0000 7f58b4ff9700 1 -- 172.21.15.2:0/1298892081 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f58a80cdd20 con 0x7f58b80ff3c0 2023-11-16T18:27:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.636+0000 7f58b6ffd700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f58a00a12c0 0x7f58a00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.636+0000 7f58b6ffd700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f58a00a12c0 0x7f58a00a3770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f58b81979f0 tx=0x7f58ac009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58c9fd3b80 1 Processor -- start 2023-11-16T18:27:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58c9fd3b80 1 -- start start 2023-11-16T18:27:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58c9fd3b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c2d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58c9fd3b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c27970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58c9fd3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5c2df60 0x55c6b5c27ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58c9fd3b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b5b29000 con 0x55c6b5a7f6b0 2023-11-16T18:27:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58c9fd3b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b5b7b840 con 0x55c6b5c2daf0 2023-11-16T18:27:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58c9fd3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b5b74cd0 con 0x55c6b5c2df60 2023-11-16T18:27:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58b6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c27970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58b77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c2d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58b6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c27970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42178/0 (socket says 172.21.15.2:42178) 2023-11-16T18:27:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58b6ffd700 1 -- 172.21.15.2:0/348782525 learned_addr learned my addr 172.21.15.2:0/348782525 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58b77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c2d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51738/0 (socket says 172.21.15.2:51738) 2023-11-16T18:27:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.637+0000 7f58b7fff700 1 --2- 172.21.15.2:0/348782525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5c2df60 0x55c6b5c27ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.638+0000 7f58b6ffd700 1 -- 172.21.15.2:0/348782525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5c2df60 msgr2=0x55c6b5c27ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.638+0000 7f58b6ffd700 1 --2- 172.21.15.2:0/348782525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5c2df60 0x55c6b5c27ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.638+0000 7f58b6ffd700 1 -- 172.21.15.2:0/348782525 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5a7f6b0 msgr2=0x55c6b5c2d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.638+0000 7f58b6ffd700 1 --2- 172.21.15.2:0/348782525 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c2d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.638+0000 7f58b6ffd700 1 -- 172.21.15.2:0/348782525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6b59f65d0 con 0x55c6b5c2daf0 2023-11-16T18:27:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.638+0000 7f58b6ffd700 1 --2- 172.21.15.2:0/348782525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c27970 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f58ac004390 tx=0x7f58ac014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f41705aed693fe19 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.638+0000 7f589dffb700 1 -- 172.21.15.2:0/348782525 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac003d10 con 0x55c6b5c2daf0 2023-11-16T18:27:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.638+0000 7f589dffb700 1 -- 172.21.15.2:0/348782525 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58ac003e70 con 0x55c6b5c2daf0 2023-11-16T18:27:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.638+0000 7f589dffb700 1 -- 172.21.15.2:0/348782525 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac01e850 con 0x55c6b5c2daf0 2023-11-16T18:27:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.639+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/348782525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 msgr2=0x55c6b5c27970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.639+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/348782525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c27970 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f58ac004390 tx=0x7f58ac014000 comp rx=0 tx=0).stop 2023-11-16T18:27:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.639+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/348782525 shutdown_connections 2023-11-16T18:27:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.639+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/348782525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5c2df60 0x55c6b5c27ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.639+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/348782525 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c2d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.639+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/348782525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c27970 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.639+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/348782525 >> 172.21.15.2:0/348782525 conn(0x55c6b5c18f50 msgr2=0x55c6b5c1e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.639+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/348782525 shutdown_connections 2023-11-16T18:27:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.639+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/348782525 wait complete. 2023-11-16T18:27:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.639+0000 7f58c9fd3b80 1 -- start start 2023-11-16T18:27:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58c9fd3b80 1 Processor -- start 2023-11-16T18:27:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58c9fd3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c33ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58c9fd3b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c310f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58c9fd3b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2df60 0x55c6b5c317e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58c9fd3b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b5b16880 con 0x55c6b5c2daf0 2023-11-16T18:27:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58c9fd3b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b5b16520 con 0x55c6b5c2df60 2023-11-16T18:27:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58c9fd3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b59f88c0 con 0x55c6b5a7f6b0 2023-11-16T18:27:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58c9fd3b80 10 client.0 _ll_register_callbacks cb 0x55c6b5b34720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58b6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c310f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58b7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2df60 0x55c6b5c317e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58b7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2df60 0x55c6b5c317e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42186/0 (socket says 172.21.15.2:42186) 2023-11-16T18:27:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58b6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c310f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51746/0 (socket says 172.21.15.2:51746) 2023-11-16T18:27:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58b7fff700 1 -- 172.21.15.2:0/3580739123 learned_addr learned my addr 172.21.15.2:0/3580739123 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58b77fe700 1 --2- 172.21.15.2:0/3580739123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c33ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58b7fff700 1 -- 172.21.15.2:0/3580739123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5a7f6b0 msgr2=0x55c6b5c33ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58b7fff700 1 --2- 172.21.15.2:0/3580739123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c33ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58b7fff700 1 -- 172.21.15.2:0/3580739123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 msgr2=0x55c6b5c310f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58b7fff700 1 --2- 172.21.15.2:0/3580739123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c310f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58b7fff700 1 -- 172.21.15.2:0/3580739123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ac017070 con 0x55c6b5c2df60 2023-11-16T18:27:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.640+0000 7f58b6ffd700 1 --2- 172.21.15.2:0/3580739123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c310f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f58b7fff700 1 --2- 172.21.15.2:0/3580739123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2df60 0x55c6b5c317e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f58b00092b0 tx=0x7f58b00092e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e336fc1a50e9e8b7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f5896ffd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:27:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f5896ffd700 1 -- 172.21.15.2:0/3580739123 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b000eb70 con 0x55c6b5c2df60 2023-11-16T18:27:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f5896ffd700 1 -- 172.21.15.2:0/3580739123 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f58b000ecd0 con 0x55c6b5c2df60 2023-11-16T18:27:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/3580739123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c6b5c16710 con 0x55c6b5c2df60 2023-11-16T18:27:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f5896ffd700 1 -- 172.21.15.2:0/3580739123 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58b000eed0 con 0x55c6b5c2df60 2023-11-16T18:27:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f58c9fd3b80 10 client.25808 fetch_fsmap learned FSMap version 24 2023-11-16T18:27:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/3580739123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c6b5b3b130 con 0x55c6b5c2df60 2023-11-16T18:27:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f5896ffd700 1 -- 172.21.15.2:0/3580739123 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c6b5c16710 con 0x55c6b5c2df60 2023-11-16T18:27:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f5896ffd700 1 -- 172.21.15.2:0/3580739123 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f58b001e690 con 0x55c6b5c2df60 2023-11-16T18:27:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f58c9fd3b80 10 client.25808 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:27:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f58c9fd3b80 10 client.25808 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f58c9fd3b80 20 client.25808 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/3580739123 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c6b5c41500 0x55c6b5c43910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f58c9fd3b80 4 client.25808 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:27:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.641+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/3580739123 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6b5b37040 con 0x55c6b5c41500 2023-11-16T18:27:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.642+0000 7f58b77fe700 1 --2- 172.21.15.2:0/3580739123 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c6b5c41500 0x55c6b5c43910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.642+0000 7f58b77fe700 1 --2- 172.21.15.2:0/3580739123 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c6b5c41500 0x55c6b5c43910 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e27d1aa57cea46f server_cookie=c3a1ce545971fb in_seq=0 out_seq=0 2023-11-16T18:27:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.642+0000 7f5896ffd700 10 client.25808 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.644+0000 7f5896ffd700 1 -- 172.21.15.2:0/3580739123 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c6b5b37040 con 0x55c6b5c41500 2023-11-16T18:27:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.644+0000 7f5896ffd700 10 client.25808 handle_command_reply: tid=0 2023-11-16T18:27:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.644+0000 7f58c9fd3b80 1 client.25808 shutdown 2023-11-16T18:27:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.644+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/3580739123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2df60 msgr2=0x55c6b5c317e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.644+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/3580739123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2df60 0x55c6b5c317e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f58b00092b0 tx=0x7f58b00092e0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.645+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/3580739123 shutdown_connections 2023-11-16T18:27:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.645+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/3580739123 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c6b5c41500 0x55c6b5c43910 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.645+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/3580739123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c33ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.645+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/3580739123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2df60 0x55c6b5c317e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.645+0000 7f5896ffd700 0 client.25808 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.645+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/3580739123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c310f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.645+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/3580739123 >> 172.21.15.2:0/3580739123 conn(0x55c6b5c18f50 msgr2=0x55c6b5c1d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.645+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/3580739123 shutdown_connections 2023-11-16T18:27:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.645+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/3580739123 wait complete. 2023-11-16T18:27:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.645+0000 7f58c9fd3b80 20 client.25808 trim_cache size 0 max 16384 2023-11-16T18:27:04.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.660+0000 7f58c9fd3b80 1 Processor -- start 2023-11-16T18:27:04.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.660+0000 7f58c9fd3b80 1 -- start start 2023-11-16T18:27:04.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.660+0000 7f58c9fd3b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c38520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.660+0000 7f58c9fd3b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c33a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.660+0000 7f58c9fd3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5c2df60 0x55c6b5c27970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.660+0000 7f58c9fd3b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b5c31e00 con 0x55c6b5c2daf0 2023-11-16T18:27:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.660+0000 7f58c9fd3b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b5c32580 con 0x55c6b5a7f6b0 2023-11-16T18:27:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.660+0000 7f58c9fd3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b5c1ef40 con 0x55c6b5c2df60 2023-11-16T18:27:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.660+0000 7f58b6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c33a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.660+0000 7f58b77fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c38520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.660+0000 7f58b6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c33a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51754/0 (socket says 172.21.15.2:51754) 2023-11-16T18:27:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.660+0000 7f58b6ffd700 1 -- 172.21.15.2:0/948196624 learned_addr learned my addr 172.21.15.2:0/948196624 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.660+0000 7f58b7fff700 1 --2- 172.21.15.2:0/948196624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5c2df60 0x55c6b5c27970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.661+0000 7f58b77fe700 1 -- 172.21.15.2:0/948196624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5c2df60 msgr2=0x55c6b5c27970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.661+0000 7f58b77fe700 1 --2- 172.21.15.2:0/948196624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5c2df60 0x55c6b5c27970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.661+0000 7f58b77fe700 1 -- 172.21.15.2:0/948196624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 msgr2=0x55c6b5c33a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.661+0000 7f58b77fe700 1 --2- 172.21.15.2:0/948196624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c33a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.661+0000 7f58b77fe700 1 -- 172.21.15.2:0/948196624 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6b5b3ae80 con 0x55c6b5a7f6b0 2023-11-16T18:27:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.661+0000 7f58b6ffd700 1 --2- 172.21.15.2:0/948196624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c33a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.661+0000 7f58b77fe700 1 --2- 172.21.15.2:0/948196624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c38520 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f58a80cb0a0 tx=0x7f58a8096f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a956987fc3140e3e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.661+0000 7f589e7fc700 1 -- 172.21.15.2:0/948196624 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a80a2e70 con 0x55c6b5a7f6b0 2023-11-16T18:27:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.661+0000 7f589e7fc700 1 -- 172.21.15.2:0/948196624 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58a8098bf0 con 0x55c6b5a7f6b0 2023-11-16T18:27:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.661+0000 7f589e7fc700 1 -- 172.21.15.2:0/948196624 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a809ad10 con 0x55c6b5a7f6b0 2023-11-16T18:27:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.662+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/948196624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5a7f6b0 msgr2=0x55c6b5c38520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.662+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/948196624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c38520 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f58a80cb0a0 tx=0x7f58a8096f20 comp rx=0 tx=0).stop 2023-11-16T18:27:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.662+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/948196624 shutdown_connections 2023-11-16T18:27:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.662+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/948196624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5c2df60 0x55c6b5c27970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.662+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/948196624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c38520 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.662+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/948196624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c33a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.662+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/948196624 >> 172.21.15.2:0/948196624 conn(0x55c6b5c18f50 msgr2=0x55c6b5c2b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.662+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/948196624 shutdown_connections 2023-11-16T18:27:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.662+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/948196624 wait complete. 2023-11-16T18:27:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.662+0000 7f58c9fd3b80 1 -- start start 2023-11-16T18:27:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58c9fd3b80 1 Processor -- start 2023-11-16T18:27:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58c9fd3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c8dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58c9fd3b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c8e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58c9fd3b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2df60 0x55c6b5c37960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58c9fd3b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b5c38030 con 0x55c6b5c2df60 2023-11-16T18:27:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58c9fd3b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b5c381a0 con 0x55c6b5c2daf0 2023-11-16T18:27:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58c9fd3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b5c38360 con 0x55c6b5a7f6b0 2023-11-16T18:27:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58c9fd3b80 10 client.0 _ll_register_callbacks cb 0x7f58b0007980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58b7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2df60 0x55c6b5c37960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58b6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c8e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58b7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2df60 0x55c6b5c37960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51762/0 (socket says 172.21.15.2:51762) 2023-11-16T18:27:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58b6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c8e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42198/0 (socket says 172.21.15.2:42198) 2023-11-16T18:27:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58b7fff700 1 -- 172.21.15.2:0/2775677429 learned_addr learned my addr 172.21.15.2:0/2775677429 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58b77fe700 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c8dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58b7fff700 1 -- 172.21.15.2:0/2775677429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5a7f6b0 msgr2=0x55c6b5c8dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.663+0000 7f58b7fff700 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c8dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f58b7fff700 1 -- 172.21.15.2:0/2775677429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 msgr2=0x55c6b5c8e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f58b7fff700 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c8e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f58b7fff700 1 -- 172.21.15.2:0/2775677429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6b59f65d0 con 0x55c6b5c2df60 2023-11-16T18:27:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f58b77fe700 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c8dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f58b6ffd700 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c8e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f58b7fff700 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2df60 0x55c6b5c37960 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f58b0009f00 tx=0x7f58b000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b08f84327db789 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f5896ffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:27:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f5896ffd700 1 -- 172.21.15.2:0/2775677429 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b001e5d0 con 0x55c6b5c2df60 2023-11-16T18:27:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f5896ffd700 1 -- 172.21.15.2:0/2775677429 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f58b001e730 con 0x55c6b5c2df60 2023-11-16T18:27:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/2775677429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f58ac003e70 con 0x55c6b5c2df60 2023-11-16T18:27:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f5896ffd700 1 -- 172.21.15.2:0/2775677429 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58b0003a00 con 0x55c6b5c2df60 2023-11-16T18:27:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f58c9fd3b80 10 client.16263 fetch_fsmap learned FSMap version 24 2023-11-16T18:27:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/2775677429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f58b810f030 con 0x55c6b5c2df60 2023-11-16T18:27:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.664+0000 7f5896ffd700 1 -- 172.21.15.2:0/2775677429 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58ac003e70 con 0x55c6b5c2df60 2023-11-16T18:27:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.665+0000 7f5896ffd700 1 -- 172.21.15.2:0/2775677429 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f58b0005730 con 0x55c6b5c2df60 2023-11-16T18:27:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.665+0000 7f58c9fd3b80 10 client.16263 fetch_fsmap finished waiting for FSMap version 24 2023-11-16T18:27:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.665+0000 7f58c9fd3b80 10 client.16263 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.665+0000 7f58c9fd3b80 20 client.16263 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.665+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c6b5c95ae0 0x55c6b5c97ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.665+0000 7f58c9fd3b80 4 client.16263 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:27:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.665+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/2775677429 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c6b5b43600 con 0x55c6b5c95ae0 2023-11-16T18:27:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.665+0000 7f58b77fe700 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c6b5c95ae0 0x55c6b5c97ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.665+0000 7f5896ffd700 10 client.16263 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:04.665+0000 7f58b77fe700 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c6b5c95ae0 0x55c6b5c97ef0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3b02492f3a48096 server_cookie=68e2c54cdc852317 in_seq=0 out_seq=0 2023-11-16T18:27:05.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:05 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:05.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:05 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:05.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:05 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:05.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:05 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:05.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:05 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:05.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:05 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:06.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:06 smithi040 ceph-mon[100251]: pgmap v70: 129 pgs: 129 active+clean; 6.8 MiB data, 94 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 433 KiB/s wr, 31 op/s 2023-11-16T18:27:06.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:06 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:06.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:06 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:06.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:06 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:27:06.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:06 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:27:06.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:06 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:06.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:06 smithi188 ceph-mon[100639]: pgmap v70: 129 pgs: 129 active+clean; 6.8 MiB data, 94 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 433 KiB/s wr, 31 op/s 2023-11-16T18:27:06.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:06 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:06.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:06 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:06.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:06 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:27:06.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:06 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:27:06.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:06 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:06.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:06 smithi002 ceph-mon[98807]: pgmap v70: 129 pgs: 129 active+clean; 6.8 MiB data, 94 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 433 KiB/s wr, 31 op/s 2023-11-16T18:27:06.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:06 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:06.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:06 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:06.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:06 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:27:06.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:06 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:27:06.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:06 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:27:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.318+0000 7f5896ffd700 1 -- 172.21.15.2:0/2775677429 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c6b5b43600 con 0x55c6b5c95ae0 2023-11-16T18:27:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.318+0000 7f5896ffd700 10 client.16263 handle_command_reply: tid=0 2023-11-16T18:27:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.318+0000 7f58c9fd3b80 1 client.16263 shutdown 2023-11-16T18:27:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/2775677429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2df60 msgr2=0x55c6b5c37960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2df60 0x55c6b5c37960 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f58b0009f00 tx=0x7f58b000ed80 comp rx=0 tx=0).stop 2023-11-16T18:27:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/2775677429 shutdown_connections 2023-11-16T18:27:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55c6b5c95ae0 0x55c6b5c97ef0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c6b5a7f6b0 0x55c6b5c8dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f5896ffd700 0 client.16263 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c6b5c2df60 0x55c6b5c37960 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58c9fd3b80 1 --2- 172.21.15.2:0/2775677429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b5c2daf0 0x55c6b5c8e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/2775677429 >> 172.21.15.2:0/2775677429 conn(0x55c6b5c18f50 msgr2=0x55c6b5c27970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/2775677429 shutdown_connections 2023-11-16T18:27:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58c9fd3b80 1 -- 172.21.15.2:0/2775677429 wait complete. 2023-11-16T18:27:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58c9fd3b80 20 client.16263 trim_cache size 0 max 16384 2023-11-16T18:27:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:27:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:27:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:27:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:27:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58bde47700 1 -- 172.21.15.2:0/1298892081 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f58a00a12c0 msgr2=0x7f58a00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58bde47700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f58a00a12c0 0x7f58a00a3770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f58b81979f0 tx=0x7f58ac009260 comp rx=0 tx=0).stop 2023-11-16T18:27:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58bde47700 1 -- 172.21.15.2:0/1298892081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58b80ff3c0 msgr2=0x7f58b81963e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58bde47700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58b80ff3c0 0x7f58b81963e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f58a80098e0 tx=0x7f58a800f690 comp rx=0 tx=0).stop 2023-11-16T18:27:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58bde47700 1 -- 172.21.15.2:0/1298892081 shutdown_connections 2023-11-16T18:27:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58bde47700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f58a00a12c0 0x7f58a00a3770 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58bde47700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b8104400 0x7f58b819ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58bde47700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58b80ff3c0 0x7f58b81963e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.319+0000 7f58bde47700 1 --2- 172.21.15.2:0/1298892081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b80ffd60 0x7f58b8196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.320+0000 7f58bde47700 1 -- 172.21.15.2:0/1298892081 >> 172.21.15.2:0/1298892081 conn(0x7f58b80763a0 msgr2=0x7f58b8103040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.320+0000 7f58bde47700 1 -- 172.21.15.2:0/1298892081 shutdown_connections 2023-11-16T18:27:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:07.320+0000 7f58bde47700 1 -- 172.21.15.2:0/1298892081 wait complete. 2023-11-16T18:27:07.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:07 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:27:07.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:07 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:27:07.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:07 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:27:07.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:07 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:27:07.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:07 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:27:07.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:07 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:27:07.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:07 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:27:07.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:07 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:27:07.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:07 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:27:07.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:07 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:27:07.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:07 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:27:07.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:07 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:27:08.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:08 smithi040 ceph-mon[100251]: pgmap v71: 129 pgs: 129 active+clean; 16 MiB data, 106 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 502 op/s 2023-11-16T18:27:08.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:08 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:27:08.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:08 smithi188 ceph-mon[100639]: pgmap v71: 129 pgs: 129 active+clean; 16 MiB data, 106 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 502 op/s 2023-11-16T18:27:08.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:08 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:27:08.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:08 smithi002 ceph-mon[98807]: pgmap v71: 129 pgs: 129 active+clean; 16 MiB data, 106 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 502 op/s 2023-11-16T18:27:08.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:08 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:27:10.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:10 smithi040 ceph-mon[100251]: pgmap v72: 129 pgs: 129 active+clean; 16 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 831 op/s 2023-11-16T18:27:10.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:10 smithi188 ceph-mon[100639]: pgmap v72: 129 pgs: 129 active+clean; 16 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 831 op/s 2023-11-16T18:27:10.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:10 smithi002 ceph-mon[98807]: pgmap v72: 129 pgs: 129 active+clean; 16 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 831 op/s 2023-11-16T18:27:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:12 smithi040 ceph-mon[100251]: pgmap v73: 129 pgs: 129 active+clean; 16 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 831 op/s 2023-11-16T18:27:12.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:12 smithi188 ceph-mon[100639]: pgmap v73: 129 pgs: 129 active+clean; 16 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 831 op/s 2023-11-16T18:27:12.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:12 smithi002 ceph-mon[98807]: pgmap v73: 129 pgs: 129 active+clean; 16 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 831 op/s 2023-11-16T18:27:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:14 smithi040 ceph-mon[100251]: pgmap v74: 129 pgs: 129 active+clean; 19 MiB data, 207 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 1.01k op/s 2023-11-16T18:27:14.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:14 smithi188 ceph-mon[100639]: pgmap v74: 129 pgs: 129 active+clean; 19 MiB data, 207 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 1.01k op/s 2023-11-16T18:27:14.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:14 smithi002 ceph-mon[98807]: pgmap v74: 129 pgs: 129 active+clean; 19 MiB data, 207 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 1.01k op/s 2023-11-16T18:27:16.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:16 smithi040 ceph-mon[100251]: pgmap v75: 129 pgs: 129 active+clean; 19 MiB data, 207 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 981 op/s 2023-11-16T18:27:16.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:16 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:27:16.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:16 smithi188 ceph-mon[100639]: pgmap v75: 129 pgs: 129 active+clean; 19 MiB data, 207 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 981 op/s 2023-11-16T18:27:16.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:16 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:27:16.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:16 smithi002 ceph-mon[98807]: pgmap v75: 129 pgs: 129 active+clean; 19 MiB data, 207 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 981 op/s 2023-11-16T18:27:16.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:16 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:27:17.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:17 smithi188 ceph-mon[100639]: pgmap v76: 129 pgs: 129 active+clean; 29 MiB data, 216 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 989 op/s 2023-11-16T18:27:17.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:17 smithi002 ceph-mon[98807]: pgmap v76: 129 pgs: 129 active+clean; 29 MiB data, 216 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 989 op/s 2023-11-16T18:27:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:17 smithi040 ceph-mon[100251]: pgmap v76: 129 pgs: 129 active+clean; 29 MiB data, 216 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 989 op/s 2023-11-16T18:27:20.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:20 smithi002 ceph-mon[98807]: pgmap v77: 129 pgs: 129 active+clean; 37 MiB data, 220 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 522 op/s 2023-11-16T18:27:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.442+0000 7fd1e7a81700 1 -- 172.21.15.2:0/3767843969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1e00ffda0 msgr2=0x7fd1e0103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.442+0000 7fd1e7a81700 1 --2- 172.21.15.2:0/3767843969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1e00ffda0 0x7fd1e0103d60 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00099c0 tx=0x7fd1d0009cd0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.442+0000 7fd1e7a81700 1 -- 172.21.15.2:0/3767843969 shutdown_connections 2023-11-16T18:27:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.442+0000 7fd1e7a81700 1 --2- 172.21.15.2:0/3767843969 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1e0104440 0x7fd1e010c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.442+0000 7fd1e7a81700 1 --2- 172.21.15.2:0/3767843969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1e00ff400 0x7fd1e00ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.442+0000 7fd1e7a81700 1 --2- 172.21.15.2:0/3767843969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1e00ffda0 0x7fd1e0103d60 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.442+0000 7fd1e7a81700 1 -- 172.21.15.2:0/3767843969 >> 172.21.15.2:0/3767843969 conn(0x7fd1e0075fa0 msgr2=0x7fd1e00763a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.442+0000 7fd1e7a81700 1 -- 172.21.15.2:0/3767843969 shutdown_connections 2023-11-16T18:27:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.442+0000 7fd1e7a81700 1 -- 172.21.15.2:0/3767843969 wait complete. 2023-11-16T18:27:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.443+0000 7fd1e7a81700 1 Processor -- start 2023-11-16T18:27:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.443+0000 7fd1e7a81700 1 -- start start 2023-11-16T18:27:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.443+0000 7fd1e7a81700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1e00ff400 0x7fd1e0100e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.443+0000 7fd1e7a81700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1e00ffda0 0x7fd1e0101340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.443+0000 7fd1e7a81700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1e0104440 0x7fd1e01a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.443+0000 7fd1e7a81700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e0101b70 con 0x7fd1e00ffda0 2023-11-16T18:27:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.443+0000 7fd1e7a81700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e01a4b20 con 0x7fd1e0104440 2023-11-16T18:27:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.443+0000 7fd1e7a81700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e01a4c60 con 0x7fd1e00ff400 2023-11-16T18:27:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.443+0000 7fd1e501c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1e00ffda0 0x7fd1e0101340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.443+0000 7fd1e501c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1e00ffda0 0x7fd1e0101340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51774/0 (socket says 172.21.15.2:51774) 2023-11-16T18:27:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.443+0000 7fd1e501c700 1 -- 172.21.15.2:0/4005484880 learned_addr learned my addr 172.21.15.2:0/4005484880 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.443+0000 7fd1e601e700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1e0104440 0x7fd1e01a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.443+0000 7fd1e581d700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1e00ff400 0x7fd1e0100e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.444+0000 7fd1e501c700 1 -- 172.21.15.2:0/4005484880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1e00ff400 msgr2=0x7fd1e0100e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.444+0000 7fd1e501c700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1e00ff400 0x7fd1e0100e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.444+0000 7fd1e501c700 1 -- 172.21.15.2:0/4005484880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1e0104440 msgr2=0x7fd1e01a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.444+0000 7fd1e501c700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1e0104440 0x7fd1e01a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.444+0000 7fd1e501c700 1 -- 172.21.15.2:0/4005484880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc009520 con 0x7fd1e00ffda0 2023-11-16T18:27:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.444+0000 7fd1e501c700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1e00ffda0 0x7fd1e0101340 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0009990 tx=0x7fd1d000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.444+0000 7fd1d6ffd700 1 -- 172.21.15.2:0/4005484880 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d001d070 con 0x7fd1e00ffda0 2023-11-16T18:27:20.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.444+0000 7fd1d6ffd700 1 -- 172.21.15.2:0/4005484880 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1d0005020 con 0x7fd1e00ffda0 2023-11-16T18:27:20.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.444+0000 7fd1e7a81700 1 -- 172.21.15.2:0/4005484880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d00095f0 con 0x7fd1e00ffda0 2023-11-16T18:27:20.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.444+0000 7fd1d6ffd700 1 -- 172.21.15.2:0/4005484880 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0004a70 con 0x7fd1e00ffda0 2023-11-16T18:27:20.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.444+0000 7fd1e7a81700 1 -- 172.21.15.2:0/4005484880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e010f230 con 0x7fd1e00ffda0 2023-11-16T18:27:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.446+0000 7fd1d6ffd700 1 -- 172.21.15.2:0/4005484880 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fd1d0005400 con 0x7fd1e00ffda0 2023-11-16T18:27:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.447+0000 7fd1d6ffd700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd1cc0a1340 0x7fd1cc0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.447+0000 7fd1d6ffd700 1 -- 172.21.15.2:0/4005484880 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd1d00cd9e0 con 0x7fd1e00ffda0 2023-11-16T18:27:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.447+0000 7fd1e581d700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd1cc0a1340 0x7fd1cc0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.448+0000 7fd1e581d700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd1cc0a1340 0x7fd1cc0a37f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc009df0 tx=0x7fd1dc009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.448+0000 7fd1f3c0db80 1 Processor -- start 2023-11-16T18:27:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.448+0000 7fd1f3c0db80 1 -- start start 2023-11-16T18:27:20.450 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.448+0000 7fd1f3c0db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae4ece6b0 0x562ae50781e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.450 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.448+0000 7fd1f3c0db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078720 0x562ae5072560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.450 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.448+0000 7fd1f3c0db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae5078b90 0x562ae5072b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.448+0000 7fd1f3c0db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ae4f78000 con 0x562ae5078b90 2023-11-16T18:27:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.448+0000 7fd1f3c0db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ae4fca840 con 0x562ae5078720 2023-11-16T18:27:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.448+0000 7fd1f3c0db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ae4fc3cd0 con 0x562ae4ece6b0 2023-11-16T18:27:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.448+0000 7fd1e581d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae4ece6b0 0x562ae50781e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.448+0000 7fd1e581d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae4ece6b0 0x562ae50781e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43310/0 (socket says 172.21.15.2:43310) 2023-11-16T18:27:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.448+0000 7fd1e601e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae5078b90 0x562ae5072b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.448+0000 7fd1e581d700 1 -- 172.21.15.2:0/2582374228 learned_addr learned my addr 172.21.15.2:0/2582374228 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.449+0000 7fd1e581d700 1 -- 172.21.15.2:0/2582374228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078720 msgr2=0x562ae5072560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.449+0000 7fd1e581d700 1 --2- 172.21.15.2:0/2582374228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078720 0x562ae5072560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.449+0000 7fd1e581d700 1 -- 172.21.15.2:0/2582374228 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae5078b90 msgr2=0x562ae5072b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.449+0000 7fd1e581d700 1 --2- 172.21.15.2:0/2582374228 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae5078b90 0x562ae5072b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.449+0000 7fd1e581d700 1 -- 172.21.15.2:0/2582374228 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ae4e455d0 con 0x562ae4ece6b0 2023-11-16T18:27:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.449+0000 7fd1e581d700 1 --2- 172.21.15.2:0/2582374228 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae4ece6b0 0x562ae50781e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0094f0 tx=0x7fd1dc014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1edcfc69810b544e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.449+0000 7fd1cbfff700 1 -- 172.21.15.2:0/2582374228 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc026630 con 0x562ae4ece6b0 2023-11-16T18:27:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.449+0000 7fd1cbfff700 1 -- 172.21.15.2:0/2582374228 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1dc021070 con 0x562ae4ece6b0 2023-11-16T18:27:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.449+0000 7fd1cbfff700 1 -- 172.21.15.2:0/2582374228 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc01d670 con 0x562ae4ece6b0 2023-11-16T18:27:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.450+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/2582374228 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae4ece6b0 msgr2=0x562ae50781e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.450+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/2582374228 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae4ece6b0 0x562ae50781e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0094f0 tx=0x7fd1dc014000 comp rx=0 tx=0).stop 2023-11-16T18:27:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.450+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/2582374228 shutdown_connections 2023-11-16T18:27:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.450+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/2582374228 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae4ece6b0 0x562ae50781e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.450+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/2582374228 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae5078b90 0x562ae5072b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.450+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/2582374228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078720 0x562ae5072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.450+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/2582374228 >> 172.21.15.2:0/2582374228 conn(0x562ae5067f50 msgr2=0x562ae506d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.450+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/2582374228 shutdown_connections 2023-11-16T18:27:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.450+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/2582374228 wait complete. 2023-11-16T18:27:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.450+0000 7fd1f3c0db80 1 -- start start 2023-11-16T18:27:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.450+0000 7fd1f3c0db80 1 Processor -- start 2023-11-16T18:27:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1f3c0db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 0x562ae507ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1f3c0db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078720 0x562ae507bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1f3c0db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078b90 0x562ae507c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1f3c0db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ae4f65880 con 0x562ae4ece6b0 2023-11-16T18:27:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1f3c0db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ae4f65520 con 0x562ae5078720 2023-11-16T18:27:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1f3c0db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ae4e478c0 con 0x562ae5078b90 2023-11-16T18:27:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1f3c0db80 10 client.0 _ll_register_callbacks cb 0x562ae4f83720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1e581d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 0x562ae507ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1e581d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 0x562ae507ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51786/0 (socket says 172.21.15.2:51786) 2023-11-16T18:27:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1e581d700 1 -- 172.21.15.2:0/2811645010 learned_addr learned my addr 172.21.15.2:0/2811645010 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1e601e700 1 --2- 172.21.15.2:0/2811645010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078b90 0x562ae507c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1e501c700 1 --2- 172.21.15.2:0/2811645010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078720 0x562ae507bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1e581d700 1 -- 172.21.15.2:0/2811645010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078b90 msgr2=0x562ae507c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1e581d700 1 --2- 172.21.15.2:0/2811645010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078b90 0x562ae507c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1e581d700 1 -- 172.21.15.2:0/2811645010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078720 msgr2=0x562ae507bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1e581d700 1 --2- 172.21.15.2:0/2811645010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078720 0x562ae507bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.451+0000 7fd1e581d700 1 -- 172.21.15.2:0/2811645010 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1dc017070 con 0x562ae4ece6b0 2023-11-16T18:27:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.452+0000 7fd1e581d700 1 --2- 172.21.15.2:0/2811645010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 0x562ae507ea90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc011ab0 tx=0x7fd1dc00eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d5751cc0a51c30 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.452+0000 7fd1c97fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:27:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.452+0000 7fd1c97fa700 1 -- 172.21.15.2:0/2811645010 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc01c6a0 con 0x562ae4ece6b0 2023-11-16T18:27:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.452+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/2811645010 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562ae5065710 con 0x562ae4ece6b0 2023-11-16T18:27:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.452+0000 7fd1c97fa700 1 -- 172.21.15.2:0/2811645010 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd1dc01c800 con 0x562ae4ece6b0 2023-11-16T18:27:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.452+0000 7fd1c97fa700 1 -- 172.21.15.2:0/2811645010 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1dc005040 con 0x562ae4ece6b0 2023-11-16T18:27:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.452+0000 7fd1f3c0db80 10 client.16275 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.452+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/2811645010 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562ae4f8a130 con 0x562ae4ece6b0 2023-11-16T18:27:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.452+0000 7fd1c97fa700 1 -- 172.21.15.2:0/2811645010 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562ae5065710 con 0x562ae4ece6b0 2023-11-16T18:27:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.452+0000 7fd1c97fa700 1 -- 172.21.15.2:0/2811645010 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fd1dc003cf0 con 0x562ae4ece6b0 2023-11-16T18:27:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.452+0000 7fd1f3c0db80 10 client.16275 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.452+0000 7fd1f3c0db80 10 client.16275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.453+0000 7fd1f3c0db80 20 client.16275 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.453+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/2811645010 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x562ae508bf00 0x562ae508e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.453+0000 7fd1f3c0db80 4 client.16275 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:27:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.453+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/2811645010 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ae4f86040 con 0x562ae508bf00 2023-11-16T18:27:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.453+0000 7fd1e501c700 1 --2- 172.21.15.2:0/2811645010 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x562ae508bf00 0x562ae508e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.453+0000 7fd1e501c700 1 --2- 172.21.15.2:0/2811645010 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x562ae508bf00 0x562ae508e310 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55479c8acbcfbf41 server_cookie=6fdfc034d60a6952 in_seq=0 out_seq=0 2023-11-16T18:27:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.453+0000 7fd1c97fa700 10 client.16275 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1c97fa700 1 -- 172.21.15.2:0/2811645010 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562ae4f86040 con 0x562ae508bf00 2023-11-16T18:27:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1c97fa700 10 client.16275 handle_command_reply: tid=0 2023-11-16T18:27:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1f3c0db80 1 client.16275 shutdown 2023-11-16T18:27:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/2811645010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 msgr2=0x562ae507ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/2811645010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 0x562ae507ea90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc011ab0 tx=0x7fd1dc00eec0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/2811645010 shutdown_connections 2023-11-16T18:27:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/2811645010 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x562ae508bf00 0x562ae508e310 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/2811645010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078b90 0x562ae507c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/2811645010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 0x562ae507ea90 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1c97fa700 0 client.16275 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/2811645010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078720 0x562ae507bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/2811645010 >> 172.21.15.2:0/2811645010 conn(0x562ae5067f50 msgr2=0x562ae506c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/2811645010 shutdown_connections 2023-11-16T18:27:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/2811645010 wait complete. 2023-11-16T18:27:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.454+0000 7fd1f3c0db80 20 client.16275 trim_cache size 0 max 16384 2023-11-16T18:27:20.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.470+0000 7fd1f3c0db80 1 Processor -- start 2023-11-16T18:27:20.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.470+0000 7fd1f3c0db80 1 -- start start 2023-11-16T18:27:20.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.470+0000 7fd1f3c0db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 0x562ae507e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.470+0000 7fd1f3c0db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078720 0x562ae5076e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.470+0000 7fd1f3c0db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078b90 0x562ae50773a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.470+0000 7fd1f3c0db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ae5092c30 con 0x562ae4ece6b0 2023-11-16T18:27:20.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.470+0000 7fd1f3c0db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ae5092da0 con 0x562ae5078b90 2023-11-16T18:27:20.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.470+0000 7fd1f3c0db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ae50765f0 con 0x562ae5078720 2023-11-16T18:27:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.470+0000 7fd1e581d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 0x562ae507e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.471+0000 7fd1e581d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 0x562ae507e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51794/0 (socket says 172.21.15.2:51794) 2023-11-16T18:27:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.471+0000 7fd1e581d700 1 -- 172.21.15.2:0/216107533 learned_addr learned my addr 172.21.15.2:0/216107533 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.471+0000 7fd1e501c700 1 --2- 172.21.15.2:0/216107533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078720 0x562ae5076e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.471+0000 7fd1e601e700 1 --2- 172.21.15.2:0/216107533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078b90 0x562ae50773a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.471+0000 7fd1e581d700 1 -- 172.21.15.2:0/216107533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078720 msgr2=0x562ae5076e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:20.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.471+0000 7fd1e581d700 1 --2- 172.21.15.2:0/216107533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078720 0x562ae5076e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.471+0000 7fd1e581d700 1 -- 172.21.15.2:0/216107533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078b90 msgr2=0x562ae50773a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:20.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.471+0000 7fd1e581d700 1 --2- 172.21.15.2:0/216107533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078b90 0x562ae50773a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.471+0000 7fd1e581d700 1 -- 172.21.15.2:0/216107533 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ae50696f0 con 0x562ae4ece6b0 2023-11-16T18:27:20.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.471+0000 7fd1e581d700 1 --2- 172.21.15.2:0/216107533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 0x562ae507e630 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0094f0 tx=0x7fd1dc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e88df5eddfa832 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:20.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.471+0000 7fd1d4dac700 1 -- 172.21.15.2:0/216107533 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc0266c0 con 0x562ae4ece6b0 2023-11-16T18:27:20.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.471+0000 7fd1d4dac700 1 -- 172.21.15.2:0/216107533 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1dc01cd70 con 0x562ae4ece6b0 2023-11-16T18:27:20.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.471+0000 7fd1d4dac700 1 -- 172.21.15.2:0/216107533 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc005310 con 0x562ae4ece6b0 2023-11-16T18:27:20.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.472+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/216107533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 msgr2=0x562ae507e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:20.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.472+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/216107533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 0x562ae507e630 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0094f0 tx=0x7fd1dc00ec80 comp rx=0 tx=0).stop 2023-11-16T18:27:20.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.472+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/216107533 shutdown_connections 2023-11-16T18:27:20.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.472+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/216107533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae5078b90 0x562ae50773a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.472+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/216107533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae4ece6b0 0x562ae507e630 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.472+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/216107533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078720 0x562ae5076e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.472+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/216107533 >> 172.21.15.2:0/216107533 conn(0x562ae5067f50 msgr2=0x562ae5075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.472+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/216107533 shutdown_connections 2023-11-16T18:27:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.472+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/216107533 wait complete. 2023-11-16T18:27:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.472+0000 7fd1f3c0db80 1 -- start start 2023-11-16T18:27:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.472+0000 7fd1f3c0db80 1 Processor -- start 2023-11-16T18:27:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1f3c0db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae4ece6b0 0x562ae5082250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1f3c0db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078720 0x562ae5082790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1f3c0db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae5078b90 0x562ae50ded80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1f3c0db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ae5082f70 con 0x562ae5078b90 2023-11-16T18:27:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1f3c0db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ae50830e0 con 0x562ae4ece6b0 2023-11-16T18:27:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1f3c0db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ae50df310 con 0x562ae5078720 2023-11-16T18:27:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1f3c0db80 10 client.0 _ll_register_callbacks cb 0x7fd1d8009450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1e601e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae5078b90 0x562ae50ded80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1e581d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae4ece6b0 0x562ae5082250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1e601e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae5078b90 0x562ae50ded80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51804/0 (socket says 172.21.15.2:51804) 2023-11-16T18:27:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1e601e700 1 -- 172.21.15.2:0/1218926426 learned_addr learned my addr 172.21.15.2:0/1218926426 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1e501c700 1 --2- 172.21.15.2:0/1218926426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078720 0x562ae5082790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1e581d700 1 -- 172.21.15.2:0/1218926426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078720 msgr2=0x562ae5082790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1e581d700 1 --2- 172.21.15.2:0/1218926426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078720 0x562ae5082790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1e581d700 1 -- 172.21.15.2:0/1218926426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae5078b90 msgr2=0x562ae50ded80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1e581d700 1 --2- 172.21.15.2:0/1218926426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae5078b90 0x562ae50ded80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1e581d700 1 -- 172.21.15.2:0/1218926426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ae4f8a130 con 0x562ae4ece6b0 2023-11-16T18:27:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.473+0000 7fd1e601e700 1 --2- 172.21.15.2:0/1218926426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae5078b90 0x562ae50ded80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1e581d700 1 --2- 172.21.15.2:0/1218926426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae4ece6b0 0x562ae5082250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc009bd0 tx=0x7fd1dc00eec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21587fca6cf9f3e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1c97fa700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:27:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1c97fa700 1 -- 172.21.15.2:0/1218926426 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc01cee0 con 0x562ae4ece6b0 2023-11-16T18:27:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/1218926426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1dc021070 con 0x562ae4ece6b0 2023-11-16T18:27:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1c97fa700 1 -- 172.21.15.2:0/1218926426 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd1dc01c8b0 con 0x562ae4ece6b0 2023-11-16T18:27:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1c97fa700 1 -- 172.21.15.2:0/1218926426 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1dc01ca70 con 0x562ae4ece6b0 2023-11-16T18:27:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1f3c0db80 10 client.25829 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/1218926426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562ae4df1d80 con 0x562ae4ece6b0 2023-11-16T18:27:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1c97fa700 1 -- 172.21.15.2:0/1218926426 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1dc021070 con 0x562ae4ece6b0 2023-11-16T18:27:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1c97fa700 1 -- 172.21.15.2:0/1218926426 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fd1dc036070 con 0x562ae4ece6b0 2023-11-16T18:27:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1f3c0db80 10 client.25829 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1f3c0db80 10 client.25829 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1f3c0db80 20 client.25829 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/1218926426 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x562ae50e0600 0x562ae50e29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1f3c0db80 4 client.25829 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:27:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.474+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/1218926426 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562ae4f92600 con 0x562ae50e0600 2023-11-16T18:27:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.475+0000 7fd1e501c700 1 --2- 172.21.15.2:0/1218926426 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x562ae50e0600 0x562ae50e29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.475+0000 7fd1c97fa700 10 client.25829 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:20.475+0000 7fd1e501c700 1 --2- 172.21.15.2:0/1218926426 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x562ae50e0600 0x562ae50e29e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f94b239802f2e3c9 server_cookie=6411ad79f8e50dc in_seq=0 out_seq=0 2023-11-16T18:27:20.493 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:20 smithi040 ceph-mon[100251]: pgmap v77: 129 pgs: 129 active+clean; 37 MiB data, 220 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 522 op/s 2023-11-16T18:27:20.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:20 smithi188 ceph-mon[100639]: pgmap v77: 129 pgs: 129 active+clean; 37 MiB data, 220 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 522 op/s 2023-11-16T18:27:21.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:21 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:27:21.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:21 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:27:21.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:21 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:27:22.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.153+0000 7fd1c97fa700 1 -- 172.21.15.2:0/1218926426 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562ae4f92600 con 0x562ae50e0600 2023-11-16T18:27:22.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.153+0000 7fd1c97fa700 10 client.25829 handle_command_reply: tid=0 2023-11-16T18:27:22.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.153+0000 7fd1f3c0db80 1 client.25829 shutdown 2023-11-16T18:27:22.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.154+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/1218926426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae4ece6b0 msgr2=0x562ae5082250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:22.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.154+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/1218926426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae4ece6b0 0x562ae5082250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc009bd0 tx=0x7fd1dc00eec0 comp rx=0 tx=0).stop 2023-11-16T18:27:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.154+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/1218926426 shutdown_connections 2023-11-16T18:27:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.154+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/1218926426 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x562ae50e0600 0x562ae50e29e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.154+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/1218926426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562ae5078b90 0x562ae50ded80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.154+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/1218926426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ae4ece6b0 0x562ae5082250 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.154+0000 7fd1c97fa700 0 client.25829 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.154+0000 7fd1f3c0db80 1 --2- 172.21.15.2:0/1218926426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562ae5078720 0x562ae5082790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.154+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/1218926426 >> 172.21.15.2:0/1218926426 conn(0x562ae5067f50 msgr2=0x562ae5076e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.154+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/1218926426 shutdown_connections 2023-11-16T18:27:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.154+0000 7fd1f3c0db80 1 -- 172.21.15.2:0/1218926426 wait complete. 2023-11-16T18:27:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.155+0000 7fd1f3c0db80 20 client.25829 trim_cache size 0 max 16384 2023-11-16T18:27:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:27:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:27:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:27:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:27:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.155+0000 7fd1e7a81700 1 -- 172.21.15.2:0/4005484880 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd1cc0a1340 msgr2=0x7fd1cc0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.155+0000 7fd1e7a81700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd1cc0a1340 0x7fd1cc0a37f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc009df0 tx=0x7fd1dc009260 comp rx=0 tx=0).stop 2023-11-16T18:27:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.155+0000 7fd1e7a81700 1 -- 172.21.15.2:0/4005484880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1e00ffda0 msgr2=0x7fd1e0101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.155+0000 7fd1e7a81700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1e00ffda0 0x7fd1e0101340 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0009990 tx=0x7fd1d000f690 comp rx=0 tx=0).stop 2023-11-16T18:27:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.155+0000 7fd1e7a81700 1 -- 172.21.15.2:0/4005484880 shutdown_connections 2023-11-16T18:27:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.155+0000 7fd1e7a81700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd1cc0a1340 0x7fd1cc0a37f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.155+0000 7fd1e7a81700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1e00ff400 0x7fd1e0100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.155+0000 7fd1e7a81700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1e0104440 0x7fd1e01a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.155+0000 7fd1e7a81700 1 --2- 172.21.15.2:0/4005484880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1e00ffda0 0x7fd1e0101340 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.155+0000 7fd1e7a81700 1 -- 172.21.15.2:0/4005484880 >> 172.21.15.2:0/4005484880 conn(0x7fd1e0075fa0 msgr2=0x7fd1e0068b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.155+0000 7fd1e7a81700 1 -- 172.21.15.2:0/4005484880 shutdown_connections 2023-11-16T18:27:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:22.155+0000 7fd1e7a81700 1 -- 172.21.15.2:0/4005484880 wait complete. 2023-11-16T18:27:22.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:22 smithi002 ceph-mon[98807]: pgmap v78: 129 pgs: 129 active+clean; 37 MiB data, 220 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 189 op/s 2023-11-16T18:27:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:22 smithi040 ceph-mon[100251]: pgmap v78: 129 pgs: 129 active+clean; 37 MiB data, 220 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 189 op/s 2023-11-16T18:27:22.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:22 smithi188 ceph-mon[100639]: pgmap v78: 129 pgs: 129 active+clean; 37 MiB data, 220 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 189 op/s 2023-11-16T18:27:24.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:24 smithi002 ceph-mon[98807]: pgmap v79: 129 pgs: 129 active+clean; 54 MiB data, 243 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 760 op/s 2023-11-16T18:27:24.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:24 smithi040 ceph-mon[100251]: pgmap v79: 129 pgs: 129 active+clean; 54 MiB data, 243 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 760 op/s 2023-11-16T18:27:24.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:24 smithi188 ceph-mon[100639]: pgmap v79: 129 pgs: 129 active+clean; 54 MiB data, 243 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 760 op/s 2023-11-16T18:27:26.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:26 smithi002 ceph-mon[98807]: pgmap v80: 129 pgs: 129 active+clean; 54 MiB data, 243 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 583 op/s 2023-11-16T18:27:26.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:26 smithi040 ceph-mon[100251]: pgmap v80: 129 pgs: 129 active+clean; 54 MiB data, 243 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 583 op/s 2023-11-16T18:27:26.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:26 smithi188 ceph-mon[100639]: pgmap v80: 129 pgs: 129 active+clean; 54 MiB data, 243 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 583 op/s 2023-11-16T18:27:27.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:27 smithi188 ceph-mon[100639]: pgmap v81: 129 pgs: 129 active+clean; 62 MiB data, 255 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 589 op/s 2023-11-16T18:27:27.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:27 smithi002 ceph-mon[98807]: pgmap v81: 129 pgs: 129 active+clean; 62 MiB data, 255 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 589 op/s 2023-11-16T18:27:27.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:27 smithi040 ceph-mon[100251]: pgmap v81: 129 pgs: 129 active+clean; 62 MiB data, 255 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 589 op/s 2023-11-16T18:27:29.803 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:27:29.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.948+0000 7fdc49b4f700 1 -- 172.21.15.2:0/2870295386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc44100420 msgr2=0x7fdc4410bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:29.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.948+0000 7fdc49b4f700 1 --2- 172.21.15.2:0/2870295386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc44100420 0x7fdc4410bfd0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00b0a0 tx=0x7fdc3c00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.948+0000 7fdc49b4f700 1 -- 172.21.15.2:0/2870295386 shutdown_connections 2023-11-16T18:27:29.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.948+0000 7fdc49b4f700 1 --2- 172.21.15.2:0/2870295386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc44100420 0x7fdc4410bfd0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.948+0000 7fdc49b4f700 1 --2- 172.21.15.2:0/2870295386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc44105250 0x7fdc44105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.948+0000 7fdc49b4f700 1 --2- 172.21.15.2:0/2870295386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc440ffa70 0x7fdc440ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.948+0000 7fdc49b4f700 1 -- 172.21.15.2:0/2870295386 >> 172.21.15.2:0/2870295386 conn(0x7fdc440763a0 msgr2=0x7fdc440767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:29.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.948+0000 7fdc49b4f700 1 -- 172.21.15.2:0/2870295386 shutdown_connections 2023-11-16T18:27:29.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.948+0000 7fdc49b4f700 1 -- 172.21.15.2:0/2870295386 wait complete. 2023-11-16T18:27:29.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.949+0000 7fdc49b4f700 1 Processor -- start 2023-11-16T18:27:29.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.949+0000 7fdc49b4f700 1 -- start start 2023-11-16T18:27:29.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.949+0000 7fdc49b4f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc440ffa70 0x7fdc44197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:29.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.949+0000 7fdc49b4f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc44100420 0x7fdc44197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:29.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.949+0000 7fdc49b4f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc44105250 0x7fdc441a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:29.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.949+0000 7fdc49b4f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc441a28e0 con 0x7fdc44105250 2023-11-16T18:27:29.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.949+0000 7fdc49b4f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc441a2a20 con 0x7fdc440ffa70 2023-11-16T18:27:29.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.949+0000 7fdc49b4f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc441a2b60 con 0x7fdc44100420 2023-11-16T18:27:29.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.949+0000 7fdc43fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc44105250 0x7fdc441a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:29.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.949+0000 7fdc437fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc440ffa70 0x7fdc44197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:29.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.949+0000 7fdc43fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc44105250 0x7fdc441a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51816/0 (socket says 172.21.15.2:51816) 2023-11-16T18:27:29.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.949+0000 7fdc43fff700 1 -- 172.21.15.2:0/3493408792 learned_addr learned my addr 172.21.15.2:0/3493408792 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:29.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.950+0000 7fdc437fe700 1 -- 172.21.15.2:0/3493408792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc44100420 msgr2=0x7fdc44197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:29.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.950+0000 7fdc437fe700 1 --2- 172.21.15.2:0/3493408792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc44100420 0x7fdc44197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.950+0000 7fdc437fe700 1 -- 172.21.15.2:0/3493408792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc44105250 msgr2=0x7fdc441a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:29.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.950+0000 7fdc437fe700 1 --2- 172.21.15.2:0/3493408792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc44105250 0x7fdc441a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.950+0000 7fdc437fe700 1 -- 172.21.15.2:0/3493408792 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c009d00 con 0x7fdc440ffa70 2023-11-16T18:27:29.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.950+0000 7fdc43fff700 1 --2- 172.21.15.2:0/3493408792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc44105250 0x7fdc441a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:29.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.950+0000 7fdc437fe700 1 --2- 172.21.15.2:0/3493408792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc440ffa70 0x7fdc44197320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc34009c80 tx=0x7fdc34009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:29.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.950+0000 7fdc40ff9700 1 -- 172.21.15.2:0/3493408792 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34015070 con 0x7fdc440ffa70 2023-11-16T18:27:29.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.950+0000 7fdc49b4f700 1 -- 172.21.15.2:0/3493408792 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4410f1e0 con 0x7fdc440ffa70 2023-11-16T18:27:29.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.950+0000 7fdc40ff9700 1 -- 172.21.15.2:0/3493408792 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc34019410 con 0x7fdc440ffa70 2023-11-16T18:27:29.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.950+0000 7fdc49b4f700 1 -- 172.21.15.2:0/3493408792 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4410f050 con 0x7fdc440ffa70 2023-11-16T18:27:29.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.950+0000 7fdc40ff9700 1 -- 172.21.15.2:0/3493408792 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc340185d0 con 0x7fdc440ffa70 2023-11-16T18:27:29.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.953+0000 7fdc40ff9700 1 -- 172.21.15.2:0/3493408792 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fdc34019880 con 0x7fdc440ffa70 2023-11-16T18:27:29.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.954+0000 7fdc40ff9700 1 --2- 172.21.15.2:0/3493408792 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdc2c0a1340 0x7fdc2c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.954+0000 7fdc42ffd700 1 --2- 172.21.15.2:0/3493408792 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdc2c0a1340 0x7fdc2c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.954+0000 7fdc40ff9700 1 -- 172.21.15.2:0/3493408792 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fdc340d0630 con 0x7fdc440ffa70 2023-11-16T18:27:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.955+0000 7fdc42ffd700 1 --2- 172.21.15.2:0/3493408792 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdc2c0a1340 0x7fdc2c0a37f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdc38005950 tx=0x7fdc380058e0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.955+0000 7fdc55cdbb80 1 Processor -- start 2023-11-16T18:27:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.955+0000 7fdc55cdbb80 1 -- start start 2023-11-16T18:27:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.955+0000 7fdc55cdbb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b7b60 0x55df5b7b7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.955+0000 7fdc55cdbb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b8470 0x55df5b7bc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.955+0000 7fdc55cdbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7bc8b0 0x55df5b7bcd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.955+0000 7fdc55cdbb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5b6bd060 con 0x55df5b7b7b60 2023-11-16T18:27:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.955+0000 7fdc55cdbb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5b70f8a0 con 0x55df5b7b8470 2023-11-16T18:27:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.955+0000 7fdc55cdbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5b708d30 con 0x55df5b7bc8b0 2023-11-16T18:27:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.955+0000 7fdc437fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b7b60 0x55df5b7b7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.955+0000 7fdc437fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b7b60 0x55df5b7b7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51822/0 (socket says 172.21.15.2:51822) 2023-11-16T18:27:29.983 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.955+0000 7fdc437fe700 1 -- 172.21.15.2:0/1166217977 learned_addr learned my addr 172.21.15.2:0/1166217977 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:29.983 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.955+0000 7fdc42ffd700 1 --2- 172.21.15.2:0/1166217977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b8470 0x55df5b7bc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:29.983 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.956+0000 7fdc437fe700 1 -- 172.21.15.2:0/1166217977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7bc8b0 msgr2=0x55df5b7bcd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:29.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.956+0000 7fdc437fe700 1 --2- 172.21.15.2:0/1166217977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7bc8b0 0x55df5b7bcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.956+0000 7fdc437fe700 1 -- 172.21.15.2:0/1166217977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b8470 msgr2=0x55df5b7bc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:29.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.956+0000 7fdc437fe700 1 --2- 172.21.15.2:0/1166217977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b8470 0x55df5b7bc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.956+0000 7fdc437fe700 1 -- 172.21.15.2:0/1166217977 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df5b58a630 con 0x55df5b7b7b60 2023-11-16T18:27:29.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.956+0000 7fdc42ffd700 1 --2- 172.21.15.2:0/1166217977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b8470 0x55df5b7bc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:27:29.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.956+0000 7fdc437fe700 1 --2- 172.21.15.2:0/1166217977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b7b60 0x55df5b7b7f30 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fdc340a52b0 tx=0x7fdc3409cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b86075f8a845dfd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:29.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.956+0000 7fdc29ffb700 1 -- 172.21.15.2:0/1166217977 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34088070 con 0x55df5b7b7b60 2023-11-16T18:27:29.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.956+0000 7fdc29ffb700 1 -- 172.21.15.2:0/1166217977 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc340a57e0 con 0x55df5b7b7b60 2023-11-16T18:27:29.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.956+0000 7fdc29ffb700 1 -- 172.21.15.2:0/1166217977 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3409d5f0 con 0x55df5b7b7b60 2023-11-16T18:27:29.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.957+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/1166217977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b7b60 msgr2=0x55df5b7b7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.957+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/1166217977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b7b60 0x55df5b7b7f30 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fdc340a52b0 tx=0x7fdc3409cd70 comp rx=0 tx=0).stop 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.957+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/1166217977 shutdown_connections 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.957+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/1166217977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7bc8b0 0x55df5b7bcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.957+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/1166217977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b7b60 0x55df5b7b7f30 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.957+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/1166217977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b8470 0x55df5b7bc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.957+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/1166217977 >> 172.21.15.2:0/1166217977 conn(0x55df5b7ad360 msgr2=0x55df5b7b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.957+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/1166217977 shutdown_connections 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.957+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/1166217977 wait complete. 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc55cdbb80 1 -- start start 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc55cdbb80 1 Processor -- start 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc55cdbb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b7b60 0x55df5b7c3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc55cdbb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b8470 0x55df5b7c1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc55cdbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7bc8b0 0x55df5b7c1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc55cdbb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5b6aa8e0 con 0x55df5b7b8470 2023-11-16T18:27:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc55cdbb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5b6aa580 con 0x55df5b7b7b60 2023-11-16T18:27:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc55cdbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5b58c920 con 0x55df5b7bc8b0 2023-11-16T18:27:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc55cdbb80 10 client.0 _ll_register_callbacks cb 0x55df5b6c8780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc42ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b8470 0x55df5b7c1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc42ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b8470 0x55df5b7c1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51830/0 (socket says 172.21.15.2:51830) 2023-11-16T18:27:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc42ffd700 1 -- 172.21.15.2:0/3184070232 learned_addr learned my addr 172.21.15.2:0/3184070232 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc43fff700 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7bc8b0 0x55df5b7c1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.958+0000 7fdc437fe700 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b7b60 0x55df5b7c3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.959+0000 7fdc43fff700 1 -- 172.21.15.2:0/3184070232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b7b60 msgr2=0x55df5b7c3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.959+0000 7fdc43fff700 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b7b60 0x55df5b7c3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.959+0000 7fdc43fff700 1 -- 172.21.15.2:0/3184070232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b8470 msgr2=0x55df5b7c1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.959+0000 7fdc43fff700 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b8470 0x55df5b7c1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.959+0000 7fdc43fff700 1 -- 172.21.15.2:0/3184070232 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4410f050 con 0x55df5b7bc8b0 2023-11-16T18:27:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.959+0000 7fdc42ffd700 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b8470 0x55df5b7c1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.959+0000 7fdc43fff700 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7bc8b0 0x55df5b7c1870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009040 tx=0x7fdc3c00be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=248d0dbb3ecd51d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.959+0000 7fdc22ffd700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:27:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.959+0000 7fdc22ffd700 1 -- 172.21.15.2:0/3184070232 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c00e040 con 0x55df5b7bc8b0 2023-11-16T18:27:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.959+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3184070232 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55df5b7aab20 con 0x55df5b7bc8b0 2023-11-16T18:27:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.959+0000 7fdc22ffd700 1 -- 172.21.15.2:0/3184070232 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fdc3c003e10 con 0x55df5b7bc8b0 2023-11-16T18:27:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.960+0000 7fdc22ffd700 1 -- 172.21.15.2:0/3184070232 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc3c0074e0 con 0x55df5b7bc8b0 2023-11-16T18:27:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.960+0000 7fdc55cdbb80 10 client.25762 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.960+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3184070232 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55df5b6cf190 con 0x55df5b7bc8b0 2023-11-16T18:27:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.960+0000 7fdc22ffd700 1 -- 172.21.15.2:0/3184070232 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55df5b7aab20 con 0x55df5b7bc8b0 2023-11-16T18:27:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.960+0000 7fdc22ffd700 1 -- 172.21.15.2:0/3184070232 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fdc3c00b0d0 con 0x55df5b7bc8b0 2023-11-16T18:27:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.960+0000 7fdc55cdbb80 10 client.25762 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.960+0000 7fdc55cdbb80 10 client.25762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.960+0000 7fdc55cdbb80 20 client.25762 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.960+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55df5b7d1920 0x55df5b7d3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.960+0000 7fdc55cdbb80 4 client.25762 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:27:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.960+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3184070232 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55df5b6cb0a0 con 0x55df5b7d1920 2023-11-16T18:27:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.960+0000 7fdc437fe700 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55df5b7d1920 0x55df5b7d3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.961+0000 7fdc437fe700 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55df5b7d1920 0x55df5b7d3d30 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92a7975d26e8b9db server_cookie=405aaebe01a30588 in_seq=0 out_seq=0 2023-11-16T18:27:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.961+0000 7fdc22ffd700 10 client.25762 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.962+0000 7fdc22ffd700 1 -- 172.21.15.2:0/3184070232 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55df5b6cb0a0 con 0x55df5b7d1920 2023-11-16T18:27:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.962+0000 7fdc22ffd700 10 client.25762 handle_command_reply: tid=0 2023-11-16T18:27:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.962+0000 7fdc55cdbb80 1 client.25762 shutdown 2023-11-16T18:27:29.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.963+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3184070232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7bc8b0 msgr2=0x55df5b7c1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:29.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.963+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7bc8b0 0x55df5b7c1870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009040 tx=0x7fdc3c00be40 comp rx=0 tx=0).stop 2023-11-16T18:27:29.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.963+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3184070232 shutdown_connections 2023-11-16T18:27:29.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.963+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55df5b7d1920 0x55df5b7d3d30 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.963+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7bc8b0 0x55df5b7c1870 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.963+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b7b60 0x55df5b7c3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.963+0000 7fdc22ffd700 0 client.25762 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:29.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.963+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/3184070232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7b8470 0x55df5b7c1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:29.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.963+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3184070232 >> 172.21.15.2:0/3184070232 conn(0x55df5b7ad360 msgr2=0x55df5b7bdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:29.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.963+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3184070232 shutdown_connections 2023-11-16T18:27:29.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.963+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3184070232 wait complete. 2023-11-16T18:27:29.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.963+0000 7fdc55cdbb80 20 client.25762 trim_cache size 0 max 16384 2023-11-16T18:27:30.003 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.982+0000 7fdc55cdbb80 1 Processor -- start 2023-11-16T18:27:30.003 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc55cdbb80 1 -- start start 2023-11-16T18:27:30.003 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc55cdbb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b7b60 0x55df5b7c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.003 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc55cdbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7b8470 0x55df5b7ba930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.003 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc55cdbb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7bc8b0 0x55df5b7b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.003 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc55cdbb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5b7b8bc0 con 0x55df5b7bc8b0 2023-11-16T18:27:30.003 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc55cdbb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5b7bf7a0 con 0x55df5b7b7b60 2023-11-16T18:27:30.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc55cdbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5b7bb3e0 con 0x55df5b7b8470 2023-11-16T18:27:30.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc43fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7bc8b0 0x55df5b7b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc42ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7b8470 0x55df5b7ba930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc42ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7b8470 0x55df5b7ba930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43368/0 (socket says 172.21.15.2:43368) 2023-11-16T18:27:30.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc43fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7bc8b0 0x55df5b7b2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51840/0 (socket says 172.21.15.2:51840) 2023-11-16T18:27:30.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc42ffd700 1 -- 172.21.15.2:0/2642082752 learned_addr learned my addr 172.21.15.2:0/2642082752 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:30.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc437fe700 1 --2- 172.21.15.2:0/2642082752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b7b60 0x55df5b7c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc43fff700 1 -- 172.21.15.2:0/2642082752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7b8470 msgr2=0x55df5b7ba930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc43fff700 1 --2- 172.21.15.2:0/2642082752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7b8470 0x55df5b7ba930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc43fff700 1 -- 172.21.15.2:0/2642082752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b7b60 msgr2=0x55df5b7c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc43fff700 1 --2- 172.21.15.2:0/2642082752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b7b60 0x55df5b7c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.983+0000 7fdc43fff700 1 -- 172.21.15.2:0/2642082752 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df5b6ceee0 con 0x55df5b7bc8b0 2023-11-16T18:27:30.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.984+0000 7fdc43fff700 1 --2- 172.21.15.2:0/2642082752 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7bc8b0 0x55df5b7b2560 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c0062a0 tx=0x7fdc3c00beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164da2f62cd59749 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.984+0000 7fdc2a7fc700 1 -- 172.21.15.2:0/2642082752 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c00e070 con 0x55df5b7bc8b0 2023-11-16T18:27:30.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.984+0000 7fdc2a7fc700 1 -- 172.21.15.2:0/2642082752 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc3c007960 con 0x55df5b7bc8b0 2023-11-16T18:27:30.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.984+0000 7fdc2a7fc700 1 -- 172.21.15.2:0/2642082752 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c025ae0 con 0x55df5b7bc8b0 2023-11-16T18:27:30.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.985+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/2642082752 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7bc8b0 msgr2=0x55df5b7b2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.985+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/2642082752 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7bc8b0 0x55df5b7b2560 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c0062a0 tx=0x7fdc3c00beb0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.985+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/2642082752 shutdown_connections 2023-11-16T18:27:30.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.985+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/2642082752 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7bc8b0 0x55df5b7b2560 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.985+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/2642082752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b7b60 0x55df5b7c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.985+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/2642082752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7b8470 0x55df5b7ba930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.985+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/2642082752 >> 172.21.15.2:0/2642082752 conn(0x55df5b7ad360 msgr2=0x55df5b7cdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.985+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/2642082752 shutdown_connections 2023-11-16T18:27:30.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.985+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/2642082752 wait complete. 2023-11-16T18:27:30.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.985+0000 7fdc55cdbb80 1 -- start start 2023-11-16T18:27:30.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.985+0000 7fdc55cdbb80 1 Processor -- start 2023-11-16T18:27:30.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.985+0000 7fdc55cdbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7b7b60 0x55df5b7c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc55cdbb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b8470 0x55df5b7c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc55cdbb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7bc8b0 0x55df5b8247d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc55cdbb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5b7c5830 con 0x55df5b7bc8b0 2023-11-16T18:27:30.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc55cdbb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5b824d10 con 0x55df5b7b8470 2023-11-16T18:27:30.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc55cdbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5b824e50 con 0x55df5b7b7b60 2023-11-16T18:27:30.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc55cdbb80 10 client.0 _ll_register_callbacks cb 0x7fdc3c009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:30.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc437fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7b7b60 0x55df5b7c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc42ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b8470 0x55df5b7c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc437fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7b7b60 0x55df5b7c49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43372/0 (socket says 172.21.15.2:43372) 2023-11-16T18:27:30.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc437fe700 1 -- 172.21.15.2:0/3869041447 learned_addr learned my addr 172.21.15.2:0/3869041447 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:30.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc43fff700 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7bc8b0 0x55df5b8247d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc42ffd700 1 -- 172.21.15.2:0/3869041447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7b7b60 msgr2=0x55df5b7c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc42ffd700 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7b7b60 0x55df5b7c49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc42ffd700 1 -- 172.21.15.2:0/3869041447 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7bc8b0 msgr2=0x55df5b8247d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc42ffd700 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7bc8b0 0x55df5b8247d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc42ffd700 1 -- 172.21.15.2:0/3869041447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55df5b6cf190 con 0x55df5b7b8470 2023-11-16T18:27:30.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc437fe700 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7b7b60 0x55df5b7c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:30.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc43fff700 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7bc8b0 0x55df5b8247d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:30.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc42ffd700 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b8470 0x55df5b7c4ef0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdc380094f0 tx=0x7fdc38004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d441447bc0ecaa0e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.986+0000 7fdc22ffd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:27:30.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc22ffd700 1 -- 172.21.15.2:0/3869041447 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3802b070 con 0x55df5b7b8470 2023-11-16T18:27:30.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc22ffd700 1 -- 172.21.15.2:0/3869041447 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fdc38004830 con 0x55df5b7b8470 2023-11-16T18:27:30.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3869041447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc340a57e0 con 0x55df5b7b8470 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc22ffd700 1 -- 172.21.15.2:0/3869041447 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc380049f0 con 0x55df5b7b8470 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc55cdbb80 10 client.25847 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3869041447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55df5b51fc60 con 0x55df5b7b8470 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc22ffd700 1 -- 172.21.15.2:0/3869041447 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc340a57e0 con 0x55df5b7b8470 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc22ffd700 1 -- 172.21.15.2:0/3869041447 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fdc3802e020 con 0x55df5b7b8470 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc55cdbb80 10 client.25847 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc55cdbb80 10 client.25847 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc55cdbb80 20 client.25847 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55df5b826700 0x55df5b828b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc55cdbb80 4 client.25847 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.987+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3869041447 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55df5b6d7660 con 0x55df5b826700 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.988+0000 7fdc437fe700 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55df5b826700 0x55df5b828b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.988+0000 7fdc22ffd700 10 client.25847 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.988+0000 7fdc437fe700 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55df5b826700 0x55df5b828b10 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5e00fc19ad749c5 server_cookie=8638fd887638782e in_seq=0 out_seq=0 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.989+0000 7fdc22ffd700 1 -- 172.21.15.2:0/3869041447 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55df5b6d7660 con 0x55df5b826700 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.989+0000 7fdc22ffd700 10 client.25847 handle_command_reply: tid=0 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.989+0000 7fdc55cdbb80 1 client.25847 shutdown 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.989+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3869041447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b8470 msgr2=0x55df5b7c4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.989+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b8470 0x55df5b7c4ef0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdc380094f0 tx=0x7fdc38004750 comp rx=0 tx=0).stop 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.989+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3869041447 shutdown_connections 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.989+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55df5b826700 0x55df5b828b10 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.989+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df5b7b7b60 0x55df5b7c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.989+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df5b7bc8b0 0x55df5b8247d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.989+0000 7fdc55cdbb80 1 --2- 172.21.15.2:0/3869041447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df5b7b8470 0x55df5b7c4ef0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.989+0000 7fdc22ffd700 0 client.25847 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.989+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3869041447 >> 172.21.15.2:0/3869041447 conn(0x55df5b7ad360 msgr2=0x55df5b7b6960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.989+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3869041447 shutdown_connections 2023-11-16T18:27:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc55cdbb80 1 -- 172.21.15.2:0/3869041447 wait complete. 2023-11-16T18:27:30.010 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc55cdbb80 20 client.25847 trim_cache size 0 max 16384 2023-11-16T18:27:30.010 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:27:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "no active scrubs running", 2023-11-16T18:27:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": {} 2023-11-16T18:27:30.010 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:27:30.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc49b4f700 1 -- 172.21.15.2:0/3493408792 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdc2c0a1340 msgr2=0x7fdc2c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc49b4f700 1 --2- 172.21.15.2:0/3493408792 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdc2c0a1340 0x7fdc2c0a37f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdc38005950 tx=0x7fdc380058e0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc49b4f700 1 -- 172.21.15.2:0/3493408792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc440ffa70 msgr2=0x7fdc44197320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc49b4f700 1 --2- 172.21.15.2:0/3493408792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc440ffa70 0x7fdc44197320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc34009c80 tx=0x7fdc34009f90 comp rx=0 tx=0).stop 2023-11-16T18:27:30.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc49b4f700 1 -- 172.21.15.2:0/3493408792 shutdown_connections 2023-11-16T18:27:30.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc49b4f700 1 --2- 172.21.15.2:0/3493408792 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdc2c0a1340 0x7fdc2c0a37f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc49b4f700 1 --2- 172.21.15.2:0/3493408792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc44105250 0x7fdc441a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc49b4f700 1 --2- 172.21.15.2:0/3493408792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc440ffa70 0x7fdc44197320 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc49b4f700 1 --2- 172.21.15.2:0/3493408792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc44100420 0x7fdc44197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc49b4f700 1 -- 172.21.15.2:0/3493408792 >> 172.21.15.2:0/3493408792 conn(0x7fdc440763a0 msgr2=0x7fdc440fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc49b4f700 1 -- 172.21.15.2:0/3493408792 shutdown_connections 2023-11-16T18:27:30.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:29.990+0000 7fdc49b4f700 1 -- 172.21.15.2:0/3493408792 wait complete. 2023-11-16T18:27:30.013 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T18:27:30.013 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:27:30.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:30 smithi002 ceph-mon[98807]: pgmap v82: 129 pgs: 129 active+clean; 64 MiB data, 269 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 620 op/s 2023-11-16T18:27:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.153+0000 7f133c5d8700 1 -- 172.21.15.2:0/4103789723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1334104970 msgr2=0x7f1334104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.153+0000 7f133c5d8700 1 --2- 172.21.15.2:0/4103789723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1334104970 0x7f1334104d40 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f133000aaa0 tx=0x7f133000adb0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.153+0000 7f133c5d8700 1 -- 172.21.15.2:0/4103789723 shutdown_connections 2023-11-16T18:27:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.153+0000 7f133c5d8700 1 --2- 172.21.15.2:0/4103789723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13340ff310 0x7f133410b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.153+0000 7f133c5d8700 1 --2- 172.21.15.2:0/4103789723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1334104970 0x7f1334104d40 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.153+0000 7f133c5d8700 1 --2- 172.21.15.2:0/4103789723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13340fe960 0x7f13340fedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.153+0000 7f133c5d8700 1 -- 172.21.15.2:0/4103789723 >> 172.21.15.2:0/4103789723 conn(0x7f13340758c0 msgr2=0x7f1334075cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.153+0000 7f133c5d8700 1 -- 172.21.15.2:0/4103789723 shutdown_connections 2023-11-16T18:27:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.153+0000 7f133c5d8700 1 -- 172.21.15.2:0/4103789723 wait complete. 2023-11-16T18:27:30.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.154+0000 7f133c5d8700 1 Processor -- start 2023-11-16T18:27:30.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.154+0000 7f133c5d8700 1 -- start start 2023-11-16T18:27:30.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.154+0000 7f133c5d8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13340fe960 0x7f13341963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.154+0000 7f133c5d8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13340ff310 0x7f1334196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.154+0000 7f133c5d8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1334104970 0x7f133419ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.154+0000 7f133c5d8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133419b3d0 con 0x7f13340fe960 2023-11-16T18:27:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.154+0000 7f133c5d8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133419b540 con 0x7f1334104970 2023-11-16T18:27:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.154+0000 7f133c5d8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133419b700 con 0x7f13340ff310 2023-11-16T18:27:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f133a374700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13340fe960 0x7f13341963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f133a374700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13340fe960 0x7f13341963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51856/0 (socket says 172.21.15.2:51856) 2023-11-16T18:27:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f133a374700 1 -- 172.21.15.2:0/2684256456 learned_addr learned my addr 172.21.15.2:0/2684256456 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f1339b73700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13340ff310 0x7f1334196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f133ab75700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1334104970 0x7f133419ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f133a374700 1 -- 172.21.15.2:0/2684256456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13340ff310 msgr2=0x7f1334196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f133a374700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13340ff310 0x7f1334196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f133a374700 1 -- 172.21.15.2:0/2684256456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1334104970 msgr2=0x7f133419ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f133a374700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1334104970 0x7f133419ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f133a374700 1 -- 172.21.15.2:0/2684256456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133000a700 con 0x7f13340fe960 2023-11-16T18:27:30.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f133a374700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13340fe960 0x7f13341963f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1330000c00 tx=0x7f133000cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f132b7fe700 1 -- 172.21.15.2:0/2684256456 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13300050d0 con 0x7f13340fe960 2023-11-16T18:27:30.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f132b7fe700 1 -- 172.21.15.2:0/2684256456 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13300056f0 con 0x7f13340fe960 2023-11-16T18:27:30.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f132b7fe700 1 -- 172.21.15.2:0/2684256456 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1330016420 con 0x7f13340fe960 2023-11-16T18:27:30.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.155+0000 7f133c5d8700 1 -- 172.21.15.2:0/2684256456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133410e170 con 0x7f13340fe960 2023-11-16T18:27:30.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.156+0000 7f133c5d8700 1 -- 172.21.15.2:0/2684256456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133410dfe0 con 0x7f13340fe960 2023-11-16T18:27:30.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.156+0000 7f133c5d8700 1 -- 172.21.15.2:0/2684256456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133410e300 con 0x7f13340fe960 2023-11-16T18:27:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.158+0000 7f132b7fe700 1 -- 172.21.15.2:0/2684256456 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f133000e5c0 con 0x7f13340fe960 2023-11-16T18:27:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.158+0000 7f132b7fe700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f13200a12c0 0x7f13200a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.158+0000 7f132b7fe700 1 -- 172.21.15.2:0/2684256456 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f1330012070 con 0x7f13340fe960 2023-11-16T18:27:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.158+0000 7f1339b73700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f13200a12c0 0x7f13200a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.159+0000 7f1339b73700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f13200a12c0 0x7f13200a3770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1334197a00 tx=0x7f1324009190 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.162+0000 7f132b7fe700 1 -- 172.21.15.2:0/2684256456 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1330098200 con 0x7f13340fe960 2023-11-16T18:27:30.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.442+0000 7f133c5d8700 1 -- 172.21.15.2:0/2684256456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1334067260 con 0x7f13340fe960 2023-11-16T18:27:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.443+0000 7f132b7fe700 1 -- 172.21.15.2:0/2684256456 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10780 (secure 0 0 0) 0x7f133009b820 con 0x7f13340fe960 2023-11-16T18:27:30.445 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:27:30.445 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:27:20.064309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:27:30.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.446+0000 7f133c5d8700 1 -- 172.21.15.2:0/2684256456 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f13200a12c0 msgr2=0x7f13200a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.446+0000 7f133c5d8700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f13200a12c0 0x7f13200a3770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1334197a00 tx=0x7f1324009190 comp rx=0 tx=0).stop 2023-11-16T18:27:30.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.446+0000 7f133c5d8700 1 -- 172.21.15.2:0/2684256456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13340fe960 msgr2=0x7f13341963f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.446+0000 7f133c5d8700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13340fe960 0x7f13341963f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1330000c00 tx=0x7f133000cda0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.446+0000 7f133c5d8700 1 -- 172.21.15.2:0/2684256456 shutdown_connections 2023-11-16T18:27:30.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.446+0000 7f133c5d8700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f13200a12c0 0x7f13200a3770 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.446+0000 7f133c5d8700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1334104970 0x7f133419ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.446+0000 7f133c5d8700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13340fe960 0x7f13341963f0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.446+0000 7f133c5d8700 1 --2- 172.21.15.2:0/2684256456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13340ff310 0x7f1334196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.446+0000 7f133c5d8700 1 -- 172.21.15.2:0/2684256456 >> 172.21.15.2:0/2684256456 conn(0x7f13340758c0 msgr2=0x7f13340fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.446+0000 7f133c5d8700 1 -- 172.21.15.2:0/2684256456 shutdown_connections 2023-11-16T18:27:30.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.446+0000 7f133c5d8700 1 -- 172.21.15.2:0/2684256456 wait complete. 2023-11-16T18:27:30.449 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 27 2023-11-16T18:27:30.463 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-11-16T18:27:30.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:30 smithi040 ceph-mon[100251]: pgmap v82: 129 pgs: 129 active+clean; 64 MiB data, 269 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 620 op/s 2023-11-16T18:27:30.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:30 smithi188 ceph-mon[100639]: pgmap v82: 129 pgs: 129 active+clean; 64 MiB data, 269 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 620 op/s 2023-11-16T18:27:30.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.610+0000 7f7400628700 1 -- 172.21.15.2:0/2364003299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f80ffa70 msgr2=0x7f73f80ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.610+0000 7f7400628700 1 --2- 172.21.15.2:0/2364003299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f80ffa70 0x7f73f80ffee0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f73e80098f0 tx=0x7f73e8009c00 comp rx=0 tx=0).stop 2023-11-16T18:27:30.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.610+0000 7f7400628700 1 -- 172.21.15.2:0/2364003299 shutdown_connections 2023-11-16T18:27:30.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.610+0000 7f7400628700 1 --2- 172.21.15.2:0/2364003299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8100420 0x7f73f810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.610+0000 7f7400628700 1 --2- 172.21.15.2:0/2364003299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f8105250 0x7f73f8105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.610+0000 7f7400628700 1 --2- 172.21.15.2:0/2364003299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f80ffa70 0x7f73f80ffee0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.610+0000 7f7400628700 1 -- 172.21.15.2:0/2364003299 >> 172.21.15.2:0/2364003299 conn(0x7f73f80763a0 msgr2=0x7f73f80767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.610+0000 7f7400628700 1 -- 172.21.15.2:0/2364003299 shutdown_connections 2023-11-16T18:27:30.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.610+0000 7f7400628700 1 -- 172.21.15.2:0/2364003299 wait complete. 2023-11-16T18:27:30.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.611+0000 7f7400628700 1 Processor -- start 2023-11-16T18:27:30.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.611+0000 7f7400628700 1 -- start start 2023-11-16T18:27:30.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.611+0000 7f7400628700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80ffa70 0x7f73f8196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.611+0000 7f7400628700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f8100420 0x7f73f81968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.611+0000 7f7400628700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8105250 0x7f73f819ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.611+0000 7f7400628700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f819b350 con 0x7f73f8100420 2023-11-16T18:27:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.611+0000 7f7400628700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f819b4c0 con 0x7f73f80ffa70 2023-11-16T18:27:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.611+0000 7f7400628700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f819b680 con 0x7f73f8105250 2023-11-16T18:27:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.611+0000 7f73fe3c4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80ffa70 0x7f73f8196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.611+0000 7f73fe3c4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80ffa70 0x7f73f8196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42308/0 (socket says 172.21.15.2:42308) 2023-11-16T18:27:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.611+0000 7f73fe3c4700 1 -- 172.21.15.2:0/261518879 learned_addr learned my addr 172.21.15.2:0/261518879 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.611+0000 7f73fdbc3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f8100420 0x7f73f81968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.612+0000 7f73febc5700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8105250 0x7f73f819ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.612+0000 7f73fe3c4700 1 -- 172.21.15.2:0/261518879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8105250 msgr2=0x7f73f819ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.612+0000 7f73fe3c4700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8105250 0x7f73f819ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.612+0000 7f73fe3c4700 1 -- 172.21.15.2:0/261518879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f8100420 msgr2=0x7f73f81968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.612+0000 7f73fe3c4700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f8100420 0x7f73f81968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.612+0000 7f73fe3c4700 1 -- 172.21.15.2:0/261518879 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e8009520 con 0x7f73f80ffa70 2023-11-16T18:27:30.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.612+0000 7f73febc5700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8105250 0x7f73f819ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:27:30.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.612+0000 7f73fe3c4700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80ffa70 0x7f73f8196370 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f73f400e9c0 tx=0x7f73f400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.612+0000 7f73ef7fe700 1 -- 172.21.15.2:0/261518879 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f400cbd0 con 0x7f73f80ffa70 2023-11-16T18:27:30.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.612+0000 7f7400628700 1 -- 172.21.15.2:0/261518879 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f810f1e0 con 0x7f73f80ffa70 2023-11-16T18:27:30.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.612+0000 7f7400628700 1 -- 172.21.15.2:0/261518879 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f810f050 con 0x7f73f80ffa70 2023-11-16T18:27:30.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.612+0000 7f73ef7fe700 1 -- 172.21.15.2:0/261518879 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73f400eeb0 con 0x7f73f80ffa70 2023-11-16T18:27:30.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.612+0000 7f73ef7fe700 1 -- 172.21.15.2:0/261518879 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f40054b0 con 0x7f73f80ffa70 2023-11-16T18:27:30.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.616+0000 7f73ef7fe700 1 -- 172.21.15.2:0/261518879 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f73f4005610 con 0x7f73f80ffa70 2023-11-16T18:27:30.618 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.617+0000 7f740c7b4b80 1 Processor -- start 2023-11-16T18:27:30.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.617+0000 7f740c7b4b80 1 -- start start 2023-11-16T18:27:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.617+0000 7f740c7b4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db39961370 0x55db39961740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.617+0000 7f740c7b4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 0x55db39965b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.617+0000 7f740c7b4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db399660c0 0x55db39966510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.617+0000 7f740c7b4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3985e060 con 0x55db399660c0 2023-11-16T18:27:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.617+0000 7f740c7b4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db398b08a0 con 0x55db39961370 2023-11-16T18:27:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.617+0000 7f740c7b4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db398a9d30 con 0x55db39961c80 2023-11-16T18:27:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73febc5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db39961370 0x55db39961740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73febc5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db39961370 0x55db39961740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42314/0 (socket says 172.21.15.2:42314) 2023-11-16T18:27:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73fe3c4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 0x55db39965b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73febc5700 1 -- 172.21.15.2:0/2742217403 learned_addr learned my addr 172.21.15.2:0/2742217403 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73fdbc3700 1 --2- 172.21.15.2:0/2742217403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db399660c0 0x55db39966510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73ef7fe700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f73e40a1340 0x7f73e40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73febc5700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f73e40a1340 0x7f73e40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73fdbc3700 1 -- 172.21.15.2:0/2742217403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 msgr2=0x55db39965b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73fdbc3700 1 --2- 172.21.15.2:0/2742217403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 0x55db39965b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73fdbc3700 1 -- 172.21.15.2:0/2742217403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db39961370 msgr2=0x55db39961740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73fdbc3700 1 --2- 172.21.15.2:0/2742217403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db39961370 0x55db39961740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73fdbc3700 1 -- 172.21.15.2:0/2742217403 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db3972b630 con 0x55db399660c0 2023-11-16T18:27:30.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73ef7fe700 1 -- 172.21.15.2:0/261518879 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f73f4014070 con 0x7f73f80ffa70 2023-11-16T18:27:30.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73fdbc3700 1 --2- 172.21.15.2:0/2742217403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db399660c0 0x55db39966510 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009810 tx=0x7f73e8005b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e797260f4a6697 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.618+0000 7f73febc5700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f73e40a1340 0x7f73e40a37f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009550 tx=0x7f73f0004b40 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.619+0000 7f73ecff9700 1 -- 172.21.15.2:0/2742217403 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e800f720 con 0x55db399660c0 2023-11-16T18:27:30.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.619+0000 7f73ecff9700 1 -- 172.21.15.2:0/2742217403 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73e801d040 con 0x55db399660c0 2023-11-16T18:27:30.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.619+0000 7f73ecff9700 1 -- 172.21.15.2:0/2742217403 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e8005460 con 0x55db399660c0 2023-11-16T18:27:30.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.619+0000 7f740c7b4b80 1 -- 172.21.15.2:0/2742217403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db399660c0 msgr2=0x55db39966510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.619+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/2742217403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db399660c0 0x55db39966510 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009810 tx=0x7f73e8005b80 comp rx=0 tx=0).stop 2023-11-16T18:27:30.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.619+0000 7f740c7b4b80 1 -- 172.21.15.2:0/2742217403 shutdown_connections 2023-11-16T18:27:30.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.619+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/2742217403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db399660c0 0x55db39966510 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.619+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/2742217403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db39961370 0x55db39961740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.619+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/2742217403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 0x55db39965b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.619+0000 7f740c7b4b80 1 -- 172.21.15.2:0/2742217403 >> 172.21.15.2:0/2742217403 conn(0x55db3994e360 msgr2=0x55db39953b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.620+0000 7f740c7b4b80 1 -- 172.21.15.2:0/2742217403 shutdown_connections 2023-11-16T18:27:30.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.620+0000 7f740c7b4b80 1 -- 172.21.15.2:0/2742217403 wait complete. 2023-11-16T18:27:30.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.620+0000 7f740c7b4b80 1 -- start start 2023-11-16T18:27:30.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.620+0000 7f740c7b4b80 1 Processor -- start 2023-11-16T18:27:30.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.620+0000 7f740c7b4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961370 0x55db3996d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.620+0000 7f740c7b4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 0x55db3996a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f740c7b4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db399660c0 0x55db3996af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f740c7b4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3984b8e0 con 0x55db39961370 2023-11-16T18:27:30.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f740c7b4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3984b580 con 0x55db399660c0 2023-11-16T18:27:30.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f740c7b4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3972d920 con 0x55db39961c80 2023-11-16T18:27:30.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73fe3c4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961370 0x55db3996d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73fe3c4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961370 0x55db3996d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51884/0 (socket says 172.21.15.2:51884) 2023-11-16T18:27:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73fdbc3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 0x55db3996a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73fe3c4700 1 -- 172.21.15.2:0/1598088120 learned_addr learned my addr 172.21.15.2:0/1598088120 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f740c7b4b80 10 client.0 _ll_register_callbacks cb 0x55db39869780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73febc5700 1 --2- 172.21.15.2:0/1598088120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db399660c0 0x55db3996af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73fe3c4700 1 -- 172.21.15.2:0/1598088120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 msgr2=0x55db3996a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73fe3c4700 1 --2- 172.21.15.2:0/1598088120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 0x55db3996a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73fe3c4700 1 -- 172.21.15.2:0/1598088120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db399660c0 msgr2=0x55db3996af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73fe3c4700 1 --2- 172.21.15.2:0/1598088120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db399660c0 0x55db3996af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73fe3c4700 1 -- 172.21.15.2:0/1598088120 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f810c8d0 con 0x55db39961370 2023-11-16T18:27:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73fe3c4700 1 --2- 172.21.15.2:0/1598088120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961370 0x55db3996d750 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f73f400c5b0 tx=0x7f73f4099840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6aec32adc72104 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73e1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:27:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73e1ffb700 1 -- 172.21.15.2:0/1598088120 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f40adde0 con 0x55db39961370 2023-11-16T18:27:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73e1ffb700 1 -- 172.21.15.2:0/1598088120 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f73f4099ee0 con 0x55db39961370 2023-11-16T18:27:30.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f740c7b4b80 1 -- 172.21.15.2:0/1598088120 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55db3994bb20 con 0x55db39961370 2023-11-16T18:27:30.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f73e1ffb700 1 -- 172.21.15.2:0/1598088120 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73f4095ee0 con 0x55db39961370 2023-11-16T18:27:30.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f740c7b4b80 10 client.16317 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:30.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.621+0000 7f740c7b4b80 1 -- 172.21.15.2:0/1598088120 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55db39870190 con 0x55db39961370 2023-11-16T18:27:30.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.622+0000 7f73e1ffb700 1 -- 172.21.15.2:0/1598088120 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55db3994bb20 con 0x55db39961370 2023-11-16T18:27:30.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.622+0000 7f73e1ffb700 1 -- 172.21.15.2:0/1598088120 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f73f40993d0 con 0x55db39961370 2023-11-16T18:27:30.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.622+0000 7f740c7b4b80 10 client.16317 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:30.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.622+0000 7f740c7b4b80 10 client.16317 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2023-11-16T18:27:30.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.622+0000 7f740c7b4b80 20 client.16317 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:30.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.622+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/1598088120 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55db3997afe0 0x55db3997d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.622+0000 7f740c7b4b80 4 client.16317 mds_command: new command op to 24446 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:27:30.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.622+0000 7f740c7b4b80 1 -- 172.21.15.2:0/1598088120 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db3986c0a0 con 0x55db3997afe0 2023-11-16T18:27:30.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.622+0000 7f73fdbc3700 1 --2- 172.21.15.2:0/1598088120 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55db3997afe0 0x55db3997d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.623+0000 7f73e1ffb700 10 client.16317 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:27:30.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.623+0000 7f73fdbc3700 1 --2- 172.21.15.2:0/1598088120 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55db3997afe0 0x55db3997d3f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f4f6a23984521e87 server_cookie=f57f001ef3c9b3ea in_seq=0 out_seq=0 2023-11-16T18:27:30.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.623+0000 7f73e1ffb700 1 -- 172.21.15.2:0/1598088120 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55db3986c0a0 con 0x55db3997afe0 2023-11-16T18:27:30.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.623+0000 7f73e1ffb700 10 client.16317 handle_command_reply: tid=0 2023-11-16T18:27:30.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.623+0000 7f740c7b4b80 1 client.16317 shutdown 2023-11-16T18:27:30.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.623+0000 7f740c7b4b80 1 -- 172.21.15.2:0/1598088120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961370 msgr2=0x55db3996d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.623+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/1598088120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961370 0x55db3996d750 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f73f400c5b0 tx=0x7f73f4099840 comp rx=0 tx=0).stop 2023-11-16T18:27:30.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.624+0000 7f740c7b4b80 1 -- 172.21.15.2:0/1598088120 shutdown_connections 2023-11-16T18:27:30.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.624+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/1598088120 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55db3997afe0 0x55db3997d3f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.624+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/1598088120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db399660c0 0x55db3996af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.624+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/1598088120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961370 0x55db3996d750 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.624+0000 7f73e1ffb700 0 client.16317 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:27:30.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.624+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/1598088120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 0x55db3996a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.624+0000 7f740c7b4b80 1 -- 172.21.15.2:0/1598088120 >> 172.21.15.2:0/1598088120 conn(0x55db3994e360 msgr2=0x55db39967570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.624+0000 7f740c7b4b80 1 -- 172.21.15.2:0/1598088120 shutdown_connections 2023-11-16T18:27:30.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.624+0000 7f740c7b4b80 1 -- 172.21.15.2:0/1598088120 wait complete. 2023-11-16T18:27:30.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.624+0000 7f740c7b4b80 20 client.16317 trim_cache size 0 max 16384 2023-11-16T18:27:30.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.637+0000 7f740c7b4b80 1 Processor -- start 2023-11-16T18:27:30.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.637+0000 7f740c7b4b80 1 -- start start 2023-11-16T18:27:30.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f740c7b4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961370 0x55db39953c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f740c7b4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 0x55db3996d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f740c7b4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db399660c0 0x55db39976960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f740c7b4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3996df80 con 0x55db39961370 2023-11-16T18:27:30.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f740c7b4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3994dee0 con 0x55db399660c0 2023-11-16T18:27:30.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f740c7b4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db39955440 con 0x55db39961c80 2023-11-16T18:27:30.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f73fe3c4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961370 0x55db39953c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f73fdbc3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 0x55db3996d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f73fdbc3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 0x55db3996d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43424/0 (socket says 172.21.15.2:43424) 2023-11-16T18:27:30.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f73febc5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db399660c0 0x55db39976960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f73fdbc3700 1 -- 172.21.15.2:0/494308677 learned_addr learned my addr 172.21.15.2:0/494308677 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:30.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f73febc5700 1 -- 172.21.15.2:0/494308677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 msgr2=0x55db3996d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f73febc5700 1 --2- 172.21.15.2:0/494308677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 0x55db3996d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f73febc5700 1 -- 172.21.15.2:0/494308677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961370 msgr2=0x55db39953c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f73febc5700 1 --2- 172.21.15.2:0/494308677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961370 0x55db39953c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.638+0000 7f73febc5700 1 -- 172.21.15.2:0/494308677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db3986fee0 con 0x55db399660c0 2023-11-16T18:27:30.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.639+0000 7f73febc5700 1 --2- 172.21.15.2:0/494308677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db399660c0 0x55db39976960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73f0016560 tx=0x7f73f0018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e21e947c82d65b91 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.639+0000 7f73ed7fa700 1 -- 172.21.15.2:0/494308677 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f0029070 con 0x55db399660c0 2023-11-16T18:27:30.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.639+0000 7f73ed7fa700 1 -- 172.21.15.2:0/494308677 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73f001f050 con 0x55db399660c0 2023-11-16T18:27:30.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.639+0000 7f73ed7fa700 1 -- 172.21.15.2:0/494308677 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f0019430 con 0x55db399660c0 2023-11-16T18:27:30.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.640+0000 7f740c7b4b80 1 -- 172.21.15.2:0/494308677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db399660c0 msgr2=0x55db39976960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.640+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/494308677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db399660c0 0x55db39976960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73f0016560 tx=0x7f73f0018000 comp rx=0 tx=0).stop 2023-11-16T18:27:30.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.640+0000 7f740c7b4b80 1 -- 172.21.15.2:0/494308677 shutdown_connections 2023-11-16T18:27:30.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.640+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/494308677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db399660c0 0x55db39976960 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.640+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/494308677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961370 0x55db39953c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.640+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/494308677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db39961c80 0x55db3996d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.640+0000 7f740c7b4b80 1 -- 172.21.15.2:0/494308677 >> 172.21.15.2:0/494308677 conn(0x55db3994e360 msgr2=0x55db399640c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.640+0000 7f740c7b4b80 1 -- 172.21.15.2:0/494308677 shutdown_connections 2023-11-16T18:27:30.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.640+0000 7f740c7b4b80 1 -- 172.21.15.2:0/494308677 wait complete. 2023-11-16T18:27:30.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.640+0000 7f740c7b4b80 1 -- start start 2023-11-16T18:27:30.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.640+0000 7f740c7b4b80 1 Processor -- start 2023-11-16T18:27:30.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f740c7b4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db39961370 0x55db399c3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f740c7b4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961c80 0x55db399c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f740c7b4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db399660c0 0x55db399c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f740c7b4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db399c7490 con 0x55db39961c80 2023-11-16T18:27:30.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f740c7b4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db399c7600 con 0x55db39961370 2023-11-16T18:27:30.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f740c7b4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db399c77c0 con 0x55db399660c0 2023-11-16T18:27:30.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f740c7b4b80 10 client.0 _ll_register_callbacks cb 0x55db39869780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:30.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f73fdbc3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961c80 0x55db399c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f73fdbc3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961c80 0x55db399c3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51900/0 (socket says 172.21.15.2:51900) 2023-11-16T18:27:30.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f73fdbc3700 1 -- 172.21.15.2:0/795339902 learned_addr learned my addr 172.21.15.2:0/795339902 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:30.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f73fe3c4700 1 --2- 172.21.15.2:0/795339902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db39961370 0x55db399c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f73febc5700 1 --2- 172.21.15.2:0/795339902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db399660c0 0x55db399c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f73fdbc3700 1 -- 172.21.15.2:0/795339902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db399660c0 msgr2=0x55db399c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f73fdbc3700 1 --2- 172.21.15.2:0/795339902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db399660c0 0x55db399c6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f73fdbc3700 1 -- 172.21.15.2:0/795339902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db39961370 msgr2=0x55db399c3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f73fdbc3700 1 --2- 172.21.15.2:0/795339902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db39961370 0x55db399c3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f73fdbc3700 1 -- 172.21.15.2:0/795339902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f0007d60 con 0x55db39961c80 2023-11-16T18:27:30.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f73fdbc3700 1 --2- 172.21.15.2:0/795339902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961c80 0x55db399c3560 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f73e8005a80 tx=0x7f73e80092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12165ea6ad18a3b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.641+0000 7f73e1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:27:30.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f73e1ffb700 1 -- 172.21.15.2:0/795339902 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e8031040 con 0x55db39961c80 2023-11-16T18:27:30.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f73e1ffb700 1 -- 172.21.15.2:0/795339902 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f73e80334b0 con 0x55db39961c80 2023-11-16T18:27:30.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f740c7b4b80 1 -- 172.21.15.2:0/795339902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73e801d040 con 0x55db39961c80 2023-11-16T18:27:30.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f73e1ffb700 1 -- 172.21.15.2:0/795339902 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73e8033860 con 0x55db39961c80 2023-11-16T18:27:30.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f740c7b4b80 10 client.16323 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:30.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f740c7b4b80 1 -- 172.21.15.2:0/795339902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55db396c0c60 con 0x55db39961c80 2023-11-16T18:27:30.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f73e1ffb700 1 -- 172.21.15.2:0/795339902 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73e801d040 con 0x55db39961c80 2023-11-16T18:27:30.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f73e1ffb700 1 -- 172.21.15.2:0/795339902 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f73e8009c70 con 0x55db39961c80 2023-11-16T18:27:30.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f740c7b4b80 10 client.16323 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:30.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f740c7b4b80 10 client.16323 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2023-11-16T18:27:30.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f740c7b4b80 20 client.16323 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:30.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/795339902 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55db399cfd30 0x55db399d2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f740c7b4b80 4 client.16323 mds_command: new command op to 24446 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:27:30.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.642+0000 7f740c7b4b80 1 -- 172.21.15.2:0/795339902 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55db39878660 con 0x55db399cfd30 2023-11-16T18:27:30.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f73fe3c4700 1 --2- 172.21.15.2:0/795339902 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55db399cfd30 0x55db399d2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f73fe3c4700 1 --2- 172.21.15.2:0/795339902 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55db399cfd30 0x55db399d2140 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=87bffd95fc456626 server_cookie=f056ac1ad3235978 in_seq=0 out_seq=0 2023-11-16T18:27:30.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f73e1ffb700 10 client.16323 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:27:30.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f73e1ffb700 1 -- 172.21.15.2:0/795339902 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55db39878660 con 0x55db399cfd30 2023-11-16T18:27:30.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f73e1ffb700 10 client.16323 handle_command_reply: tid=0 2023-11-16T18:27:30.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f740c7b4b80 1 client.16323 shutdown 2023-11-16T18:27:30.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f740c7b4b80 1 -- 172.21.15.2:0/795339902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961c80 msgr2=0x55db399c3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/795339902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961c80 0x55db399c3560 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f73e8005a80 tx=0x7f73e80092d0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f740c7b4b80 1 -- 172.21.15.2:0/795339902 shutdown_connections 2023-11-16T18:27:30.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/795339902 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x55db399cfd30 0x55db399d2140 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/795339902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db399660c0 0x55db399c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/795339902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db39961370 0x55db399c3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f73e1ffb700 0 client.16323 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:27:30.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f740c7b4b80 1 --2- 172.21.15.2:0/795339902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55db39961c80 0x55db399c3560 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.643+0000 7f740c7b4b80 1 -- 172.21.15.2:0/795339902 >> 172.21.15.2:0/795339902 conn(0x55db3994e360 msgr2=0x55db39960170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f740c7b4b80 1 -- 172.21.15.2:0/795339902 shutdown_connections 2023-11-16T18:27:30.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f740c7b4b80 1 -- 172.21.15.2:0/795339902 wait complete. 2023-11-16T18:27:30.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f740c7b4b80 20 client.16323 trim_cache size 0 max 16384 2023-11-16T18:27:30.673 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:27:30.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f7400628700 1 -- 172.21.15.2:0/261518879 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f73e40a1340 msgr2=0x7f73e40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f7400628700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f73e40a1340 0x7f73e40a37f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009550 tx=0x7f73f0004b40 comp rx=0 tx=0).stop 2023-11-16T18:27:30.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f7400628700 1 -- 172.21.15.2:0/261518879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80ffa70 msgr2=0x7f73f8196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f7400628700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80ffa70 0x7f73f8196370 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f73f400e9c0 tx=0x7f73f400ecd0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f7400628700 1 -- 172.21.15.2:0/261518879 shutdown_connections 2023-11-16T18:27:30.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f7400628700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f73e40a1340 0x7f73e40a37f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f7400628700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8105250 0x7f73f819ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f7400628700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80ffa70 0x7f73f8196370 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f7400628700 1 --2- 172.21.15.2:0/261518879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f8100420 0x7f73f81968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f7400628700 1 -- 172.21.15.2:0/261518879 >> 172.21.15.2:0/261518879 conn(0x7f73f80763a0 msgr2=0x7f73f80fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f7400628700 1 -- 172.21.15.2:0/261518879 shutdown_connections 2023-11-16T18:27:30.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.644+0000 7f7400628700 1 -- 172.21.15.2:0/261518879 wait complete. 2023-11-16T18:27:30.676 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-11-16T18:27:30.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.825+0000 7f6b8019b700 1 -- 172.21.15.2:0/3563831401 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b780ff8f0 msgr2=0x7f6b780ffd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.825+0000 7f6b8019b700 1 --2- 172.21.15.2:0/3563831401 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b780ff8f0 0x7f6b780ffd60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6b680099c0 tx=0x7f6b68009cd0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.825+0000 7f6b8019b700 1 -- 172.21.15.2:0/3563831401 shutdown_connections 2023-11-16T18:27:30.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.825+0000 7f6b8019b700 1 --2- 172.21.15.2:0/3563831401 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b78104140 0x7f6b78104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.825+0000 7f6b8019b700 1 --2- 172.21.15.2:0/3563831401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b781002a0 0x7f6b7810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.825+0000 7f6b8019b700 1 --2- 172.21.15.2:0/3563831401 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b780ff8f0 0x7f6b780ffd60 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.825+0000 7f6b8019b700 1 -- 172.21.15.2:0/3563831401 >> 172.21.15.2:0/3563831401 conn(0x7f6b78075880 msgr2=0x7f6b78075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.825+0000 7f6b8019b700 1 -- 172.21.15.2:0/3563831401 shutdown_connections 2023-11-16T18:27:30.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.825+0000 7f6b8019b700 1 -- 172.21.15.2:0/3563831401 wait complete. 2023-11-16T18:27:30.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b8019b700 1 Processor -- start 2023-11-16T18:27:30.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b8019b700 1 -- start start 2023-11-16T18:27:30.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b8019b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b780ff8f0 0x7f6b7810ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b8019b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b781002a0 0x7f6b7810b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b8019b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b78104140 0x7f6b781a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b8019b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b781a19e0 con 0x7f6b781002a0 2023-11-16T18:27:30.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b8019b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b781a1b50 con 0x7f6b780ff8f0 2023-11-16T18:27:30.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b8019b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b781a1d10 con 0x7f6b78104140 2023-11-16T18:27:30.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b7d736700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b781002a0 0x7f6b7810b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b7df37700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b780ff8f0 0x7f6b7810ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b7d736700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b781002a0 0x7f6b7810b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51914/0 (socket says 172.21.15.2:51914) 2023-11-16T18:27:30.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b7d736700 1 -- 172.21.15.2:0/1957853349 learned_addr learned my addr 172.21.15.2:0/1957853349 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:30.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b7e738700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b78104140 0x7f6b781a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b7d736700 1 -- 172.21.15.2:0/1957853349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b78104140 msgr2=0x7f6b781a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.826+0000 7f6b7d736700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b78104140 0x7f6b781a1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.827+0000 7f6b7d736700 1 -- 172.21.15.2:0/1957853349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b780ff8f0 msgr2=0x7f6b7810ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.827+0000 7f6b7d736700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b780ff8f0 0x7f6b7810ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.827+0000 7f6b7d736700 1 -- 172.21.15.2:0/1957853349 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b680095f0 con 0x7f6b781002a0 2023-11-16T18:27:30.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.827+0000 7f6b7df37700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b780ff8f0 0x7f6b7810ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:30.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.827+0000 7f6b7d736700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b781002a0 0x7f6b7810b090 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6b68000c00 tx=0x7f6b68005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.827+0000 7f6b6effd700 1 -- 172.21.15.2:0/1957853349 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6801d070 con 0x7f6b781002a0 2023-11-16T18:27:30.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.827+0000 7f6b8019b700 1 -- 172.21.15.2:0/1957853349 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7810e150 con 0x7f6b781002a0 2023-11-16T18:27:30.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.827+0000 7f6b6effd700 1 -- 172.21.15.2:0/1957853349 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b68003fa0 con 0x7f6b781002a0 2023-11-16T18:27:30.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.827+0000 7f6b6effd700 1 -- 172.21.15.2:0/1957853349 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68022e10 con 0x7f6b781002a0 2023-11-16T18:27:30.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.827+0000 7f6b8019b700 1 -- 172.21.15.2:0/1957853349 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7810dfc0 con 0x7f6b781002a0 2023-11-16T18:27:30.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.830+0000 7f6b6effd700 1 -- 172.21.15.2:0/1957853349 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f6b68022500 con 0x7f6b781002a0 2023-11-16T18:27:30.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.831+0000 7f6b6effd700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b640a1330 0x7f6b640a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.831+0000 7f6b7df37700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b640a1330 0x7f6b640a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.831+0000 7f6b6effd700 1 -- 172.21.15.2:0/1957853349 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6b680cdde0 con 0x7f6b781002a0 2023-11-16T18:27:30.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.831+0000 7f6b8c327b80 1 Processor -- start 2023-11-16T18:27:30.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b8c327b80 1 -- start start 2023-11-16T18:27:30.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b7df37700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b640a1330 0x7f6b640a37e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6b74006fd0 tx=0x7f6b74009190 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b8c327b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e38b60 0x558b50e38f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b8c327b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 0x558b50e3d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b8c327b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 0x558b50e3dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b8c327b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b50d3e060 con 0x558b50e39470 2023-11-16T18:27:30.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b8c327b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b50d908a0 con 0x558b50e38b60 2023-11-16T18:27:30.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b8c327b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b50d89d30 con 0x558b50e3d8b0 2023-11-16T18:27:30.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b7d736700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 0x558b50e3d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b7d736700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 0x558b50e3d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51922/0 (socket says 172.21.15.2:51922) 2023-11-16T18:27:30.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b7d736700 1 -- 172.21.15.2:0/3660203638 learned_addr learned my addr 172.21.15.2:0/3660203638 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b7e738700 1 --2- 172.21.15.2:0/3660203638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 0x558b50e3dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b7df37700 1 --2- 172.21.15.2:0/3660203638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e38b60 0x558b50e38f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b7d736700 1 -- 172.21.15.2:0/3660203638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 msgr2=0x558b50e3dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b7d736700 1 --2- 172.21.15.2:0/3660203638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 0x558b50e3dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b7d736700 1 -- 172.21.15.2:0/3660203638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e38b60 msgr2=0x558b50e38f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b7d736700 1 --2- 172.21.15.2:0/3660203638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e38b60 0x558b50e38f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b7d736700 1 -- 172.21.15.2:0/3660203638 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b50c0b630 con 0x558b50e39470 2023-11-16T18:27:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.832+0000 7f6b7e738700 1 --2- 172.21.15.2:0/3660203638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 0x558b50e3dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.833+0000 7f6b7d736700 1 --2- 172.21.15.2:0/3660203638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 0x558b50e3d370 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6b680976d0 tx=0x7f6b68009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b361aca8d5bc8c9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.833+0000 7f6b63fff700 1 -- 172.21.15.2:0/3660203638 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6809ec90 con 0x558b50e39470 2023-11-16T18:27:30.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.833+0000 7f6b63fff700 1 -- 172.21.15.2:0/3660203638 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b680a3020 con 0x558b50e39470 2023-11-16T18:27:30.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.833+0000 7f6b63fff700 1 -- 172.21.15.2:0/3660203638 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6809a3b0 con 0x558b50e39470 2023-11-16T18:27:30.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.834+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3660203638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 msgr2=0x558b50e3d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.834+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3660203638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 0x558b50e3d370 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6b680976d0 tx=0x7f6b68009cd0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.834+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3660203638 shutdown_connections 2023-11-16T18:27:30.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.834+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3660203638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 0x558b50e3dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.834+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3660203638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e38b60 0x558b50e38f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.834+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3660203638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 0x558b50e3d370 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.834+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3660203638 >> 172.21.15.2:0/3660203638 conn(0x558b50e2e360 msgr2=0x558b50e33b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.834+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3660203638 shutdown_connections 2023-11-16T18:27:30.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.834+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3660203638 wait complete. 2023-11-16T18:27:30.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.834+0000 7f6b8c327b80 1 -- start start 2023-11-16T18:27:30.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.834+0000 7f6b8c327b80 1 Processor -- start 2023-11-16T18:27:30.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b8c327b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 0x558b50e44ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b8c327b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e39470 0x558b50e421e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b8c327b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 0x558b50e42720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b8c327b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b50d2b8e0 con 0x558b50e38b60 2023-11-16T18:27:30.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b8c327b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b50d2b580 con 0x558b50e39470 2023-11-16T18:27:30.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b8c327b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b50c0d920 con 0x558b50e3d8b0 2023-11-16T18:27:30.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b8c327b80 10 client.0 _ll_register_callbacks cb 0x558b50d49780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:30.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b7df37700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 0x558b50e44ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b7df37700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 0x558b50e44ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51926/0 (socket says 172.21.15.2:51926) 2023-11-16T18:27:30.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b7df37700 1 -- 172.21.15.2:0/1935281156 learned_addr learned my addr 172.21.15.2:0/1935281156 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:30.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b7d736700 1 --2- 172.21.15.2:0/1935281156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e39470 0x558b50e421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b7df37700 1 -- 172.21.15.2:0/1935281156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 msgr2=0x558b50e42720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b7df37700 1 --2- 172.21.15.2:0/1935281156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 0x558b50e42720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b7df37700 1 -- 172.21.15.2:0/1935281156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e39470 msgr2=0x558b50e421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b7df37700 1 --2- 172.21.15.2:0/1935281156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e39470 0x558b50e421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b7df37700 1 -- 172.21.15.2:0/1935281156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7810dfc0 con 0x558b50e38b60 2023-11-16T18:27:30.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.835+0000 7f6b7d736700 1 --2- 172.21.15.2:0/1935281156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e39470 0x558b50e421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:30.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.836+0000 7f6b7df37700 1 --2- 172.21.15.2:0/1935281156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 0x558b50e44ec0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6b74015a00 tx=0x7f6b740047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c703d5d07f07ba server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.836+0000 7f6b617fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:27:30.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.836+0000 7f6b617fa700 1 -- 172.21.15.2:0/1935281156 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b74021470 con 0x558b50e38b60 2023-11-16T18:27:30.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.836+0000 7f6b617fa700 1 -- 172.21.15.2:0/1935281156 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6b740215d0 con 0x558b50e38b60 2023-11-16T18:27:30.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.836+0000 7f6b8c327b80 1 -- 172.21.15.2:0/1935281156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558b50e2bb20 con 0x558b50e38b60 2023-11-16T18:27:30.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.836+0000 7f6b617fa700 1 -- 172.21.15.2:0/1935281156 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b740217d0 con 0x558b50e38b60 2023-11-16T18:27:30.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.836+0000 7f6b8c327b80 10 client.16335 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:30.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.836+0000 7f6b8c327b80 1 -- 172.21.15.2:0/1935281156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558b50d50190 con 0x558b50e38b60 2023-11-16T18:27:30.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.836+0000 7f6b617fa700 1 -- 172.21.15.2:0/1935281156 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558b50e2bb20 con 0x558b50e38b60 2023-11-16T18:27:30.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.837+0000 7f6b617fa700 1 -- 172.21.15.2:0/1935281156 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f6b74013b20 con 0x558b50e38b60 2023-11-16T18:27:30.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.837+0000 7f6b8c327b80 10 client.16335 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:30.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.837+0000 7f6b8c327b80 10 client.16335 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-11-16T18:27:30.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.837+0000 7f6b8c327b80 20 client.16335 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:30.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.837+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/1935281156 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x558b50e63510 0x558b50e65920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.837+0000 7f6b8c327b80 4 client.16335 mds_command: new command op to 24460 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:27:30.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.837+0000 7f6b8c327b80 1 -- 172.21.15.2:0/1935281156 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558b50d4c0a0 con 0x558b50e63510 2023-11-16T18:27:30.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.837+0000 7f6b7d736700 1 --2- 172.21.15.2:0/1935281156 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x558b50e63510 0x558b50e65920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.837+0000 7f6b617fa700 10 client.16335 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:27:30.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.837+0000 7f6b7d736700 1 --2- 172.21.15.2:0/1935281156 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x558b50e63510 0x558b50e65920 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=be43eb4b410c76ae server_cookie=6fe55587b310cab0 in_seq=0 out_seq=0 2023-11-16T18:27:30.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.840+0000 7f6b617fa700 1 -- 172.21.15.2:0/1935281156 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x558b50d4c0a0 con 0x558b50e63510 2023-11-16T18:27:30.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.840+0000 7f6b617fa700 10 client.16335 handle_command_reply: tid=0 2023-11-16T18:27:30.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.840+0000 7f6b8c327b80 1 client.16335 shutdown 2023-11-16T18:27:30.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.840+0000 7f6b8c327b80 1 -- 172.21.15.2:0/1935281156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 msgr2=0x558b50e44ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.840+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/1935281156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 0x558b50e44ec0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6b74015a00 tx=0x7f6b740047e0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.840+0000 7f6b8c327b80 1 -- 172.21.15.2:0/1935281156 shutdown_connections 2023-11-16T18:27:30.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.840+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/1935281156 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x558b50e63510 0x558b50e65920 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.840+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/1935281156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 0x558b50e42720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.840+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/1935281156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 0x558b50e44ec0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.840+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/1935281156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e39470 0x558b50e421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.840+0000 7f6b617fa700 0 client.16335 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:27:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.840+0000 7f6b8c327b80 1 -- 172.21.15.2:0/1935281156 >> 172.21.15.2:0/1935281156 conn(0x558b50e2e360 msgr2=0x558b50e3ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.840+0000 7f6b8c327b80 1 -- 172.21.15.2:0/1935281156 shutdown_connections 2023-11-16T18:27:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.841+0000 7f6b8c327b80 1 -- 172.21.15.2:0/1935281156 wait complete. 2023-11-16T18:27:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.841+0000 7f6b8c327b80 20 client.16335 trim_cache size 0 max 16384 2023-11-16T18:27:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b8c327b80 1 Processor -- start 2023-11-16T18:27:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b8c327b80 1 -- start start 2023-11-16T18:27:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b8c327b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e38b60 0x558b50e449f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b8c327b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 0x558b50e3b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b8c327b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e3d8b0 0x558b50e33640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b8c327b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b50e437e0 con 0x558b50e39470 2023-11-16T18:27:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b8c327b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b50e39bc0 con 0x558b50e3d8b0 2023-11-16T18:27:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b8c327b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b50e60be0 con 0x558b50e38b60 2023-11-16T18:27:30.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b7d736700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 0x558b50e3b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b7e738700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e3d8b0 0x558b50e33640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b7d736700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 0x558b50e3b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51938/0 (socket says 172.21.15.2:51938) 2023-11-16T18:27:30.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b7e738700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e3d8b0 0x558b50e33640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42374/0 (socket says 172.21.15.2:42374) 2023-11-16T18:27:30.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b7d736700 1 -- 172.21.15.2:0/3520172328 learned_addr learned my addr 172.21.15.2:0/3520172328 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:30.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.857+0000 7f6b7df37700 1 --2- 172.21.15.2:0/3520172328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e38b60 0x558b50e449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.858+0000 7f6b7d736700 1 -- 172.21.15.2:0/3520172328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e38b60 msgr2=0x558b50e449f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.858+0000 7f6b7d736700 1 --2- 172.21.15.2:0/3520172328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e38b60 0x558b50e449f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.858+0000 7f6b7d736700 1 -- 172.21.15.2:0/3520172328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e3d8b0 msgr2=0x558b50e33640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.858+0000 7f6b7d736700 1 --2- 172.21.15.2:0/3520172328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e3d8b0 0x558b50e33640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.858+0000 7f6b7d736700 1 -- 172.21.15.2:0/3520172328 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b50d4fee0 con 0x558b50e39470 2023-11-16T18:27:30.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.858+0000 7f6b7e738700 1 --2- 172.21.15.2:0/3520172328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e3d8b0 0x558b50e33640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:30.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.858+0000 7f6b7d736700 1 --2- 172.21.15.2:0/3520172328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 0x558b50e3b8b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6b680cb740 tx=0x7f6b6801a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f5ff7154d2f2734 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.858+0000 7f6b6cff9700 1 -- 172.21.15.2:0/3520172328 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b680b2070 con 0x558b50e39470 2023-11-16T18:27:30.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.858+0000 7f6b6cff9700 1 -- 172.21.15.2:0/3520172328 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b6809a5c0 con 0x558b50e39470 2023-11-16T18:27:30.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.858+0000 7f6b6cff9700 1 -- 172.21.15.2:0/3520172328 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b680a2df0 con 0x558b50e39470 2023-11-16T18:27:30.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.859+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3520172328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 msgr2=0x558b50e3b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.859+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3520172328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 0x558b50e3b8b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6b680cb740 tx=0x7f6b6801a040 comp rx=0 tx=0).stop 2023-11-16T18:27:30.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.859+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3520172328 shutdown_connections 2023-11-16T18:27:30.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.859+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3520172328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e38b60 0x558b50e449f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.859+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3520172328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e3d8b0 0x558b50e33640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.859+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3520172328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e39470 0x558b50e3b8b0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.859+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3520172328 >> 172.21.15.2:0/3520172328 conn(0x558b50e2e360 msgr2=0x558b50e5f860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.859+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3520172328 shutdown_connections 2023-11-16T18:27:30.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.859+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3520172328 wait complete. 2023-11-16T18:27:30.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.859+0000 7f6b8c327b80 1 -- start start 2023-11-16T18:27:30.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.859+0000 7f6b8c327b80 1 Processor -- start 2023-11-16T18:27:30.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b8c327b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 0x558b50e57d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b8c327b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e39470 0x558b50e582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b8c327b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 0x558b50e58800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b8c327b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b50eb6390 con 0x558b50e38b60 2023-11-16T18:27:30.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b8c327b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b50eb64d0 con 0x558b50e39470 2023-11-16T18:27:30.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b8c327b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b50eb6690 con 0x558b50e3d8b0 2023-11-16T18:27:30.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b8c327b80 10 client.0 _ll_register_callbacks cb 0x7f6b7400ef60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:30.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b7df37700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 0x558b50e57d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b7df37700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 0x558b50e57d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51940/0 (socket says 172.21.15.2:51940) 2023-11-16T18:27:30.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b7df37700 1 -- 172.21.15.2:0/3273681986 learned_addr learned my addr 172.21.15.2:0/3273681986 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:30.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b7d736700 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e39470 0x558b50e582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.889 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b7e738700 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 0x558b50e58800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b7df37700 1 -- 172.21.15.2:0/3273681986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 msgr2=0x558b50e58800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b7df37700 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 0x558b50e58800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b7df37700 1 -- 172.21.15.2:0/3273681986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e39470 msgr2=0x558b50e582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:30.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b7df37700 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e39470 0x558b50e582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b7df37700 1 -- 172.21.15.2:0/3273681986 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b70003c50 con 0x558b50e38b60 2023-11-16T18:27:30.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.860+0000 7f6b7d736700 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e39470 0x558b50e582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:30.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.861+0000 7f6b7df37700 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 0x558b50e57d80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6b74015a00 tx=0x7f6b74014a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10fe77eda0ed863c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:30.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.861+0000 7f6b617fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:27:30.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.861+0000 7f6b617fa700 1 -- 172.21.15.2:0/3273681986 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b74004230 con 0x558b50e38b60 2023-11-16T18:27:30.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.861+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3273681986 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b680a3020 con 0x558b50e38b60 2023-11-16T18:27:30.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.861+0000 7f6b617fa700 1 -- 172.21.15.2:0/3273681986 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6b74014530 con 0x558b50e38b60 2023-11-16T18:27:30.890 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.861+0000 7f6b617fa700 1 -- 172.21.15.2:0/3273681986 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b74004390 con 0x558b50e38b60 2023-11-16T18:27:30.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.861+0000 7f6b8c327b80 10 client.16341 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:30.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.861+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3273681986 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558b50c0b630 con 0x558b50e38b60 2023-11-16T18:27:30.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.861+0000 7f6b617fa700 1 -- 172.21.15.2:0/3273681986 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b680a3020 con 0x558b50e38b60 2023-11-16T18:27:30.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.862+0000 7f6b617fa700 1 -- 172.21.15.2:0/3273681986 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f6b7403c030 con 0x558b50e38b60 2023-11-16T18:27:30.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.862+0000 7f6b8c327b80 10 client.16341 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:30.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.862+0000 7f6b8c327b80 10 client.16341 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-11-16T18:27:30.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.862+0000 7f6b8c327b80 20 client.16341 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:30.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.862+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x558b50eb7da0 0x558b50eba1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:30.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.862+0000 7f6b8c327b80 4 client.16341 mds_command: new command op to 24460 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:27:30.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.862+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3273681986 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558b50d58660 con 0x558b50eb7da0 2023-11-16T18:27:30.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.862+0000 7f6b7d736700 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x558b50eb7da0 0x558b50eba1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:30.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.863+0000 7f6b617fa700 10 client.16341 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:27:30.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.863+0000 7f6b7d736700 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x558b50eb7da0 0x558b50eba1b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=84fac63243eb3466 server_cookie=45a261b9b8f37a25 in_seq=0 out_seq=0 2023-11-16T18:27:30.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.866+0000 7f6b617fa700 1 -- 172.21.15.2:0/3273681986 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558b50d58660 con 0x558b50eb7da0 2023-11-16T18:27:30.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.866+0000 7f6b617fa700 10 client.16341 handle_command_reply: tid=0 2023-11-16T18:27:30.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.866+0000 7f6b8c327b80 1 client.16341 shutdown 2023-11-16T18:27:30.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.867+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3273681986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 msgr2=0x558b50e57d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.867+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 0x558b50e57d80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6b74015a00 tx=0x7f6b74014a20 comp rx=0 tx=0).stop 2023-11-16T18:27:30.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.867+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3273681986 shutdown_connections 2023-11-16T18:27:30.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.867+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x558b50eb7da0 0x558b50eba1b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.867+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558b50e3d8b0 0x558b50e58800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.867+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558b50e38b60 0x558b50e57d80 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.867+0000 7f6b8c327b80 1 --2- 172.21.15.2:0/3273681986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b50e39470 0x558b50e582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.867+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3273681986 >> 172.21.15.2:0/3273681986 conn(0x558b50e2e360 msgr2=0x558b50e3b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.867+0000 7f6b617fa700 0 client.16341 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:27:30.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.867+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3273681986 shutdown_connections 2023-11-16T18:27:30.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.867+0000 7f6b8c327b80 1 -- 172.21.15.2:0/3273681986 wait complete. 2023-11-16T18:27:30.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.867+0000 7f6b8c327b80 20 client.16341 trim_cache size 0 max 16384 2023-11-16T18:27:30.893 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:27:30.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.868+0000 7f6b8019b700 1 -- 172.21.15.2:0/1957853349 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b640a1330 msgr2=0x7f6b640a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.868+0000 7f6b8019b700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b640a1330 0x7f6b640a37e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6b74006fd0 tx=0x7f6b74009190 comp rx=0 tx=0).stop 2023-11-16T18:27:30.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.868+0000 7f6b8019b700 1 -- 172.21.15.2:0/1957853349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b781002a0 msgr2=0x7f6b7810b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:30.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.868+0000 7f6b8019b700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b781002a0 0x7f6b7810b090 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6b68000c00 tx=0x7f6b68005740 comp rx=0 tx=0).stop 2023-11-16T18:27:30.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.868+0000 7f6b8019b700 1 -- 172.21.15.2:0/1957853349 shutdown_connections 2023-11-16T18:27:30.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.868+0000 7f6b8019b700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6b640a1330 0x7f6b640a37e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.868+0000 7f6b8019b700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b78104140 0x7f6b781a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.868+0000 7f6b8019b700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b780ff8f0 0x7f6b7810ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.868+0000 7f6b8019b700 1 --2- 172.21.15.2:0/1957853349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b781002a0 0x7f6b7810b090 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:30.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.868+0000 7f6b8019b700 1 -- 172.21.15.2:0/1957853349 >> 172.21.15.2:0/1957853349 conn(0x7f6b78075880 msgr2=0x7f6b780fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:30.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.868+0000 7f6b8019b700 1 -- 172.21.15.2:0/1957853349 shutdown_connections 2023-11-16T18:27:30.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:30.868+0000 7f6b8019b700 1 -- 172.21.15.2:0/1957853349 wait complete. 2023-11-16T18:27:30.895 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T18:27:31.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.042+0000 7f5617b8b700 1 -- 172.21.15.2:0/1764815567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5610105590 msgr2=0x7f561010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:31.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.042+0000 7f5617b8b700 1 --2- 172.21.15.2:0/1764815567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5610105590 0x7f561010d160 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f560800b0d0 tx=0x7f560800b3e0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.043+0000 7f5617b8b700 1 -- 172.21.15.2:0/1764815567 shutdown_connections 2023-11-16T18:27:31.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.043+0000 7f5617b8b700 1 --2- 172.21.15.2:0/1764815567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5610105590 0x7f561010d160 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.043+0000 7f5617b8b700 1 --2- 172.21.15.2:0/1764815567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5610100660 0x7f5610100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.043+0000 7f5617b8b700 1 --2- 172.21.15.2:0/1764815567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5610101000 0x7f5610104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.043+0000 7f5617b8b700 1 -- 172.21.15.2:0/1764815567 >> 172.21.15.2:0/1764815567 conn(0x7f5610077380 msgr2=0x7f5610077780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:31.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.043+0000 7f5617b8b700 1 -- 172.21.15.2:0/1764815567 shutdown_connections 2023-11-16T18:27:31.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.043+0000 7f5617b8b700 1 -- 172.21.15.2:0/1764815567 wait complete. 2023-11-16T18:27:31.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.044+0000 7f5617b8b700 1 Processor -- start 2023-11-16T18:27:31.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.044+0000 7f5617b8b700 1 -- start start 2023-11-16T18:27:31.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.044+0000 7f5617b8b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5610100660 0x7f56101986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.044+0000 7f5617b8b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5610101000 0x7f5610198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.044+0000 7f5617b8b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5610105590 0x7f561019bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.044+0000 7f5617b8b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561019c590 con 0x7f5610101000 2023-11-16T18:27:31.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.044+0000 7f5617b8b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561019c700 con 0x7f5610100660 2023-11-16T18:27:31.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.044+0000 7f5617b8b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561019c8c0 con 0x7f5610105590 2023-11-16T18:27:31.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.044+0000 7f5615126700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5610101000 0x7f5610198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.044+0000 7f5615927700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5610100660 0x7f56101986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.044+0000 7f5615126700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5610101000 0x7f5610198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51956/0 (socket says 172.21.15.2:51956) 2023-11-16T18:27:31.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.044+0000 7f5615126700 1 -- 172.21.15.2:0/1779422127 learned_addr learned my addr 172.21.15.2:0/1779422127 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:31.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.044+0000 7f5616128700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5610105590 0x7f561019bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.045+0000 7f5615126700 1 -- 172.21.15.2:0/1779422127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5610105590 msgr2=0x7f561019bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:31.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.045+0000 7f5615126700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5610105590 0x7f561019bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.045+0000 7f5615126700 1 -- 172.21.15.2:0/1779422127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5610100660 msgr2=0x7f56101986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:31.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.045+0000 7f5615126700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5610100660 0x7f56101986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.045+0000 7f5615126700 1 -- 172.21.15.2:0/1779422127 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5608009d00 con 0x7f5610101000 2023-11-16T18:27:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.045+0000 7f5616128700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5610105590 0x7f561019bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:27:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.045+0000 7f5615126700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5610101000 0x7f5610198be0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f560000e940 tx=0x7f560000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.045+0000 7f5606ffd700 1 -- 172.21.15.2:0/1779422127 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600009730 con 0x7f5610101000 2023-11-16T18:27:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.045+0000 7f5606ffd700 1 -- 172.21.15.2:0/1779422127 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f560000ee90 con 0x7f5610101000 2023-11-16T18:27:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.045+0000 7f5606ffd700 1 -- 172.21.15.2:0/1779422127 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600010ea0 con 0x7f5610101000 2023-11-16T18:27:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.045+0000 7f5617b8b700 1 -- 172.21.15.2:0/1779422127 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56100743f0 con 0x7f5610101000 2023-11-16T18:27:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.045+0000 7f5617b8b700 1 -- 172.21.15.2:0/1779422127 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5610074260 con 0x7f5610101000 2023-11-16T18:27:31.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.048+0000 7f5606ffd700 1 -- 172.21.15.2:0/1779422127 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f560000cbd0 con 0x7f5610101000 2023-11-16T18:27:31.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.049+0000 7f5606ffd700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f55fc0a12c0 0x7f55fc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.049+0000 7f5606ffd700 1 -- 172.21.15.2:0/1779422127 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f5600014070 con 0x7f5610101000 2023-11-16T18:27:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.049+0000 7f5615927700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f55fc0a12c0 0x7f55fc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.050+0000 7f5623d17b80 1 Processor -- start 2023-11-16T18:27:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.050+0000 7f5623d17b80 1 -- start start 2023-11-16T18:27:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.050+0000 7f5615927700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f55fc0a12c0 0x7f55fc0a3770 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f560c009c60 tx=0x7f560c009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.050+0000 7f5623d17b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa979370 0x5555aa979740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.050+0000 7f5623d17b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979c80 0x5555aa97db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.050+0000 7f5623d17b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa97e0c0 0x5555aa97e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.050+0000 7f5623d17b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555aa876060 con 0x5555aa97e0c0 2023-11-16T18:27:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.050+0000 7f5623d17b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555aa8c88a0 con 0x5555aa979c80 2023-11-16T18:27:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.050+0000 7f5623d17b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555aa8c1d30 con 0x5555aa979370 2023-11-16T18:27:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.050+0000 7f5615927700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa979370 0x5555aa979740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.050+0000 7f5616128700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa97e0c0 0x5555aa97e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.051+0000 7f5616128700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa97e0c0 0x5555aa97e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51966/0 (socket says 172.21.15.2:51966) 2023-11-16T18:27:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.051+0000 7f5615126700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979c80 0x5555aa97db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.051+0000 7f5616128700 1 -- 172.21.15.2:0/1082007093 learned_addr learned my addr 172.21.15.2:0/1082007093 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.051+0000 7f5615126700 1 -- 172.21.15.2:0/1082007093 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa979370 msgr2=0x5555aa979740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.051+0000 7f5615126700 1 --2- 172.21.15.2:0/1082007093 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa979370 0x5555aa979740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.051+0000 7f5615126700 1 -- 172.21.15.2:0/1082007093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa97e0c0 msgr2=0x5555aa97e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.051+0000 7f5615126700 1 --2- 172.21.15.2:0/1082007093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa97e0c0 0x5555aa97e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.051+0000 7f5615126700 1 -- 172.21.15.2:0/1082007093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555aa743630 con 0x5555aa979c80 2023-11-16T18:27:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.051+0000 7f5616128700 1 --2- 172.21.15.2:0/1082007093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa97e0c0 0x5555aa97e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.051+0000 7f5615126700 1 --2- 172.21.15.2:0/1082007093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979c80 0x5555aa97db80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f56000ca7a0 tx=0x7f5600016330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fedc89710618a6df server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.051+0000 7f55fbfff700 1 -- 172.21.15.2:0/1082007093 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56000a2570 con 0x5555aa979c80 2023-11-16T18:27:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.051+0000 7f55fbfff700 1 -- 172.21.15.2:0/1082007093 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56000984c0 con 0x5555aa979c80 2023-11-16T18:27:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.051+0000 7f55fbfff700 1 -- 172.21.15.2:0/1082007093 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56000a20e0 con 0x5555aa979c80 2023-11-16T18:27:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.052+0000 7f5623d17b80 1 -- 172.21.15.2:0/1082007093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979c80 msgr2=0x5555aa97db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.052+0000 7f5623d17b80 1 --2- 172.21.15.2:0/1082007093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979c80 0x5555aa97db80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f56000ca7a0 tx=0x7f5600016330 comp rx=0 tx=0).stop 2023-11-16T18:27:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.052+0000 7f5623d17b80 1 -- 172.21.15.2:0/1082007093 shutdown_connections 2023-11-16T18:27:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.052+0000 7f5623d17b80 1 --2- 172.21.15.2:0/1082007093 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa979370 0x5555aa979740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.052+0000 7f5623d17b80 1 --2- 172.21.15.2:0/1082007093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa97e0c0 0x5555aa97e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.052+0000 7f5623d17b80 1 --2- 172.21.15.2:0/1082007093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979c80 0x5555aa97db80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.052+0000 7f5623d17b80 1 -- 172.21.15.2:0/1082007093 >> 172.21.15.2:0/1082007093 conn(0x5555aa966360 msgr2=0x5555aa96bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.052+0000 7f5623d17b80 1 -- 172.21.15.2:0/1082007093 shutdown_connections 2023-11-16T18:27:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.052+0000 7f5623d17b80 1 -- 172.21.15.2:0/1082007093 wait complete. 2023-11-16T18:27:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.053+0000 7f5623d17b80 1 -- start start 2023-11-16T18:27:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.053+0000 7f5623d17b80 1 Processor -- start 2023-11-16T18:27:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.053+0000 7f5623d17b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 0x5555aa982c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.053+0000 7f5623d17b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa979c80 0x5555aa983190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.053+0000 7f5623d17b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa97e0c0 0x5555aa9836d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.053+0000 7f5623d17b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555aa8638e0 con 0x5555aa979370 2023-11-16T18:27:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.053+0000 7f5623d17b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555aa863580 con 0x5555aa97e0c0 2023-11-16T18:27:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.053+0000 7f5623d17b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555aa745920 con 0x5555aa979c80 2023-11-16T18:27:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.053+0000 7f5623d17b80 10 client.0 _ll_register_callbacks cb 0x5555aa881780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.053+0000 7f5615927700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 0x5555aa982c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f5615927700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 0x5555aa982c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51968/0 (socket says 172.21.15.2:51968) 2023-11-16T18:27:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f5615927700 1 -- 172.21.15.2:0/3462528846 learned_addr learned my addr 172.21.15.2:0/3462528846 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f5616128700 1 --2- 172.21.15.2:0/3462528846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa97e0c0 0x5555aa9836d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f5615927700 1 -- 172.21.15.2:0/3462528846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa979c80 msgr2=0x5555aa983190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f5615927700 1 --2- 172.21.15.2:0/3462528846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa979c80 0x5555aa983190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f5615927700 1 -- 172.21.15.2:0/3462528846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa97e0c0 msgr2=0x5555aa9836d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f5615927700 1 --2- 172.21.15.2:0/3462528846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa97e0c0 0x5555aa9836d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f5615927700 1 -- 172.21.15.2:0/3462528846 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5610074260 con 0x5555aa979370 2023-11-16T18:27:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f5616128700 1 --2- 172.21.15.2:0/3462528846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa97e0c0 0x5555aa9836d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f5615927700 1 --2- 172.21.15.2:0/3462528846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 0x5555aa982c50 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f560c005910 tx=0x7f560c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b0cceb55f97b70 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f55f97fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:27:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f55f97fa700 1 -- 172.21.15.2:0/3462528846 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c022070 con 0x5555aa979370 2023-11-16T18:27:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f55f97fa700 1 -- 172.21.15.2:0/3462528846 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f560c01d720 con 0x5555aa979370 2023-11-16T18:27:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.054+0000 7f5623d17b80 1 -- 172.21.15.2:0/3462528846 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5555aa963b20 con 0x5555aa979370 2023-11-16T18:27:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.055+0000 7f55f97fa700 1 -- 172.21.15.2:0/3462528846 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f560c01d8e0 con 0x5555aa979370 2023-11-16T18:27:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.055+0000 7f5623d17b80 10 client.16353 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.055+0000 7f5623d17b80 1 -- 172.21.15.2:0/3462528846 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5555aa888190 con 0x5555aa979370 2023-11-16T18:27:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.055+0000 7f55f97fa700 1 -- 172.21.15.2:0/3462528846 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5555aa963b20 con 0x5555aa979370 2023-11-16T18:27:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.055+0000 7f55f97fa700 1 -- 172.21.15.2:0/3462528846 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f560c027b70 con 0x5555aa979370 2023-11-16T18:27:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.055+0000 7f5623d17b80 10 client.16353 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.055+0000 7f5623d17b80 10 client.16353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.055+0000 7f5623d17b80 20 client.16353 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.055+0000 7f5623d17b80 1 --2- 172.21.15.2:0/3462528846 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5555aa993280 0x5555aa995690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.055+0000 7f5623d17b80 4 client.16353 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:27:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.055+0000 7f5623d17b80 1 -- 172.21.15.2:0/3462528846 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5555aa8840a0 con 0x5555aa993280 2023-11-16T18:27:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.055+0000 7f5615126700 1 --2- 172.21.15.2:0/3462528846 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5555aa993280 0x5555aa995690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.056+0000 7f55f97fa700 10 client.16353 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.056+0000 7f5615126700 1 --2- 172.21.15.2:0/3462528846 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5555aa993280 0x5555aa995690 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99f098699d8cb617 server_cookie=3528548f00ee5829 in_seq=0 out_seq=0 2023-11-16T18:27:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f55f97fa700 1 -- 172.21.15.2:0/3462528846 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5555aa8840a0 con 0x5555aa993280 2023-11-16T18:27:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f55f97fa700 10 client.16353 handle_command_reply: tid=0 2023-11-16T18:27:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f5623d17b80 1 client.16353 shutdown 2023-11-16T18:27:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f5623d17b80 1 -- 172.21.15.2:0/3462528846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 msgr2=0x5555aa982c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f5623d17b80 1 --2- 172.21.15.2:0/3462528846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 0x5555aa982c50 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f560c005910 tx=0x7f560c0047c0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f5623d17b80 1 -- 172.21.15.2:0/3462528846 shutdown_connections 2023-11-16T18:27:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f5623d17b80 1 --2- 172.21.15.2:0/3462528846 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5555aa993280 0x5555aa995690 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f5623d17b80 1 --2- 172.21.15.2:0/3462528846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa97e0c0 0x5555aa9836d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f55f97fa700 0 client.16353 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f5623d17b80 1 --2- 172.21.15.2:0/3462528846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 0x5555aa982c50 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f5623d17b80 1 --2- 172.21.15.2:0/3462528846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa979c80 0x5555aa983190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f5623d17b80 1 -- 172.21.15.2:0/3462528846 >> 172.21.15.2:0/3462528846 conn(0x5555aa966360 msgr2=0x5555aa97f570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f5623d17b80 1 -- 172.21.15.2:0/3462528846 shutdown_connections 2023-11-16T18:27:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.059+0000 7f5623d17b80 1 -- 172.21.15.2:0/3462528846 wait complete. 2023-11-16T18:27:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.060+0000 7f5623d17b80 20 client.16353 trim_cache size 0 max 16384 2023-11-16T18:27:31.079 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.077+0000 7f5623d17b80 1 Processor -- start 2023-11-16T18:27:31.079 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.077+0000 7f5623d17b80 1 -- start start 2023-11-16T18:27:31.080 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.077+0000 7f5623d17b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 0x5555aa97b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.080 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.077+0000 7f5623d17b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979c80 0x5555aa98a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.080 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.077+0000 7f5623d17b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa97e0c0 0x5555aa983540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.077+0000 7f5623d17b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555aa97cd90 con 0x5555aa979370 2023-11-16T18:27:31.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.077+0000 7f5623d17b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555aa985c20 con 0x5555aa979c80 2023-11-16T18:27:31.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.077+0000 7f5623d17b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555aa98a6a0 con 0x5555aa97e0c0 2023-11-16T18:27:31.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.077+0000 7f5615927700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 0x5555aa97b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.077+0000 7f5615927700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 0x5555aa97b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51976/0 (socket says 172.21.15.2:51976) 2023-11-16T18:27:31.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.078+0000 7f5615927700 1 -- 172.21.15.2:0/2677018779 learned_addr learned my addr 172.21.15.2:0/2677018779 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:31.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.078+0000 7f5615126700 1 --2- 172.21.15.2:0/2677018779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979c80 0x5555aa98a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.078+0000 7f5615927700 1 -- 172.21.15.2:0/2677018779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa97e0c0 msgr2=0x5555aa983540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T18:27:31.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.078+0000 7f5615927700 1 --2- 172.21.15.2:0/2677018779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa97e0c0 0x5555aa983540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.078+0000 7f5615927700 1 -- 172.21.15.2:0/2677018779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979c80 msgr2=0x5555aa98a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:31.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.078+0000 7f5615927700 1 --2- 172.21.15.2:0/2677018779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979c80 0x5555aa98a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.078+0000 7f5615927700 1 -- 172.21.15.2:0/2677018779 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555aa887ee0 con 0x5555aa979370 2023-11-16T18:27:31.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.078+0000 7f5615126700 1 --2- 172.21.15.2:0/2677018779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979c80 0x5555aa98a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:31.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.078+0000 7f5615927700 1 --2- 172.21.15.2:0/2677018779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 0x5555aa97b9b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f560c009fd0 tx=0x7f560c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b895ff54226e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:31.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.078+0000 7f5604dac700 1 -- 172.21.15.2:0/2677018779 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c022070 con 0x5555aa979370 2023-11-16T18:27:31.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.078+0000 7f5604dac700 1 -- 172.21.15.2:0/2677018779 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f560c003fa0 con 0x5555aa979370 2023-11-16T18:27:31.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.078+0000 7f5604dac700 1 -- 172.21.15.2:0/2677018779 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c01dbe0 con 0x5555aa979370 2023-11-16T18:27:31.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.079+0000 7f5623d17b80 1 -- 172.21.15.2:0/2677018779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 msgr2=0x5555aa97b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:31.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.079+0000 7f5623d17b80 1 --2- 172.21.15.2:0/2677018779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 0x5555aa97b9b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f560c009fd0 tx=0x7f560c01c4b0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.079+0000 7f5623d17b80 1 -- 172.21.15.2:0/2677018779 shutdown_connections 2023-11-16T18:27:31.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.079+0000 7f5623d17b80 1 --2- 172.21.15.2:0/2677018779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa97e0c0 0x5555aa983540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.079+0000 7f5623d17b80 1 --2- 172.21.15.2:0/2677018779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979370 0x5555aa97b9b0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.079+0000 7f5623d17b80 1 --2- 172.21.15.2:0/2677018779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979c80 0x5555aa98a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.079+0000 7f5623d17b80 1 -- 172.21.15.2:0/2677018779 >> 172.21.15.2:0/2677018779 conn(0x5555aa966360 msgr2=0x5555aa999ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:31.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.079+0000 7f5623d17b80 1 -- 172.21.15.2:0/2677018779 shutdown_connections 2023-11-16T18:27:31.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.079+0000 7f5623d17b80 1 -- 172.21.15.2:0/2677018779 wait complete. 2023-11-16T18:27:31.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.079+0000 7f5623d17b80 1 -- start start 2023-11-16T18:27:31.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.080+0000 7f5623d17b80 1 Processor -- start 2023-11-16T18:27:31.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.080+0000 7f5623d17b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979370 0x5555aa9dfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.080+0000 7f5623d17b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979c80 0x5555aa9e0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.080+0000 7f5623d17b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa97e0c0 0x5555aa989610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.080+0000 7f5623d17b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555aa989bf0 con 0x5555aa979c80 2023-11-16T18:27:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.080+0000 7f5623d17b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555aa989d60 con 0x5555aa979370 2023-11-16T18:27:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.080+0000 7f5623d17b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555aa989f20 con 0x5555aa97e0c0 2023-11-16T18:27:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.080+0000 7f5623d17b80 10 client.0 _ll_register_callbacks cb 0x7f560c0044e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.080+0000 7f5615126700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979c80 0x5555aa9e0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.080+0000 7f5615927700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979370 0x5555aa9dfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.080+0000 7f5615126700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979c80 0x5555aa9e0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51982/0 (socket says 172.21.15.2:51982) 2023-11-16T18:27:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.080+0000 7f5616128700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa97e0c0 0x5555aa989610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.080+0000 7f5615126700 1 -- 172.21.15.2:0/2160429034 learned_addr learned my addr 172.21.15.2:0/2160429034 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f5615927700 1 -- 172.21.15.2:0/2160429034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa97e0c0 msgr2=0x5555aa989610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f5615927700 1 --2- 172.21.15.2:0/2160429034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa97e0c0 0x5555aa989610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f5615927700 1 -- 172.21.15.2:0/2160429034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979c80 msgr2=0x5555aa9e0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:31.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f5615927700 1 --2- 172.21.15.2:0/2160429034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979c80 0x5555aa9e0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f5615927700 1 -- 172.21.15.2:0/2160429034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5555aa743630 con 0x5555aa979370 2023-11-16T18:27:31.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f5615126700 1 --2- 172.21.15.2:0/2160429034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979c80 0x5555aa9e0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:31.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f5615927700 1 --2- 172.21.15.2:0/2160429034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979370 0x5555aa9dfd10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f560c009ed0 tx=0x7f560c009350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a11558bc7ca6f65 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:31.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f55f97fa700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:27:31.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f55f97fa700 1 -- 172.21.15.2:0/2160429034 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c022030 con 0x5555aa979370 2023-11-16T18:27:31.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f55f97fa700 1 -- 172.21.15.2:0/2160429034 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f560c023070 con 0x5555aa979370 2023-11-16T18:27:31.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f5623d17b80 1 -- 172.21.15.2:0/2160429034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f56000984c0 con 0x5555aa979370 2023-11-16T18:27:31.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f55f97fa700 1 -- 172.21.15.2:0/2160429034 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f560c01c840 con 0x5555aa979370 2023-11-16T18:27:31.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f5623d17b80 10 client.25907 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:31.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.081+0000 7f5623d17b80 1 -- 172.21.15.2:0/2160429034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5555aa888190 con 0x5555aa979370 2023-11-16T18:27:31.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.082+0000 7f55f97fa700 1 -- 172.21.15.2:0/2160429034 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56000984c0 con 0x5555aa979370 2023-11-16T18:27:31.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.082+0000 7f55f97fa700 1 -- 172.21.15.2:0/2160429034 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f560c027ae0 con 0x5555aa979370 2023-11-16T18:27:31.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.082+0000 7f5623d17b80 10 client.25907 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:31.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.082+0000 7f5623d17b80 10 client.25907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:31.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.082+0000 7f5623d17b80 20 client.25907 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:31.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.082+0000 7f5623d17b80 1 --2- 172.21.15.2:0/2160429034 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5555aa993280 0x5555aa9e97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:31.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.082+0000 7f5623d17b80 4 client.25907 mds_command: new command op to 24473 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:27:31.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.082+0000 7f5623d17b80 1 -- 172.21.15.2:0/2160429034 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5555aa890660 con 0x5555aa993280 2023-11-16T18:27:31.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.082+0000 7f5615126700 1 --2- 172.21.15.2:0/2160429034 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5555aa993280 0x5555aa9e97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:31.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.083+0000 7f5615126700 1 --2- 172.21.15.2:0/2160429034 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5555aa993280 0x5555aa9e97f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb4dcc3694004c48 server_cookie=831af4e931205860 in_seq=0 out_seq=0 2023-11-16T18:27:31.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.083+0000 7f55f97fa700 10 client.25907 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:31.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f55f97fa700 1 -- 172.21.15.2:0/2160429034 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5555aa890660 con 0x5555aa993280 2023-11-16T18:27:31.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f55f97fa700 10 client.25907 handle_command_reply: tid=0 2023-11-16T18:27:31.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f5623d17b80 1 client.25907 shutdown 2023-11-16T18:27:31.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f5623d17b80 1 -- 172.21.15.2:0/2160429034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979370 msgr2=0x5555aa9dfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:31.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f5623d17b80 1 --2- 172.21.15.2:0/2160429034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979370 0x5555aa9dfd10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f560c009ed0 tx=0x7f560c009350 comp rx=0 tx=0).stop 2023-11-16T18:27:31.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f5623d17b80 1 -- 172.21.15.2:0/2160429034 shutdown_connections 2023-11-16T18:27:31.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f5623d17b80 1 --2- 172.21.15.2:0/2160429034 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5555aa993280 0x5555aa9e97f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f5623d17b80 1 --2- 172.21.15.2:0/2160429034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5555aa97e0c0 0x5555aa989610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f55f97fa700 0 client.25907 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:31.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f5623d17b80 1 --2- 172.21.15.2:0/2160429034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5555aa979370 0x5555aa9dfd10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f5623d17b80 1 --2- 172.21.15.2:0/2160429034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5555aa979c80 0x5555aa9e0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f5623d17b80 1 -- 172.21.15.2:0/2160429034 >> 172.21.15.2:0/2160429034 conn(0x5555aa966360 msgr2=0x5555aa983540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:31.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f5623d17b80 1 -- 172.21.15.2:0/2160429034 shutdown_connections 2023-11-16T18:27:31.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.087+0000 7f5623d17b80 1 -- 172.21.15.2:0/2160429034 wait complete. 2023-11-16T18:27:31.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.088+0000 7f5623d17b80 20 client.25907 trim_cache size 0 max 16384 2023-11-16T18:27:31.095 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:27:31.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.088+0000 7f5617b8b700 1 -- 172.21.15.2:0/1779422127 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f55fc0a12c0 msgr2=0x7f55fc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:31.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.088+0000 7f5617b8b700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f55fc0a12c0 0x7f55fc0a3770 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f560c009c60 tx=0x7f560c009260 comp rx=0 tx=0).stop 2023-11-16T18:27:31.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.088+0000 7f5617b8b700 1 -- 172.21.15.2:0/1779422127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5610101000 msgr2=0x7f5610198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:31.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.088+0000 7f5617b8b700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5610101000 0x7f5610198be0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f560000e940 tx=0x7f560000ec50 comp rx=0 tx=0).stop 2023-11-16T18:27:31.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.088+0000 7f5617b8b700 1 -- 172.21.15.2:0/1779422127 shutdown_connections 2023-11-16T18:27:31.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.088+0000 7f5617b8b700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f55fc0a12c0 0x7f55fc0a3770 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.088+0000 7f5617b8b700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5610105590 0x7f561019bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.088+0000 7f5617b8b700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5610100660 0x7f56101986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.088+0000 7f5617b8b700 1 --2- 172.21.15.2:0/1779422127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5610101000 0x7f5610198be0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:31.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.088+0000 7f5617b8b700 1 -- 172.21.15.2:0/1779422127 >> 172.21.15.2:0/1779422127 conn(0x7f5610077380 msgr2=0x7f56100fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:31.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.088+0000 7f5617b8b700 1 -- 172.21.15.2:0/1779422127 shutdown_connections 2023-11-16T18:27:31.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:31.088+0000 7f5617b8b700 1 -- 172.21.15.2:0/1779422127 wait complete. 2023-11-16T18:27:31.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:31 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2684256456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:27:31.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:31 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2684256456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:27:31.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:31 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2684256456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:27:32.105 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T18:27:32.105 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T18:27:32.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.260+0000 7f393a3d1700 1 -- 172.21.15.2:0/1260876805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39340ffdb0 msgr2=0x7f3934103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.260+0000 7f393a3d1700 1 --2- 172.21.15.2:0/1260876805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39340ffdb0 0x7f3934103d70 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f39280099c0 tx=0x7f3928009cd0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.260+0000 7f393a3d1700 1 -- 172.21.15.2:0/1260876805 shutdown_connections 2023-11-16T18:27:32.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.260+0000 7f393a3d1700 1 --2- 172.21.15.2:0/1260876805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934104450 0x7f393410c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.260+0000 7f393a3d1700 1 --2- 172.21.15.2:0/1260876805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39340ff410 0x7f39340ff7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.260+0000 7f393a3d1700 1 --2- 172.21.15.2:0/1260876805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39340ffdb0 0x7f3934103d70 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.260+0000 7f393a3d1700 1 -- 172.21.15.2:0/1260876805 >> 172.21.15.2:0/1260876805 conn(0x7f3934076370 msgr2=0x7f3934076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:32.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.260+0000 7f393a3d1700 1 -- 172.21.15.2:0/1260876805 shutdown_connections 2023-11-16T18:27:32.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.260+0000 7f393a3d1700 1 -- 172.21.15.2:0/1260876805 wait complete. 2023-11-16T18:27:32.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.261+0000 7f393a3d1700 1 Processor -- start 2023-11-16T18:27:32.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.261+0000 7f393a3d1700 1 -- start start 2023-11-16T18:27:32.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f393a3d1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39340ff410 0x7f39341963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f393a3d1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39340ffdb0 0x7f3934196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f393a3d1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934104450 0x7f393419ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f393a3d1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393419b3d0 con 0x7f39340ffdb0 2023-11-16T18:27:32.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f393a3d1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393419b540 con 0x7f39340ff410 2023-11-16T18:27:32.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f393a3d1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393419b700 con 0x7f3934104450 2023-11-16T18:27:32.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f39337fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39340ffdb0 0x7f3934196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f3933fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39340ff410 0x7f39341963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f39337fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39340ffdb0 0x7f3934196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52002/0 (socket says 172.21.15.2:52002) 2023-11-16T18:27:32.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f39337fe700 1 -- 172.21.15.2:0/4105547582 learned_addr learned my addr 172.21.15.2:0/4105547582 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:32.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f393896e700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934104450 0x7f393419ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f39337fe700 1 -- 172.21.15.2:0/4105547582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934104450 msgr2=0x7f393419ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f39337fe700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934104450 0x7f393419ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f39337fe700 1 -- 172.21.15.2:0/4105547582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39340ff410 msgr2=0x7f39341963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f39337fe700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39340ff410 0x7f39341963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f39337fe700 1 -- 172.21.15.2:0/4105547582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39280095f0 con 0x7f39340ffdb0 2023-11-16T18:27:32.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.262+0000 7f3933fff700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39340ff410 0x7f39341963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:32.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.263+0000 7f39337fe700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39340ffdb0 0x7f3934196930 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3928009990 tx=0x7f392800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:32.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.263+0000 7f39317fa700 1 -- 172.21.15.2:0/4105547582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392801d070 con 0x7f39340ffdb0 2023-11-16T18:27:32.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.263+0000 7f39317fa700 1 -- 172.21.15.2:0/4105547582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3928005020 con 0x7f39340ffdb0 2023-11-16T18:27:32.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.263+0000 7f393a3d1700 1 -- 172.21.15.2:0/4105547582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39341103e0 con 0x7f39340ffdb0 2023-11-16T18:27:32.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.263+0000 7f39317fa700 1 -- 172.21.15.2:0/4105547582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928004a70 con 0x7f39340ffdb0 2023-11-16T18:27:32.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.264+0000 7f393a3d1700 1 -- 172.21.15.2:0/4105547582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3934110250 con 0x7f39340ffdb0 2023-11-16T18:27:32.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.267+0000 7f394655db80 1 Processor -- start 2023-11-16T18:27:32.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.267+0000 7f394655db80 1 -- start start 2023-11-16T18:27:32.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.267+0000 7f394655db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fa7be0 0x555ec6fa7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f394655db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec6fac3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f394655db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fac930 0x555ec6facd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f394655db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ec6ead0e0 con 0x555ec6fa84f0 2023-11-16T18:27:32.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f394655db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ec6eff920 con 0x555ec6fa7be0 2023-11-16T18:27:32.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f394655db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ec6ef8db0 con 0x555ec6fac930 2023-11-16T18:27:32.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f3933fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec6fac3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f393896e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fa7be0 0x555ec6fa7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f393896e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fa7be0 0x555ec6fa7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42444/0 (socket says 172.21.15.2:42444) 2023-11-16T18:27:32.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f393896e700 1 -- 172.21.15.2:0/2006553679 learned_addr learned my addr 172.21.15.2:0/2006553679 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:32.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f3933fff700 1 --2- 172.21.15.2:0/2006553679 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec6fac3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52008/0 (socket says 172.21.15.2:52008) 2023-11-16T18:27:32.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f393896e700 1 -- 172.21.15.2:0/2006553679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fac930 msgr2=0x555ec6facd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T18:27:32.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f393896e700 1 --2- 172.21.15.2:0/2006553679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fac930 0x555ec6facd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f393896e700 1 -- 172.21.15.2:0/2006553679 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 msgr2=0x555ec6fac3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f393896e700 1 --2- 172.21.15.2:0/2006553679 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec6fac3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f393896e700 1 -- 172.21.15.2:0/2006553679 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ec6d7a6b0 con 0x555ec6fa7be0 2023-11-16T18:27:32.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.268+0000 7f393896e700 1 --2- 172.21.15.2:0/2006553679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fa7be0 0x555ec6fa7fb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f392c00a430 tx=0x7f392c009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7010b35e709ef003 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:32.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.269+0000 7f391a7fc700 1 -- 172.21.15.2:0/2006553679 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c010040 con 0x555ec6fa7be0 2023-11-16T18:27:32.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.269+0000 7f391a7fc700 1 -- 172.21.15.2:0/2006553679 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f392c014ae0 con 0x555ec6fa7be0 2023-11-16T18:27:32.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.269+0000 7f391a7fc700 1 -- 172.21.15.2:0/2006553679 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c013a70 con 0x555ec6fa7be0 2023-11-16T18:27:32.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.269+0000 7f39317fa700 1 -- 172.21.15.2:0/4105547582 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f3928005400 con 0x7f39340ffdb0 2023-11-16T18:27:32.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.270+0000 7f394655db80 1 -- 172.21.15.2:0/2006553679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fa7be0 msgr2=0x555ec6fa7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.270+0000 7f394655db80 1 --2- 172.21.15.2:0/2006553679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fa7be0 0x555ec6fa7fb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f392c00a430 tx=0x7f392c009650 comp rx=0 tx=0).stop 2023-11-16T18:27:32.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.270+0000 7f39317fa700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f391c0a1410 0x7f391c0a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.270+0000 7f394655db80 1 -- 172.21.15.2:0/2006553679 shutdown_connections 2023-11-16T18:27:32.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.270+0000 7f394655db80 1 --2- 172.21.15.2:0/2006553679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fac930 0x555ec6facd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.270+0000 7f394655db80 1 --2- 172.21.15.2:0/2006553679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fa7be0 0x555ec6fa7fb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.270+0000 7f394655db80 1 --2- 172.21.15.2:0/2006553679 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec6fac3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.270+0000 7f394655db80 1 -- 172.21.15.2:0/2006553679 >> 172.21.15.2:0/2006553679 conn(0x555ec6f9d3e0 msgr2=0x555ec6fa2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:32.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.270+0000 7f393896e700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f391c0a1410 0x7f391c0a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.270+0000 7f39317fa700 1 -- 172.21.15.2:0/4105547582 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f39280cd9e0 con 0x7f39340ffdb0 2023-11-16T18:27:32.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.270+0000 7f394655db80 1 -- 172.21.15.2:0/2006553679 shutdown_connections 2023-11-16T18:27:32.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.270+0000 7f394655db80 1 -- 172.21.15.2:0/2006553679 wait complete. 2023-11-16T18:27:32.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.271+0000 7f393896e700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f391c0a1410 0x7f391c0a38c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f392c007070 tx=0x7f392c00c040 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:32.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.271+0000 7f394655db80 1 -- start start 2023-11-16T18:27:32.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.271+0000 7f394655db80 1 Processor -- start 2023-11-16T18:27:32.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.271+0000 7f394655db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa7be0 0x555ec6fb4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.271+0000 7f394655db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fa84f0 0x555ec6fb1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f394655db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 0x555ec6fb1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f394655db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ec6e9a960 con 0x555ec6fa7be0 2023-11-16T18:27:32.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f394655db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ec6e9a600 con 0x555ec6fac930 2023-11-16T18:27:32.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f394655db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ec6d7c9a0 con 0x555ec6fa84f0 2023-11-16T18:27:32.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f394655db80 10 client.0 _ll_register_callbacks cb 0x555ec6eb8800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:32.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f3933fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa7be0 0x555ec6fb4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f393896e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 0x555ec6fb1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f3933fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa7be0 0x555ec6fb4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52012/0 (socket says 172.21.15.2:52012) 2023-11-16T18:27:32.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f393896e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 0x555ec6fb1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42454/0 (socket says 172.21.15.2:42454) 2023-11-16T18:27:32.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f3933fff700 1 -- 172.21.15.2:0/3936739164 learned_addr learned my addr 172.21.15.2:0/3936739164 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:32.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f39337fe700 1 --2- 172.21.15.2:0/3936739164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fa84f0 0x555ec6fb1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f3933fff700 1 -- 172.21.15.2:0/3936739164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fa84f0 msgr2=0x555ec6fb1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f3933fff700 1 --2- 172.21.15.2:0/3936739164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fa84f0 0x555ec6fb1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f3933fff700 1 -- 172.21.15.2:0/3936739164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 msgr2=0x555ec6fb1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f3933fff700 1 --2- 172.21.15.2:0/3936739164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 0x555ec6fb1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f3933fff700 1 -- 172.21.15.2:0/3936739164 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392c009c60 con 0x555ec6fa7be0 2023-11-16T18:27:32.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f393896e700 1 --2- 172.21.15.2:0/3936739164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 0x555ec6fb1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:27:32.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f3933fff700 1 --2- 172.21.15.2:0/3936739164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa7be0 0x555ec6fb4050 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f3924011d10 tx=0x7f392400f6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=534ce1756b99a882 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:32.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.272+0000 7f39137fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:27:32.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f39137fe700 1 -- 172.21.15.2:0/3936739164 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924016070 con 0x555ec6fa7be0 2023-11-16T18:27:32.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f39137fe700 1 -- 172.21.15.2:0/3936739164 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f392400fe40 con 0x555ec6fa7be0 2023-11-16T18:27:32.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f394655db80 1 -- 172.21.15.2:0/3936739164 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555ec6f9aba0 con 0x555ec6fa7be0 2023-11-16T18:27:32.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f39137fe700 1 -- 172.21.15.2:0/3936739164 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3924004050 con 0x555ec6fa7be0 2023-11-16T18:27:32.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f394655db80 10 client.16371 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:32.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f394655db80 1 -- 172.21.15.2:0/3936739164 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555ec6ebf210 con 0x555ec6fa7be0 2023-11-16T18:27:32.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f39137fe700 1 -- 172.21.15.2:0/3936739164 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555ec6f9aba0 con 0x555ec6fa7be0 2023-11-16T18:27:32.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f39137fe700 1 -- 172.21.15.2:0/3936739164 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f3924019370 con 0x555ec6fa7be0 2023-11-16T18:27:32.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f394655db80 10 client.16371 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:32.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f394655db80 10 client.16371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:32.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f394655db80 20 client.16371 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:32.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f394655db80 1 --2- 172.21.15.2:0/3936739164 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555ec6fc19e0 0x555ec6fc3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f394655db80 4 client.16371 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:27:32.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f394655db80 1 -- 172.21.15.2:0/3936739164 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ec6ebb120 con 0x555ec6fc19e0 2023-11-16T18:27:32.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.273+0000 7f39337fe700 1 --2- 172.21.15.2:0/3936739164 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555ec6fc19e0 0x555ec6fc3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.274+0000 7f39137fe700 10 client.16371 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:32.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.274+0000 7f39337fe700 1 --2- 172.21.15.2:0/3936739164 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555ec6fc19e0 0x555ec6fc3df0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a9f4722f3733f24 server_cookie=a6885bb9ea5d69c7 in_seq=0 out_seq=0 2023-11-16T18:27:32.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f39137fe700 1 -- 172.21.15.2:0/3936739164 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555ec6ebb120 con 0x555ec6fc19e0 2023-11-16T18:27:32.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f39137fe700 10 client.16371 handle_command_reply: tid=0 2023-11-16T18:27:32.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f394655db80 1 client.16371 shutdown 2023-11-16T18:27:32.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f394655db80 1 -- 172.21.15.2:0/3936739164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa7be0 msgr2=0x555ec6fb4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f394655db80 1 --2- 172.21.15.2:0/3936739164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa7be0 0x555ec6fb4050 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f3924011d10 tx=0x7f392400f6a0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f394655db80 1 -- 172.21.15.2:0/3936739164 shutdown_connections 2023-11-16T18:27:32.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f394655db80 1 --2- 172.21.15.2:0/3936739164 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555ec6fc19e0 0x555ec6fc3df0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f394655db80 1 --2- 172.21.15.2:0/3936739164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 0x555ec6fb1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f39137fe700 0 client.16371 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:32.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f394655db80 1 --2- 172.21.15.2:0/3936739164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa7be0 0x555ec6fb4050 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f394655db80 1 --2- 172.21.15.2:0/3936739164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fa84f0 0x555ec6fb1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f394655db80 1 -- 172.21.15.2:0/3936739164 >> 172.21.15.2:0/3936739164 conn(0x555ec6f9d3e0 msgr2=0x555ec6fa69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:32.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f394655db80 1 -- 172.21.15.2:0/3936739164 shutdown_connections 2023-11-16T18:27:32.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f394655db80 1 -- 172.21.15.2:0/3936739164 wait complete. 2023-11-16T18:27:32.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.276+0000 7f394655db80 20 client.16371 trim_cache size 0 max 16384 2023-11-16T18:27:32.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f394655db80 1 Processor -- start 2023-11-16T18:27:32.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f394655db80 1 -- start start 2023-11-16T18:27:32.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f394655db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fa7be0 0x555ec6fb8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f394655db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec6fb3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f394655db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 0x555ec6fae750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f394655db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ec6fb2690 con 0x555ec6fa84f0 2023-11-16T18:27:32.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f394655db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ec6f9cf60 con 0x555ec6fac930 2023-11-16T18:27:32.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f394655db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ec6fa89f0 con 0x555ec6fa7be0 2023-11-16T18:27:32.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f39337fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec6fb3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f3933fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fa7be0 0x555ec6fb8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f393896e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 0x555ec6fae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f3933fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fa7be0 0x555ec6fb8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43550/0 (socket says 172.21.15.2:43550) 2023-11-16T18:27:32.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f3933fff700 1 -- 172.21.15.2:0/909170993 learned_addr learned my addr 172.21.15.2:0/909170993 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:32.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.295+0000 7f393896e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 0x555ec6fae750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42462/0 (socket says 172.21.15.2:42462) 2023-11-16T18:27:32.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.296+0000 7f39337fe700 1 -- 172.21.15.2:0/909170993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fa7be0 msgr2=0x555ec6fb8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.296+0000 7f39337fe700 1 --2- 172.21.15.2:0/909170993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fa7be0 0x555ec6fb8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.296+0000 7f39337fe700 1 -- 172.21.15.2:0/909170993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 msgr2=0x555ec6fae750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.296+0000 7f39337fe700 1 --2- 172.21.15.2:0/909170993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 0x555ec6fae750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.296+0000 7f39337fe700 1 -- 172.21.15.2:0/909170993 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ec6ebef60 con 0x555ec6fa84f0 2023-11-16T18:27:32.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.296+0000 7f393896e700 1 --2- 172.21.15.2:0/909170993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 0x555ec6fae750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:32.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.296+0000 7f3933fff700 1 --2- 172.21.15.2:0/909170993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fa7be0 0x555ec6fb8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:32.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.296+0000 7f39337fe700 1 --2- 172.21.15.2:0/909170993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec6fb3bf0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f39280cad90 tx=0x7f3928005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9452d705c3fc9d6c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:32.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.296+0000 7f391affd700 1 -- 172.21.15.2:0/909170993 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392809e800 con 0x555ec6fa84f0 2023-11-16T18:27:32.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.296+0000 7f391affd700 1 -- 172.21.15.2:0/909170993 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39280a2b90 con 0x555ec6fa84f0 2023-11-16T18:27:32.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.296+0000 7f391affd700 1 -- 172.21.15.2:0/909170993 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39280b88d0 con 0x555ec6fa84f0 2023-11-16T18:27:32.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.297+0000 7f394655db80 1 -- 172.21.15.2:0/909170993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 msgr2=0x555ec6fb3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.297+0000 7f394655db80 1 --2- 172.21.15.2:0/909170993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec6fb3bf0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f39280cad90 tx=0x7f3928005c90 comp rx=0 tx=0).stop 2023-11-16T18:27:32.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.297+0000 7f394655db80 1 -- 172.21.15.2:0/909170993 shutdown_connections 2023-11-16T18:27:32.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.297+0000 7f394655db80 1 --2- 172.21.15.2:0/909170993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fa7be0 0x555ec6fb8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.297+0000 7f394655db80 1 --2- 172.21.15.2:0/909170993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fac930 0x555ec6fae750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.297+0000 7f394655db80 1 --2- 172.21.15.2:0/909170993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec6fb3bf0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.297+0000 7f394655db80 1 -- 172.21.15.2:0/909170993 >> 172.21.15.2:0/909170993 conn(0x555ec6f9d3e0 msgr2=0x555ec6fa26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:32.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.297+0000 7f394655db80 1 -- 172.21.15.2:0/909170993 shutdown_connections 2023-11-16T18:27:32.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.297+0000 7f394655db80 1 -- 172.21.15.2:0/909170993 wait complete. 2023-11-16T18:27:32.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.297+0000 7f394655db80 1 -- start start 2023-11-16T18:27:32.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.298+0000 7f394655db80 1 Processor -- start 2023-11-16T18:27:32.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.298+0000 7f394655db80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fa7be0 0x555ec70137b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.298+0000 7f394655db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec7013cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.298+0000 7f394655db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fac930 0x555ec700dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.298+0000 7f394655db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ec700e200 con 0x555ec6fa84f0 2023-11-16T18:27:32.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.298+0000 7f394655db80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ec700e370 con 0x555ec6fa7be0 2023-11-16T18:27:32.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.298+0000 7f394655db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ec700e530 con 0x555ec6fac930 2023-11-16T18:27:32.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.298+0000 7f394655db80 10 client.0 _ll_register_callbacks cb 0x7f39240117e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:32.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.298+0000 7f39337fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec7013cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.298+0000 7f39337fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec7013cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52028/0 (socket says 172.21.15.2:52028) 2023-11-16T18:27:32.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.298+0000 7f39337fe700 1 -- 172.21.15.2:0/2383096838 learned_addr learned my addr 172.21.15.2:0/2383096838 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:32.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.298+0000 7f3933fff700 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fa7be0 0x555ec70137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.298+0000 7f393896e700 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fac930 0x555ec700dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f39337fe700 1 -- 172.21.15.2:0/2383096838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fac930 msgr2=0x555ec700dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f39337fe700 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fac930 0x555ec700dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f39337fe700 1 -- 172.21.15.2:0/2383096838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fa7be0 msgr2=0x555ec70137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f39337fe700 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fa7be0 0x555ec70137b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f39337fe700 1 -- 172.21.15.2:0/2383096838 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ec6ebf210 con 0x555ec6fa84f0 2023-11-16T18:27:32.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f3933fff700 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fa7be0 0x555ec70137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:32.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f39337fe700 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec7013cf0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f392800b580 tx=0x7f392809e800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208926dc16887444 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:32.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f39137fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:27:32.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f39137fe700 1 -- 172.21.15.2:0/2383096838 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39280989d0 con 0x555ec6fa84f0 2023-11-16T18:27:32.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f39137fe700 1 -- 172.21.15.2:0/2383096838 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f3928096ee0 con 0x555ec6fa84f0 2023-11-16T18:27:32.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f394655db80 1 -- 172.21.15.2:0/2383096838 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f392c014ae0 con 0x555ec6fa84f0 2023-11-16T18:27:32.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f39137fe700 1 -- 172.21.15.2:0/2383096838 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39280970a0 con 0x555ec6fa84f0 2023-11-16T18:27:32.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f394655db80 10 client.16377 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:32.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f394655db80 1 -- 172.21.15.2:0/2383096838 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555ec6d7a6b0 con 0x555ec6fa84f0 2023-11-16T18:27:32.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.299+0000 7f39137fe700 1 -- 172.21.15.2:0/2383096838 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f392c014ae0 con 0x555ec6fa84f0 2023-11-16T18:27:32.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.300+0000 7f39137fe700 1 -- 172.21.15.2:0/2383096838 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f392809aa30 con 0x555ec6fa84f0 2023-11-16T18:27:32.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.300+0000 7f394655db80 10 client.16377 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:32.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.300+0000 7f394655db80 10 client.16377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:32.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.300+0000 7f394655db80 20 client.16377 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:32.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.300+0000 7f394655db80 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555ec701b2b0 0x555ec701d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.300+0000 7f394655db80 4 client.16377 mds_command: new command op to 24473 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T18:27:32.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.300+0000 7f394655db80 1 -- 172.21.15.2:0/2383096838 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x555ec6ec76e0 con 0x555ec701b2b0 2023-11-16T18:27:32.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.300+0000 7f3933fff700 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555ec701b2b0 0x555ec701d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.301+0000 7f3933fff700 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555ec701b2b0 0x555ec701d6c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc885cf266fe57e3 server_cookie=59e0a8af7143f405 in_seq=0 out_seq=0 2023-11-16T18:27:32.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.301+0000 7f39137fe700 10 client.16377 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:32.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f39137fe700 1 -- 172.21.15.2:0/2383096838 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555ec6ec76e0 con 0x555ec701b2b0 2023-11-16T18:27:32.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f39137fe700 10 client.16377 handle_command_reply: tid=0 2023-11-16T18:27:32.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f394655db80 1 client.16377 shutdown 2023-11-16T18:27:32.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f394655db80 1 -- 172.21.15.2:0/2383096838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 msgr2=0x555ec7013cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f394655db80 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec7013cf0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f392800b580 tx=0x7f392809e800 comp rx=0 tx=0).stop 2023-11-16T18:27:32.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f394655db80 1 -- 172.21.15.2:0/2383096838 shutdown_connections 2023-11-16T18:27:32.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f394655db80 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555ec701b2b0 0x555ec701d6c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f394655db80 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ec6fac930 0x555ec700dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f394655db80 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec6fa7be0 0x555ec70137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f39137fe700 0 client.16377 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:32.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f394655db80 1 --2- 172.21.15.2:0/2383096838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555ec6fa84f0 0x555ec7013cf0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f394655db80 1 -- 172.21.15.2:0/2383096838 >> 172.21.15.2:0/2383096838 conn(0x555ec6f9d3e0 msgr2=0x555ec6fa15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:32.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f394655db80 1 -- 172.21.15.2:0/2383096838 shutdown_connections 2023-11-16T18:27:32.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f394655db80 1 -- 172.21.15.2:0/2383096838 wait complete. 2023-11-16T18:27:32.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.302+0000 7f394655db80 20 client.16377 trim_cache size 0 max 16384 2023-11-16T18:27:32.309 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:27:32.309 INFO:teuthology.orchestra.run.smithi002.stdout: "return_code": 0, 2023-11-16T18:27:32.310 INFO:teuthology.orchestra.run.smithi002.stdout: "scrub_tag": "9607aa23-76ae-4198-9378-3740d8aff22c", 2023-11-16T18:27:32.310 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": "asynchronous" 2023-11-16T18:27:32.310 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:27:32.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.303+0000 7f393a3d1700 1 -- 172.21.15.2:0/4105547582 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f391c0a1410 msgr2=0x7f391c0a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.303+0000 7f393a3d1700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f391c0a1410 0x7f391c0a38c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f392c007070 tx=0x7f392c00c040 comp rx=0 tx=0).stop 2023-11-16T18:27:32.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.303+0000 7f393a3d1700 1 -- 172.21.15.2:0/4105547582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39340ffdb0 msgr2=0x7f3934196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.303+0000 7f393a3d1700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39340ffdb0 0x7f3934196930 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3928009990 tx=0x7f392800f690 comp rx=0 tx=0).stop 2023-11-16T18:27:32.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.303+0000 7f393a3d1700 1 -- 172.21.15.2:0/4105547582 shutdown_connections 2023-11-16T18:27:32.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.303+0000 7f393a3d1700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f391c0a1410 0x7f391c0a38c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.303+0000 7f393a3d1700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934104450 0x7f393419ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.303+0000 7f393a3d1700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39340ff410 0x7f39341963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.303+0000 7f393a3d1700 1 --2- 172.21.15.2:0/4105547582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39340ffdb0 0x7f3934196930 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.303+0000 7f393a3d1700 1 -- 172.21.15.2:0/4105547582 >> 172.21.15.2:0/4105547582 conn(0x7f3934076370 msgr2=0x7f39340fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:32.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.303+0000 7f393a3d1700 1 -- 172.21.15.2:0/4105547582 shutdown_connections 2023-11-16T18:27:32.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.303+0000 7f393a3d1700 1 -- 172.21.15.2:0/4105547582 wait complete. 2023-11-16T18:27:32.320 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:27:32.346 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:32 smithi002 ceph-mon[98807]: pgmap v83: 129 pgs: 129 active+clean; 64 MiB data, 269 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 616 op/s 2023-11-16T18:27:32.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.459+0000 7faefa2fb700 1 -- 172.21.15.2:0/1371920090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faef4100420 msgr2=0x7faef410bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.459+0000 7faefa2fb700 1 --2- 172.21.15.2:0/1371920090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faef4100420 0x7faef410bfd0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7faeec00b0a0 tx=0x7faeec00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.459+0000 7faefa2fb700 1 -- 172.21.15.2:0/1371920090 shutdown_connections 2023-11-16T18:27:32.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.459+0000 7faefa2fb700 1 --2- 172.21.15.2:0/1371920090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faef4100420 0x7faef410bfd0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.459+0000 7faefa2fb700 1 --2- 172.21.15.2:0/1371920090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faef4105250 0x7faef4105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.459+0000 7faefa2fb700 1 --2- 172.21.15.2:0/1371920090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faef40ffa70 0x7faef40ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.459+0000 7faefa2fb700 1 -- 172.21.15.2:0/1371920090 >> 172.21.15.2:0/1371920090 conn(0x7faef40763a0 msgr2=0x7faef40767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:32.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.459+0000 7faefa2fb700 1 -- 172.21.15.2:0/1371920090 shutdown_connections 2023-11-16T18:27:32.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.459+0000 7faefa2fb700 1 -- 172.21.15.2:0/1371920090 wait complete. 2023-11-16T18:27:32.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.459+0000 7faefa2fb700 1 Processor -- start 2023-11-16T18:27:32.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.460+0000 7faefa2fb700 1 -- start start 2023-11-16T18:27:32.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.460+0000 7faefa2fb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faef40ffa70 0x7faef4197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.460+0000 7faefa2fb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faef4100420 0x7faef4197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.460+0000 7faefa2fb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faef4105250 0x7faef41a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.460+0000 7faefa2fb700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef41a28e0 con 0x7faef4105250 2023-11-16T18:27:32.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.460+0000 7faefa2fb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef41a2a20 con 0x7faef4100420 2023-11-16T18:27:32.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.460+0000 7faefa2fb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef41a2b60 con 0x7faef40ffa70 2023-11-16T18:27:32.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.460+0000 7faef37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faef4100420 0x7faef4197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.460+0000 7faef8898700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faef4105250 0x7faef41a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.460+0000 7faef37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faef4100420 0x7faef4197860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42480/0 (socket says 172.21.15.2:42480) 2023-11-16T18:27:32.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.460+0000 7faef8898700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faef4105250 0x7faef41a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52044/0 (socket says 172.21.15.2:52044) 2023-11-16T18:27:32.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.460+0000 7faef37fe700 1 -- 172.21.15.2:0/2650037972 learned_addr learned my addr 172.21.15.2:0/2650037972 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:32.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.461+0000 7faef8898700 1 -- 172.21.15.2:0/2650037972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faef40ffa70 msgr2=0x7faef4197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.461+0000 7faef8898700 1 --2- 172.21.15.2:0/2650037972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faef40ffa70 0x7faef4197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.461+0000 7faef8898700 1 -- 172.21.15.2:0/2650037972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faef4100420 msgr2=0x7faef4197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.461+0000 7faef8898700 1 --2- 172.21.15.2:0/2650037972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faef4100420 0x7faef4197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.461+0000 7faef8898700 1 -- 172.21.15.2:0/2650037972 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee8009520 con 0x7faef4105250 2023-11-16T18:27:32.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.461+0000 7faef37fe700 1 --2- 172.21.15.2:0/2650037972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faef4100420 0x7faef4197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:32.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.461+0000 7faef8898700 1 --2- 172.21.15.2:0/2650037972 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faef4105250 0x7faef41a23a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7faeec015000 tx=0x7faeec00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:32.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.461+0000 7faef17fa700 1 -- 172.21.15.2:0/2650037972 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec00e040 con 0x7faef4105250 2023-11-16T18:27:32.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.461+0000 7faef17fa700 1 -- 172.21.15.2:0/2650037972 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faeec007950 con 0x7faef4105250 2023-11-16T18:27:32.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.461+0000 7faefa2fb700 1 -- 172.21.15.2:0/2650037972 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeec009d00 con 0x7faef4105250 2023-11-16T18:27:32.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.461+0000 7faef17fa700 1 -- 172.21.15.2:0/2650037972 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec005040 con 0x7faef4105250 2023-11-16T18:27:32.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.461+0000 7faefa2fb700 1 -- 172.21.15.2:0/2650037972 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef410f1e0 con 0x7faef4105250 2023-11-16T18:27:32.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.464+0000 7faef17fa700 1 -- 172.21.15.2:0/2650037972 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7faeec01d470 con 0x7faef4105250 2023-11-16T18:27:32.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.465+0000 7faef17fa700 1 --2- 172.21.15.2:0/2650037972 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7faedc0a12c0 0x7faedc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.465+0000 7faef3fff700 1 --2- 172.21.15.2:0/2650037972 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7faedc0a12c0 0x7faedc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.465+0000 7faef17fa700 1 -- 172.21.15.2:0/2650037972 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7faeec0ce390 con 0x7faef4105250 2023-11-16T18:27:32.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.465+0000 7faef3fff700 1 --2- 172.21.15.2:0/2650037972 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7faedc0a12c0 0x7faedc0a3770 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faee4009b60 tx=0x7faee4009620 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:32.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.465+0000 7faf06487b80 1 Processor -- start 2023-11-16T18:27:32.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faf06487b80 1 -- start start 2023-11-16T18:27:32.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faf06487b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5efb60 0x55834e5eff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faf06487b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f0470 0x55834e5f4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faf06487b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5f48b0 0x55834e5f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faf06487b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55834e4f5060 con 0x55834e5efb60 2023-11-16T18:27:32.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faf06487b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55834e5478a0 con 0x55834e5f48b0 2023-11-16T18:27:32.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faf06487b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55834e540d30 con 0x55834e5f0470 2023-11-16T18:27:32.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faef3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5efb60 0x55834e5eff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faef3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5efb60 0x55834e5eff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52048/0 (socket says 172.21.15.2:52048) 2023-11-16T18:27:32.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faef3fff700 1 -- 172.21.15.2:0/3559402981 learned_addr learned my addr 172.21.15.2:0/3559402981 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:32.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faef8898700 1 --2- 172.21.15.2:0/3559402981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5f48b0 0x55834e5f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faef37fe700 1 --2- 172.21.15.2:0/3559402981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f0470 0x55834e5f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faef3fff700 1 -- 172.21.15.2:0/3559402981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f0470 msgr2=0x55834e5f4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faef3fff700 1 --2- 172.21.15.2:0/3559402981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f0470 0x55834e5f4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faef3fff700 1 -- 172.21.15.2:0/3559402981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5f48b0 msgr2=0x55834e5f4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faef3fff700 1 --2- 172.21.15.2:0/3559402981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5f48b0 0x55834e5f4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faef3fff700 1 -- 172.21.15.2:0/3559402981 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55834e3c2630 con 0x55834e5efb60 2023-11-16T18:27:32.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faef8898700 1 --2- 172.21.15.2:0/3559402981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5f48b0 0x55834e5f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:32.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.466+0000 7faef3fff700 1 --2- 172.21.15.2:0/3559402981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5efb60 0x55834e5eff30 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7faee400eb80 tx=0x7faee4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80fa163c41ad712f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:32.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.467+0000 7faeda7fc700 1 -- 172.21.15.2:0/3559402981 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee4010ac0 con 0x55834e5efb60 2023-11-16T18:27:32.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.467+0000 7faeda7fc700 1 -- 172.21.15.2:0/3559402981 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faee400ee90 con 0x55834e5efb60 2023-11-16T18:27:32.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.467+0000 7faeda7fc700 1 -- 172.21.15.2:0/3559402981 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee401c680 con 0x55834e5efb60 2023-11-16T18:27:32.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.468+0000 7faf06487b80 1 -- 172.21.15.2:0/3559402981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5efb60 msgr2=0x55834e5eff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.468+0000 7faf06487b80 1 --2- 172.21.15.2:0/3559402981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5efb60 0x55834e5eff30 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7faee400eb80 tx=0x7faee4011000 comp rx=0 tx=0).stop 2023-11-16T18:27:32.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.468+0000 7faf06487b80 1 -- 172.21.15.2:0/3559402981 shutdown_connections 2023-11-16T18:27:32.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.468+0000 7faf06487b80 1 --2- 172.21.15.2:0/3559402981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5f48b0 0x55834e5f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.468+0000 7faf06487b80 1 --2- 172.21.15.2:0/3559402981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5efb60 0x55834e5eff30 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.468+0000 7faf06487b80 1 --2- 172.21.15.2:0/3559402981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f0470 0x55834e5f4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.468+0000 7faf06487b80 1 -- 172.21.15.2:0/3559402981 >> 172.21.15.2:0/3559402981 conn(0x55834e5e5360 msgr2=0x55834e5eab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:32.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.468+0000 7faf06487b80 1 -- 172.21.15.2:0/3559402981 shutdown_connections 2023-11-16T18:27:32.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.468+0000 7faf06487b80 1 -- 172.21.15.2:0/3559402981 wait complete. 2023-11-16T18:27:32.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.469+0000 7faf06487b80 1 -- start start 2023-11-16T18:27:32.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.469+0000 7faf06487b80 1 Processor -- start 2023-11-16T18:27:32.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.469+0000 7faf06487b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5efb60 0x55834e5fc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.469+0000 7faf06487b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 0x55834e5f9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.469+0000 7faf06487b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f48b0 0x55834e5f9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.469+0000 7faf06487b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55834e4e28e0 con 0x55834e5f0470 2023-11-16T18:27:32.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.469+0000 7faf06487b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55834e4e2580 con 0x55834e5efb60 2023-11-16T18:27:32.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.469+0000 7faf06487b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55834e3c4920 con 0x55834e5f48b0 2023-11-16T18:27:32.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.469+0000 7faf06487b80 10 client.0 _ll_register_callbacks cb 0x55834e500780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:32.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faef37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 0x55834e5f9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faef3fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5efb60 0x55834e5fc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faef37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 0x55834e5f9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52056/0 (socket says 172.21.15.2:52056) 2023-11-16T18:27:32.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faef37fe700 1 -- 172.21.15.2:0/4276132747 learned_addr learned my addr 172.21.15.2:0/4276132747 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:32.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faef3fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5efb60 0x55834e5fc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42492/0 (socket says 172.21.15.2:42492) 2023-11-16T18:27:32.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faef8898700 1 --2- 172.21.15.2:0/4276132747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f48b0 0x55834e5f9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faef3fff700 1 -- 172.21.15.2:0/4276132747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f48b0 msgr2=0x55834e5f9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faef3fff700 1 --2- 172.21.15.2:0/4276132747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f48b0 0x55834e5f9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faef3fff700 1 -- 172.21.15.2:0/4276132747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 msgr2=0x55834e5f9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faef3fff700 1 --2- 172.21.15.2:0/4276132747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 0x55834e5f9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faef3fff700 1 -- 172.21.15.2:0/4276132747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee400e830 con 0x55834e5efb60 2023-11-16T18:27:32.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faef37fe700 1 --2- 172.21.15.2:0/4276132747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 0x55834e5f9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:32.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faef3fff700 1 --2- 172.21.15.2:0/4276132747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5efb60 0x55834e5fc010 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faee4025cb0 tx=0x7faee4025690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5900679c6f246aa1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:32.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faed37fe700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:27:32.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faed37fe700 1 -- 172.21.15.2:0/4276132747 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee4034070 con 0x55834e5efb60 2023-11-16T18:27:32.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faf06487b80 1 -- 172.21.15.2:0/4276132747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55834e5e2b20 con 0x55834e5efb60 2023-11-16T18:27:32.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.470+0000 7faed37fe700 1 -- 172.21.15.2:0/4276132747 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7faee4010ad0 con 0x55834e5efb60 2023-11-16T18:27:32.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.471+0000 7faed37fe700 1 -- 172.21.15.2:0/4276132747 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7faee4029050 con 0x55834e5efb60 2023-11-16T18:27:32.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.471+0000 7faf06487b80 10 client.25937 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:32.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.471+0000 7faf06487b80 1 -- 172.21.15.2:0/4276132747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55834e507190 con 0x55834e5efb60 2023-11-16T18:27:32.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.471+0000 7faed37fe700 1 -- 172.21.15.2:0/4276132747 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55834e5e2b20 con 0x55834e5efb60 2023-11-16T18:27:32.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.471+0000 7faed37fe700 1 -- 172.21.15.2:0/4276132747 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7faee4037030 con 0x55834e5efb60 2023-11-16T18:27:32.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.471+0000 7faf06487b80 10 client.25937 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:32.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.471+0000 7faf06487b80 10 client.25937 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:32.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.471+0000 7faf06487b80 20 client.25937 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:32.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.471+0000 7faf06487b80 1 --2- 172.21.15.2:0/4276132747 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55834e6097e0 0x55834e60bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.471+0000 7faf06487b80 4 client.25937 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:27:32.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.471+0000 7faf06487b80 1 -- 172.21.15.2:0/4276132747 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55834e5030a0 con 0x55834e6097e0 2023-11-16T18:27:32.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.472+0000 7faef37fe700 1 --2- 172.21.15.2:0/4276132747 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55834e6097e0 0x55834e60bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.473+0000 7faed37fe700 10 client.25937 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:32.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.473+0000 7faef37fe700 1 --2- 172.21.15.2:0/4276132747 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55834e6097e0 0x55834e60bbf0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fac1c945a61fb38 server_cookie=cb9237842aaff975 in_seq=0 out_seq=0 2023-11-16T18:27:32.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faed37fe700 1 -- 172.21.15.2:0/4276132747 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55834e5030a0 con 0x55834e6097e0 2023-11-16T18:27:32.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faed37fe700 10 client.25937 handle_command_reply: tid=0 2023-11-16T18:27:32.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faf06487b80 1 client.25937 shutdown 2023-11-16T18:27:32.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faf06487b80 1 -- 172.21.15.2:0/4276132747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5efb60 msgr2=0x55834e5fc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faf06487b80 1 --2- 172.21.15.2:0/4276132747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5efb60 0x55834e5fc010 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faee4025cb0 tx=0x7faee4025690 comp rx=0 tx=0).stop 2023-11-16T18:27:32.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faf06487b80 1 -- 172.21.15.2:0/4276132747 shutdown_connections 2023-11-16T18:27:32.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faf06487b80 1 --2- 172.21.15.2:0/4276132747 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55834e6097e0 0x55834e60bbf0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faf06487b80 1 --2- 172.21.15.2:0/4276132747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f48b0 0x55834e5f9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faed37fe700 0 client.25937 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:32.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faf06487b80 1 --2- 172.21.15.2:0/4276132747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5efb60 0x55834e5fc010 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faf06487b80 1 --2- 172.21.15.2:0/4276132747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 0x55834e5f9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faf06487b80 1 -- 172.21.15.2:0/4276132747 >> 172.21.15.2:0/4276132747 conn(0x55834e5e5360 msgr2=0x55834e5f5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:32.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faf06487b80 1 -- 172.21.15.2:0/4276132747 shutdown_connections 2023-11-16T18:27:32.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faf06487b80 1 -- 172.21.15.2:0/4276132747 wait complete. 2023-11-16T18:27:32.484 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.481+0000 7faf06487b80 20 client.25937 trim_cache size 0 max 16384 2023-11-16T18:27:32.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:32 smithi040 ceph-mon[100251]: pgmap v83: 129 pgs: 129 active+clean; 64 MiB data, 269 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 616 op/s 2023-11-16T18:27:32.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.500+0000 7faf06487b80 1 Processor -- start 2023-11-16T18:27:32.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.500+0000 7faf06487b80 1 -- start start 2023-11-16T18:27:32.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.500+0000 7faf06487b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5efb60 0x55834e5eac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.500+0000 7faf06487b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 0x55834e600a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.500+0000 7faf06487b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f48b0 0x55834e5fbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.500+0000 7faf06487b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55834e5f77f0 con 0x55834e5f0470 2023-11-16T18:27:32.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.500+0000 7faf06487b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55834e5f3420 con 0x55834e5efb60 2023-11-16T18:27:32.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.500+0000 7faf06487b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55834e5f0900 con 0x55834e5f48b0 2023-11-16T18:27:32.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faef37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 0x55834e600a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faef37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 0x55834e600a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52062/0 (socket says 172.21.15.2:52062) 2023-11-16T18:27:32.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faef37fe700 1 -- 172.21.15.2:0/2149255172 learned_addr learned my addr 172.21.15.2:0/2149255172 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:32.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faef3fff700 1 --2- 172.21.15.2:0/2149255172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5efb60 0x55834e5eac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.507 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faef8898700 1 --2- 172.21.15.2:0/2149255172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f48b0 0x55834e5fbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.507 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faef37fe700 1 -- 172.21.15.2:0/2149255172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f48b0 msgr2=0x55834e5fbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.507 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faef37fe700 1 --2- 172.21.15.2:0/2149255172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f48b0 0x55834e5fbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.507 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faef37fe700 1 -- 172.21.15.2:0/2149255172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5efb60 msgr2=0x55834e5eac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.507 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faef37fe700 1 --2- 172.21.15.2:0/2149255172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5efb60 0x55834e5eac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faef37fe700 1 -- 172.21.15.2:0/2149255172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55834e506ee0 con 0x55834e5f0470 2023-11-16T18:27:32.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faef3fff700 1 --2- 172.21.15.2:0/2149255172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5efb60 0x55834e5eac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:32.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faef37fe700 1 --2- 172.21.15.2:0/2149255172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 0x55834e600a00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7faee8004790 tx=0x7faee8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56cc8d27435d638d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:32.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faedaffd700 1 -- 172.21.15.2:0/2149255172 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee8017040 con 0x55834e5f0470 2023-11-16T18:27:32.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faedaffd700 1 -- 172.21.15.2:0/2149255172 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faee8018040 con 0x55834e5f0470 2023-11-16T18:27:32.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.501+0000 7faedaffd700 1 -- 172.21.15.2:0/2149255172 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee8004830 con 0x55834e5f0470 2023-11-16T18:27:32.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.502+0000 7faf06487b80 1 -- 172.21.15.2:0/2149255172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 msgr2=0x55834e600a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.502+0000 7faf06487b80 1 --2- 172.21.15.2:0/2149255172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 0x55834e600a00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7faee8004790 tx=0x7faee8009260 comp rx=0 tx=0).stop 2023-11-16T18:27:32.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.502+0000 7faf06487b80 1 -- 172.21.15.2:0/2149255172 shutdown_connections 2023-11-16T18:27:32.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.502+0000 7faf06487b80 1 --2- 172.21.15.2:0/2149255172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f48b0 0x55834e5fbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.502+0000 7faf06487b80 1 --2- 172.21.15.2:0/2149255172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5efb60 0x55834e5eac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.502+0000 7faf06487b80 1 --2- 172.21.15.2:0/2149255172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5f0470 0x55834e600a00 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.502+0000 7faf06487b80 1 -- 172.21.15.2:0/2149255172 >> 172.21.15.2:0/2149255172 conn(0x55834e5e5360 msgr2=0x55834e5f6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:32.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.502+0000 7faf06487b80 1 -- 172.21.15.2:0/2149255172 shutdown_connections 2023-11-16T18:27:32.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.502+0000 7faf06487b80 1 -- 172.21.15.2:0/2149255172 wait complete. 2023-11-16T18:27:32.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faf06487b80 1 -- start start 2023-11-16T18:27:32.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faf06487b80 1 Processor -- start 2023-11-16T18:27:32.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faf06487b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5efb60 0x55834e5fe9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faf06487b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f0470 0x55834e5fef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faf06487b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5f48b0 0x55834e65c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faf06487b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55834e5ff810 con 0x55834e5efb60 2023-11-16T18:27:32.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faf06487b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55834e65cba0 con 0x55834e5f48b0 2023-11-16T18:27:32.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faf06487b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55834e65cce0 con 0x55834e5f0470 2023-11-16T18:27:32.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faf06487b80 10 client.0 _ll_register_callbacks cb 0x7faeec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:32.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faef3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5efb60 0x55834e5fe9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faef8898700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5f48b0 0x55834e65c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faef37fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f0470 0x55834e5fef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faef3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5efb60 0x55834e5fe9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52068/0 (socket says 172.21.15.2:52068) 2023-11-16T18:27:32.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.503+0000 7faef3fff700 1 -- 172.21.15.2:0/348640402 learned_addr learned my addr 172.21.15.2:0/348640402 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:32.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.504+0000 7faef8898700 1 -- 172.21.15.2:0/348640402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f0470 msgr2=0x55834e5fef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.504+0000 7faef8898700 1 --2- 172.21.15.2:0/348640402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f0470 0x55834e5fef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.504+0000 7faef8898700 1 -- 172.21.15.2:0/348640402 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5efb60 msgr2=0x55834e5fe9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:32.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.504+0000 7faef8898700 1 --2- 172.21.15.2:0/348640402 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5efb60 0x55834e5fe9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.504+0000 7faef8898700 1 -- 172.21.15.2:0/348640402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55834e507190 con 0x55834e5f48b0 2023-11-16T18:27:32.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.504+0000 7faef8898700 1 --2- 172.21.15.2:0/348640402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5f48b0 0x55834e65c660 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faeec0cb770 tx=0x7faeec0099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efdbf51b6405b49f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:32.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.504+0000 7faed37fe700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:27:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.504+0000 7faed37fe700 1 -- 172.21.15.2:0/348640402 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec09f1b0 con 0x55834e5f48b0 2023-11-16T18:27:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.504+0000 7faf06487b80 1 -- 172.21.15.2:0/348640402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faee400ee90 con 0x55834e5f48b0 2023-11-16T18:27:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.504+0000 7faed37fe700 1 -- 172.21.15.2:0/348640402 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7faeec0a3540 con 0x55834e5f48b0 2023-11-16T18:27:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.504+0000 7faed37fe700 1 -- 172.21.15.2:0/348640402 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7faeec004ce0 con 0x55834e5f48b0 2023-11-16T18:27:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.504+0000 7faf06487b80 10 client.25943 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.504+0000 7faf06487b80 1 -- 172.21.15.2:0/348640402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7faee8012a00 con 0x55834e5f48b0 2023-11-16T18:27:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.505+0000 7faed37fe700 1 -- 172.21.15.2:0/348640402 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faeec0b9020 con 0x55834e5f48b0 2023-11-16T18:27:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.505+0000 7faed37fe700 1 -- 172.21.15.2:0/348640402 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7faeec09a670 con 0x55834e5f48b0 2023-11-16T18:27:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.505+0000 7faf06487b80 10 client.25943 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.505+0000 7faf06487b80 10 client.25943 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.505+0000 7faf06487b80 20 client.25943 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.505+0000 7faf06487b80 1 --2- 172.21.15.2:0/348640402 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55834e6097e0 0x55834e660140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.505+0000 7faf06487b80 4 client.25943 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:27:32.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.505+0000 7faf06487b80 1 -- 172.21.15.2:0/348640402 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55834e50f660 con 0x55834e6097e0 2023-11-16T18:27:32.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.505+0000 7faef3fff700 1 --2- 172.21.15.2:0/348640402 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55834e6097e0 0x55834e660140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:32.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.505+0000 7faed37fe700 10 client.25943 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:32.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.505+0000 7faef3fff700 1 --2- 172.21.15.2:0/348640402 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55834e6097e0 0x55834e660140 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6047feb6ac64b10 server_cookie=b901e6b4739dd8a3 in_seq=0 out_seq=0 2023-11-16T18:27:32.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faed37fe700 1 -- 172.21.15.2:0/348640402 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55834e50f660 con 0x55834e6097e0 2023-11-16T18:27:32.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faed37fe700 10 client.25943 handle_command_reply: tid=0 2023-11-16T18:27:32.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faf06487b80 1 client.25943 shutdown 2023-11-16T18:27:32.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faf06487b80 1 -- 172.21.15.2:0/348640402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5f48b0 msgr2=0x55834e65c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faf06487b80 1 --2- 172.21.15.2:0/348640402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5f48b0 0x55834e65c660 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faeec0cb770 tx=0x7faeec0099e0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faf06487b80 1 -- 172.21.15.2:0/348640402 shutdown_connections 2023-11-16T18:27:32.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faf06487b80 1 --2- 172.21.15.2:0/348640402 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55834e6097e0 0x55834e660140 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faf06487b80 1 --2- 172.21.15.2:0/348640402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55834e5f48b0 0x55834e65c660 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faf06487b80 1 --2- 172.21.15.2:0/348640402 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55834e5efb60 0x55834e5fe9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faed37fe700 0 client.25943 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faf06487b80 1 --2- 172.21.15.2:0/348640402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55834e5f0470 0x55834e5fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faf06487b80 1 -- 172.21.15.2:0/348640402 >> 172.21.15.2:0/348640402 conn(0x55834e5e5360 msgr2=0x55834e5f28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faf06487b80 1 -- 172.21.15.2:0/348640402 shutdown_connections 2023-11-16T18:27:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faf06487b80 1 -- 172.21.15.2:0/348640402 wait complete. 2023-11-16T18:27:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.510+0000 7faf06487b80 20 client.25943 trim_cache size 0 max 16384 2023-11-16T18:27:32.514 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:27:32.514 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "scrub active (4323 inodes in the stack)", 2023-11-16T18:27:32.515 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": { 2023-11-16T18:27:32.515 INFO:teuthology.orchestra.run.smithi002.stdout: "9607aa23-76ae-4198-9378-3740d8aff22c": { 2023-11-16T18:27:32.515 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/", 2023-11-16T18:27:32.515 INFO:teuthology.orchestra.run.smithi002.stdout: "tag": "9607aa23-76ae-4198-9378-3740d8aff22c", 2023-11-16T18:27:32.515 INFO:teuthology.orchestra.run.smithi002.stdout: "options": "recursive,force" 2023-11-16T18:27:32.515 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:27:32.515 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:27:32.515 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:27:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.511+0000 7faefa2fb700 1 -- 172.21.15.2:0/2650037972 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7faedc0a12c0 msgr2=0x7faedc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.511+0000 7faefa2fb700 1 --2- 172.21.15.2:0/2650037972 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7faedc0a12c0 0x7faedc0a3770 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faee4009b60 tx=0x7faee4009620 comp rx=0 tx=0).stop 2023-11-16T18:27:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.511+0000 7faefa2fb700 1 -- 172.21.15.2:0/2650037972 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faef4105250 msgr2=0x7faef41a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.511+0000 7faefa2fb700 1 --2- 172.21.15.2:0/2650037972 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faef4105250 0x7faef41a23a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7faeec015000 tx=0x7faeec00bea0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.511+0000 7faefa2fb700 1 -- 172.21.15.2:0/2650037972 shutdown_connections 2023-11-16T18:27:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.511+0000 7faefa2fb700 1 --2- 172.21.15.2:0/2650037972 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7faedc0a12c0 0x7faedc0a3770 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.511+0000 7faefa2fb700 1 --2- 172.21.15.2:0/2650037972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faef40ffa70 0x7faef4197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.511+0000 7faefa2fb700 1 --2- 172.21.15.2:0/2650037972 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faef4105250 0x7faef41a23a0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.511+0000 7faefa2fb700 1 --2- 172.21.15.2:0/2650037972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faef4100420 0x7faef4197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:32.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.511+0000 7faefa2fb700 1 -- 172.21.15.2:0/2650037972 >> 172.21.15.2:0/2650037972 conn(0x7faef40763a0 msgr2=0x7faef40fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:32.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.511+0000 7faefa2fb700 1 -- 172.21.15.2:0/2650037972 shutdown_connections 2023-11-16T18:27:32.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:27:32.511+0000 7faefa2fb700 1 -- 172.21.15.2:0/2650037972 wait complete. 2023-11-16T18:27:32.527 INFO:tasks.cephfs.filesystem:scrub status for tag:9607aa23-76ae-4198-9378-3740d8aff22c - {'path': '/', 'tag': '9607aa23-76ae-4198-9378-3740d8aff22c', 'options': 'recursive,force'} 2023-11-16T18:27:32.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:32 smithi188 ceph-mon[100639]: pgmap v83: 129 pgs: 129 active+clean; 64 MiB data, 269 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 616 op/s 2023-11-16T18:27:34.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:34 smithi002 ceph-mon[98807]: scrub queued for path: / 2023-11-16T18:27:34.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:34 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:34.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:34 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:27:34.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:34 smithi002 ceph-mon[98807]: pgmap v84: 129 pgs: 129 active+clean; 84 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 748 op/s 2023-11-16T18:27:34.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:34 smithi040 ceph-mon[100251]: scrub queued for path: / 2023-11-16T18:27:34.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:34 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:34.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:34 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:27:34.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:34 smithi040 ceph-mon[100251]: pgmap v84: 129 pgs: 129 active+clean; 84 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 748 op/s 2023-11-16T18:27:34.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:34 smithi188 ceph-mon[100639]: scrub queued for path: / 2023-11-16T18:27:34.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:34 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:34.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:34 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:27:34.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:34 smithi188 ceph-mon[100639]: pgmap v84: 129 pgs: 129 active+clean; 84 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 748 op/s 2023-11-16T18:27:35.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.281+0000 7f74cfe60700 1 -- 172.21.15.2:0/2727332597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c8105520 msgr2=0x7f74c810d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:35.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.281+0000 7f74cfe60700 1 --2- 172.21.15.2:0/2727332597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c8105520 0x7f74c810d0f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f74c000b0d0 tx=0x7f74c000b3e0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.281+0000 7f74cfe60700 1 -- 172.21.15.2:0/2727332597 shutdown_connections 2023-11-16T18:27:35.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.281+0000 7f74cfe60700 1 --2- 172.21.15.2:0/2727332597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c81004e0 0x7f74c81008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.281+0000 7f74cfe60700 1 --2- 172.21.15.2:0/2727332597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c8105520 0x7f74c810d0f0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.281+0000 7f74cfe60700 1 --2- 172.21.15.2:0/2727332597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c8100e80 0x7f74c8104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.281+0000 7f74cfe60700 1 -- 172.21.15.2:0/2727332597 >> 172.21.15.2:0/2727332597 conn(0x7f74c80fbd50 msgr2=0x7f74c80fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:35.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.282+0000 7f74cfe60700 1 -- 172.21.15.2:0/2727332597 shutdown_connections 2023-11-16T18:27:35.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.282+0000 7f74cfe60700 1 -- 172.21.15.2:0/2727332597 wait complete. 2023-11-16T18:27:35.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.282+0000 7f74cfe60700 1 Processor -- start 2023-11-16T18:27:35.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.282+0000 7f74cfe60700 1 -- start start 2023-11-16T18:27:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.282+0000 7f74cfe60700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c81004e0 0x7f74c81942a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.282+0000 7f74cfe60700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c8100e80 0x7f74c81947e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.282+0000 7f74cfe60700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c8105520 0x7f74c8198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.282+0000 7f74cfe60700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8199280 con 0x7f74c8105520 2023-11-16T18:27:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.282+0000 7f74cfe60700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c81993f0 con 0x7f74c8100e80 2023-11-16T18:27:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.282+0000 7f74cfe60700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c81995b0 con 0x7f74c81004e0 2023-11-16T18:27:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74ce3fd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c8105520 0x7f74c8198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74cd3fb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c8100e80 0x7f74c81947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74ce3fd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c8105520 0x7f74c8198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52084/0 (socket says 172.21.15.2:52084) 2023-11-16T18:27:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74ce3fd700 1 -- 172.21.15.2:0/285934206 learned_addr learned my addr 172.21.15.2:0/285934206 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74cdbfc700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c81004e0 0x7f74c81942a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74ce3fd700 1 -- 172.21.15.2:0/285934206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c81004e0 msgr2=0x7f74c81942a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74ce3fd700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c81004e0 0x7f74c81942a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74ce3fd700 1 -- 172.21.15.2:0/285934206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c8100e80 msgr2=0x7f74c81947e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74ce3fd700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c8100e80 0x7f74c81947e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74ce3fd700 1 -- 172.21.15.2:0/285934206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c0009d00 con 0x7f74c8105520 2023-11-16T18:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74cd3fb700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c8100e80 0x7f74c81947e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74ce3fd700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c8105520 0x7f74c8198c40 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f74c0010c20 tx=0x7f74c0010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74beffd700 1 -- 172.21.15.2:0/285934206 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c0004c60 con 0x7f74c8105520 2023-11-16T18:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74beffd700 1 -- 172.21.15.2:0/285934206 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74c0004dc0 con 0x7f74c8105520 2023-11-16T18:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74cfe60700 1 -- 172.21.15.2:0/285934206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c810e160 con 0x7f74c8105520 2023-11-16T18:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.283+0000 7f74beffd700 1 -- 172.21.15.2:0/285934206 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c00052b0 con 0x7f74c8105520 2023-11-16T18:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.284+0000 7f74cfe60700 1 -- 172.21.15.2:0/285934206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c810dfd0 con 0x7f74c8105520 2023-11-16T18:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.286+0000 7f74beffd700 1 -- 172.21.15.2:0/285934206 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f74c00074e0 con 0x7f74c8105520 2023-11-16T18:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.286+0000 7f74beffd700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f74b40a1340 0x7f74b40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.287+0000 7f74beffd700 1 -- 172.21.15.2:0/285934206 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f74c00d2470 con 0x7f74c8105520 2023-11-16T18:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.287+0000 7f74cdbfc700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f74b40a1340 0x7f74b40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.287+0000 7f74cdbfc700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f74b40a1340 0x7f74b40a37f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009c60 tx=0x7f74c4009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.287+0000 7f74dbfecb80 1 Processor -- start 2023-11-16T18:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.287+0000 7f74dbfecb80 1 -- start start 2023-11-16T18:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74dbfecb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125a9b81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74dbfecb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8720 0x56125a9b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74dbfecb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74dbfecb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125a8b8000 con 0x56125a9b8b90 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74dbfecb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125a90a840 con 0x56125a9b8720 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74dbfecb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125a903cd0 con 0x56125a80e6b0 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74cd3fb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8720 0x56125a9b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74cdbfc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125a9b81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74cd3fb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8720 0x56125a9b2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42528/0 (socket says 172.21.15.2:42528) 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74cd3fb700 1 -- 172.21.15.2:0/2026646747 learned_addr learned my addr 172.21.15.2:0/2026646747 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74cdbfc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125a9b81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43618/0 (socket says 172.21.15.2:43618) 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74ce3fd700 1 --2- 172.21.15.2:0/2026646747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74cd3fb700 1 -- 172.21.15.2:0/2026646747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 msgr2=0x56125a9b81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74cd3fb700 1 --2- 172.21.15.2:0/2026646747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125a9b81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74cd3fb700 1 -- 172.21.15.2:0/2026646747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 msgr2=0x56125a9b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74cd3fb700 1 --2- 172.21.15.2:0/2026646747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74cd3fb700 1 -- 172.21.15.2:0/2026646747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56125a7855d0 con 0x56125a9b8720 2023-11-16T18:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.288+0000 7f74cdbfc700 1 --2- 172.21.15.2:0/2026646747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125a9b81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.289+0000 7f74cd3fb700 1 --2- 172.21.15.2:0/2026646747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8720 0x56125a9b2560 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f74b80114f0 tx=0x7f74b8011800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf1eb21b74065fae server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.289+0000 7f74b3fff700 1 -- 172.21.15.2:0/2026646747 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b8004230 con 0x56125a9b8720 2023-11-16T18:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.289+0000 7f74b3fff700 1 -- 172.21.15.2:0/2026646747 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74b8004390 con 0x56125a9b8720 2023-11-16T18:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.289+0000 7f74b3fff700 1 -- 172.21.15.2:0/2026646747 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b800e9d0 con 0x56125a9b8720 2023-11-16T18:27:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.290+0000 7f74dbfecb80 1 -- 172.21.15.2:0/2026646747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8720 msgr2=0x56125a9b2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.290+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/2026646747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8720 0x56125a9b2560 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f74b80114f0 tx=0x7f74b8011800 comp rx=0 tx=0).stop 2023-11-16T18:27:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.290+0000 7f74dbfecb80 1 -- 172.21.15.2:0/2026646747 shutdown_connections 2023-11-16T18:27:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.290+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/2026646747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125a9b81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.290+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/2026646747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.290+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/2026646747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8720 0x56125a9b2560 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.290+0000 7f74dbfecb80 1 -- 172.21.15.2:0/2026646747 >> 172.21.15.2:0/2026646747 conn(0x56125a9a7f50 msgr2=0x56125a9ad270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.290+0000 7f74dbfecb80 1 -- 172.21.15.2:0/2026646747 shutdown_connections 2023-11-16T18:27:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.290+0000 7f74dbfecb80 1 -- 172.21.15.2:0/2026646747 wait complete. 2023-11-16T18:27:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.290+0000 7f74dbfecb80 1 -- start start 2023-11-16T18:27:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.290+0000 7f74dbfecb80 1 Processor -- start 2023-11-16T18:27:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74dbfecb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125a9beb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74dbfecb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8720 0x56125a9bbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74dbfecb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9bc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74dbfecb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125a8a5880 con 0x56125a9b8b90 2023-11-16T18:27:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74dbfecb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125a8a5520 con 0x56125a9b8720 2023-11-16T18:27:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74dbfecb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125a7878c0 con 0x56125a80e6b0 2023-11-16T18:27:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74dbfecb80 10 client.0 _ll_register_callbacks cb 0x56125a8c3720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74ce3fd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9bc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74cd3fb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8720 0x56125a9bbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74ce3fd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9bc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52098/0 (socket says 172.21.15.2:52098) 2023-11-16T18:27:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74ce3fd700 1 -- 172.21.15.2:0/3717810333 learned_addr learned my addr 172.21.15.2:0/3717810333 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74cdbfc700 1 --2- 172.21.15.2:0/3717810333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125a9beb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74ce3fd700 1 -- 172.21.15.2:0/3717810333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 msgr2=0x56125a9beb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74ce3fd700 1 --2- 172.21.15.2:0/3717810333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125a9beb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74ce3fd700 1 -- 172.21.15.2:0/3717810333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8720 msgr2=0x56125a9bbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74ce3fd700 1 --2- 172.21.15.2:0/3717810333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8720 0x56125a9bbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74ce3fd700 1 -- 172.21.15.2:0/3717810333 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b8009c90 con 0x56125a9b8b90 2023-11-16T18:27:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.291+0000 7f74cd3fb700 1 --2- 172.21.15.2:0/3717810333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8720 0x56125a9bbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.292+0000 7f74ce3fd700 1 --2- 172.21.15.2:0/3717810333 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9bc4d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f74c009f110 tx=0x7f74c0007da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef2b3bd04251066 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.292+0000 7f74b17fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:27:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.292+0000 7f74b17fa700 1 -- 172.21.15.2:0/3717810333 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c009ef50 con 0x56125a9b8b90 2023-11-16T18:27:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.292+0000 7f74b17fa700 1 -- 172.21.15.2:0/3717810333 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f74c008a0b0 con 0x56125a9b8b90 2023-11-16T18:27:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.292+0000 7f74dbfecb80 1 -- 172.21.15.2:0/3717810333 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56125a9a5710 con 0x56125a9b8b90 2023-11-16T18:27:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.292+0000 7f74b17fa700 1 -- 172.21.15.2:0/3717810333 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74c009ed70 con 0x56125a9b8b90 2023-11-16T18:27:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.292+0000 7f74dbfecb80 10 client.16407 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.292+0000 7f74dbfecb80 1 -- 172.21.15.2:0/3717810333 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56125a8ca130 con 0x56125a9b8b90 2023-11-16T18:27:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.292+0000 7f74b17fa700 1 -- 172.21.15.2:0/3717810333 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56125a9a5710 con 0x56125a9b8b90 2023-11-16T18:27:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.292+0000 7f74b17fa700 1 -- 172.21.15.2:0/3717810333 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f74c00d2070 con 0x56125a9b8b90 2023-11-16T18:27:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.293+0000 7f74dbfecb80 10 client.16407 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.293+0000 7f74dbfecb80 10 client.16407 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.293+0000 7f74dbfecb80 20 client.16407 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.293+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/3717810333 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56125a9cbe00 0x56125a9ce210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.293+0000 7f74dbfecb80 4 client.16407 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:27:35.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.293+0000 7f74dbfecb80 1 -- 172.21.15.2:0/3717810333 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56125a8c6040 con 0x56125a9cbe00 2023-11-16T18:27:35.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.293+0000 7f74cdbfc700 1 --2- 172.21.15.2:0/3717810333 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56125a9cbe00 0x56125a9ce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.293+0000 7f74b17fa700 10 client.16407 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:35.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.293+0000 7f74cdbfc700 1 --2- 172.21.15.2:0/3717810333 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56125a9cbe00 0x56125a9ce210 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c97e1d6147808035 server_cookie=434eeff9d99f5642 in_seq=0 out_seq=0 2023-11-16T18:27:35.305 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.299+0000 7f74b17fa700 1 -- 172.21.15.2:0/3717810333 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56125a8c6040 con 0x56125a9cbe00 2023-11-16T18:27:35.305 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.299+0000 7f74b17fa700 10 client.16407 handle_command_reply: tid=0 2023-11-16T18:27:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.300+0000 7f74dbfecb80 1 client.16407 shutdown 2023-11-16T18:27:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.300+0000 7f74dbfecb80 1 -- 172.21.15.2:0/3717810333 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 msgr2=0x56125a9bc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.300+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/3717810333 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9bc4d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f74c009f110 tx=0x7f74c0007da0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.300+0000 7f74dbfecb80 1 -- 172.21.15.2:0/3717810333 shutdown_connections 2023-11-16T18:27:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.300+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/3717810333 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56125a9cbe00 0x56125a9ce210 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.300+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/3717810333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125a9beb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.300+0000 7f74b17fa700 0 client.16407 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.300+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/3717810333 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9bc4d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.300+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/3717810333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8720 0x56125a9bbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.300+0000 7f74dbfecb80 1 -- 172.21.15.2:0/3717810333 >> 172.21.15.2:0/3717810333 conn(0x56125a9a7f50 msgr2=0x56125a9ac7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.300+0000 7f74dbfecb80 1 -- 172.21.15.2:0/3717810333 shutdown_connections 2023-11-16T18:27:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.300+0000 7f74dbfecb80 1 -- 172.21.15.2:0/3717810333 wait complete. 2023-11-16T18:27:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.300+0000 7f74dbfecb80 20 client.16407 trim_cache size 0 max 16384 2023-11-16T18:27:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.316+0000 7f74dbfecb80 1 Processor -- start 2023-11-16T18:27:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74dbfecb80 1 -- start start 2023-11-16T18:27:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74dbfecb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a80e6b0 0x56125a9c1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74dbfecb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a9b8720 0x56125a9b75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74dbfecb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9acf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74dbfecb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125a9ad7e0 con 0x56125a9b8b90 2023-11-16T18:27:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74dbfecb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125a9a9fd0 con 0x56125a80e6b0 2023-11-16T18:27:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74dbfecb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125a9bd220 con 0x56125a9b8720 2023-11-16T18:27:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74ce3fd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9acf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74ce3fd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9acf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52106/0 (socket says 172.21.15.2:52106) 2023-11-16T18:27:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74ce3fd700 1 -- 172.21.15.2:0/557643116 learned_addr learned my addr 172.21.15.2:0/557643116 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74cd3fb700 1 --2- 172.21.15.2:0/557643116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a9b8720 0x56125a9b75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74cdbfc700 1 --2- 172.21.15.2:0/557643116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a80e6b0 0x56125a9c1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74ce3fd700 1 -- 172.21.15.2:0/557643116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a9b8720 msgr2=0x56125a9b75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74ce3fd700 1 --2- 172.21.15.2:0/557643116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a9b8720 0x56125a9b75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74ce3fd700 1 -- 172.21.15.2:0/557643116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a80e6b0 msgr2=0x56125a9c1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74ce3fd700 1 --2- 172.21.15.2:0/557643116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a80e6b0 0x56125a9c1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.317+0000 7f74ce3fd700 1 -- 172.21.15.2:0/557643116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56125a8c9e80 con 0x56125a9b8b90 2023-11-16T18:27:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.318+0000 7f74ce3fd700 1 --2- 172.21.15.2:0/557643116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9acf60 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f74c009ec90 tx=0x7f74c009ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62278b9d9be1a0f4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.318+0000 7f74bcff9700 1 -- 172.21.15.2:0/557643116 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c009ed70 con 0x56125a9b8b90 2023-11-16T18:27:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.318+0000 7f74bcff9700 1 -- 172.21.15.2:0/557643116 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74c009eaa0 con 0x56125a9b8b90 2023-11-16T18:27:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.318+0000 7f74bcff9700 1 -- 172.21.15.2:0/557643116 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c009f410 con 0x56125a9b8b90 2023-11-16T18:27:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.318+0000 7f74dbfecb80 1 -- 172.21.15.2:0/557643116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 msgr2=0x56125a9acf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.318+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/557643116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9acf60 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f74c009ec90 tx=0x7f74c009ecc0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 -- 172.21.15.2:0/557643116 shutdown_connections 2023-11-16T18:27:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/557643116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8b90 0x56125a9acf60 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/557643116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a80e6b0 0x56125a9c1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/557643116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a9b8720 0x56125a9b75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 -- 172.21.15.2:0/557643116 >> 172.21.15.2:0/557643116 conn(0x56125a9a7f50 msgr2=0x56125a9b6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 -- 172.21.15.2:0/557643116 shutdown_connections 2023-11-16T18:27:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 -- 172.21.15.2:0/557643116 wait complete. 2023-11-16T18:27:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 -- start start 2023-11-16T18:27:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 Processor -- start 2023-11-16T18:27:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125aa186f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8720 0x56125aa18c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8b90 0x56125a9c0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125a9c11e0 con 0x56125a9b8720 2023-11-16T18:27:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125a9c1350 con 0x56125a9b8b90 2023-11-16T18:27:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125a9c1510 con 0x56125a80e6b0 2023-11-16T18:27:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.319+0000 7f74dbfecb80 10 client.0 _ll_register_callbacks cb 0x7f74c000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74cd3fb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8720 0x56125aa18c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74cd3fb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8720 0x56125aa18c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52108/0 (socket says 172.21.15.2:52108) 2023-11-16T18:27:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74ce3fd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8b90 0x56125a9c0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74cdbfc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125aa186f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74cd3fb700 1 -- 172.21.15.2:0/1672823343 learned_addr learned my addr 172.21.15.2:0/1672823343 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74cd3fb700 1 -- 172.21.15.2:0/1672823343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 msgr2=0x56125aa186f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74cd3fb700 1 --2- 172.21.15.2:0/1672823343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125aa186f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74cd3fb700 1 -- 172.21.15.2:0/1672823343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8b90 msgr2=0x56125a9c0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74cd3fb700 1 --2- 172.21.15.2:0/1672823343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8b90 0x56125a9c0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74cd3fb700 1 -- 172.21.15.2:0/1672823343 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56125a8ca130 con 0x56125a9b8720 2023-11-16T18:27:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74cd3fb700 1 --2- 172.21.15.2:0/1672823343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8720 0x56125aa18c30 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f74b801b040 tx=0x7f74b80038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7077da1cd7439efe server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74b17fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:27:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74b17fa700 1 -- 172.21.15.2:0/1672823343 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b80041b0 con 0x56125a9b8720 2023-11-16T18:27:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74b17fa700 1 -- 172.21.15.2:0/1672823343 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f74b8005040 con 0x56125a9b8720 2023-11-16T18:27:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74dbfecb80 1 -- 172.21.15.2:0/1672823343 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74b8004390 con 0x56125a9b8720 2023-11-16T18:27:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.320+0000 7f74b17fa700 1 -- 172.21.15.2:0/1672823343 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74b8005240 con 0x56125a9b8720 2023-11-16T18:27:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.321+0000 7f74dbfecb80 10 client.16413 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.321+0000 7f74dbfecb80 1 -- 172.21.15.2:0/1672823343 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56125a71ac00 con 0x56125a9b8720 2023-11-16T18:27:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.321+0000 7f74b17fa700 1 -- 172.21.15.2:0/1672823343 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74b8004390 con 0x56125a9b8720 2023-11-16T18:27:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.321+0000 7f74b17fa700 1 -- 172.21.15.2:0/1672823343 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f74b8004500 con 0x56125a9b8720 2023-11-16T18:27:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.321+0000 7f74dbfecb80 10 client.16413 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.321+0000 7f74dbfecb80 10 client.16413 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.321+0000 7f74dbfecb80 20 client.16413 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.321+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/1672823343 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56125aa204c0 0x56125aa228d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.321+0000 7f74dbfecb80 4 client.16413 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:27:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.321+0000 7f74dbfecb80 1 -- 172.21.15.2:0/1672823343 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56125a8d2600 con 0x56125aa204c0 2023-11-16T18:27:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.321+0000 7f74cdbfc700 1 --2- 172.21.15.2:0/1672823343 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56125aa204c0 0x56125aa228d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.322+0000 7f74b17fa700 10 client.16413 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:35.322+0000 7f74cdbfc700 1 --2- 172.21.15.2:0/1672823343 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56125aa204c0 0x56125aa228d0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d818f73e3dbfe1b8 server_cookie=a6caafe66e9c8468 in_seq=0 out_seq=0 2023-11-16T18:27:36.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:36 smithi002 ceph-mon[98807]: pgmap v85: 129 pgs: 129 active+clean; 84 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 176 op/s 2023-11-16T18:27:36.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:36 smithi040 ceph-mon[100251]: pgmap v85: 129 pgs: 129 active+clean; 84 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 176 op/s 2023-11-16T18:27:36.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:36 smithi188 ceph-mon[100639]: pgmap v85: 129 pgs: 129 active+clean; 84 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 176 op/s 2023-11-16T18:27:37.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:37 smithi002 ceph-mon[98807]: pgmap v86: 129 pgs: 129 active+clean; 84 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 326 op/s 2023-11-16T18:27:37.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:37 smithi040 ceph-mon[100251]: pgmap v86: 129 pgs: 129 active+clean; 84 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 326 op/s 2023-11-16T18:27:37.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:37 smithi188 ceph-mon[100639]: pgmap v86: 129 pgs: 129 active+clean; 84 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 326 op/s 2023-11-16T18:27:38.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:27:39.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:27:39.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:27:39.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.183+0000 7f74b17fa700 1 -- 172.21.15.2:0/1672823343 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56125a8d2600 con 0x56125aa204c0 2023-11-16T18:27:39.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.183+0000 7f74b17fa700 10 client.16413 handle_command_reply: tid=0 2023-11-16T18:27:39.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.184+0000 7f74dbfecb80 1 client.16413 shutdown 2023-11-16T18:27:39.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.184+0000 7f74dbfecb80 1 -- 172.21.15.2:0/1672823343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8720 msgr2=0x56125aa18c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.184+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/1672823343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8720 0x56125aa18c30 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f74b801b040 tx=0x7f74b80038b0 comp rx=0 tx=0).stop 2023-11-16T18:27:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.184+0000 7f74dbfecb80 1 -- 172.21.15.2:0/1672823343 shutdown_connections 2023-11-16T18:27:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.184+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/1672823343 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56125aa204c0 0x56125aa228d0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.184+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/1672823343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56125a80e6b0 0x56125aa186f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.184+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/1672823343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56125a9b8b90 0x56125a9c0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.184+0000 7f74dbfecb80 1 --2- 172.21.15.2:0/1672823343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56125a9b8720 0x56125aa18c30 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.184+0000 7f74dbfecb80 1 -- 172.21.15.2:0/1672823343 >> 172.21.15.2:0/1672823343 conn(0x56125a9a7f50 msgr2=0x56125a9ba2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.184+0000 7f74b17fa700 0 client.16413 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.184+0000 7f74dbfecb80 1 -- 172.21.15.2:0/1672823343 shutdown_connections 2023-11-16T18:27:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.184+0000 7f74dbfecb80 1 -- 172.21.15.2:0/1672823343 wait complete. 2023-11-16T18:27:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.184+0000 7f74dbfecb80 20 client.16413 trim_cache size 0 max 16384 2023-11-16T18:27:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:27:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:27:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:27:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:27:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.185+0000 7f74cfe60700 1 -- 172.21.15.2:0/285934206 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f74b40a1340 msgr2=0x7f74b40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.185+0000 7f74cfe60700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f74b40a1340 0x7f74b40a37f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009c60 tx=0x7f74c4009260 comp rx=0 tx=0).stop 2023-11-16T18:27:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.185+0000 7f74cfe60700 1 -- 172.21.15.2:0/285934206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c8105520 msgr2=0x7f74c8198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.185+0000 7f74cfe60700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c8105520 0x7f74c8198c40 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f74c0010c20 tx=0x7f74c0010c50 comp rx=0 tx=0).stop 2023-11-16T18:27:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.185+0000 7f74cfe60700 1 -- 172.21.15.2:0/285934206 shutdown_connections 2023-11-16T18:27:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.185+0000 7f74cfe60700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f74b40a1340 0x7f74b40a37f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.185+0000 7f74cfe60700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c81004e0 0x7f74c81942a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.185+0000 7f74cfe60700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74c8105520 0x7f74c8198c40 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.185+0000 7f74cfe60700 1 --2- 172.21.15.2:0/285934206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74c8100e80 0x7f74c81947e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.185+0000 7f74cfe60700 1 -- 172.21.15.2:0/285934206 >> 172.21.15.2:0/285934206 conn(0x7f74c80fbd50 msgr2=0x7f74c80fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.185+0000 7f74cfe60700 1 -- 172.21.15.2:0/285934206 shutdown_connections 2023-11-16T18:27:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:39.185+0000 7f74cfe60700 1 -- 172.21.15.2:0/285934206 wait complete. 2023-11-16T18:27:39.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:39 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:27:39.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:39 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:39.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:39 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:27:39.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:39 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:39.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:39 smithi040 ceph-mon[100251]: pgmap v87: 129 pgs: 129 active+clean; 99 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.2 MiB/s wr, 1.51k op/s 2023-11-16T18:27:40.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:39 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:27:40.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:39 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:40.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:39 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:27:40.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:39 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:40.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:39 smithi188 ceph-mon[100639]: pgmap v87: 129 pgs: 129 active+clean; 99 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.2 MiB/s wr, 1.51k op/s 2023-11-16T18:27:40.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:39 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:27:40.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:39 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:40.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:39 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:27:40.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:39 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:27:40.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:39 smithi002 ceph-mon[98807]: pgmap v87: 129 pgs: 129 active+clean; 99 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.2 MiB/s wr, 1.51k op/s 2023-11-16T18:27:42.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:42 smithi002 ceph-mon[98807]: pgmap v88: 129 pgs: 129 active+clean; 99 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 1.48k op/s 2023-11-16T18:27:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:42 smithi040 ceph-mon[100251]: pgmap v88: 129 pgs: 129 active+clean; 99 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 1.48k op/s 2023-11-16T18:27:42.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:42 smithi188 ceph-mon[100639]: pgmap v88: 129 pgs: 129 active+clean; 99 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 1.48k op/s 2023-11-16T18:27:44.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:44 smithi002 ceph-mon[98807]: pgmap v89: 129 pgs: 129 active+clean; 100 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.4 MiB/s wr, 2.97k op/s 2023-11-16T18:27:44.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:44 smithi040 ceph-mon[100251]: pgmap v89: 129 pgs: 129 active+clean; 100 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.4 MiB/s wr, 2.97k op/s 2023-11-16T18:27:44.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:44 smithi188 ceph-mon[100639]: pgmap v89: 129 pgs: 129 active+clean; 100 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.4 MiB/s wr, 2.97k op/s 2023-11-16T18:27:46.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:46 smithi002 ceph-mon[98807]: pgmap v90: 129 pgs: 129 active+clean; 100 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 2.84k op/s 2023-11-16T18:27:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:46 smithi040 ceph-mon[100251]: pgmap v90: 129 pgs: 129 active+clean; 100 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 2.84k op/s 2023-11-16T18:27:46.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:46 smithi188 ceph-mon[100639]: pgmap v90: 129 pgs: 129 active+clean; 100 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 2.84k op/s 2023-11-16T18:27:47.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:47 smithi188 ceph-mon[100639]: pgmap v91: 129 pgs: 129 active+clean; 108 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.9 MiB/s wr, 2.86k op/s 2023-11-16T18:27:47.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:47 smithi002 ceph-mon[98807]: pgmap v91: 129 pgs: 129 active+clean; 108 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.9 MiB/s wr, 2.86k op/s 2023-11-16T18:27:47.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:47 smithi040 ceph-mon[100251]: pgmap v91: 129 pgs: 129 active+clean; 108 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.9 MiB/s wr, 2.86k op/s 2023-11-16T18:27:50.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:50 smithi002 ceph-mon[98807]: pgmap v92: 129 pgs: 129 active+clean; 136 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.3 MiB/s wr, 2.87k op/s 2023-11-16T18:27:50.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:50 smithi040 ceph-mon[100251]: pgmap v92: 129 pgs: 129 active+clean; 136 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.3 MiB/s wr, 2.87k op/s 2023-11-16T18:27:50.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:50 smithi188 ceph-mon[100639]: pgmap v92: 129 pgs: 129 active+clean; 136 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.3 MiB/s wr, 2.87k op/s 2023-11-16T18:27:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.305+0000 7f123d118700 1 -- 172.21.15.2:0/2320672815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12380ff3c0 msgr2=0x7f12380ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.305+0000 7f123d118700 1 --2- 172.21.15.2:0/2320672815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12380ff3c0 0x7f12380ff790 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1228009920 tx=0x7f1228009c30 comp rx=0 tx=0).stop 2023-11-16T18:27:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.305+0000 7f123d118700 1 -- 172.21.15.2:0/2320672815 shutdown_connections 2023-11-16T18:27:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.305+0000 7f123d118700 1 --2- 172.21.15.2:0/2320672815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12380ff3c0 0x7f12380ff790 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.305+0000 7f123d118700 1 --2- 172.21.15.2:0/2320672815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1238104400 0x7f123810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.305+0000 7f123d118700 1 --2- 172.21.15.2:0/2320672815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f12380ffd60 0x7f1238103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.305+0000 7f123d118700 1 -- 172.21.15.2:0/2320672815 >> 172.21.15.2:0/2320672815 conn(0x7f12380763a0 msgr2=0x7f12380767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.305+0000 7f123d118700 1 -- 172.21.15.2:0/2320672815 shutdown_connections 2023-11-16T18:27:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.305+0000 7f123d118700 1 -- 172.21.15.2:0/2320672815 wait complete. 2023-11-16T18:27:52.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.306+0000 7f123d118700 1 Processor -- start 2023-11-16T18:27:52.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.306+0000 7f123d118700 1 -- start start 2023-11-16T18:27:52.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123d118700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12380ff3c0 0x7f1238195090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123d118700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f12380ffd60 0x7f12381955d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123d118700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1238104400 0x7f1238195b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123d118700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12381a48d0 con 0x7f12380ffd60 2023-11-16T18:27:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123d118700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12381a4a40 con 0x7f12380ff3c0 2023-11-16T18:27:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123d118700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12381a4c00 con 0x7f1238104400 2023-11-16T18:27:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f1236d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12380ff3c0 0x7f1238195090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f12380ffd60 0x7f12381955d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f1236d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12380ff3c0 0x7f1238195090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42560/0 (socket says 172.21.15.2:42560) 2023-11-16T18:27:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f1236d9d700 1 -- 172.21.15.2:0/1092758282 learned_addr learned my addr 172.21.15.2:0/1092758282 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123659c700 1 -- 172.21.15.2:0/1092758282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1238104400 msgr2=0x7f1238195b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123759e700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1238104400 0x7f1238195b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123659c700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1238104400 0x7f1238195b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123659c700 1 -- 172.21.15.2:0/1092758282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12380ff3c0 msgr2=0x7f1238195090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123659c700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12380ff3c0 0x7f1238195090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123659c700 1 -- 172.21.15.2:0/1092758282 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c0095f0 con 0x7f12380ffd60 2023-11-16T18:27:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f1236d9d700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12380ff3c0 0x7f1238195090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123759e700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1238104400 0x7f1238195b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:27:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.307+0000 7f123659c700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f12380ffd60 0x7f12381955d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1238100130 tx=0x7f122c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.308+0000 7f121ffff700 1 -- 172.21.15.2:0/1092758282 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c00cc60 con 0x7f12380ffd60 2023-11-16T18:27:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.308+0000 7f121ffff700 1 -- 172.21.15.2:0/1092758282 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f122c0044e0 con 0x7f12380ffd60 2023-11-16T18:27:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.308+0000 7f123d118700 1 -- 172.21.15.2:0/1092758282 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1228009520 con 0x7f12380ffd60 2023-11-16T18:27:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.308+0000 7f121ffff700 1 -- 172.21.15.2:0/1092758282 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c010430 con 0x7f12380ffd60 2023-11-16T18:27:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.308+0000 7f123d118700 1 -- 172.21.15.2:0/1092758282 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123810f1c0 con 0x7f12380ffd60 2023-11-16T18:27:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.311+0000 7f121ffff700 1 -- 172.21.15.2:0/1092758282 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f122c003f40 con 0x7f12380ffd60 2023-11-16T18:27:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.311+0000 7f121ffff700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f12200a12c0 0x7f12200a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.311+0000 7f121ffff700 1 -- 172.21.15.2:0/1092758282 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f122c014070 con 0x7f12380ffd60 2023-11-16T18:27:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.311+0000 7f1236d9d700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f12200a12c0 0x7f12200a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f12492a4b80 1 Processor -- start 2023-11-16T18:27:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f12492a4b80 1 -- start start 2023-11-16T18:27:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f1236d9d700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f12200a12c0 0x7f12200a3770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f122800c010 tx=0x7f1228019040 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f12492a4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 0x55dd65d291e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f12492a4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29720 0x55dd65d23560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f12492a4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65d29b90 0x55dd65d23b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f12492a4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65c29000 con 0x55dd65b7f6b0 2023-11-16T18:27:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f12492a4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65c7b840 con 0x55dd65d29720 2023-11-16T18:27:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f12492a4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65c74cd0 con 0x55dd65d29b90 2023-11-16T18:27:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f1236d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 0x55dd65d291e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f1236d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 0x55dd65d291e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52130/0 (socket says 172.21.15.2:52130) 2023-11-16T18:27:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f1236d9d700 1 -- 172.21.15.2:0/3130681793 learned_addr learned my addr 172.21.15.2:0/3130681793 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f123659c700 1 --2- 172.21.15.2:0/3130681793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29720 0x55dd65d23560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f1236d9d700 1 -- 172.21.15.2:0/3130681793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65d29b90 msgr2=0x55dd65d23b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f1236d9d700 1 --2- 172.21.15.2:0/3130681793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65d29b90 0x55dd65d23b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f1236d9d700 1 -- 172.21.15.2:0/3130681793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29720 msgr2=0x55dd65d23560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.312+0000 7f1236d9d700 1 --2- 172.21.15.2:0/3130681793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29720 0x55dd65d23560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.313+0000 7f1236d9d700 1 -- 172.21.15.2:0/3130681793 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd65af65d0 con 0x55dd65b7f6b0 2023-11-16T18:27:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.313+0000 7f1236d9d700 1 --2- 172.21.15.2:0/3130681793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 0x55dd65d291e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1228005950 tx=0x7f122801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110d58fcc2658347 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.313+0000 7f121d7fa700 1 -- 172.21.15.2:0/3130681793 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122800b860 con 0x55dd65b7f6b0 2023-11-16T18:27:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.313+0000 7f121d7fa700 1 -- 172.21.15.2:0/3130681793 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f122801d040 con 0x55dd65b7f6b0 2023-11-16T18:27:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.313+0000 7f121d7fa700 1 -- 172.21.15.2:0/3130681793 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1228017780 con 0x55dd65b7f6b0 2023-11-16T18:27:52.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.314+0000 7f12492a4b80 1 -- 172.21.15.2:0/3130681793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 msgr2=0x55dd65d291e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.314+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3130681793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 0x55dd65d291e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1228005950 tx=0x7f122801f000 comp rx=0 tx=0).stop 2023-11-16T18:27:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.314+0000 7f12492a4b80 1 -- 172.21.15.2:0/3130681793 shutdown_connections 2023-11-16T18:27:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.314+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3130681793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65d29b90 0x55dd65d23b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.314+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3130681793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 0x55dd65d291e0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.314+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3130681793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29720 0x55dd65d23560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.314+0000 7f12492a4b80 1 -- 172.21.15.2:0/3130681793 >> 172.21.15.2:0/3130681793 conn(0x55dd65d18f50 msgr2=0x55dd65d1e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.314+0000 7f12492a4b80 1 -- 172.21.15.2:0/3130681793 shutdown_connections 2023-11-16T18:27:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.314+0000 7f12492a4b80 1 -- 172.21.15.2:0/3130681793 wait complete. 2023-11-16T18:27:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.314+0000 7f12492a4b80 1 -- start start 2023-11-16T18:27:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.314+0000 7f12492a4b80 1 Processor -- start 2023-11-16T18:27:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f12492a4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 0x55dd65d32a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f12492a4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65d29720 0x55dd65d32fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f12492a4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29b90 0x55dd65d2cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f12492a4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65c16880 con 0x55dd65b7f6b0 2023-11-16T18:27:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f12492a4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65c16520 con 0x55dd65d29b90 2023-11-16T18:27:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f12492a4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65af88c0 con 0x55dd65d29720 2023-11-16T18:27:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f12492a4b80 10 client.0 _ll_register_callbacks cb 0x55dd65c34720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f1236d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 0x55dd65d32a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f1236d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 0x55dd65d32a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52136/0 (socket says 172.21.15.2:52136) 2023-11-16T18:27:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f123659c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65d29720 0x55dd65d32fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f1236d9d700 1 -- 172.21.15.2:0/3389224791 learned_addr learned my addr 172.21.15.2:0/3389224791 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f123759e700 1 --2- 172.21.15.2:0/3389224791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29b90 0x55dd65d2cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f1236d9d700 1 -- 172.21.15.2:0/3389224791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65d29720 msgr2=0x55dd65d32fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f1236d9d700 1 --2- 172.21.15.2:0/3389224791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65d29720 0x55dd65d32fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f1236d9d700 1 -- 172.21.15.2:0/3389224791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29b90 msgr2=0x55dd65d2cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f1236d9d700 1 --2- 172.21.15.2:0/3389224791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29b90 0x55dd65d2cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f1236d9d700 1 -- 172.21.15.2:0/3389224791 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123810c810 con 0x55dd65b7f6b0 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f123659c700 1 --2- 172.21.15.2:0/3389224791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65d29720 0x55dd65d32fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f1236d9d700 1 --2- 172.21.15.2:0/3389224791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 0x55dd65d32a80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f1228017440 tx=0x7f122800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e23a44e338381f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f121aff5700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f121aff5700 1 -- 172.21.15.2:0/3389224791 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1228005610 con 0x55dd65b7f6b0 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f121aff5700 1 -- 172.21.15.2:0/3389224791 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f1228003680 con 0x55dd65b7f6b0 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.315+0000 7f12492a4b80 1 -- 172.21.15.2:0/3389224791 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd65d16710 con 0x55dd65b7f6b0 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.316+0000 7f121aff5700 1 -- 172.21.15.2:0/3389224791 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1228003840 con 0x55dd65b7f6b0 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.316+0000 7f12492a4b80 10 client.16425 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.316+0000 7f12492a4b80 1 -- 172.21.15.2:0/3389224791 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55dd65c3b130 con 0x55dd65b7f6b0 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.316+0000 7f121aff5700 1 -- 172.21.15.2:0/3389224791 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dd65d16710 con 0x55dd65b7f6b0 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.316+0000 7f121aff5700 1 -- 172.21.15.2:0/3389224791 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f1228017b40 con 0x55dd65b7f6b0 2023-11-16T18:27:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.316+0000 7f12492a4b80 10 client.16425 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.316+0000 7f12492a4b80 10 client.16425 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.316+0000 7f12492a4b80 20 client.16425 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.316+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3389224791 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55dd65d3ca40 0x55dd65d3ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.316+0000 7f12492a4b80 4 client.16425 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:27:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.316+0000 7f12492a4b80 1 -- 172.21.15.2:0/3389224791 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd65c37040 con 0x55dd65d3ca40 2023-11-16T18:27:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.316+0000 7f123659c700 1 --2- 172.21.15.2:0/3389224791 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55dd65d3ca40 0x55dd65d3ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.317+0000 7f121aff5700 10 client.16425 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.317+0000 7f123659c700 1 --2- 172.21.15.2:0/3389224791 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55dd65d3ca40 0x55dd65d3ee20 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e583b3113cb73379 server_cookie=e56cecb98ea31797 in_seq=0 out_seq=0 2023-11-16T18:27:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f121aff5700 1 -- 172.21.15.2:0/3389224791 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55dd65c37040 con 0x55dd65d3ca40 2023-11-16T18:27:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f121aff5700 10 client.16425 handle_command_reply: tid=0 2023-11-16T18:27:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f12492a4b80 1 client.16425 shutdown 2023-11-16T18:27:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f12492a4b80 1 -- 172.21.15.2:0/3389224791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 msgr2=0x55dd65d32a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3389224791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 0x55dd65d32a80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f1228017440 tx=0x7f122800f520 comp rx=0 tx=0).stop 2023-11-16T18:27:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f12492a4b80 1 -- 172.21.15.2:0/3389224791 shutdown_connections 2023-11-16T18:27:52.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3389224791 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55dd65d3ca40 0x55dd65d3ee20 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3389224791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29b90 0x55dd65d2cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f121aff5700 0 client.16425 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:52.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3389224791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65b7f6b0 0x55dd65d32a80 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3389224791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65d29720 0x55dd65d32fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f12492a4b80 1 -- 172.21.15.2:0/3389224791 >> 172.21.15.2:0/3389224791 conn(0x55dd65d18f50 msgr2=0x55dd65d1d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f12492a4b80 1 -- 172.21.15.2:0/3389224791 shutdown_connections 2023-11-16T18:27:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.319+0000 7f12492a4b80 1 -- 172.21.15.2:0/3389224791 wait complete. 2023-11-16T18:27:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.320+0000 7f12492a4b80 20 client.16425 trim_cache size 0 max 16384 2023-11-16T18:27:52.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.335+0000 7f12492a4b80 1 Processor -- start 2023-11-16T18:27:52.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.335+0000 7f12492a4b80 1 -- start start 2023-11-16T18:27:52.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.335+0000 7f12492a4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65b7f6b0 0x55dd65d23560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.335+0000 7f12492a4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65d29720 0x55dd65d1df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.335+0000 7f12492a4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29b90 0x55dd65d27dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.335+0000 7f12492a4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65d27790 con 0x55dd65d29720 2023-11-16T18:27:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.335+0000 7f12492a4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65d2b290 con 0x55dd65d29b90 2023-11-16T18:27:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.335+0000 7f12492a4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65d279c0 con 0x55dd65b7f6b0 2023-11-16T18:27:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.335+0000 7f123659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65d29720 0x55dd65d1df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.335+0000 7f1236d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65b7f6b0 0x55dd65d23560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.335+0000 7f123659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65d29720 0x55dd65d1df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52146/0 (socket says 172.21.15.2:52146) 2023-11-16T18:27:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.335+0000 7f123659c700 1 -- 172.21.15.2:0/3519502217 learned_addr learned my addr 172.21.15.2:0/3519502217 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.335+0000 7f123759e700 1 --2- 172.21.15.2:0/3519502217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29b90 0x55dd65d27dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.336+0000 7f1236d9d700 1 -- 172.21.15.2:0/3519502217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29b90 msgr2=0x55dd65d27dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.336+0000 7f1236d9d700 1 --2- 172.21.15.2:0/3519502217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29b90 0x55dd65d27dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.336+0000 7f1236d9d700 1 -- 172.21.15.2:0/3519502217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65d29720 msgr2=0x55dd65d1df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.336+0000 7f1236d9d700 1 --2- 172.21.15.2:0/3519502217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65d29720 0x55dd65d1df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.336+0000 7f1236d9d700 1 -- 172.21.15.2:0/3519502217 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd65c3ae80 con 0x55dd65b7f6b0 2023-11-16T18:27:52.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.336+0000 7f123659c700 1 --2- 172.21.15.2:0/3519502217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65d29720 0x55dd65d1df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:27:52.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.336+0000 7f1236d9d700 1 --2- 172.21.15.2:0/3519502217 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65b7f6b0 0x55dd65d23560 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f122800bb10 tx=0x7f1228009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba72579b67e4b1e3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:52.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.336+0000 7f121dffb700 1 -- 172.21.15.2:0/3519502217 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1228005610 con 0x55dd65b7f6b0 2023-11-16T18:27:52.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.336+0000 7f121dffb700 1 -- 172.21.15.2:0/3519502217 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f122800bbf0 con 0x55dd65b7f6b0 2023-11-16T18:27:52.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.336+0000 7f121dffb700 1 -- 172.21.15.2:0/3519502217 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12280038e0 con 0x55dd65b7f6b0 2023-11-16T18:27:52.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.337+0000 7f12492a4b80 1 -- 172.21.15.2:0/3519502217 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65b7f6b0 msgr2=0x55dd65d23560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:52.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.337+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3519502217 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65b7f6b0 0x55dd65d23560 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f122800bb10 tx=0x7f1228009c30 comp rx=0 tx=0).stop 2023-11-16T18:27:52.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.337+0000 7f12492a4b80 1 -- 172.21.15.2:0/3519502217 shutdown_connections 2023-11-16T18:27:52.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.337+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3519502217 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65b7f6b0 0x55dd65d23560 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.337+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3519502217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29b90 0x55dd65d27dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.337+0000 7f12492a4b80 1 --2- 172.21.15.2:0/3519502217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65d29720 0x55dd65d1df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.337+0000 7f12492a4b80 1 -- 172.21.15.2:0/3519502217 >> 172.21.15.2:0/3519502217 conn(0x55dd65d18f50 msgr2=0x55dd65d2d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.337+0000 7f12492a4b80 1 -- 172.21.15.2:0/3519502217 shutdown_connections 2023-11-16T18:27:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.337+0000 7f12492a4b80 1 -- 172.21.15.2:0/3519502217 wait complete. 2023-11-16T18:27:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.337+0000 7f12492a4b80 1 -- start start 2023-11-16T18:27:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.337+0000 7f12492a4b80 1 Processor -- start 2023-11-16T18:27:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f12492a4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65b7f6b0 0x55dd65d32e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f12492a4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29720 0x55dd65d30630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f12492a4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65d29b90 0x55dd65d30bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f12492a4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65d312e0 con 0x55dd65d29b90 2023-11-16T18:27:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f12492a4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65d31450 con 0x55dd65d29720 2023-11-16T18:27:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f12492a4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65d8f8d0 con 0x55dd65b7f6b0 2023-11-16T18:27:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f12492a4b80 10 client.0 _ll_register_callbacks cb 0x7f122c00ce40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:27:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f1236d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65b7f6b0 0x55dd65d32e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f123759e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65d29b90 0x55dd65d30bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f123659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29720 0x55dd65d30630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f1236d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65b7f6b0 0x55dd65d32e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43680/0 (socket says 172.21.15.2:43680) 2023-11-16T18:27:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f1236d9d700 1 -- 172.21.15.2:0/2857785785 learned_addr learned my addr 172.21.15.2:0/2857785785 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:27:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f123759e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65d29b90 0x55dd65d30bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52154/0 (socket says 172.21.15.2:52154) 2023-11-16T18:27:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f1236d9d700 1 -- 172.21.15.2:0/2857785785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29720 msgr2=0x55dd65d30630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f1236d9d700 1 --2- 172.21.15.2:0/2857785785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29720 0x55dd65d30630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f1236d9d700 1 -- 172.21.15.2:0/2857785785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65d29b90 msgr2=0x55dd65d30bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:27:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f1236d9d700 1 --2- 172.21.15.2:0/2857785785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65d29b90 0x55dd65d30bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f1236d9d700 1 -- 172.21.15.2:0/2857785785 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123810f1c0 con 0x55dd65b7f6b0 2023-11-16T18:27:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f1236d9d700 1 --2- 172.21.15.2:0/2857785785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65b7f6b0 0x55dd65d32e20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1228005a80 tx=0x7f12280374f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12b60f3492689785 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:27:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f121aff5700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:27:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f121aff5700 1 -- 172.21.15.2:0/2857785785 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12280378e0 con 0x55dd65b7f6b0 2023-11-16T18:27:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f12492a4b80 1 -- 172.21.15.2:0/2857785785 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f122801d040 con 0x55dd65b7f6b0 2023-11-16T18:27:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.338+0000 7f121aff5700 1 -- 172.21.15.2:0/2857785785 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f1228037a40 con 0x55dd65b7f6b0 2023-11-16T18:27:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.339+0000 7f121aff5700 1 -- 172.21.15.2:0/2857785785 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1228037c00 con 0x55dd65b7f6b0 2023-11-16T18:27:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.339+0000 7f12492a4b80 10 client.25867 fetch_fsmap learned FSMap version 27 2023-11-16T18:27:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.339+0000 7f12492a4b80 1 -- 172.21.15.2:0/2857785785 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55dd65c3b130 con 0x55dd65b7f6b0 2023-11-16T18:27:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.339+0000 7f121aff5700 1 -- 172.21.15.2:0/2857785785 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f122801d040 con 0x55dd65b7f6b0 2023-11-16T18:27:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.339+0000 7f121aff5700 1 -- 172.21.15.2:0/2857785785 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f1228017520 con 0x55dd65b7f6b0 2023-11-16T18:27:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.339+0000 7f12492a4b80 10 client.25867 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:27:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.339+0000 7f12492a4b80 10 client.25867 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:27:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.339+0000 7f12492a4b80 20 client.25867 populate_metadata read hostname 'smithi002' 2023-11-16T18:27:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.339+0000 7f12492a4b80 1 --2- 172.21.15.2:0/2857785785 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55dd65d90a50 0x55dd65d92e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:27:52.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.339+0000 7f12492a4b80 4 client.25867 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:27:52.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.339+0000 7f12492a4b80 1 -- 172.21.15.2:0/2857785785 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55dd65c43600 con 0x55dd65d90a50 2023-11-16T18:27:52.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.339+0000 7f123659c700 1 --2- 172.21.15.2:0/2857785785 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55dd65d90a50 0x55dd65d92e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:27:52.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.340+0000 7f123659c700 1 --2- 172.21.15.2:0/2857785785 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55dd65d90a50 0x55dd65d92e30 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82fba0269fb25087 server_cookie=b248eafbd955a4e9 in_seq=0 out_seq=0 2023-11-16T18:27:52.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:52.340+0000 7f121aff5700 10 client.25867 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:52.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:52 smithi002 ceph-mon[98807]: pgmap v93: 129 pgs: 129 active+clean; 136 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.1 MiB/s wr, 1.68k op/s 2023-11-16T18:27:52.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:52 smithi040 ceph-mon[100251]: pgmap v93: 129 pgs: 129 active+clean; 136 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.1 MiB/s wr, 1.68k op/s 2023-11-16T18:27:52.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:52 smithi188 ceph-mon[100639]: pgmap v93: 129 pgs: 129 active+clean; 136 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.1 MiB/s wr, 1.68k op/s 2023-11-16T18:27:54.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:54 smithi002 ceph-mon[98807]: pgmap v94: 129 pgs: 129 active+clean; 149 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 6.2 MiB/s wr, 1.81k op/s 2023-11-16T18:27:54.737 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:54 smithi188 ceph-mon[100639]: pgmap v94: 129 pgs: 129 active+clean; 149 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 6.2 MiB/s wr, 1.81k op/s 2023-11-16T18:27:54.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:54 smithi040 ceph-mon[100251]: pgmap v94: 129 pgs: 129 active+clean; 149 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 6.2 MiB/s wr, 1.81k op/s 2023-11-16T18:27:56.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.104+0000 7f121aff5700 1 -- 172.21.15.2:0/2857785785 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55dd65c43600 con 0x55dd65d90a50 2023-11-16T18:27:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.104+0000 7f121aff5700 10 client.25867 handle_command_reply: tid=0 2023-11-16T18:27:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.104+0000 7f12492a4b80 1 client.25867 shutdown 2023-11-16T18:27:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.104+0000 7f12492a4b80 1 -- 172.21.15.2:0/2857785785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65b7f6b0 msgr2=0x55dd65d32e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.104+0000 7f12492a4b80 1 --2- 172.21.15.2:0/2857785785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65b7f6b0 0x55dd65d32e20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1228005a80 tx=0x7f12280374f0 comp rx=0 tx=0).stop 2023-11-16T18:27:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.104+0000 7f12492a4b80 1 -- 172.21.15.2:0/2857785785 shutdown_connections 2023-11-16T18:27:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.104+0000 7f12492a4b80 1 --2- 172.21.15.2:0/2857785785 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55dd65d90a50 0x55dd65d92e30 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.104+0000 7f12492a4b80 1 --2- 172.21.15.2:0/2857785785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd65b7f6b0 0x55dd65d32e20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.104+0000 7f12492a4b80 1 --2- 172.21.15.2:0/2857785785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55dd65d29b90 0x55dd65d30bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.104+0000 7f12492a4b80 1 --2- 172.21.15.2:0/2857785785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd65d29720 0x55dd65d30630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.104+0000 7f12492a4b80 1 -- 172.21.15.2:0/2857785785 >> 172.21.15.2:0/2857785785 conn(0x55dd65d18f50 msgr2=0x55dd65d27dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.104+0000 7f121aff5700 0 client.25867 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:27:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.105+0000 7f12492a4b80 1 -- 172.21.15.2:0/2857785785 shutdown_connections 2023-11-16T18:27:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.105+0000 7f12492a4b80 1 -- 172.21.15.2:0/2857785785 wait complete. 2023-11-16T18:27:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.105+0000 7f12492a4b80 20 client.25867 trim_cache size 0 max 16384 2023-11-16T18:27:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:27:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:27:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:27:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:27:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.105+0000 7f123d118700 1 -- 172.21.15.2:0/1092758282 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f12200a12c0 msgr2=0x7f12200a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.105+0000 7f123d118700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f12200a12c0 0x7f12200a3770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f122800c010 tx=0x7f1228019040 comp rx=0 tx=0).stop 2023-11-16T18:27:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.105+0000 7f123d118700 1 -- 172.21.15.2:0/1092758282 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f12380ffd60 msgr2=0x7f12381955d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:27:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.105+0000 7f123d118700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f12380ffd60 0x7f12381955d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1238100130 tx=0x7f122c00ee70 comp rx=0 tx=0).stop 2023-11-16T18:27:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.105+0000 7f123d118700 1 -- 172.21.15.2:0/1092758282 shutdown_connections 2023-11-16T18:27:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.106+0000 7f123d118700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f12200a12c0 0x7f12200a3770 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.106+0000 7f123d118700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1238104400 0x7f1238195b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.106+0000 7f123d118700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12380ff3c0 0x7f1238195090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.106+0000 7f123d118700 1 --2- 172.21.15.2:0/1092758282 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f12380ffd60 0x7f12381955d0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:27:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.106+0000 7f123d118700 1 -- 172.21.15.2:0/1092758282 >> 172.21.15.2:0/1092758282 conn(0x7f12380763a0 msgr2=0x7f12380fca20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:27:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.106+0000 7f123d118700 1 -- 172.21.15.2:0/1092758282 shutdown_connections 2023-11-16T18:27:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:27:56.106+0000 7f123d118700 1 -- 172.21.15.2:0/1092758282 wait complete. 2023-11-16T18:27:56.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:56 smithi002 ceph-mon[98807]: pgmap v95: 129 pgs: 129 active+clean; 149 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 319 op/s 2023-11-16T18:27:56.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:56 smithi040 ceph-mon[100251]: pgmap v95: 129 pgs: 129 active+clean; 149 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 319 op/s 2023-11-16T18:27:56.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:56 smithi188 ceph-mon[100639]: pgmap v95: 129 pgs: 129 active+clean; 149 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 319 op/s 2023-11-16T18:27:57.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:27:57 smithi188 ceph-mon[100639]: pgmap v96: 129 pgs: 129 active+clean; 157 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.1 MiB/s wr, 410 op/s 2023-11-16T18:27:57.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:27:57 smithi002 ceph-mon[98807]: pgmap v96: 129 pgs: 129 active+clean; 157 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.1 MiB/s wr, 410 op/s 2023-11-16T18:27:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:27:57 smithi040 ceph-mon[100251]: pgmap v96: 129 pgs: 129 active+clean; 157 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.1 MiB/s wr, 410 op/s 2023-11-16T18:28:00.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:00 smithi002 ceph-mon[98807]: pgmap v97: 129 pgs: 129 active+clean; 168 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.3 MiB/s wr, 1.35k op/s 2023-11-16T18:28:00.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:00 smithi040 ceph-mon[100251]: pgmap v97: 129 pgs: 129 active+clean; 168 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.3 MiB/s wr, 1.35k op/s 2023-11-16T18:28:00.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:00 smithi188 ceph-mon[100639]: pgmap v97: 129 pgs: 129 active+clean; 168 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.3 MiB/s wr, 1.35k op/s 2023-11-16T18:28:02.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:02 smithi002 ceph-mon[98807]: pgmap v98: 129 pgs: 129 active+clean; 168 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 1.19k op/s 2023-11-16T18:28:02.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:02 smithi040 ceph-mon[100251]: pgmap v98: 129 pgs: 129 active+clean; 168 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 1.19k op/s 2023-11-16T18:28:02.528 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:28:02.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:02 smithi188 ceph-mon[100639]: pgmap v98: 129 pgs: 129 active+clean; 168 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 1.19k op/s 2023-11-16T18:28:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.675+0000 7fc9a9e2c700 1 -- 172.21.15.2:0/3676569709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9a4105250 msgr2=0x7fc9a4105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.675+0000 7fc9a9e2c700 1 --2- 172.21.15.2:0/3676569709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9a4105250 0x7fc9a4105620 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc994009990 tx=0x7fc994009ca0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.675+0000 7fc9a9e2c700 1 -- 172.21.15.2:0/3676569709 shutdown_connections 2023-11-16T18:28:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.675+0000 7fc9a9e2c700 1 --2- 172.21.15.2:0/3676569709 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9a4100420 0x7fc9a410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.675+0000 7fc9a9e2c700 1 --2- 172.21.15.2:0/3676569709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9a4105250 0x7fc9a4105620 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.675+0000 7fc9a9e2c700 1 --2- 172.21.15.2:0/3676569709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9a40ffa70 0x7fc9a40ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.675+0000 7fc9a9e2c700 1 -- 172.21.15.2:0/3676569709 >> 172.21.15.2:0/3676569709 conn(0x7fc9a40763a0 msgr2=0x7fc9a40767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.675+0000 7fc9a9e2c700 1 -- 172.21.15.2:0/3676569709 shutdown_connections 2023-11-16T18:28:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.675+0000 7fc9a9e2c700 1 -- 172.21.15.2:0/3676569709 wait complete. 2023-11-16T18:28:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.676+0000 7fc9a9e2c700 1 Processor -- start 2023-11-16T18:28:02.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.676+0000 7fc9a9e2c700 1 -- start start 2023-11-16T18:28:02.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.676+0000 7fc9a9e2c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9a40ffa70 0x7fc9a4196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.676+0000 7fc9a9e2c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9a4100420 0x7fc9a4196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.676+0000 7fc9a9e2c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9a4105250 0x7fc9a419add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.676+0000 7fc9a9e2c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a419b410 con 0x7fc9a40ffa70 2023-11-16T18:28:02.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.676+0000 7fc9a9e2c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a419b580 con 0x7fc9a4100420 2023-11-16T18:28:02.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.676+0000 7fc9a9e2c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a419b740 con 0x7fc9a4105250 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.676+0000 7fc9a37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9a40ffa70 0x7fc9a4196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.676+0000 7fc9a2ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9a4100420 0x7fc9a4196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.676+0000 7fc9a37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9a40ffa70 0x7fc9a4196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52162/0 (socket says 172.21.15.2:52162) 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.676+0000 7fc9a37fe700 1 -- 172.21.15.2:0/970145922 learned_addr learned my addr 172.21.15.2:0/970145922 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.676+0000 7fc9a3fff700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9a4105250 0x7fc9a419add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.677+0000 7fc9a37fe700 1 -- 172.21.15.2:0/970145922 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9a4105250 msgr2=0x7fc9a419add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.677+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9a4105250 0x7fc9a419add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.677+0000 7fc9a37fe700 1 -- 172.21.15.2:0/970145922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9a4100420 msgr2=0x7fc9a4196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.677+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9a4100420 0x7fc9a4196970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.677+0000 7fc9a37fe700 1 -- 172.21.15.2:0/970145922 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9940095f0 con 0x7fc9a40ffa70 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.677+0000 7fc9a3fff700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9a4105250 0x7fc9a419add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.677+0000 7fc9a2ffd700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9a4100420 0x7fc9a4196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.677+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9a40ffa70 0x7fc9a4196430 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fc994005910 tx=0x7fc994004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.677+0000 7fc9a0ff9700 1 -- 172.21.15.2:0/970145922 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994021070 con 0x7fc9a40ffa70 2023-11-16T18:28:02.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.677+0000 7fc9a0ff9700 1 -- 172.21.15.2:0/970145922 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc99400bcf0 con 0x7fc9a40ffa70 2023-11-16T18:28:02.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.677+0000 7fc9a9e2c700 1 -- 172.21.15.2:0/970145922 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a410f200 con 0x7fc9a40ffa70 2023-11-16T18:28:02.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.677+0000 7fc9a0ff9700 1 -- 172.21.15.2:0/970145922 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994027680 con 0x7fc9a40ffa70 2023-11-16T18:28:02.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.677+0000 7fc9a9e2c700 1 -- 172.21.15.2:0/970145922 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a410f070 con 0x7fc9a40ffa70 2023-11-16T18:28:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.679+0000 7fc9a0ff9700 1 -- 172.21.15.2:0/970145922 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fc994025070 con 0x7fc9a40ffa70 2023-11-16T18:28:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.680+0000 7fc9a0ff9700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc98c0a12c0 0x7fc98c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.680+0000 7fc9a0ff9700 1 -- 172.21.15.2:0/970145922 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc9940d24c0 con 0x7fc9a40ffa70 2023-11-16T18:28:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.680+0000 7fc9a2ffd700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc98c0a12c0 0x7fc98c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.680+0000 7fc9a2ffd700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc98c0a12c0 0x7fc98c0a3770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4197a40 tx=0x7fc998009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.681+0000 7fc9b5fb8b80 1 Processor -- start 2023-11-16T18:28:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.681+0000 7fc9b5fb8b80 1 -- start start 2023-11-16T18:28:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.681+0000 7fc9b5fb8b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34ab60 0x55714c34af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.681+0000 7fc9b5fb8b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34b470 0x55714c34f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.681+0000 7fc9b5fb8b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34f8b0 0x55714c34fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.681+0000 7fc9b5fb8b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55714c250060 con 0x55714c34f8b0 2023-11-16T18:28:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.681+0000 7fc9b5fb8b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55714c2a28a0 con 0x55714c34b470 2023-11-16T18:28:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.681+0000 7fc9b5fb8b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55714c29bd30 con 0x55714c34ab60 2023-11-16T18:28:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.681+0000 7fc9a3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34f8b0 0x55714c34fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.681+0000 7fc9a2ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34b470 0x55714c34f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.681+0000 7fc9a37fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34ab60 0x55714c34af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.681+0000 7fc9a3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34f8b0 0x55714c34fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52174/0 (socket says 172.21.15.2:52174) 2023-11-16T18:28:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.681+0000 7fc9a3fff700 1 -- 172.21.15.2:0/2088314873 learned_addr learned my addr 172.21.15.2:0/2088314873 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.682+0000 7fc9a37fe700 1 -- 172.21.15.2:0/2088314873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34b470 msgr2=0x55714c34f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.682+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/2088314873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34b470 0x55714c34f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.682+0000 7fc9a37fe700 1 -- 172.21.15.2:0/2088314873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34f8b0 msgr2=0x55714c34fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:02.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.682+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/2088314873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34f8b0 0x55714c34fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.682+0000 7fc9a37fe700 1 -- 172.21.15.2:0/2088314873 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55714c11d630 con 0x55714c34ab60 2023-11-16T18:28:02.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.682+0000 7fc9a3fff700 1 --2- 172.21.15.2:0/2088314873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34f8b0 0x55714c34fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:28:02.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.682+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/2088314873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34ab60 0x55714c34af30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc9940cf8a0 tx=0x7fc994009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deb020187f59457e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:02.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.682+0000 7fc989ffb700 1 -- 172.21.15.2:0/2088314873 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99409ec50 con 0x55714c34ab60 2023-11-16T18:28:02.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.682+0000 7fc989ffb700 1 -- 172.21.15.2:0/2088314873 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc99400be60 con 0x55714c34ab60 2023-11-16T18:28:02.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.682+0000 7fc989ffb700 1 -- 172.21.15.2:0/2088314873 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9940d2100 con 0x55714c34ab60 2023-11-16T18:28:02.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.683+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/2088314873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34ab60 msgr2=0x55714c34af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:02.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.683+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/2088314873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34ab60 0x55714c34af30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc9940cf8a0 tx=0x7fc994009ca0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.683+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/2088314873 shutdown_connections 2023-11-16T18:28:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.683+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/2088314873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34ab60 0x55714c34af30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.683+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/2088314873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34f8b0 0x55714c34fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.683+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/2088314873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34b470 0x55714c34f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.683+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/2088314873 >> 172.21.15.2:0/2088314873 conn(0x55714c340360 msgr2=0x55714c345b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.683+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/2088314873 shutdown_connections 2023-11-16T18:28:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.683+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/2088314873 wait complete. 2023-11-16T18:28:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.683+0000 7fc9b5fb8b80 1 -- start start 2023-11-16T18:28:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.683+0000 7fc9b5fb8b80 1 Processor -- start 2023-11-16T18:28:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9b5fb8b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34ab60 0x55714c356f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9b5fb8b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34b470 0x55714c354120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9b5fb8b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34f8b0 0x55714c354660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9b5fb8b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55714c23d8e0 con 0x55714c34ab60 2023-11-16T18:28:02.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9b5fb8b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55714c23d580 con 0x55714c34f8b0 2023-11-16T18:28:02.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9b5fb8b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55714c11f920 con 0x55714c34b470 2023-11-16T18:28:02.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9b5fb8b80 10 client.0 _ll_register_callbacks cb 0x55714c25b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:28:02.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9a37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34ab60 0x55714c356f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9a2ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34b470 0x55714c354120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9a37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34ab60 0x55714c356f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52176/0 (socket says 172.21.15.2:52176) 2023-11-16T18:28:02.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9a2ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34b470 0x55714c354120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43706/0 (socket says 172.21.15.2:43706) 2023-11-16T18:28:02.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9a37fe700 1 -- 172.21.15.2:0/1878007537 learned_addr learned my addr 172.21.15.2:0/1878007537 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:02.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9a3fff700 1 --2- 172.21.15.2:0/1878007537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34f8b0 0x55714c354660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9a2ffd700 1 -- 172.21.15.2:0/1878007537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34f8b0 msgr2=0x55714c354660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:02.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9a2ffd700 1 --2- 172.21.15.2:0/1878007537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34f8b0 0x55714c354660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9a2ffd700 1 -- 172.21.15.2:0/1878007537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34ab60 msgr2=0x55714c356f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:02.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9a2ffd700 1 --2- 172.21.15.2:0/1878007537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34ab60 0x55714c356f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9a2ffd700 1 -- 172.21.15.2:0/1878007537 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a410f070 con 0x55714c34b470 2023-11-16T18:28:02.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc9a2ffd700 1 --2- 172.21.15.2:0/1878007537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34b470 0x55714c354120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc998003aa0 tx=0x7fc9980047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45702ddff9e83093 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:02.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc982ffd700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:28:02.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.684+0000 7fc982ffd700 1 -- 172.21.15.2:0/1878007537 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99801fa10 con 0x55714c34b470 2023-11-16T18:28:02.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc982ffd700 1 -- 172.21.15.2:0/1878007537 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc99801fb70 con 0x55714c34b470 2023-11-16T18:28:02.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/1878007537 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55714c33db20 con 0x55714c34b470 2023-11-16T18:28:02.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc982ffd700 1 -- 172.21.15.2:0/1878007537 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc99801fd30 con 0x55714c34b470 2023-11-16T18:28:02.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc9b5fb8b80 10 client.25879 fetch_fsmap learned FSMap version 27 2023-11-16T18:28:02.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/1878007537 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55714c262190 con 0x55714c34b470 2023-11-16T18:28:02.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc982ffd700 1 -- 172.21.15.2:0/1878007537 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55714c33db20 con 0x55714c34b470 2023-11-16T18:28:02.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc982ffd700 1 -- 172.21.15.2:0/1878007537 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fc9980273e0 con 0x55714c34b470 2023-11-16T18:28:02.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc9b5fb8b80 10 client.25879 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:28:02.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc9b5fb8b80 10 client.25879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:28:02.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc9b5fb8b80 20 client.25879 populate_metadata read hostname 'smithi002' 2023-11-16T18:28:02.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/1878007537 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55714c364630 0x55714c366a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc9b5fb8b80 4 client.25879 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:28:02.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/1878007537 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55714c25e0a0 con 0x55714c364630 2023-11-16T18:28:02.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.685+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/1878007537 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55714c364630 0x55714c366a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.693 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.686+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/1878007537 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55714c364630 0x55714c366a40 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e065a9ed635b1dd1 server_cookie=a169b83c41d8a81a in_seq=0 out_seq=0 2023-11-16T18:28:02.693 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.686+0000 7fc982ffd700 10 client.25879 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:02.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.689+0000 7fc982ffd700 1 -- 172.21.15.2:0/1878007537 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55714c25e0a0 con 0x55714c364630 2023-11-16T18:28:02.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.689+0000 7fc982ffd700 10 client.25879 handle_command_reply: tid=0 2023-11-16T18:28:02.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.689+0000 7fc9b5fb8b80 1 client.25879 shutdown 2023-11-16T18:28:02.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.689+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/1878007537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34b470 msgr2=0x55714c354120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:02.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.689+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/1878007537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34b470 0x55714c354120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc998003aa0 tx=0x7fc9980047c0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.689+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/1878007537 shutdown_connections 2023-11-16T18:28:02.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.689+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/1878007537 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55714c364630 0x55714c366a40 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.689+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/1878007537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34f8b0 0x55714c354660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.689+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/1878007537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34ab60 0x55714c356f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.689+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/1878007537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34b470 0x55714c354120 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.689+0000 7fc982ffd700 0 client.25879 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:02.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.689+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/1878007537 >> 172.21.15.2:0/1878007537 conn(0x55714c340360 msgr2=0x55714c350d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:02.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.690+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/1878007537 shutdown_connections 2023-11-16T18:28:02.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.690+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/1878007537 wait complete. 2023-11-16T18:28:02.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.690+0000 7fc9b5fb8b80 20 client.25879 trim_cache size 0 max 16384 2023-11-16T18:28:02.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.707+0000 7fc9b5fb8b80 1 Processor -- start 2023-11-16T18:28:02.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.707+0000 7fc9b5fb8b80 1 -- start start 2023-11-16T18:28:02.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.707+0000 7fc9b5fb8b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34ab60 0x55714c35b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.707+0000 7fc9b5fb8b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 0x55714c356a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.707+0000 7fc9b5fb8b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34f8b0 0x55714c345650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.707+0000 7fc9b5fb8b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55714c361b90 con 0x55714c34b470 2023-11-16T18:28:02.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.707+0000 7fc9b5fb8b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55714c360f10 con 0x55714c34ab60 2023-11-16T18:28:02.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.707+0000 7fc9b5fb8b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55714c347350 con 0x55714c34f8b0 2023-11-16T18:28:02.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.707+0000 7fc9a2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 0x55714c356a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.707+0000 7fc9a37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34ab60 0x55714c35b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.707+0000 7fc9a2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 0x55714c356a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52186/0 (socket says 172.21.15.2:52186) 2023-11-16T18:28:02.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.707+0000 7fc9a37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34ab60 0x55714c35b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42622/0 (socket says 172.21.15.2:42622) 2023-11-16T18:28:02.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.707+0000 7fc9a2ffd700 1 -- 172.21.15.2:0/1124277624 learned_addr learned my addr 172.21.15.2:0/1124277624 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:02.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.708+0000 7fc9a37fe700 1 -- 172.21.15.2:0/1124277624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34f8b0 msgr2=0x55714c345650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:02.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.708+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/1124277624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34f8b0 0x55714c345650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.708+0000 7fc9a37fe700 1 -- 172.21.15.2:0/1124277624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 msgr2=0x55714c356a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:02.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.708+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/1124277624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 0x55714c356a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.708+0000 7fc9a37fe700 1 -- 172.21.15.2:0/1124277624 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55714c261ee0 con 0x55714c34ab60 2023-11-16T18:28:02.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.708+0000 7fc9a2ffd700 1 --2- 172.21.15.2:0/1124277624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 0x55714c356a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:28:02.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.708+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/1124277624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34ab60 0x55714c35b750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc99400bff0 tx=0x7fc994000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14f687f036d3b42d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:02.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.708+0000 7fc98a7fc700 1 -- 172.21.15.2:0/1124277624 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9940a76f0 con 0x55714c34ab60 2023-11-16T18:28:02.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.708+0000 7fc98a7fc700 1 -- 172.21.15.2:0/1124277624 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc99409ec10 con 0x55714c34ab60 2023-11-16T18:28:02.710 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.708+0000 7fc98a7fc700 1 -- 172.21.15.2:0/1124277624 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99409f770 con 0x55714c34ab60 2023-11-16T18:28:02.713 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.709+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/1124277624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34ab60 msgr2=0x55714c35b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:02.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.709+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/1124277624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34ab60 0x55714c35b750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc99400bff0 tx=0x7fc994000bc0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.709+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/1124277624 shutdown_connections 2023-11-16T18:28:02.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.709+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/1124277624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34f8b0 0x55714c345650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.709+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/1124277624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34ab60 0x55714c35b750 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.709+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/1124277624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 0x55714c356a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.709+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/1124277624 >> 172.21.15.2:0/1124277624 conn(0x55714c340360 msgr2=0x55714c3540e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:02.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.709+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/1124277624 shutdown_connections 2023-11-16T18:28:02.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.709+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/1124277624 wait complete. 2023-11-16T18:28:02.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.709+0000 7fc9b5fb8b80 1 -- start start 2023-11-16T18:28:02.714 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.709+0000 7fc9b5fb8b80 1 Processor -- start 2023-11-16T18:28:02.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9b5fb8b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34ab60 0x55714c357860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9b5fb8b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 0x55714c357da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9b5fb8b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34f8b0 0x55714c3b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9b5fb8b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55714c3b1a20 con 0x55714c34b470 2023-11-16T18:28:02.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9b5fb8b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55714c3b1b90 con 0x55714c34ab60 2023-11-16T18:28:02.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9b5fb8b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55714c3b1d50 con 0x55714c34f8b0 2023-11-16T18:28:02.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9b5fb8b80 10 client.0 _ll_register_callbacks cb 0x7fc998003680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:28:02.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9a2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 0x55714c357da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9a37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34ab60 0x55714c357860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9a2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 0x55714c357da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52192/0 (socket says 172.21.15.2:52192) 2023-11-16T18:28:02.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9a2ffd700 1 -- 172.21.15.2:0/64315350 learned_addr learned my addr 172.21.15.2:0/64315350 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:02.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9a3fff700 1 --2- 172.21.15.2:0/64315350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34f8b0 0x55714c3b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.715 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9a2ffd700 1 -- 172.21.15.2:0/64315350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34f8b0 msgr2=0x55714c3b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9a2ffd700 1 --2- 172.21.15.2:0/64315350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34f8b0 0x55714c3b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9a2ffd700 1 -- 172.21.15.2:0/64315350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34ab60 msgr2=0x55714c357860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9a2ffd700 1 --2- 172.21.15.2:0/64315350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34ab60 0x55714c357860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9a2ffd700 1 -- 172.21.15.2:0/64315350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55714c262190 con 0x55714c34b470 2023-11-16T18:28:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/64315350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34ab60 0x55714c357860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:28:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc9a2ffd700 1 --2- 172.21.15.2:0/64315350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 0x55714c357da0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fc998027740 tx=0x7fc99800fd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e6a1cd65785af9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.710+0000 7fc982ffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:28:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc982ffd700 1 -- 172.21.15.2:0/64315350 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99800f5d0 con 0x55714c34b470 2023-11-16T18:28:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc982ffd700 1 -- 172.21.15.2:0/64315350 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc99800f730 con 0x55714c34b470 2023-11-16T18:28:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/64315350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc99400be60 con 0x55714c34b470 2023-11-16T18:28:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc982ffd700 1 -- 172.21.15.2:0/64315350 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc99800f8f0 con 0x55714c34b470 2023-11-16T18:28:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc9b5fb8b80 10 client.16449 fetch_fsmap learned FSMap version 27 2023-11-16T18:28:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/64315350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55714c11d630 con 0x55714c34b470 2023-11-16T18:28:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc982ffd700 1 -- 172.21.15.2:0/64315350 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc99400be60 con 0x55714c34b470 2023-11-16T18:28:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc982ffd700 1 -- 172.21.15.2:0/64315350 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fc99801fb60 con 0x55714c34b470 2023-11-16T18:28:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc9b5fb8b80 10 client.16449 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:28:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc9b5fb8b80 10 client.16449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:28:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc9b5fb8b80 20 client.16449 populate_metadata read hostname 'smithi002' 2023-11-16T18:28:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/64315350 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55714c3b95a0 0x55714c3bb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc9b5fb8b80 4 client.16449 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:28:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/64315350 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55714c26a660 con 0x55714c3b95a0 2023-11-16T18:28:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.711+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/64315350 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55714c3b95a0 0x55714c3bb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.712+0000 7fc982ffd700 10 client.16449 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.712+0000 7fc9a37fe700 1 --2- 172.21.15.2:0/64315350 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55714c3b95a0 0x55714c3bb9b0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5942b66ef0a4eb3e server_cookie=a1f178dd60a836f in_seq=0 out_seq=0 2023-11-16T18:28:02.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.715+0000 7fc982ffd700 1 -- 172.21.15.2:0/64315350 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+266 (crc 0 0 0) 0x55714c26a660 con 0x55714c3b95a0 2023-11-16T18:28:02.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.715+0000 7fc982ffd700 10 client.16449 handle_command_reply: tid=0 2023-11-16T18:28:02.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.715+0000 7fc9b5fb8b80 1 client.16449 shutdown 2023-11-16T18:28:02.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.715+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/64315350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 msgr2=0x55714c357da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:02.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.715+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/64315350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 0x55714c357da0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fc998027740 tx=0x7fc99800fd00 comp rx=0 tx=0).stop 2023-11-16T18:28:02.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.715+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/64315350 shutdown_connections 2023-11-16T18:28:02.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.715+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/64315350 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55714c3b95a0 0x55714c3bb9b0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.715+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/64315350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55714c34f8b0 0x55714c3b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.715+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/64315350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55714c34ab60 0x55714c357860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.715+0000 7fc982ffd700 0 client.16449 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:02.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.715+0000 7fc9b5fb8b80 1 --2- 172.21.15.2:0/64315350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55714c34b470 0x55714c357da0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.715+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/64315350 >> 172.21.15.2:0/64315350 conn(0x55714c340360 msgr2=0x55714c345600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:02.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/64315350 shutdown_connections 2023-11-16T18:28:02.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9b5fb8b80 1 -- 172.21.15.2:0/64315350 wait complete. 2023-11-16T18:28:02.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9b5fb8b80 20 client.16449 trim_cache size 0 max 16384 2023-11-16T18:28:02.720 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:28:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "scrub active (35 inodes in the stack)", 2023-11-16T18:28:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": { 2023-11-16T18:28:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "9607aa23-76ae-4198-9378-3740d8aff22c": { 2023-11-16T18:28:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/", 2023-11-16T18:28:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "tag": "9607aa23-76ae-4198-9378-3740d8aff22c", 2023-11-16T18:28:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "options": "recursive,force" 2023-11-16T18:28:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:28:02.721 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:28:02.721 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:28:02.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9a9e2c700 1 -- 172.21.15.2:0/970145922 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc98c0a12c0 msgr2=0x7fc98c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:02.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9a9e2c700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc98c0a12c0 0x7fc98c0a3770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4197a40 tx=0x7fc998009260 comp rx=0 tx=0).stop 2023-11-16T18:28:02.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9a9e2c700 1 -- 172.21.15.2:0/970145922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9a40ffa70 msgr2=0x7fc9a4196430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:02.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9a9e2c700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9a40ffa70 0x7fc9a4196430 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fc994005910 tx=0x7fc994004710 comp rx=0 tx=0).stop 2023-11-16T18:28:02.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9a9e2c700 1 -- 172.21.15.2:0/970145922 shutdown_connections 2023-11-16T18:28:02.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9a9e2c700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc98c0a12c0 0x7fc98c0a3770 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9a9e2c700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9a4105250 0x7fc9a419add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9a9e2c700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9a40ffa70 0x7fc9a4196430 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9a9e2c700 1 --2- 172.21.15.2:0/970145922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9a4100420 0x7fc9a4196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:02.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9a9e2c700 1 -- 172.21.15.2:0/970145922 >> 172.21.15.2:0/970145922 conn(0x7fc9a40763a0 msgr2=0x7fc9a40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:02.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9a9e2c700 1 -- 172.21.15.2:0/970145922 shutdown_connections 2023-11-16T18:28:02.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:02.716+0000 7fc9a9e2c700 1 -- 172.21.15.2:0/970145922 wait complete. 2023-11-16T18:28:02.730 INFO:tasks.cephfs.filesystem:scrub status for tag:9607aa23-76ae-4198-9378-3740d8aff22c - {'path': '/', 'tag': '9607aa23-76ae-4198-9378-3740d8aff22c', 'options': 'recursive,force'} 2023-11-16T18:28:04.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:04 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:04.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:04 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:04.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:04 smithi002 ceph-mon[98807]: pgmap v99: 129 pgs: 129 active+clean; 200 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 7.4 MiB/s wr, 1.20k op/s 2023-11-16T18:28:04.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:04 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:04.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:04 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:04.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:04 smithi040 ceph-mon[100251]: pgmap v99: 129 pgs: 129 active+clean; 200 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 7.4 MiB/s wr, 1.20k op/s 2023-11-16T18:28:04.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:04 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:04.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:04 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:04.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:04 smithi188 ceph-mon[100639]: pgmap v99: 129 pgs: 129 active+clean; 200 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 7.4 MiB/s wr, 1.20k op/s 2023-11-16T18:28:06.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:06 smithi002 ceph-mon[98807]: pgmap v100: 129 pgs: 129 active+clean; 200 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.3 MiB/s wr, 1.06k op/s 2023-11-16T18:28:06.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:06 smithi040 ceph-mon[100251]: pgmap v100: 129 pgs: 129 active+clean; 200 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.3 MiB/s wr, 1.06k op/s 2023-11-16T18:28:06.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:06 smithi188 ceph-mon[100639]: pgmap v100: 129 pgs: 129 active+clean; 200 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.3 MiB/s wr, 1.06k op/s 2023-11-16T18:28:07.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:07 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:28:07.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:07 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:28:07.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:07 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:28:07.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:07 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:28:07.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:07 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:28:07.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:07 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:28:07.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:07 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:28:07.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:07 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:28:07.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:07 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:28:08.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:08 smithi002 ceph-mon[98807]: pgmap v101: 129 pgs: 129 active+clean; 212 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.3 MiB/s wr, 1.07k op/s 2023-11-16T18:28:08.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:08 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:28:08.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:08 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:28:08.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:08 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:28:08.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:08 smithi040 ceph-mon[100251]: pgmap v101: 129 pgs: 129 active+clean; 212 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.3 MiB/s wr, 1.07k op/s 2023-11-16T18:28:08.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:08 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:28:08.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:08 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:28:08.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:08 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:28:08.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:08 smithi188 ceph-mon[100639]: pgmap v101: 129 pgs: 129 active+clean; 212 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.3 MiB/s wr, 1.07k op/s 2023-11-16T18:28:08.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:08 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:28:08.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:08 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:28:08.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:08 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:28:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.224+0000 7fa49b05d700 1 -- 172.21.15.2:0/3653960917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa494105520 msgr2=0x7fa49410d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.224+0000 7fa49b05d700 1 --2- 172.21.15.2:0/3653960917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa494105520 0x7fa49410d0f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00b0d0 tx=0x7fa48c00b3e0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.224+0000 7fa49b05d700 1 -- 172.21.15.2:0/3653960917 shutdown_connections 2023-11-16T18:28:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.224+0000 7fa49b05d700 1 --2- 172.21.15.2:0/3653960917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4941004e0 0x7fa4941008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.224+0000 7fa49b05d700 1 --2- 172.21.15.2:0/3653960917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa494105520 0x7fa49410d0f0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.224+0000 7fa49b05d700 1 --2- 172.21.15.2:0/3653960917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa494100e80 0x7fa494104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.224+0000 7fa49b05d700 1 -- 172.21.15.2:0/3653960917 >> 172.21.15.2:0/3653960917 conn(0x7fa4940fbd50 msgr2=0x7fa4940fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.224+0000 7fa49b05d700 1 -- 172.21.15.2:0/3653960917 shutdown_connections 2023-11-16T18:28:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.224+0000 7fa49b05d700 1 -- 172.21.15.2:0/3653960917 wait complete. 2023-11-16T18:28:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.224+0000 7fa49b05d700 1 Processor -- start 2023-11-16T18:28:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa49b05d700 1 -- start start 2023-11-16T18:28:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa49b05d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4941004e0 0x7fa494072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa49b05d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa494100e80 0x7fa494073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa49b05d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa494105520 0x7fa4941a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa49b05d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa494073b60 con 0x7fa4941004e0 2023-11-16T18:28:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa49b05d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4941a4c10 con 0x7fa494105520 2023-11-16T18:28:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa49b05d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4941a4d50 con 0x7fa494100e80 2023-11-16T18:28:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa498df9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4941004e0 0x7fa494072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa493fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa494100e80 0x7fa494073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa498df9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4941004e0 0x7fa494072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52204/0 (socket says 172.21.15.2:52204) 2023-11-16T18:28:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa498df9700 1 -- 172.21.15.2:0/3101421936 learned_addr learned my addr 172.21.15.2:0/3101421936 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa4995fa700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa494105520 0x7fa4941a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa498df9700 1 -- 172.21.15.2:0/3101421936 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa494100e80 msgr2=0x7fa494073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa498df9700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa494100e80 0x7fa494073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa498df9700 1 -- 172.21.15.2:0/3101421936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa494105520 msgr2=0x7fa4941a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa498df9700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa494105520 0x7fa4941a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa498df9700 1 -- 172.21.15.2:0/3101421936 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48c009d00 con 0x7fa4941004e0 2023-11-16T18:28:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa4995fa700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa494105520 0x7fa4941a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:28:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.225+0000 7fa498df9700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4941004e0 0x7fa494072df0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fa48800e9a0 tx=0x7fa48800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.226+0000 7fa491ffb700 1 -- 172.21.15.2:0/3101421936 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488009790 con 0x7fa4941004e0 2023-11-16T18:28:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.226+0000 7fa491ffb700 1 -- 172.21.15.2:0/3101421936 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa488010430 con 0x7fa4941004e0 2023-11-16T18:28:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.226+0000 7fa49b05d700 1 -- 172.21.15.2:0/3101421936 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa494110340 con 0x7fa4941004e0 2023-11-16T18:28:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.226+0000 7fa491ffb700 1 -- 172.21.15.2:0/3101421936 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488018680 con 0x7fa4941004e0 2023-11-16T18:28:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.226+0000 7fa49b05d700 1 -- 172.21.15.2:0/3101421936 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4941101b0 con 0x7fa4941004e0 2023-11-16T18:28:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.228+0000 7fa491ffb700 1 -- 172.21.15.2:0/3101421936 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fa4880105a0 con 0x7fa4941004e0 2023-11-16T18:28:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.229+0000 7fa491ffb700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa4840a1340 0x7fa4840a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.229+0000 7fa491ffb700 1 -- 172.21.15.2:0/3101421936 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa488014070 con 0x7fa4941004e0 2023-11-16T18:28:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.229+0000 7fa493fff700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa4840a1340 0x7fa4840a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.229+0000 7fa493fff700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa4840a1340 0x7fa4840a37f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa4940745e0 tx=0x7fa480009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.229+0000 7fa4a71e9b80 1 Processor -- start 2023-11-16T18:28:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.229+0000 7fa4a71e9b80 1 -- start start 2023-11-16T18:28:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.229+0000 7fa4a71e9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c4376b0 0x55721c5e11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa4a71e9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1720 0x55721c5db560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa4a71e9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c5e1b90 0x55721c5dbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa4a71e9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55721c4e1000 con 0x55721c4376b0 2023-11-16T18:28:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa4a71e9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55721c533840 con 0x55721c5e1b90 2023-11-16T18:28:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa4a71e9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55721c52ccd0 con 0x55721c5e1720 2023-11-16T18:28:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa498df9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c4376b0 0x55721c5e11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa493fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1720 0x55721c5db560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa498df9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c4376b0 0x55721c5e11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52212/0 (socket says 172.21.15.2:52212) 2023-11-16T18:28:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa498df9700 1 -- 172.21.15.2:0/2450974298 learned_addr learned my addr 172.21.15.2:0/2450974298 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa4995fa700 1 --2- 172.21.15.2:0/2450974298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c5e1b90 0x55721c5dbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa493fff700 1 -- 172.21.15.2:0/2450974298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c5e1b90 msgr2=0x55721c5dbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa493fff700 1 --2- 172.21.15.2:0/2450974298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c5e1b90 0x55721c5dbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa493fff700 1 -- 172.21.15.2:0/2450974298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c4376b0 msgr2=0x55721c5e11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa493fff700 1 --2- 172.21.15.2:0/2450974298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c4376b0 0x55721c5e11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa493fff700 1 -- 172.21.15.2:0/2450974298 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55721c3ae5d0 con 0x55721c5e1720 2023-11-16T18:28:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa498df9700 1 --2- 172.21.15.2:0/2450974298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c4376b0 0x55721c5e11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:28:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa493fff700 1 --2- 172.21.15.2:0/2450974298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1720 0x55721c5db560 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa480009fd0 tx=0x7fa480014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d12b23d309d0bb09 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa47effd700 1 -- 172.21.15.2:0/2450974298 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4800119b0 con 0x55721c5e1720 2023-11-16T18:28:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa47effd700 1 -- 172.21.15.2:0/2450974298 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4800044e0 con 0x55721c5e1720 2023-11-16T18:28:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.230+0000 7fa47effd700 1 -- 172.21.15.2:0/2450974298 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4800039e0 con 0x55721c5e1720 2023-11-16T18:28:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.231+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/2450974298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1720 msgr2=0x55721c5db560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.231+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/2450974298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1720 0x55721c5db560 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa480009fd0 tx=0x7fa480014000 comp rx=0 tx=0).stop 2023-11-16T18:28:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.231+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/2450974298 shutdown_connections 2023-11-16T18:28:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.231+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/2450974298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c5e1b90 0x55721c5dbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.231+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/2450974298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c4376b0 0x55721c5e11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.231+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/2450974298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1720 0x55721c5db560 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.231+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/2450974298 >> 172.21.15.2:0/2450974298 conn(0x55721c5d0f50 msgr2=0x55721c5d6270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.231+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/2450974298 shutdown_connections 2023-11-16T18:28:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.231+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/2450974298 wait complete. 2023-11-16T18:28:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa4a71e9b80 1 -- start start 2023-11-16T18:28:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa4a71e9b80 1 Processor -- start 2023-11-16T18:28:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa4a71e9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5e7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa4a71e9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1720 0x55721c5e4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa4a71e9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1b90 0x55721c5e52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa4a71e9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55721c4ce880 con 0x55721c5e1b90 2023-11-16T18:28:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa4a71e9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55721c4ce520 con 0x55721c4376b0 2023-11-16T18:28:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa4a71e9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55721c3b08c0 con 0x55721c5e1720 2023-11-16T18:28:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa4a71e9b80 10 client.0 _ll_register_callbacks cb 0x55721c4ec720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:28:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa498df9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5e7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa493fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1720 0x55721c5e4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa4995fa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1b90 0x55721c5e52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa498df9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5e7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42656/0 (socket says 172.21.15.2:42656) 2023-11-16T18:28:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa493fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1720 0x55721c5e4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43750/0 (socket says 172.21.15.2:43750) 2023-11-16T18:28:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.232+0000 7fa498df9700 1 -- 172.21.15.2:0/1858044559 learned_addr learned my addr 172.21.15.2:0/1858044559 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa498df9700 1 -- 172.21.15.2:0/1858044559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1720 msgr2=0x55721c5e4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa498df9700 1 --2- 172.21.15.2:0/1858044559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1720 0x55721c5e4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa498df9700 1 -- 172.21.15.2:0/1858044559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1b90 msgr2=0x55721c5e52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa498df9700 1 --2- 172.21.15.2:0/1858044559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1b90 0x55721c5e52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa498df9700 1 -- 172.21.15.2:0/1858044559 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa480017070 con 0x55721c4376b0 2023-11-16T18:28:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa498df9700 1 --2- 172.21.15.2:0/1858044559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5e7a20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa4880ca670 tx=0x7fa488099d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b45dec363308f175 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa473fff700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:28:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa473fff700 1 -- 172.21.15.2:0/1858044559 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48809a3a0 con 0x55721c4376b0 2023-11-16T18:28:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/1858044559 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55721c5ce710 con 0x55721c4376b0 2023-11-16T18:28:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa473fff700 1 -- 172.21.15.2:0/1858044559 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa48809df70 con 0x55721c4376b0 2023-11-16T18:28:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa473fff700 1 -- 172.21.15.2:0/1858044559 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4880a2390 con 0x55721c4376b0 2023-11-16T18:28:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa4a71e9b80 10 client.26009 fetch_fsmap learned FSMap version 27 2023-11-16T18:28:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/1858044559 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55721c4f3130 con 0x55721c4376b0 2023-11-16T18:28:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.233+0000 7fa473fff700 1 -- 172.21.15.2:0/1858044559 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4880ca080 con 0x55721c4376b0 2023-11-16T18:28:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.234+0000 7fa473fff700 1 -- 172.21.15.2:0/1858044559 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fa488099550 con 0x55721c4376b0 2023-11-16T18:28:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.234+0000 7fa4a71e9b80 10 client.26009 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:28:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.234+0000 7fa4a71e9b80 10 client.26009 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:28:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.234+0000 7fa4a71e9b80 20 client.26009 populate_metadata read hostname 'smithi002' 2023-11-16T18:28:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.234+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/1858044559 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55721c5f4fa0 0x55721c5f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.234+0000 7fa4a71e9b80 4 client.26009 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:28:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.234+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/1858044559 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55721c4ef040 con 0x55721c5f4fa0 2023-11-16T18:28:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.234+0000 7fa493fff700 1 --2- 172.21.15.2:0/1858044559 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55721c5f4fa0 0x55721c5f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.234+0000 7fa493fff700 1 --2- 172.21.15.2:0/1858044559 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55721c5f4fa0 0x55721c5f73b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=786b4995b5accb20 server_cookie=965637229324fd39 in_seq=0 out_seq=0 2023-11-16T18:28:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.234+0000 7fa473fff700 10 client.26009 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa473fff700 1 -- 172.21.15.2:0/1858044559 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55721c4ef040 con 0x55721c5f4fa0 2023-11-16T18:28:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa473fff700 10 client.26009 handle_command_reply: tid=0 2023-11-16T18:28:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa4a71e9b80 1 client.26009 shutdown 2023-11-16T18:28:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/1858044559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 msgr2=0x55721c5e7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/1858044559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5e7a20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa4880ca670 tx=0x7fa488099d90 comp rx=0 tx=0).stop 2023-11-16T18:28:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/1858044559 shutdown_connections 2023-11-16T18:28:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/1858044559 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55721c5f4fa0 0x55721c5f73b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/1858044559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1b90 0x55721c5e52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/1858044559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5e7a20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa473fff700 0 client.26009 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/1858044559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1720 0x55721c5e4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/1858044559 >> 172.21.15.2:0/1858044559 conn(0x55721c5d0f50 msgr2=0x55721c5d57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/1858044559 shutdown_connections 2023-11-16T18:28:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/1858044559 wait complete. 2023-11-16T18:28:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.237+0000 7fa4a71e9b80 20 client.26009 trim_cache size 0 max 16384 2023-11-16T18:28:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.252+0000 7fa4a71e9b80 1 Processor -- start 2023-11-16T18:28:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.252+0000 7fa4a71e9b80 1 -- start start 2023-11-16T18:28:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.252+0000 7fa4a71e9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5ec020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.252+0000 7fa4a71e9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 0x55721c5d5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.252+0000 7fa4a71e9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1b90 0x55721c5e75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.252+0000 7fa4a71e9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55721c5e62a0 con 0x55721c5e1720 2023-11-16T18:28:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.252+0000 7fa4a71e9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55721c5e3220 con 0x55721c4376b0 2023-11-16T18:28:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.252+0000 7fa4a71e9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55721c5df970 con 0x55721c5e1b90 2023-11-16T18:28:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.252+0000 7fa493fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 0x55721c5d5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.253+0000 7fa498df9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5ec020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.253+0000 7fa493fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 0x55721c5d5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52228/0 (socket says 172.21.15.2:52228) 2023-11-16T18:28:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.253+0000 7fa493fff700 1 -- 172.21.15.2:0/1775330427 learned_addr learned my addr 172.21.15.2:0/1775330427 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.253+0000 7fa4995fa700 1 --2- 172.21.15.2:0/1775330427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1b90 0x55721c5e75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.253+0000 7fa493fff700 1 -- 172.21.15.2:0/1775330427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1b90 msgr2=0x55721c5e75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.253+0000 7fa493fff700 1 --2- 172.21.15.2:0/1775330427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1b90 0x55721c5e75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.253+0000 7fa493fff700 1 -- 172.21.15.2:0/1775330427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 msgr2=0x55721c5ec020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.253+0000 7fa493fff700 1 --2- 172.21.15.2:0/1775330427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5ec020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.253+0000 7fa493fff700 1 -- 172.21.15.2:0/1775330427 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55721c4f2e80 con 0x55721c5e1720 2023-11-16T18:28:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.253+0000 7fa493fff700 1 --2- 172.21.15.2:0/1775330427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 0x55721c5d5f60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa48000ef00 tx=0x7fa4800092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f26e47108241c714 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.253+0000 7fa47f7fe700 1 -- 172.21.15.2:0/1775330427 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48002b030 con 0x55721c5e1720 2023-11-16T18:28:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.253+0000 7fa47f7fe700 1 -- 172.21.15.2:0/1775330427 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa48002c040 con 0x55721c5e1720 2023-11-16T18:28:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.253+0000 7fa47f7fe700 1 -- 172.21.15.2:0/1775330427 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480005040 con 0x55721c5e1720 2023-11-16T18:28:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.254+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/1775330427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 msgr2=0x55721c5d5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.254+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/1775330427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 0x55721c5d5f60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa48000ef00 tx=0x7fa4800092d0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.254+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/1775330427 shutdown_connections 2023-11-16T18:28:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.254+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/1775330427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1b90 0x55721c5e75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.254+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/1775330427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5ec020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.254+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/1775330427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 0x55721c5d5f60 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.254+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/1775330427 >> 172.21.15.2:0/1775330427 conn(0x55721c5d0f50 msgr2=0x55721c5f12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.254+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/1775330427 shutdown_connections 2023-11-16T18:28:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.254+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/1775330427 wait complete. 2023-11-16T18:28:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.254+0000 7fa4a71e9b80 1 -- start start 2023-11-16T18:28:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.254+0000 7fa4a71e9b80 1 Processor -- start 2023-11-16T18:28:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.254+0000 7fa4a71e9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5ddea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.254+0000 7fa4a71e9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 0x55721c5de3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa4a71e9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1b90 0x55721c5dcc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa4a71e9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55721c5dd300 con 0x55721c5e1720 2023-11-16T18:28:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa4a71e9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55721c5dd470 con 0x55721c4376b0 2023-11-16T18:28:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa4a71e9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55721c5dd630 con 0x55721c5e1b90 2023-11-16T18:28:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa4a71e9b80 10 client.0 _ll_register_callbacks cb 0x55721c4ec720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:28:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa493fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 0x55721c5de3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa498df9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5ddea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa493fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 0x55721c5de3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52234/0 (socket says 172.21.15.2:52234) 2023-11-16T18:28:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa4995fa700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1b90 0x55721c5dcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa493fff700 1 -- 172.21.15.2:0/2161965560 learned_addr learned my addr 172.21.15.2:0/2161965560 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa493fff700 1 -- 172.21.15.2:0/2161965560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1b90 msgr2=0x55721c5dcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa493fff700 1 --2- 172.21.15.2:0/2161965560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1b90 0x55721c5dcc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa493fff700 1 -- 172.21.15.2:0/2161965560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 msgr2=0x55721c5ddea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa493fff700 1 --2- 172.21.15.2:0/2161965560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5ddea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa493fff700 1 -- 172.21.15.2:0/2161965560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55721c3ae5d0 con 0x55721c5e1720 2023-11-16T18:28:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa498df9700 1 --2- 172.21.15.2:0/2161965560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5ddea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:28:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa493fff700 1 --2- 172.21.15.2:0/2161965560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 0x55721c5de3e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa480005950 tx=0x7fa48000e760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964960427d7a406c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa473fff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:28:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa473fff700 1 -- 172.21.15.2:0/2161965560 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48002b070 con 0x55721c5e1720 2023-11-16T18:28:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa473fff700 1 -- 172.21.15.2:0/2161965560 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa48000e990 con 0x55721c5e1720 2023-11-16T18:28:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/2161965560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4800044e0 con 0x55721c5e1720 2023-11-16T18:28:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.255+0000 7fa473fff700 1 -- 172.21.15.2:0/2161965560 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa480032610 con 0x55721c5e1720 2023-11-16T18:28:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.256+0000 7fa4a71e9b80 10 client.16467 fetch_fsmap learned FSMap version 27 2023-11-16T18:28:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.256+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/2161965560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55721c343c00 con 0x55721c5e1720 2023-11-16T18:28:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.256+0000 7fa473fff700 1 -- 172.21.15.2:0/2161965560 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4800044e0 con 0x55721c5e1720 2023-11-16T18:28:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.256+0000 7fa473fff700 1 -- 172.21.15.2:0/2161965560 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fa480024030 con 0x55721c5e1720 2023-11-16T18:28:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.256+0000 7fa4a71e9b80 10 client.16467 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T18:28:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.256+0000 7fa4a71e9b80 10 client.16467 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:28:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.256+0000 7fa4a71e9b80 20 client.16467 populate_metadata read hostname 'smithi002' 2023-11-16T18:28:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.256+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/2161965560 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55721c649b70 0x55721c64bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.256+0000 7fa4a71e9b80 4 client.16467 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:28:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.256+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/2161965560 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55721c4fb600 con 0x55721c649b70 2023-11-16T18:28:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.256+0000 7fa498df9700 1 --2- 172.21.15.2:0/2161965560 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55721c649b70 0x55721c64bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.256+0000 7fa473fff700 10 client.16467 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:09.256+0000 7fa498df9700 1 --2- 172.21.15.2:0/2161965560 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55721c649b70 0x55721c64bf80 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0b9ca4e083e1d32 server_cookie=b9e4aad33f866782 in_seq=0 out_seq=0 2023-11-16T18:28:09.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:09 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:28:09.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:09 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:28:09.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:09 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:28:09.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:09 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:28:09.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:09 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:28:09.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:09 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:28:09.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:09 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:28:09.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:09 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:28:09.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:09 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:28:09.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:09 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:28:09.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:09 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:28:09.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:09 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:28:10.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:10 smithi002 ceph-mon[98807]: pgmap v102: 129 pgs: 129 active+clean; 253 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 10 MiB/s wr, 1.01k op/s 2023-11-16T18:28:10.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:10 smithi040 ceph-mon[100251]: pgmap v102: 129 pgs: 129 active+clean; 253 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 10 MiB/s wr, 1.01k op/s 2023-11-16T18:28:10.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:10 smithi188 ceph-mon[100639]: pgmap v102: 129 pgs: 129 active+clean; 253 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 10 MiB/s wr, 1.01k op/s 2023-11-16T18:28:12.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:12 smithi002 ceph-mon[98807]: pgmap v103: 129 pgs: 129 active+clean; 253 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 7.1 MiB/s wr, 44 op/s 2023-11-16T18:28:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:12 smithi040 ceph-mon[100251]: pgmap v103: 129 pgs: 129 active+clean; 253 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 7.1 MiB/s wr, 44 op/s 2023-11-16T18:28:12.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:12 smithi188 ceph-mon[100639]: pgmap v103: 129 pgs: 129 active+clean; 253 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 7.1 MiB/s wr, 44 op/s 2023-11-16T18:28:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa473fff700 1 -- 172.21.15.2:0/2161965560 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55721c4fb600 con 0x55721c649b70 2023-11-16T18:28:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa473fff700 10 client.16467 handle_command_reply: tid=0 2023-11-16T18:28:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa4a71e9b80 1 client.16467 shutdown 2023-11-16T18:28:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/2161965560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 msgr2=0x55721c5de3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/2161965560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 0x55721c5de3e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa480005950 tx=0x7fa48000e760 comp rx=0 tx=0).stop 2023-11-16T18:28:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/2161965560 shutdown_connections 2023-11-16T18:28:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/2161965560 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55721c649b70 0x55721c64bf80 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/2161965560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55721c5e1b90 0x55721c5dcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/2161965560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55721c4376b0 0x55721c5ddea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa473fff700 0 client.16467 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa4a71e9b80 1 --2- 172.21.15.2:0/2161965560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55721c5e1720 0x55721c5de3e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/2161965560 >> 172.21.15.2:0/2161965560 conn(0x55721c5d0f50 msgr2=0x55721c5e1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/2161965560 shutdown_connections 2023-11-16T18:28:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa4a71e9b80 1 -- 172.21.15.2:0/2161965560 wait complete. 2023-11-16T18:28:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.477+0000 7fa4a71e9b80 20 client.16467 trim_cache size 0 max 16384 2023-11-16T18:28:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:28:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:28:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:28:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:28:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.478+0000 7fa49b05d700 1 -- 172.21.15.2:0/3101421936 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa4840a1340 msgr2=0x7fa4840a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.478+0000 7fa49b05d700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa4840a1340 0x7fa4840a37f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa4940745e0 tx=0x7fa480009260 comp rx=0 tx=0).stop 2023-11-16T18:28:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.478+0000 7fa49b05d700 1 -- 172.21.15.2:0/3101421936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4941004e0 msgr2=0x7fa494072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.478+0000 7fa49b05d700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4941004e0 0x7fa494072df0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fa48800e9a0 tx=0x7fa48800ecb0 comp rx=0 tx=0).stop 2023-11-16T18:28:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.478+0000 7fa49b05d700 1 -- 172.21.15.2:0/3101421936 shutdown_connections 2023-11-16T18:28:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.478+0000 7fa49b05d700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa4840a1340 0x7fa4840a37f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.478+0000 7fa49b05d700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa494105520 0x7fa4941a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.478+0000 7fa49b05d700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4941004e0 0x7fa494072df0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.478+0000 7fa49b05d700 1 --2- 172.21.15.2:0/3101421936 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa494100e80 0x7fa494073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.478+0000 7fa49b05d700 1 -- 172.21.15.2:0/3101421936 >> 172.21.15.2:0/3101421936 conn(0x7fa4940fbd50 msgr2=0x7fa4940fc750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.478+0000 7fa49b05d700 1 -- 172.21.15.2:0/3101421936 shutdown_connections 2023-11-16T18:28:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:13.478+0000 7fa49b05d700 1 -- 172.21.15.2:0/3101421936 wait complete. 2023-11-16T18:28:14.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:14 smithi002 ceph-mon[98807]: pgmap v104: 129 pgs: 129 active+clean; 300 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 275 op/s 2023-11-16T18:28:14.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:14 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:14.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:14 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:14 smithi040 ceph-mon[100251]: pgmap v104: 129 pgs: 129 active+clean; 300 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 275 op/s 2023-11-16T18:28:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:14 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:14 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:14.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:14 smithi188 ceph-mon[100639]: pgmap v104: 129 pgs: 129 active+clean; 300 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 275 op/s 2023-11-16T18:28:14.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:14 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:14.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:14 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:16.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:16 smithi002 ceph-mon[98807]: pgmap v105: 129 pgs: 129 active+clean; 300 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 262 op/s 2023-11-16T18:28:16.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:16 smithi040 ceph-mon[100251]: pgmap v105: 129 pgs: 129 active+clean; 300 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 262 op/s 2023-11-16T18:28:16.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:16 smithi188 ceph-mon[100639]: pgmap v105: 129 pgs: 129 active+clean; 300 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 262 op/s 2023-11-16T18:28:17.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:17 smithi188 ceph-mon[100639]: pgmap v106: 129 pgs: 129 active+clean; 296 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 265 op/s 2023-11-16T18:28:17.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:17 smithi002 ceph-mon[98807]: pgmap v106: 129 pgs: 129 active+clean; 296 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 265 op/s 2023-11-16T18:28:17.646 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:17 smithi040 ceph-mon[100251]: pgmap v106: 129 pgs: 129 active+clean; 296 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 265 op/s 2023-11-16T18:28:17.983 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:17 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:17.643+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000038c2 [5,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/file9398 auth{0=1} v16377 ap=1 s=0 n(v0 rc2023-11-16T18:28:00.308224+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25006=0-4194304@5} caps={25006=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25006 | importingcaps=0 caps=1 replicated=1 authpin=1 scrubqueue=0 0x55d214913600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000038c2:[<0x10000000273/file9398 v16377>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x100000038c2:[<0x10000000273/file9398 v16377>,<0x10000000272/dir4 v14772>,<0x10000000271/dir3 v43468>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v22390>,<0x1000000026e/testdir v248>,<0x10000000005/tmp v349>,<0x100000001fb/client.0 v309>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v302>,<0x10000000001/sv_1 v319>,<0x10000000000/_nogroup v298>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:18.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:18 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:18.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:18 smithi040 ceph-mon[100251]: Scrub error on inode 0x100000038c2 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/file9398) see mds.h log and `damage ls` output for details 2023-11-16T18:28:18.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:18 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:18.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:18 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:18.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:18 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:18.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:18 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:18.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:18 smithi188 ceph-mon[100639]: Scrub error on inode 0x100000038c2 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/file9398) see mds.h log and `damage ls` output for details 2023-11-16T18:28:18.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:18 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:18.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:18 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:18.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:18 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:18.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:18 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:18.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:18 smithi002 ceph-mon[98807]: Scrub error on inode 0x100000038c2 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/file9398) see mds.h log and `damage ls` output for details 2023-11-16T18:28:18.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:18 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:18.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:18 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:18.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:18 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:19.399 INFO:tasks.workunit.client.0.smithi002.stdout:created hierarchy, now cleaning up 2023-11-16T18:28:19.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:19 smithi040 ceph-mon[100251]: pgmap v107: 129 pgs: 129 active+clean; 286 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 271 op/s 2023-11-16T18:28:19.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:19 smithi188 ceph-mon[100639]: pgmap v107: 129 pgs: 129 active+clean; 286 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 271 op/s 2023-11-16T18:28:19.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:19 smithi002 ceph-mon[98807]: pgmap v107: 129 pgs: 129 active+clean; 286 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 271 op/s 2023-11-16T18:28:20.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:20.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:20.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:20.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:20.864 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.747 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.748 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.749 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.750 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.750 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.750 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.750 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.750 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.750 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.750 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.750 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.750 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.750 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.751 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.752 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.752 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.752 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.752 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.752 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.752 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.752 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.752 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.752 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.753 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.753 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.753 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.753 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.753 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.753 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.753 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.754 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.754 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.754 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.754 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.754 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.754 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.755 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.756 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.756 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.756 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.756 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.756 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.756 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.756 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.756 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.756 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.756 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.757 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.757 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.757 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.757 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.757 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.757 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.757 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.757 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.757 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.757 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.757 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.757 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.757 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.758 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.758 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.758 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.758 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.758 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.758 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.758 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.793 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.794 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.794 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.794 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.794 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.794 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.794 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.794 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.794 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.794 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.794 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.794 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.795 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.795 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.795 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.795 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.795 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.795 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.795 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.795 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.795 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.795 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.795 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.795 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.796 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.796 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.796 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.796 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.796 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.796 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.796 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.796 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.797 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.797 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.797 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.797 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.797 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.797 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.797 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.797 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.797 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.797 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.798 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.798 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.798 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.798 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.798 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: pgmap v108: 129 pgs: 129 active+clean; 286 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.0 MiB/s wr, 245 op/s 2023-11-16T18:28:21.798 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.798 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.798 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.798 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.798 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.798 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.798 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.798 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.799 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.799 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.799 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.799 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.799 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.799 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.799 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.799 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.799 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.799 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.799 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.799 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.799 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.800 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.800 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.800 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.800 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.800 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.800 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.800 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.800 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.800 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.816 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.817 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.817 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.817 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.817 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.817 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.817 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.817 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.817 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.817 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.817 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.817 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.817 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.818 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.818 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.818 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.818 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.818 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.818 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.818 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.818 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.818 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.819 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.819 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.819 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.819 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.819 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.819 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.819 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.819 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.819 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.819 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.819 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.819 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.819 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.821 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.821 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.821 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.821 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.821 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.821 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.821 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.821 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:21.823 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.823 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.823 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.823 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.823 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.823 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.825 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.825 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.825 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.825 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.825 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.825 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.826 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.861 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.861 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.861 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.862 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.862 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.862 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.862 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.862 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.863 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.863 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.863 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.863 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.863 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.863 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.868 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.868 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.868 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.868 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.868 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.868 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.868 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.868 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.868 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.868 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.868 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.868 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.868 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.869 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.869 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.869 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.869 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.869 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.869 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.869 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.869 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: pgmap v108: 129 pgs: 129 active+clean; 286 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.0 MiB/s wr, 245 op/s 2023-11-16T18:28:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.958 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.958 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.958 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.958 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.958 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.958 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:21 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:21.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.995 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.995 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.995 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.998 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.998 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.998 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.998 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:21.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:21.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.000 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.000 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.000 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.000 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.000 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.000 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.000 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.000 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.000 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.000 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.000 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.000 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.000 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.020 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.020 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: pgmap v108: 129 pgs: 129 active+clean; 286 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.0 MiB/s wr, 245 op/s 2023-11-16T18:28:22.020 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.020 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.020 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.020 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.020 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.097 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.098 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:21 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.359 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:21 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:21.739+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000038a6 [5,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9950 auth{0=1} v873 ap=1 s=0 n(v0 rc2023-11-16T18:27:52.387401+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25006=0-4194304@5} caps={25006=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=25006 | importingcaps=0 caps=1 replicated=1 authpin=1 scrubqueue=0 0x55d21c8c3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000038a6:[<0x10000000274/file9950 v873>,<0x10000000273/dir5 v38269>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x100000038a6:[<0x10000000274/file9950 v873>,<0x10000000273/dir5 v43359>,<0x10000000272/dir4 v14999>,<0x10000000271/dir3 v45975>,<0x10000000270/dir2 v45165>,<0x1000000026f/dir1 v24470>,<0x1000000026e/testdir v265>,<0x10000000005/tmp v364>,<0x100000001fb/client.0 v325>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v302>,<0x10000000001/sv_1 v335>,<0x10000000000/_nogroup v298>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:22.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.736 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.739 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: Scrub error on inode 0x100000038a6 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9950) see mds.h log and `damage ls` output for details 2023-11-16T18:28:22.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.740 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.743 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.744 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.744 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.744 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.744 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.744 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.744 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.744 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:22.744 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: Scrub error on inode 0x100000038a6 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9950) see mds.h log and `damage ls` output for details 2023-11-16T18:28:22.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.826 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.826 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:22.906 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: Scrub error on inode 0x100000038a6 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9950) see mds.h log and `damage ls` output for details 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.945 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.959 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.959 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.959 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.959 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.959 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.959 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.959 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.962 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.962 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.962 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.962 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.962 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:22.962 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:22.962 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:23.233 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:22 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:22.912+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000038c3 [6,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9977 auth{0=1} v890 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25006=0-4194304@5} caps={25006=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=25006 | importingcaps=0 caps=1 replicated=1 authpin=1 scrubqueue=0 0x55d21a73cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000038c3:[<0x10000000274/file9977 v890>,<0x10000000273/dir5 v38269>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x100000038c3:[<0x10000000274/file9977 v890>,<0x10000000273/dir5 v43359>,<0x10000000272/dir4 v15025>,<0x10000000271/dir3 v46389>,<0x10000000270/dir2 v45535>,<0x1000000026f/dir1 v24470>,<0x1000000026e/testdir v265>,<0x10000000005/tmp v364>,<0x100000001fb/client.0 v325>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v302>,<0x10000000001/sv_1 v335>,<0x10000000000/_nogroup v298>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:23.733 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:23.428+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000038ac [5,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9956 auth{0=1} v884 ap=1 s=0 n(v0 rc2023-11-16T18:27:52.407401+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25006=0-4194304@5} caps={25006=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=25006 | importingcaps=0 caps=1 replicated=1 authpin=1 scrubqueue=0 0x55d216829600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000038ac:[<0x10000000274/file9956 v884>,<0x10000000273/dir5 v38269>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x100000038ac:[<0x10000000274/file9956 v884>,<0x10000000273/dir5 v43359>,<0x10000000272/dir4 v15025>,<0x10000000271/dir3 v46389>,<0x10000000270/dir2 v45535>,<0x1000000026f/dir1 v24470>,<0x1000000026e/testdir v265>,<0x10000000005/tmp v364>,<0x100000001fb/client.0 v325>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v302>,<0x10000000001/sv_1 v335>,<0x10000000000/_nogroup v298>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:23.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:23.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:23.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:23.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:23.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:23.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:23.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:23.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:23.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:23.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:23.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:23.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:23.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:23.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: Scrub error on inode 0x100000038c3 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9977) see mds.h log and `damage ls` output for details 2023-11-16T18:28:24.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.867 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.868 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.871 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.871 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.913 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.914 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.914 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.914 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.914 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.914 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.914 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.914 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.914 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.915 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.915 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.915 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.915 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.915 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.915 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.915 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.915 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.915 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.916 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.916 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.916 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.916 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.916 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.916 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.916 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.916 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.917 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.974 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.974 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.978 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.978 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.978 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.978 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.980 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: pgmap v109: 129 pgs: 129 active+clean; 312 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 462 op/s 2023-11-16T18:28:24.980 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.980 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:24.981 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:24.982 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.060 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.063 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.063 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.063 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.063 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.063 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:25.063 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:23 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.200 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:24.707+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000038af [5,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9959 auth{0=1} v853 ap=1 s=0 n(v0 rc2023-11-16T18:27:52.416401+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25006=0-4194304@5} caps={25006=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=25006 | importingcaps=0 caps=1 replicated=1 authpin=1 scrubqueue=0 0x55d21c862c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000038af:[<0x10000000274/file9959 v853>,<0x10000000273/dir5 v38269>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x100000038af:[<0x10000000274/file9959 v853>,<0x10000000273/dir5 v43359>,<0x10000000272/dir4 v15025>,<0x10000000271/dir3 v46389>,<0x10000000270/dir2 v45535>,<0x1000000026f/dir1 v24470>,<0x1000000026e/testdir v265>,<0x10000000005/tmp v364>,<0x100000001fb/client.0 v325>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v302>,<0x10000000001/sv_1 v335>,<0x10000000000/_nogroup v298>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:25.200 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:24.820+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000038a2 [5,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9946 auth{0=1} v874 ap=1 s=0 n(v0 rc2023-11-16T18:27:52.377401+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25006=0-4194304@5} caps={25006=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=25006 | importingcaps=0 caps=1 replicated=1 authpin=1 scrubqueue=0 0x55d21b727180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000038a2:[<0x10000000274/file9946 v874>,<0x10000000273/dir5 v38269>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x100000038a2:[<0x10000000274/file9946 v874>,<0x10000000273/dir5 v43359>,<0x10000000272/dir4 v15025>,<0x10000000271/dir3 v46389>,<0x10000000270/dir2 v45535>,<0x1000000026f/dir1 v24470>,<0x1000000026e/testdir v265>,<0x10000000005/tmp v364>,<0x100000001fb/client.0 v325>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v302>,<0x10000000001/sv_1 v335>,<0x10000000000/_nogroup v298>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:25.201 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-a[98783]: 2023-11-16T18:28:23.473+0000 7f30d8773700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2023-11-16T18:28:25.201 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.201 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.202 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.202 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.202 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.202 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.202 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: Scrub error on inode 0x100000038c3 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9977) see mds.h log and `damage ls` output for details 2023-11-16T18:28:25.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.297 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.297 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.297 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.297 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.297 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.297 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.297 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.297 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.297 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.297 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.297 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.298 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.298 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.298 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.298 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.298 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.298 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.298 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.298 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.298 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.298 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.298 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.298 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.298 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.300 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.300 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.300 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.300 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.300 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.332 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.332 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.333 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.333 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.333 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.333 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.333 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.333 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: pgmap v109: 129 pgs: 129 active+clean; 312 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 462 op/s 2023-11-16T18:28:25.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:25.336 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:25.336 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.724 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.727 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.727 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.727 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.727 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.727 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.727 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.727 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.727 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.727 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:23 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: Scrub error on inode 0x100000038ac (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9956) see mds.h log and `damage ls` output for details 2023-11-16T18:28:26.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2023-11-16T18:28:26.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] up:active 2023-11-16T18:28:26.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:28:26.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.909 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.910 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.911 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.911 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.911 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.911 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:26.911 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:26.911 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.037 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.037 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.037 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.037 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.037 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.037 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.037 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.037 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.037 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.038 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.038 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.038 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.038 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.038 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.038 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.038 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.039 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.039 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.039 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.039 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.039 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.039 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.039 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.039 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.039 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.123 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.123 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.123 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.123 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.123 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.123 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:24 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:27.125 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.125 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.125 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.125 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.125 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.125 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.125 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.125 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.125 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.125 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.286 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.286 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.286 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.286 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.286 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.286 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.286 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: Scrub error on inode 0x100000038c3 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9977) see mds.h log and `damage ls` output for details 2023-11-16T18:28:27.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.350 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.350 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: pgmap v109: 129 pgs: 129 active+clean; 312 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 462 op/s 2023-11-16T18:28:27.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.620 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.620 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.697 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.697 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.697 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.698 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.698 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.698 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:27.698 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:27.699 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.586 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.586 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.593 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.593 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.594 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.628 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.654 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.654 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.654 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.655 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.655 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.655 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.655 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.656 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.657 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.658 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.678 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.678 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.678 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.678 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:23 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: Scrub error on inode 0x100000038ac (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9956) see mds.h log and `damage ls` output for details 2023-11-16T18:28:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2023-11-16T18:28:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] up:active 2023-11-16T18:28:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:28:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.751 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.751 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.751 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.753 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.753 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.754 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.754 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.755 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.778 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.778 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.778 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.780 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.781 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.781 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.781 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.781 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.781 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.781 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.781 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.781 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.781 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.781 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.781 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.781 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.781 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.782 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.782 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.782 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.782 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.782 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.783 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.783 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.783 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:24 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:38.784 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:28:38.789 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.789 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.789 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.802 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: Scrub error on inode 0x100000038af (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9959) see mds.h log and `damage ls` output for details 2023-11-16T18:28:38.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: Scrub error on inode 0x100000038a2 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9946) see mds.h log and `damage ls` output for details 2023-11-16T18:28:38.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: pgmap v110: 129 pgs: 129 active+clean; 312 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.6 MiB/s wr, 231 op/s 2023-11-16T18:28:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.835 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.835 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.835 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.836 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.836 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.836 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.836 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.934 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.934 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.934 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.934 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.934 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.934 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.934 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.934 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.934 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.934 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.940 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.940 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.940 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.940 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.940 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.940 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.941 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.941 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.941 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.941 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.941 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.941 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.943 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.945 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.945 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.945 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.945 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.945 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.945 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.945 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.945 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.945 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.945 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.945 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.946 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.946 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.946 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.946 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.946 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.946 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.946 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.946 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.946 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.948 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.948 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.948 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:38.949 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.949 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.949 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.949 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: Scrub error on inode 0x100000038ac (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9956) see mds.h log and `damage ls` output for details 2023-11-16T18:28:38.949 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.949 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.949 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.949 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.949 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.949 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.949 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2023-11-16T18:28:38.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] up:active 2023-11-16T18:28:38.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:28:38.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.951 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.952 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.955 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.955 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.955 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.955 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.955 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.955 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.955 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.955 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.955 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.955 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.955 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.955 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.955 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.956 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.956 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.956 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.956 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.956 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.956 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.956 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.956 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.956 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.956 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.956 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.956 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.956 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.957 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.958 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.958 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.958 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.958 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.958 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.958 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.958 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.958 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.958 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.958 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:24 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.958 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.958 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.959 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.959 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.959 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.959 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.959 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.959 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.959 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.959 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.959 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.959 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.959 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.960 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.960 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.960 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.960 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.960 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.960 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.960 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.960 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.960 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.961 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.961 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.961 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.961 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.961 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.961 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.961 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.961 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.961 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.961 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.961 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.967 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.967 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.967 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.967 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.967 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.967 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.967 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.967 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.967 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.968 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.968 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.968 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:38.968 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:38.968 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.015 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.047 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.047 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.048 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.048 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.048 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.048 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.048 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.048 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.048 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.049 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.049 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.049 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.049 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.049 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.049 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.050 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.050 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.050 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.050 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.050 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.050 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.052 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.053 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.054 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.055 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.055 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.055 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.055 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.055 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.055 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.055 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.055 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.055 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.055 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.056 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.056 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.056 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.056 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.056 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.056 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.056 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.056 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.056 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.056 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.057 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.057 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.057 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.057 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.057 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.057 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.058 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.058 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.059 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.059 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.060 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.060 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.060 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.060 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.060 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.060 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.061 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.062 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.063 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.063 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.063 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.063 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.063 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.063 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.064 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.064 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.064 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.064 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.064 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.064 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.064 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.065 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.065 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.065 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.065 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.065 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.065 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.066 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.066 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.066 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.066 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.066 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.066 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.067 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.067 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.067 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.067 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.067 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.067 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.067 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.068 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.068 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.068 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: Scrub error on inode 0x100000038af (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9959) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.068 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.068 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.068 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.069 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.069 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.069 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.069 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.069 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.069 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.070 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.070 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.070 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.071 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.071 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.071 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.071 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.071 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.072 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.072 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.072 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.072 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.072 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.072 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.073 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.073 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.073 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.073 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.073 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.073 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.073 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.074 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.074 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.074 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.074 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.074 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.074 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.075 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.075 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.075 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.075 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.075 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.075 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.076 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.076 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.076 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.076 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.076 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.076 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.076 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.077 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.077 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.077 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.077 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.077 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.077 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.078 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.078 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.078 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.087 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.113 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.113 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.113 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.124 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.124 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.124 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.124 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.124 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.124 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.125 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.125 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.125 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.125 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.125 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.125 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.125 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.125 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.125 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.126 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.126 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.126 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.126 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.126 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.126 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.126 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.126 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.127 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.127 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.127 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.127 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.127 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.127 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.127 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.127 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.128 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.128 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.128 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.128 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.128 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.128 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.128 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.128 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: Scrub error on inode 0x100000038a2 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9946) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.128 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.129 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.129 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.129 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.129 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.129 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.129 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.129 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.178 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.178 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.194 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.194 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.194 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.194 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.194 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.194 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.194 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.194 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.194 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.194 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.194 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.194 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.195 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.196 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.197 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.198 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.199 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.199 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.199 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.199 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.199 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.199 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.214 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.215 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: pgmap v110: 129 pgs: 129 active+clean; 312 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.6 MiB/s wr, 231 op/s 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.234 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.235 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.237 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.238 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.239 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.239 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.239 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.239 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.239 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.239 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.239 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.239 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.239 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.242 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.242 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.242 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.242 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.242 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.242 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.242 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.242 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.242 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.259 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.259 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.259 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.259 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.261 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.261 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.261 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.261 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.261 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.261 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.261 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.261 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.261 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.262 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.263 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.263 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.263 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.263 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.263 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.264 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.264 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.265 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.265 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.265 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.265 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.265 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.265 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.265 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.265 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.265 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.265 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.265 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.266 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.266 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.266 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.266 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.266 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.266 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.266 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.266 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.266 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.266 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.266 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.266 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.267 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.267 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.267 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.267 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.267 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.267 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.268 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.269 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.295 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.295 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.295 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.295 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.295 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.295 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.295 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.295 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.295 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.296 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.297 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.298 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.298 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.298 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.298 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.298 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.298 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.298 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.298 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.298 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.299 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.299 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.299 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.306 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.306 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.306 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.307 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.307 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.307 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.307 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.307 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.308 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.308 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.308 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.308 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.309 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.309 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.309 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.309 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.310 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.310 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.310 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.311 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.311 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.311 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.312 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.312 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.312 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.312 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.313 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.313 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.313 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.313 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.313 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.313 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.313 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.313 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.313 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.313 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.314 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.314 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.330 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.330 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.330 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.331 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.331 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.333 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.333 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.334 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.334 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.334 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.335 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.335 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.336 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.336 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.336 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.337 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.337 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.337 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.338 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.338 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.338 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.339 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.339 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.339 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.340 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.340 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.341 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.341 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.341 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.341 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.342 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.342 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.343 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.343 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.343 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.343 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.344 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.344 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.344 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.344 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.345 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.345 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.345 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.346 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.346 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.346 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.347 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.347 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.348 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.348 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.348 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.349 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.349 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.349 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.349 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.349 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.349 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.350 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.350 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.350 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.350 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.350 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.350 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.350 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.350 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.350 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.351 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.351 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.351 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.351 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.352 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.352 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.352 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.352 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.352 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.352 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.352 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.353 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.353 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.354 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.355 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.355 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.356 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.356 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.356 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.356 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.357 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.357 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.357 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.358 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.358 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.358 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.358 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.359 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.359 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.359 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.360 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.360 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.360 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.361 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.361 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.361 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.361 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.362 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.362 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.362 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.362 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.362 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.362 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.362 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.363 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.363 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.363 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.363 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.363 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.363 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.363 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.363 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.363 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.363 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.363 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.363 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.363 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.365 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.366 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.367 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.368 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.369 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:26 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:27 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.370 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:27 smithi040 ceph-mon[100251]: pgmap v111: 129 pgs: 129 active+clean; 312 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 251 op/s 2023-11-16T18:28:39.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:28 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:28:39.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:29 smithi040 ceph-mon[100251]: pgmap v112: 129 pgs: 129 active+clean; 327 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 4.9 MiB/s wr, 459 op/s 2023-11-16T18:28:39.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:30 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:30 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:30 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:30 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:30 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:30 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.371 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:31 smithi040 ceph-mon[100251]: pgmap v113: 129 pgs: 129 active+clean; 327 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 448 op/s 2023-11-16T18:28:39.372 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:33 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:33.398+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000038a7 [6,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9951 auth{0=1} v925 ap=1 s=0 n(v0 rc2023-11-16T18:27:52.390401+0000 1=1+0) (iversion lock) caps={25006=pAsLsXsFs/-@2},l=25006(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x55d21c84cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000038a7:[<0x10000000274/file9951 v884>,<0x10000000273/dir5 v38269>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x100000038a7:[<0x10000000274/file9951 v925>,<0x10000000273/dir5 v44815>,<0x10000000272/dir4 v15292>,<0x10000000271/dir3 v46772>,<0x10000000270/dir2 v47723>,<0x1000000026f/dir1 v24857>,<0x1000000026e/testdir v265>,<0x10000000005/tmp v369>,<0x100000001fb/client.0 v330>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v321>,<0x10000000001/sv_1 v340>,<0x10000000000/_nogroup v316>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:39.372 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:33 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:33.540+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000389e [6,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9943 auth{0=1} v933 ap=1 s=0 n(v0 rc2023-11-16T18:27:52.363402+0000 1=1+0) (iversion lock) caps={25006=pAsLsXsFs/-@2},l=25006(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x55d21c77d080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000389e:[<0x10000000274/file9943 v892>,<0x10000000273/dir5 v38269>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x1000000389e:[<0x10000000274/file9943 v933>,<0x10000000273/dir5 v44815>,<0x10000000272/dir4 v15292>,<0x10000000271/dir3 v46772>,<0x10000000270/dir2 v47723>,<0x1000000026f/dir1 v24857>,<0x1000000026e/testdir v265>,<0x10000000005/tmp v369>,<0x100000001fb/client.0 v330>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v321>,<0x10000000001/sv_1 v340>,<0x10000000000/_nogroup v316>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:39.372 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:33 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:33.597+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000038a5 [6,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9949 auth{0=1} v932 ap=1 s=0 n(v0 rc2023-11-16T18:27:52.385401+0000 1=1+0) (iversion lock) caps={25006=pAsLsXsFs/-@2},l=25006(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x55d21afdd600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000038a5:[<0x10000000274/file9949 v877>,<0x10000000273/dir5 v38269>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x100000038a5:[<0x10000000274/file9949 v932>,<0x10000000273/dir5 v44815>,<0x10000000272/dir4 v15292>,<0x10000000271/dir3 v46772>,<0x10000000270/dir2 v47723>,<0x1000000026f/dir1 v24857>,<0x1000000026e/testdir v265>,<0x10000000005/tmp v369>,<0x100000001fb/client.0 v330>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v321>,<0x10000000001/sv_1 v340>,<0x10000000000/_nogroup v316>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:39.373 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.374 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.374 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.374 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.374 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.374 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.374 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.374 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.374 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.374 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.374 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.374 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.374 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: pgmap v114: 129 pgs: 129 active+clean; 333 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 6.4 MiB/s wr, 901 op/s 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:34 smithi040 ceph-mon[100251]: Scrub error on inode 0x100000038a7 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9951) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: Scrub error on inode 0x1000000389e (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9943) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: Scrub error on inode 0x100000038a5 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9949) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.375 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.376 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.376 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.376 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.376 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.376 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.376 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.376 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.377 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:35.379+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000038a9 [6,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9953 auth{0=1} v943 ap=1 s=0 n(v0 rc2023-11-16T18:27:52.396401+0000 1=1+0) (iversion lock) caps={25006=pAsLsXsFs/-@2},l=25006(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x55d21c805b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000038a9:[<0x10000000274/file9953 v905>,<0x10000000273/dir5 v38269>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x100000038a9:[<0x10000000274/file9953 v943>,<0x10000000273/dir5 v44815>,<0x10000000272/dir4 v15292>,<0x10000000271/dir3 v46898>,<0x10000000270/dir2 v47723>,<0x1000000026f/dir1 v24859>,<0x1000000026e/testdir v265>,<0x10000000005/tmp v369>,<0x100000001fb/client.0 v330>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v321>,<0x10000000001/sv_1 v340>,<0x10000000000/_nogroup v316>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:39.377 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:35.549+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000038b7 [6,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9967 auth{0=1} v895 ap=1 s=0 n(v0 rc2023-11-16T18:27:52.439400+0000 1=1+0) (iversion lock) caps={25006=pAsLsXsFs/-@2},l=25006(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x55d21b971080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000038b7:[<0x10000000274/file9967 v854>,<0x10000000273/dir5 v38269>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x100000038b7:[<0x10000000274/file9967 v895>,<0x10000000273/dir5 v44815>,<0x10000000272/dir4 v15292>,<0x10000000271/dir3 v46898>,<0x10000000270/dir2 v47723>,<0x1000000026f/dir1 v24859>,<0x1000000026e/testdir v265>,<0x10000000005/tmp v369>,<0x100000001fb/client.0 v330>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v321>,<0x10000000001/sv_1 v340>,<0x10000000000/_nogroup v316>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:39.377 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:35.740+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000038ad [6,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9957 auth{0=1} v945 ap=1 s=0 n(v0 rc2023-11-16T18:27:52.410401+0000 1=1+0) (iversion lock) caps={25006=pAsLsXsFs/-@2},l=25006(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x55d21c5a6580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000038ad:[<0x10000000274/file9957 v904>,<0x10000000273/dir5 v38269>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x100000038ad:[<0x10000000274/file9957 v945>,<0x10000000273/dir5 v44815>,<0x10000000272/dir4 v15292>,<0x10000000271/dir3 v46898>,<0x10000000270/dir2 v47723>,<0x1000000026f/dir1 v24859>,<0x1000000026e/testdir v265>,<0x10000000005/tmp v369>,<0x100000001fb/client.0 v330>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v321>,<0x10000000001/sv_1 v340>,<0x10000000000/_nogroup v316>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:39.377 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:35 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:35.783+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000038b5 [6,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9965 auth{0=1} v941 ap=1 s=0 n(v0 rc2023-11-16T18:27:52.433400+0000 1=1+0) (iversion lock) caps={25006=pAsLsXsFs/-@2},l=25006(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x55d2184e8580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000038b5:[<0x10000000274/file9965 v892>,<0x10000000273/dir5 v38269>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x100000038b5:[<0x10000000274/file9965 v941>,<0x10000000273/dir5 v44815>,<0x10000000272/dir4 v15292>,<0x10000000271/dir3 v46898>,<0x10000000270/dir2 v47723>,<0x1000000026f/dir1 v24859>,<0x1000000026e/testdir v265>,<0x10000000005/tmp v369>,<0x100000001fb/client.0 v330>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v321>,<0x10000000001/sv_1 v340>,<0x10000000000/_nogroup v316>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:39.377 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:28:36.076+0000 7fd43a2ab700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000038a1 [6,head] /volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9945 auth{0=1} v892 ap=1 s=0 n(v0 rc2023-11-16T18:27:52.374401+0000 1=1+0) (iversion lock) caps={25006=pAsLsXsFs/-@2},l=25006(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x55d21c126c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000038a1:[<0x10000000274/file9945 v842>,<0x10000000273/dir5 v38269>,<0x10000000272/dir4 v14439>,<0x10000000271/dir3 v33651>,<0x10000000270/dir2 v39593>,<0x1000000026f/dir1 v21989>,<0x1000000026e/testdir v243>,<0x10000000005/tmp v344>,<0x100000001fb/client.0 v305>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v297>,<0x10000000001/sv_1 v315>,<0x10000000000/_nogroup v294>,<0x1/volumes v291>]//[3]","memoryvalue":"(4)0x100000038a1:[<0x10000000274/file9945 v892>,<0x10000000273/dir5 v44815>,<0x10000000272/dir4 v15292>,<0x10000000271/dir3 v46898>,<0x10000000270/dir2 v47723>,<0x1000000026f/dir1 v24859>,<0x1000000026e/testdir v265>,<0x10000000005/tmp v369>,<0x100000001fb/client.0 v330>,<0x100000001fa/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e v321>,<0x10000000001/sv_1 v340>,<0x10000000000/_nogroup v316>,<0x1/volumes v279>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-16T18:28:39.380 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.381 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.382 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.382 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.382 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.382 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.382 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.382 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.382 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.382 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.382 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.382 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.382 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.382 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.383 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.384 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.384 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.384 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.384 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.384 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.384 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.384 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.384 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.384 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.384 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.384 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.384 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.384 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.385 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: pgmap v115: 129 pgs: 129 active+clean; 333 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 683 op/s 2023-11-16T18:28:39.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.387 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.387 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.387 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.387 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.387 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.387 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.387 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.387 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.387 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.387 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.387 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.387 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.387 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: Scrub error on inode 0x100000038a9 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9953) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:36 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.389 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.389 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.389 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.389 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.389 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.389 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.389 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.389 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.389 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.389 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.389 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.389 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.389 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.391 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.392 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.392 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.392 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.392 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.392 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.392 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.392 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.392 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.392 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.392 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.392 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.392 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.392 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: Scrub error on inode 0x100000038b7 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9967) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.393 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.394 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.394 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.394 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.394 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.394 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.394 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.394 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.394 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.394 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.394 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.394 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.394 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.394 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.395 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.395 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.395 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.395 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.395 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.395 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.395 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.395 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.395 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.395 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.395 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: Scrub error on inode 0x100000038ad (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9957) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.395 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.395 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.396 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: Scrub error on inode 0x100000038b5 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9965) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.397 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.398 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.398 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.398 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.398 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.398 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.398 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.398 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.398 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.398 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.398 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.398 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.398 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.398 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.399 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.400 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.400 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.400 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.400 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.400 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.400 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.400 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.400 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.400 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.400 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.400 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.400 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.400 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.401 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.401 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.401 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.401 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.401 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.401 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.401 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.401 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.401 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.401 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.401 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.401 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.402 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: Scrub error on inode 0x100000038a1 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9945) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:37 smithi040 ceph-mon[100251]: pgmap v116: 129 pgs: 129 active+clean; 336 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 716 op/s 2023-11-16T18:28:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:39.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: Scrub error on inode 0x100000038af (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9959) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: Scrub error on inode 0x100000038a2 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9946) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.444 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.444 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.444 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.444 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.444 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.444 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.444 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.444 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.444 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.444 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.444 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.444 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.444 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.449 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.449 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.449 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.449 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.449 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.449 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.449 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.449 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.449 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.449 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.449 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.449 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.449 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: pgmap v110: 129 pgs: 129 active+clean; 312 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.6 MiB/s wr, 231 op/s 2023-11-16T18:28:39.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.452 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.452 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.452 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.452 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.452 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.452 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.452 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.452 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.452 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.452 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.452 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.452 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.467 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.467 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.467 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.467 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.467 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.467 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.467 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.467 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.467 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.467 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.467 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.467 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.467 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.469 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.469 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.469 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.469 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.469 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.469 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.469 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.469 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.469 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.469 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.469 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.469 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.488 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.488 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.488 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.488 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.488 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.488 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.488 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.488 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.488 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.488 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.488 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.488 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.488 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.490 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.490 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.490 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.490 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.490 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.490 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.490 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.490 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.490 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.490 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.490 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.490 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.490 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.493 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.493 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.493 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.493 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.493 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.493 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.493 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.493 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.493 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.493 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.493 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.493 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.493 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:26 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:27 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:27 smithi188 ceph-mon[100639]: pgmap v111: 129 pgs: 129 active+clean; 312 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 251 op/s 2023-11-16T18:28:39.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:28 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:28:39.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:29 smithi188 ceph-mon[100639]: pgmap v112: 129 pgs: 129 active+clean; 327 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 4.9 MiB/s wr, 459 op/s 2023-11-16T18:28:39.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:30 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:30 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:30 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:30 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:30 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:30 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:31 smithi188 ceph-mon[100639]: pgmap v113: 129 pgs: 129 active+clean; 327 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 448 op/s 2023-11-16T18:28:39.495 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.496 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.497 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.497 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: pgmap v114: 129 pgs: 129 active+clean; 333 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 6.4 MiB/s wr, 901 op/s 2023-11-16T18:28:39.497 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:34 smithi188 ceph-mon[100639]: Scrub error on inode 0x100000038a7 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9951) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.497 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: Scrub error on inode 0x1000000389e (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9943) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.497 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: Scrub error on inode 0x100000038a5 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9949) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.497 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.497 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.497 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.497 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.497 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.497 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.497 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.497 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.498 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.498 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.498 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.498 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.498 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.498 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.498 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.498 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:35 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.498 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.498 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.498 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.498 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.502 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.502 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.502 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.502 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.502 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.502 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.502 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.502 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.502 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.502 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.502 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.502 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.502 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: pgmap v115: 129 pgs: 129 active+clean; 333 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 683 op/s 2023-11-16T18:28:39.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.506 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.506 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.506 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.506 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: Scrub error on inode 0x100000038a9 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9953) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.506 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.506 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.506 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:36 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.506 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.506 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.506 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.506 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.507 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.507 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.507 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.507 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.507 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.507 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.507 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.507 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.507 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.507 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.507 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.507 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.508 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.509 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.509 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.509 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.509 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.509 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.509 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.509 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.509 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.509 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.509 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.509 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.509 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.509 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.510 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.510 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.510 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.510 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.510 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.510 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.510 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.510 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.510 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.510 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.510 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.510 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.511 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.511 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.511 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.511 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.511 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.511 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.511 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.511 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.511 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.511 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.511 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: Scrub error on inode 0x100000038b7 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9967) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.511 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.511 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.512 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.512 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.512 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.512 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.512 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.512 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.512 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.512 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.512 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.512 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.512 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.512 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: Scrub error on inode 0x100000038ad (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9957) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: Scrub error on inode 0x100000038b5 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9965) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: Scrub error on inode 0x100000038a1 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9945) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.527 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.527 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.527 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.527 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.527 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.527 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.527 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.527 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.527 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.527 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.527 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.527 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.527 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:37 smithi188 ceph-mon[100639]: pgmap v116: 129 pgs: 129 active+clean; 336 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 716 op/s 2023-11-16T18:28:39.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:39.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.615+0000 7fb38ea39700 1 -- 172.21.15.2:0/318226111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb388104470 msgr2=0x7fb38810c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.616+0000 7fb38ea39700 1 --2- 172.21.15.2:0/318226111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb388104470 0x7fb38810c040 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb38000d360 tx=0x7fb38000d670 comp rx=0 tx=0).stop 2023-11-16T18:28:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.616+0000 7fb38ea39700 1 -- 172.21.15.2:0/318226111 shutdown_connections 2023-11-16T18:28:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.616+0000 7fb38ea39700 1 --2- 172.21.15.2:0/318226111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb388104470 0x7fb38810c040 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.616+0000 7fb38ea39700 1 --2- 172.21.15.2:0/318226111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3880ff430 0x7fb3880ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.616+0000 7fb38ea39700 1 --2- 172.21.15.2:0/318226111 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3880ffdd0 0x7fb388103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.616+0000 7fb38ea39700 1 -- 172.21.15.2:0/318226111 >> 172.21.15.2:0/318226111 conn(0x7fb388076370 msgr2=0x7fb388076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.616+0000 7fb38ea39700 1 -- 172.21.15.2:0/318226111 shutdown_connections 2023-11-16T18:28:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.616+0000 7fb38ea39700 1 -- 172.21.15.2:0/318226111 wait complete. 2023-11-16T18:28:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.616+0000 7fb38ea39700 1 Processor -- start 2023-11-16T18:28:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.617+0000 7fb38ea39700 1 -- start start 2023-11-16T18:28:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.617+0000 7fb38ea39700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3880ff430 0x7fb388194350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.617+0000 7fb38ea39700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3880ffdd0 0x7fb388194890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.617+0000 7fb38ea39700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb388104470 0x7fb388198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.617+0000 7fb38ea39700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb388199330 con 0x7fb388104470 2023-11-16T18:28:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.617+0000 7fb38ea39700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3881994a0 con 0x7fb3880ffdd0 2023-11-16T18:28:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.617+0000 7fb38ea39700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb388199660 con 0x7fb3880ff430 2023-11-16T18:28:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.617+0000 7fb38cfd6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb388104470 0x7fb388198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.617+0000 7fb38cfd6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb388104470 0x7fb388198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52248/0 (socket says 172.21.15.2:52248) 2023-11-16T18:28:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.617+0000 7fb38cfd6700 1 -- 172.21.15.2:0/810507771 learned_addr learned my addr 172.21.15.2:0/810507771 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.618+0000 7fb3877fe700 1 --2- 172.21.15.2:0/810507771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3880ffdd0 0x7fb388194890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.618+0000 7fb38cfd6700 1 -- 172.21.15.2:0/810507771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3880ff430 msgr2=0x7fb388194350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.618+0000 7fb38cfd6700 1 --2- 172.21.15.2:0/810507771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3880ff430 0x7fb388194350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.618+0000 7fb38cfd6700 1 -- 172.21.15.2:0/810507771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3880ffdd0 msgr2=0x7fb388194890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.618+0000 7fb38cfd6700 1 --2- 172.21.15.2:0/810507771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3880ffdd0 0x7fb388194890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.618+0000 7fb38cfd6700 1 -- 172.21.15.2:0/810507771 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb380009d00 con 0x7fb388104470 2023-11-16T18:28:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.618+0000 7fb38cfd6700 1 --2- 172.21.15.2:0/810507771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb388104470 0x7fb388198cf0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fb380009850 tx=0x7fb3800093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.618+0000 7fb3857fa700 1 -- 172.21.15.2:0/810507771 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb380003e10 con 0x7fb388104470 2023-11-16T18:28:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.618+0000 7fb38ea39700 1 -- 172.21.15.2:0/810507771 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38810e1f0 con 0x7fb388104470 2023-11-16T18:28:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.618+0000 7fb3857fa700 1 -- 172.21.15.2:0/810507771 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb380005530 con 0x7fb388104470 2023-11-16T18:28:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.618+0000 7fb3857fa700 1 -- 172.21.15.2:0/810507771 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38001f850 con 0x7fb388104470 2023-11-16T18:28:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.618+0000 7fb38ea39700 1 -- 172.21.15.2:0/810507771 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38810e060 con 0x7fb388104470 2023-11-16T18:28:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.621+0000 7fb3857fa700 1 -- 172.21.15.2:0/810507771 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fb3800056a0 con 0x7fb388104470 2023-11-16T18:28:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.622+0000 7fb3857fa700 1 --2- 172.21.15.2:0/810507771 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fb3700a1340 0x7fb3700a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.622+0000 7fb3857fa700 1 -- 172.21.15.2:0/810507771 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fb3800d6ca0 con 0x7fb388104470 2023-11-16T18:28:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.622+0000 7fb387fff700 1 --2- 172.21.15.2:0/810507771 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fb3700a1340 0x7fb3700a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.623+0000 7fb39abc5b80 1 Processor -- start 2023-11-16T18:28:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.623+0000 7fb39abc5b80 1 -- start start 2023-11-16T18:28:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.623+0000 7fb387fff700 1 --2- 172.21.15.2:0/810507771 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fb3700a1340 0x7fb3700a37f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb3780097b0 tx=0x7fb378006d20 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.623+0000 7fb39abc5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 0x5611048a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.623+0000 7fb39abc5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8720 0x5611048a2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.623+0000 7fb39abc5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8b90 0x5611048a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.623+0000 7fb39abc5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611047a8000 con 0x5611048a8b90 2023-11-16T18:28:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.623+0000 7fb39abc5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611047fa840 con 0x5611046fe6b0 2023-11-16T18:28:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.623+0000 7fb39abc5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611047f3cd0 con 0x5611048a8720 2023-11-16T18:28:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.624+0000 7fb3877fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8720 0x5611048a2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.624+0000 7fb38cfd6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8b90 0x5611048a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.624+0000 7fb3877fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8720 0x5611048a2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43786/0 (socket says 172.21.15.2:43786) 2023-11-16T18:28:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.624+0000 7fb3877fe700 1 -- 172.21.15.2:0/1509501073 learned_addr learned my addr 172.21.15.2:0/1509501073 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.624+0000 7fb387fff700 1 --2- 172.21.15.2:0/1509501073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 0x5611048a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.624+0000 7fb38cfd6700 1 -- 172.21.15.2:0/1509501073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8720 msgr2=0x5611048a2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.624+0000 7fb38cfd6700 1 --2- 172.21.15.2:0/1509501073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8720 0x5611048a2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.624+0000 7fb38cfd6700 1 -- 172.21.15.2:0/1509501073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 msgr2=0x5611048a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.624+0000 7fb38cfd6700 1 --2- 172.21.15.2:0/1509501073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 0x5611048a81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.624+0000 7fb38cfd6700 1 -- 172.21.15.2:0/1509501073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611046755d0 con 0x5611048a8b90 2023-11-16T18:28:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.624+0000 7fb387fff700 1 --2- 172.21.15.2:0/1509501073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 0x5611048a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:28:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.624+0000 7fb38cfd6700 1 --2- 172.21.15.2:0/1509501073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8b90 0x5611048a2b00 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb3800d4080 tx=0x7fb3800a35d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817e65211c4b6ee9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.625+0000 7fb36e7fc700 1 -- 172.21.15.2:0/1509501073 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3800a7ac0 con 0x5611048a8b90 2023-11-16T18:28:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.625+0000 7fb36e7fc700 1 -- 172.21.15.2:0/1509501073 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3800abe50 con 0x5611048a8b90 2023-11-16T18:28:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.625+0000 7fb36e7fc700 1 -- 172.21.15.2:0/1509501073 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3800a31e0 con 0x5611048a8b90 2023-11-16T18:28:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.626+0000 7fb39abc5b80 1 -- 172.21.15.2:0/1509501073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8b90 msgr2=0x5611048a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.626+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/1509501073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8b90 0x5611048a2b00 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb3800d4080 tx=0x7fb3800a35d0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.627+0000 7fb39abc5b80 1 -- 172.21.15.2:0/1509501073 shutdown_connections 2023-11-16T18:28:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.627+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/1509501073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8b90 0x5611048a2b00 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.627+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/1509501073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 0x5611048a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.627+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/1509501073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8720 0x5611048a2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.628+0000 7fb39abc5b80 1 -- 172.21.15.2:0/1509501073 >> 172.21.15.2:0/1509501073 conn(0x561104897f50 msgr2=0x56110489d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.628+0000 7fb39abc5b80 1 -- 172.21.15.2:0/1509501073 shutdown_connections 2023-11-16T18:28:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.628+0000 7fb39abc5b80 1 -- 172.21.15.2:0/1509501073 wait complete. 2023-11-16T18:28:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.629+0000 7fb39abc5b80 1 -- start start 2023-11-16T18:28:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.629+0000 7fb39abc5b80 1 Processor -- start 2023-11-16T18:28:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.629+0000 7fb39abc5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611046fe6b0 0x5611048b1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.629+0000 7fb39abc5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611048a8720 0x5611048b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.629+0000 7fb39abc5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8b90 0x5611048aba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.629+0000 7fb39abc5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561104795880 con 0x5611048a8b90 2023-11-16T18:28:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.629+0000 7fb39abc5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561104795520 con 0x5611048a8720 2023-11-16T18:28:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.629+0000 7fb39abc5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611046778c0 con 0x5611046fe6b0 2023-11-16T18:28:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.629+0000 7fb39abc5b80 10 client.0 _ll_register_callbacks cb 0x5611047b3720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:28:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.630+0000 7fb3877fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611048a8720 0x5611048b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.630+0000 7fb3877fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611048a8720 0x5611048b1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42700/0 (socket says 172.21.15.2:42700) 2023-11-16T18:28:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.630+0000 7fb3877fe700 1 -- 172.21.15.2:0/2232177962 learned_addr learned my addr 172.21.15.2:0/2232177962 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.630+0000 7fb387fff700 1 --2- 172.21.15.2:0/2232177962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611046fe6b0 0x5611048b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.630+0000 7fb3877fe700 1 -- 172.21.15.2:0/2232177962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611046fe6b0 msgr2=0x5611048b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.630+0000 7fb3877fe700 1 --2- 172.21.15.2:0/2232177962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611046fe6b0 0x5611048b1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.630+0000 7fb3877fe700 1 -- 172.21.15.2:0/2232177962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8b90 msgr2=0x5611048aba80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T18:28:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.630+0000 7fb3877fe700 1 --2- 172.21.15.2:0/2232177962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8b90 0x5611048aba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.630+0000 7fb3877fe700 1 -- 172.21.15.2:0/2232177962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38810e060 con 0x5611048a8720 2023-11-16T18:28:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.631+0000 7fb3877fe700 1 --2- 172.21.15.2:0/2232177962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611048a8720 0x5611048b1c90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0044b0 tx=0x7fb37c003960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f35a7a5624c52a9d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.631+0000 7fb3677fe700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:28:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.631+0000 7fb3677fe700 1 -- 172.21.15.2:0/2232177962 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c012040 con 0x5611048a8720 2023-11-16T18:28:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.631+0000 7fb3677fe700 1 -- 172.21.15.2:0/2232177962 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fb37c0177e0 con 0x5611048a8720 2023-11-16T18:28:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.631+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2232177962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561104895710 con 0x5611048a8720 2023-11-16T18:28:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.632+0000 7fb3677fe700 1 -- 172.21.15.2:0/2232177962 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb37c0179a0 con 0x5611048a8720 2023-11-16T18:28:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.632+0000 7fb39abc5b80 10 client.26021 fetch_fsmap learned FSMap version 28 2023-11-16T18:28:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.632+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2232177962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5611047ba130 con 0x5611048a8720 2023-11-16T18:28:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.632+0000 7fb3677fe700 1 -- 172.21.15.2:0/2232177962 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561104895710 con 0x5611048a8720 2023-11-16T18:28:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.633+0000 7fb3677fe700 1 -- 172.21.15.2:0/2232177962 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fb37c018ae0 con 0x5611048a8720 2023-11-16T18:28:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.633+0000 7fb39abc5b80 10 client.26021 fetch_fsmap finished waiting for FSMap version 28 2023-11-16T18:28:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.633+0000 7fb39abc5b80 10 client.26021 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:28:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.633+0000 7fb39abc5b80 20 client.26021 populate_metadata read hostname 'smithi002' 2023-11-16T18:28:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.633+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/2232177962 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5611048bbcf0 0x5611048be100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.633+0000 7fb39abc5b80 4 client.26021 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:28:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.633+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2232177962 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5611047b6040 con 0x5611048bbcf0 2023-11-16T18:28:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.633+0000 7fb387fff700 1 --2- 172.21.15.2:0/2232177962 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5611048bbcf0 0x5611048be100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.633+0000 7fb3677fe700 10 client.26021 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.633+0000 7fb387fff700 1 --2- 172.21.15.2:0/2232177962 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5611048bbcf0 0x5611048be100 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48a5df79e14f5862 server_cookie=f2205ab98bcaf09e in_seq=0 out_seq=0 2023-11-16T18:28:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.642+0000 7fb3677fe700 1 -- 172.21.15.2:0/2232177962 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5611047b6040 con 0x5611048bbcf0 2023-11-16T18:28:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.642+0000 7fb3677fe700 10 client.26021 handle_command_reply: tid=0 2023-11-16T18:28:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.642+0000 7fb39abc5b80 1 client.26021 shutdown 2023-11-16T18:28:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.643+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2232177962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611048a8720 msgr2=0x5611048b1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.643+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/2232177962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611048a8720 0x5611048b1c90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0044b0 tx=0x7fb37c003960 comp rx=0 tx=0).stop 2023-11-16T18:28:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.643+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2232177962 shutdown_connections 2023-11-16T18:28:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.643+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/2232177962 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5611048bbcf0 0x5611048be100 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.643+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/2232177962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611046fe6b0 0x5611048b1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.643+0000 7fb3677fe700 0 client.26021 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.643+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/2232177962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8b90 0x5611048aba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.643+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/2232177962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611048a8720 0x5611048b1c90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.643+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2232177962 >> 172.21.15.2:0/2232177962 conn(0x561104897f50 msgr2=0x56110489c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.643+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2232177962 shutdown_connections 2023-11-16T18:28:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.643+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2232177962 wait complete. 2023-11-16T18:28:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.643+0000 7fb39abc5b80 20 client.26021 trim_cache size 0 max 16384 2023-11-16T18:28:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.668+0000 7fb39abc5b80 1 Processor -- start 2023-11-16T18:28:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.668+0000 7fb39abc5b80 1 -- start start 2023-11-16T18:28:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.668+0000 7fb39abc5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 0x56110489cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.668+0000 7fb39abc5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8720 0x5611048a8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.668+0000 7fb39abc5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8b90 0x5611048a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.668+0000 7fb39abc5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611048abff0 con 0x5611048a8720 2023-11-16T18:28:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.668+0000 7fb39abc5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561104899fd0 con 0x5611046fe6b0 2023-11-16T18:28:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.668+0000 7fb39abc5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110489e3e0 con 0x5611048a8b90 2023-11-16T18:28:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.668+0000 7fb3877fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8720 0x5611048a8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.668+0000 7fb3877fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8720 0x5611048a8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52268/0 (socket says 172.21.15.2:52268) 2023-11-16T18:28:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.668+0000 7fb3877fe700 1 -- 172.21.15.2:0/65417988 learned_addr learned my addr 172.21.15.2:0/65417988 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.668+0000 7fb38cfd6700 1 --2- 172.21.15.2:0/65417988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8b90 0x5611048a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.669+0000 7fb387fff700 1 --2- 172.21.15.2:0/65417988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 0x56110489cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.669+0000 7fb3877fe700 1 -- 172.21.15.2:0/65417988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8b90 msgr2=0x5611048a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.669+0000 7fb3877fe700 1 --2- 172.21.15.2:0/65417988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8b90 0x5611048a94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.669+0000 7fb3877fe700 1 -- 172.21.15.2:0/65417988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 msgr2=0x56110489cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.669+0000 7fb3877fe700 1 --2- 172.21.15.2:0/65417988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 0x56110489cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.669+0000 7fb3877fe700 1 -- 172.21.15.2:0/65417988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611047b9e80 con 0x5611048a8720 2023-11-16T18:28:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.669+0000 7fb38cfd6700 1 --2- 172.21.15.2:0/65417988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8b90 0x5611048a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:28:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.669+0000 7fb3877fe700 1 --2- 172.21.15.2:0/65417988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8720 0x5611048a8f90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb37c018e10 tx=0x7fb37c00dba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78205b5a481b7b20 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.669+0000 7fb36effd700 1 -- 172.21.15.2:0/65417988 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c012070 con 0x5611048a8720 2023-11-16T18:28:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.669+0000 7fb36effd700 1 -- 172.21.15.2:0/65417988 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb37c00d350 con 0x5611048a8720 2023-11-16T18:28:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.669+0000 7fb36effd700 1 -- 172.21.15.2:0/65417988 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c004df0 con 0x5611048a8720 2023-11-16T18:28:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.670+0000 7fb39abc5b80 1 -- 172.21.15.2:0/65417988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8720 msgr2=0x5611048a8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.670+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/65417988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8720 0x5611048a8f90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb37c018e10 tx=0x7fb37c00dba0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.670+0000 7fb39abc5b80 1 -- 172.21.15.2:0/65417988 shutdown_connections 2023-11-16T18:28:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.670+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/65417988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8b90 0x5611048a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.670+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/65417988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 0x56110489cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.670+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/65417988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8720 0x5611048a8f90 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.670+0000 7fb39abc5b80 1 -- 172.21.15.2:0/65417988 >> 172.21.15.2:0/65417988 conn(0x561104897f50 msgr2=0x5611048a5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.670+0000 7fb39abc5b80 1 -- 172.21.15.2:0/65417988 shutdown_connections 2023-11-16T18:28:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.670+0000 7fb39abc5b80 1 -- 172.21.15.2:0/65417988 wait complete. 2023-11-16T18:28:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.671+0000 7fb39abc5b80 1 -- start start 2023-11-16T18:28:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.671+0000 7fb39abc5b80 1 Processor -- start 2023-11-16T18:28:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.671+0000 7fb39abc5b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 0x561104903d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.671+0000 7fb39abc5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8720 0x561104904270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.671+0000 7fb39abc5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8b90 0x56110490ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.671+0000 7fb39abc5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611049049c0 con 0x5611048a8720 2023-11-16T18:28:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.671+0000 7fb39abc5b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110490f100 con 0x5611046fe6b0 2023-11-16T18:28:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.671+0000 7fb39abc5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110490f2c0 con 0x5611048a8b90 2023-11-16T18:28:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.671+0000 7fb39abc5b80 10 client.0 _ll_register_callbacks cb 0x7fb37c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:28:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.672+0000 7fb3877fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8720 0x561104904270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.672+0000 7fb38cfd6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8b90 0x56110490ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.672+0000 7fb38cfd6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8b90 0x56110490ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43804/0 (socket says 172.21.15.2:43804) 2023-11-16T18:28:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.672+0000 7fb38cfd6700 1 -- 172.21.15.2:0/2361465193 learned_addr learned my addr 172.21.15.2:0/2361465193 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.672+0000 7fb38cfd6700 1 -- 172.21.15.2:0/2361465193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 msgr2=0x561104903d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.672+0000 7fb38cfd6700 1 --2- 172.21.15.2:0/2361465193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 0x561104903d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.672+0000 7fb38cfd6700 1 -- 172.21.15.2:0/2361465193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8720 msgr2=0x561104904270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.672+0000 7fb38cfd6700 1 --2- 172.21.15.2:0/2361465193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8720 0x561104904270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.672+0000 7fb38cfd6700 1 -- 172.21.15.2:0/2361465193 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611046755d0 con 0x5611048a8b90 2023-11-16T18:28:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.672+0000 7fb3877fe700 1 --2- 172.21.15.2:0/2361465193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8720 0x561104904270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:28:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.672+0000 7fb38cfd6700 1 --2- 172.21.15.2:0/2361465193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8b90 0x56110490ebc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb3800d4680 tx=0x7fb38000d670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35429dc57bfe6585 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.672+0000 7fb3677fe700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:28:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.672+0000 7fb3677fe700 1 -- 172.21.15.2:0/2361465193 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3800a3180 con 0x5611048a8b90 2023-11-16T18:28:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.673+0000 7fb3677fe700 1 -- 172.21.15.2:0/2361465193 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fb3800a7ac0 con 0x5611048a8b90 2023-11-16T18:28:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.673+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2361465193 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb3800abe50 con 0x5611048a8b90 2023-11-16T18:28:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.673+0000 7fb3677fe700 1 -- 172.21.15.2:0/2361465193 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3800a3cf0 con 0x5611048a8b90 2023-11-16T18:28:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.673+0000 7fb39abc5b80 10 client.25915 fetch_fsmap learned FSMap version 28 2023-11-16T18:28:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.673+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2361465193 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5611047ba130 con 0x5611048a8b90 2023-11-16T18:28:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.673+0000 7fb3677fe700 1 -- 172.21.15.2:0/2361465193 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3800abe50 con 0x5611048a8b90 2023-11-16T18:28:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.673+0000 7fb3677fe700 1 -- 172.21.15.2:0/2361465193 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 28) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fb3800a7d20 con 0x5611048a8b90 2023-11-16T18:28:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.673+0000 7fb39abc5b80 10 client.25915 fetch_fsmap finished waiting for FSMap version 28 2023-11-16T18:28:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.673+0000 7fb39abc5b80 10 client.25915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:28:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.673+0000 7fb39abc5b80 20 client.25915 populate_metadata read hostname 'smithi002' 2023-11-16T18:28:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.673+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/2361465193 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x561104910620 0x561104912a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.673+0000 7fb39abc5b80 4 client.25915 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:28:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.673+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2361465193 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5611047c2600 con 0x561104910620 2023-11-16T18:28:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.674+0000 7fb387fff700 1 --2- 172.21.15.2:0/2361465193 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x561104910620 0x561104912a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.675+0000 7fb3677fe700 10 client.25915 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:26.675+0000 7fb387fff700 1 --2- 172.21.15.2:0/2361465193 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x561104910620 0x561104912a30 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5eb65db2b2ab167e server_cookie=efe1f527f7c81dc2 in_seq=0 out_seq=0 2023-11-16T18:28:39.559 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.570 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.570 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.570 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.570 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.570 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.570 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.570 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.570 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.570 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.570 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.570 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.570 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.570 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.583 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.583 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.583 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.583 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.583 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.583 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:26 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.583 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:27 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.583 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:27 smithi002 ceph-mon[98807]: pgmap v111: 129 pgs: 129 active+clean; 312 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 251 op/s 2023-11-16T18:28:39.583 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:28 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:28:39.583 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:29 smithi002 ceph-mon[98807]: pgmap v112: 129 pgs: 129 active+clean; 327 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 4.9 MiB/s wr, 459 op/s 2023-11-16T18:28:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb3677fe700 1 -- 172.21.15.2:0/2361465193 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5611047c2600 con 0x561104910620 2023-11-16T18:28:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb3677fe700 10 client.25915 handle_command_reply: tid=0 2023-11-16T18:28:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb39abc5b80 1 client.25915 shutdown 2023-11-16T18:28:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2361465193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8b90 msgr2=0x56110490ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/2361465193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8b90 0x56110490ebc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb3800d4680 tx=0x7fb38000d670 comp rx=0 tx=0).stop 2023-11-16T18:28:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2361465193 shutdown_connections 2023-11-16T18:28:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/2361465193 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x561104910620 0x561104912a30 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/2361465193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611048a8b90 0x56110490ebc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb3677fe700 0 client.25915 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/2361465193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611046fe6b0 0x561104903d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb39abc5b80 1 --2- 172.21.15.2:0/2361465193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5611048a8720 0x561104904270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2361465193 >> 172.21.15.2:0/2361465193 conn(0x561104897f50 msgr2=0x5611048a8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2361465193 shutdown_connections 2023-11-16T18:28:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.996+0000 7fb39abc5b80 1 -- 172.21.15.2:0/2361465193 wait complete. 2023-11-16T18:28:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.997+0000 7fb39abc5b80 20 client.25915 trim_cache size 0 max 16384 2023-11-16T18:28:39.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:28:39.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:28:39.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:28:39.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:28:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.997+0000 7fb38ea39700 1 -- 172.21.15.2:0/810507771 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fb3700a1340 msgr2=0x7fb3700a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.997+0000 7fb38ea39700 1 --2- 172.21.15.2:0/810507771 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fb3700a1340 0x7fb3700a37f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb3780097b0 tx=0x7fb378006d20 comp rx=0 tx=0).stop 2023-11-16T18:28:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.997+0000 7fb38ea39700 1 -- 172.21.15.2:0/810507771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb388104470 msgr2=0x7fb388198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.997+0000 7fb38ea39700 1 --2- 172.21.15.2:0/810507771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb388104470 0x7fb388198cf0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fb380009850 tx=0x7fb3800093b0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.997+0000 7fb38ea39700 1 -- 172.21.15.2:0/810507771 shutdown_connections 2023-11-16T18:28:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.997+0000 7fb38ea39700 1 --2- 172.21.15.2:0/810507771 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fb3700a1340 0x7fb3700a37f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.997+0000 7fb38ea39700 1 --2- 172.21.15.2:0/810507771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3880ff430 0x7fb388194350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.997+0000 7fb38ea39700 1 --2- 172.21.15.2:0/810507771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb388104470 0x7fb388198cf0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.997+0000 7fb38ea39700 1 --2- 172.21.15.2:0/810507771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3880ffdd0 0x7fb388194890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.997+0000 7fb38ea39700 1 -- 172.21.15.2:0/810507771 >> 172.21.15.2:0/810507771 conn(0x7fb388076370 msgr2=0x7fb388102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.997+0000 7fb38ea39700 1 -- 172.21.15.2:0/810507771 shutdown_connections 2023-11-16T18:28:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:29.997+0000 7fb38ea39700 1 -- 172.21.15.2:0/810507771 wait complete. 2023-11-16T18:28:39.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:30 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:30 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:30 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:30 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:30 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:30 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:31 smithi002 ceph-mon[98807]: pgmap v113: 129 pgs: 129 active+clean; 327 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 448 op/s 2023-11-16T18:28:39.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: pgmap v114: 129 pgs: 129 active+clean; 333 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 6.4 MiB/s wr, 901 op/s 2023-11-16T18:28:39.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:34 smithi002 ceph-mon[98807]: Scrub error on inode 0x100000038a7 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9951) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: Scrub error on inode 0x1000000389e (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9943) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: Scrub error on inode 0x100000038a5 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9949) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:35 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.594 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.594 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.594 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.594 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.594 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.594 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.594 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.594 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.594 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.594 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.594 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.594 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.594 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.595 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.595 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.595 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.595 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.595 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.595 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.595 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.595 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.595 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.595 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.595 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.595 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: pgmap v115: 129 pgs: 129 active+clean; 333 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 683 op/s 2023-11-16T18:28:39.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: Scrub error on inode 0x100000038a9 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9953) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:36 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: Scrub error on inode 0x100000038b7 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9967) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: Scrub error on inode 0x100000038ad (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9957) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: Scrub error on inode 0x100000038b5 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9965) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: Scrub error on inode 0x100000038a1 (/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9945) see mds.h log and `damage ls` output for details 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:37 smithi002 ceph-mon[98807]: pgmap v116: 129 pgs: 129 active+clean; 336 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 716 op/s 2023-11-16T18:28:39.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:39.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:39.630 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:39 smithi002 ceph-mon[98807]: pgmap v117: 129 pgs: 129 active+clean; 352 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.8 MiB/s wr, 1.13k op/s 2023-11-16T18:28:39.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:39 smithi040 ceph-mon[100251]: pgmap v117: 129 pgs: 129 active+clean; 352 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.8 MiB/s wr, 1.13k op/s 2023-11-16T18:28:39.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:39 smithi188 ceph-mon[100639]: pgmap v117: 129 pgs: 129 active+clean; 352 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.8 MiB/s wr, 1.13k op/s 2023-11-16T18:28:39.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.815+0000 7f7f423de700 1 -- 172.21.15.2:0/4021436128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f3c10ce50 msgr2=0x7f7f3c10f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.815+0000 7f7f423de700 1 --2- 172.21.15.2:0/4021436128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f3c10ce50 0x7f7f3c10f230 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f7f34009f90 tx=0x7f7f3400b330 comp rx=0 tx=0).stop 2023-11-16T18:28:39.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.815+0000 7f7f423de700 1 -- 172.21.15.2:0/4021436128 shutdown_connections 2023-11-16T18:28:39.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.815+0000 7f7f423de700 1 --2- 172.21.15.2:0/4021436128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f3c10ce50 0x7f7f3c10f230 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.815+0000 7f7f423de700 1 --2- 172.21.15.2:0/4021436128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f3c0746a0 0x7f7f3c072d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.815+0000 7f7f423de700 1 --2- 172.21.15.2:0/4021436128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f3c074b00 0x7f7f3c073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.815+0000 7f7f423de700 1 -- 172.21.15.2:0/4021436128 >> 172.21.15.2:0/4021436128 conn(0x7f7f3c077500 msgr2=0x7f7f3c077900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:39.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.815+0000 7f7f423de700 1 -- 172.21.15.2:0/4021436128 shutdown_connections 2023-11-16T18:28:39.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.815+0000 7f7f423de700 1 -- 172.21.15.2:0/4021436128 wait complete. 2023-11-16T18:28:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.816+0000 7f7f423de700 1 Processor -- start 2023-11-16T18:28:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.816+0000 7f7f423de700 1 -- start start 2023-11-16T18:28:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.816+0000 7f7f423de700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f3c0746a0 0x7f7f3c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.816+0000 7f7f423de700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f3c074b00 0x7f7f3c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.816+0000 7f7f423de700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f3c10ce50 0x7f7f3c19d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f423de700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c19d660 con 0x7f7f3c0746a0 2023-11-16T18:28:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f423de700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c19d7d0 con 0x7f7f3c10ce50 2023-11-16T18:28:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f423de700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c19d990 con 0x7f7f3c074b00 2023-11-16T18:28:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f3bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f3c0746a0 0x7f7f3c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f3bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f3c0746a0 0x7f7f3c198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52286/0 (socket says 172.21.15.2:52286) 2023-11-16T18:28:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f3bfff700 1 -- 172.21.15.2:0/299239620 learned_addr learned my addr 172.21.15.2:0/299239620 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f4097b700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f3c10ce50 0x7f7f3c19d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f3b7fe700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f3c074b00 0x7f7f3c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f3bfff700 1 -- 172.21.15.2:0/299239620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f3c074b00 msgr2=0x7f7f3c198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f3bfff700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f3c074b00 0x7f7f3c198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f3bfff700 1 -- 172.21.15.2:0/299239620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f3c10ce50 msgr2=0x7f7f3c19d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f3bfff700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f3c10ce50 0x7f7f3c19d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f3bfff700 1 -- 172.21.15.2:0/299239620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f34009c40 con 0x7f7f3c0746a0 2023-11-16T18:28:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f4097b700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f3c10ce50 0x7f7f3c19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:28:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f3bfff700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f3c0746a0 0x7f7f3c198680 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c00e9c0 tx=0x7f7f2c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:39.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.817+0000 7f7f397fa700 1 -- 172.21.15.2:0/299239620 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c004a30 con 0x7f7f3c0746a0 2023-11-16T18:28:39.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.818+0000 7f7f397fa700 1 -- 172.21.15.2:0/299239620 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f2c00cc80 con 0x7f7f3c0746a0 2023-11-16T18:28:39.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.818+0000 7f7f423de700 1 -- 172.21.15.2:0/299239620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3c1124d0 con 0x7f7f3c0746a0 2023-11-16T18:28:39.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.818+0000 7f7f397fa700 1 -- 172.21.15.2:0/299239620 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c0054e0 con 0x7f7f3c0746a0 2023-11-16T18:28:39.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.818+0000 7f7f423de700 1 -- 172.21.15.2:0/299239620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3c112340 con 0x7f7f3c0746a0 2023-11-16T18:28:39.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.821+0000 7f7f397fa700 1 -- 172.21.15.2:0/299239620 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f7f2c004b90 con 0x7f7f3c0746a0 2023-11-16T18:28:39.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.821+0000 7f7f397fa700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f7f240a12c0 0x7f7f240a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.821+0000 7f7f397fa700 1 -- 172.21.15.2:0/299239620 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f7f2c014070 con 0x7f7f3c0746a0 2023-11-16T18:28:39.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.821+0000 7f7f3b7fe700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f7f240a12c0 0x7f7f240a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f3b7fe700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f7f240a12c0 0x7f7f240a3770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c199c90 tx=0x7f7f30009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:39.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f4e56ab80 1 Processor -- start 2023-11-16T18:28:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f4e56ab80 1 -- start start 2023-11-16T18:28:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f4e56ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 0x55ec7b225f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f4e56ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b22a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f4e56ab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b22a8b0 0x55ec7b22ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f4e56ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec7b12b060 con 0x55ec7b226470 2023-11-16T18:28:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f4e56ab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec7b17d8a0 con 0x55ec7b22a8b0 2023-11-16T18:28:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f4e56ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec7b176d30 con 0x55ec7b225b60 2023-11-16T18:28:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f3b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b22a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f3b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b22a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52296/0 (socket says 172.21.15.2:52296) 2023-11-16T18:28:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f3b7fe700 1 -- 172.21.15.2:0/3278886230 learned_addr learned my addr 172.21.15.2:0/3278886230 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f4097b700 1 --2- 172.21.15.2:0/3278886230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b22a8b0 0x55ec7b22ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f3b7fe700 1 -- 172.21.15.2:0/3278886230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 msgr2=0x55ec7b225f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.822+0000 7f7f3b7fe700 1 --2- 172.21.15.2:0/3278886230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 0x55ec7b225f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.823+0000 7f7f3b7fe700 1 -- 172.21.15.2:0/3278886230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b22a8b0 msgr2=0x55ec7b22ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.823+0000 7f7f3b7fe700 1 --2- 172.21.15.2:0/3278886230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b22a8b0 0x55ec7b22ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.823+0000 7f7f3b7fe700 1 -- 172.21.15.2:0/3278886230 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec7aff8630 con 0x55ec7b226470 2023-11-16T18:28:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.823+0000 7f7f3b7fe700 1 --2- 172.21.15.2:0/3278886230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b22a370 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f7f30011d40 tx=0x7f7f30014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7288fe89b5dbadf9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.823+0000 7f7f227fc700 1 -- 172.21.15.2:0/3278886230 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f30011d70 con 0x55ec7b226470 2023-11-16T18:28:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.823+0000 7f7f227fc700 1 -- 172.21.15.2:0/3278886230 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f30004020 con 0x55ec7b226470 2023-11-16T18:28:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.823+0000 7f7f227fc700 1 -- 172.21.15.2:0/3278886230 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f30003c70 con 0x55ec7b226470 2023-11-16T18:28:39.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.824+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/3278886230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 msgr2=0x55ec7b22a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.824+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/3278886230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b22a370 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f7f30011d40 tx=0x7f7f30014000 comp rx=0 tx=0).stop 2023-11-16T18:28:39.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.824+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/3278886230 shutdown_connections 2023-11-16T18:28:39.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.824+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/3278886230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 0x55ec7b225f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.824+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/3278886230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b22a8b0 0x55ec7b22ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.824+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/3278886230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b22a370 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.824+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/3278886230 >> 172.21.15.2:0/3278886230 conn(0x55ec7b21b360 msgr2=0x55ec7b220b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:39.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.824+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/3278886230 shutdown_connections 2023-11-16T18:28:39.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.824+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/3278886230 wait complete. 2023-11-16T18:28:39.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.824+0000 7f7f4e56ab80 1 -- start start 2023-11-16T18:28:39.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.824+0000 7f7f4e56ab80 1 Processor -- start 2023-11-16T18:28:39.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4e56ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 0x55ec7b232040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4e56ab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b226470 0x55ec7b22f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4e56ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b22a8b0 0x55ec7b22f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4e56ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec7b1188e0 con 0x55ec7b22a8b0 2023-11-16T18:28:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4e56ab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec7b118580 con 0x55ec7b226470 2023-11-16T18:28:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4e56ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec7affa920 con 0x55ec7b225b60 2023-11-16T18:28:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4e56ab80 10 client.0 _ll_register_callbacks cb 0x55ec7b136780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:28:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4097b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b22a8b0 0x55ec7b22f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f3b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b226470 0x55ec7b22f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4097b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b22a8b0 0x55ec7b22f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52304/0 (socket says 172.21.15.2:52304) 2023-11-16T18:28:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4097b700 1 -- 172.21.15.2:0/1608029702 learned_addr learned my addr 172.21.15.2:0/1608029702 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f3b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b226470 0x55ec7b22f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42740/0 (socket says 172.21.15.2:42740) 2023-11-16T18:28:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f3bfff700 1 --2- 172.21.15.2:0/1608029702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 0x55ec7b232040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4097b700 1 -- 172.21.15.2:0/1608029702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 msgr2=0x55ec7b232040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4097b700 1 --2- 172.21.15.2:0/1608029702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 0x55ec7b232040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4097b700 1 -- 172.21.15.2:0/1608029702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b226470 msgr2=0x55ec7b22f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4097b700 1 --2- 172.21.15.2:0/1608029702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b226470 0x55ec7b22f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f4097b700 1 -- 172.21.15.2:0/1608029702 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f30017070 con 0x55ec7b22a8b0 2023-11-16T18:28:39.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.825+0000 7f7f3b7fe700 1 --2- 172.21.15.2:0/1608029702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b226470 0x55ec7b22f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:28:39.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f4097b700 1 --2- 172.21.15.2:0/1608029702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b22a8b0 0x55ec7b22f940 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7f34015000 tx=0x7f7f34003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=befe891675bcf51b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:39.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f1b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:28:39.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f1b7fe700 1 -- 172.21.15.2:0/1608029702 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3400e040 con 0x55ec7b22a8b0 2023-11-16T18:28:39.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/1608029702 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec7b218b20 con 0x55ec7b22a8b0 2023-11-16T18:28:39.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f1b7fe700 1 -- 172.21.15.2:0/1608029702 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f7f340093a0 con 0x55ec7b22a8b0 2023-11-16T18:28:39.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f1b7fe700 1 -- 172.21.15.2:0/1608029702 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f34003e20 con 0x55ec7b22a8b0 2023-11-16T18:28:39.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f4e56ab80 10 client.16494 fetch_fsmap learned FSMap version 29 2023-11-16T18:28:39.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/1608029702 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ec7b13d190 con 0x55ec7b22a8b0 2023-11-16T18:28:39.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f1b7fe700 1 -- 172.21.15.2:0/1608029702 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ec7b218b20 con 0x55ec7b22a8b0 2023-11-16T18:28:39.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f1b7fe700 1 -- 172.21.15.2:0/1608029702 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f7f34019070 con 0x55ec7b22a8b0 2023-11-16T18:28:39.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f4e56ab80 10 client.16494 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:28:39.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f4e56ab80 10 client.16494 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:28:39.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f4e56ab80 20 client.16494 populate_metadata read hostname 'smithi002' 2023-11-16T18:28:39.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/1608029702 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ec7b23f8c0 0x55ec7b241cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f4e56ab80 4 client.16494 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:28:39.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.826+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/1608029702 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec7b1390a0 con 0x55ec7b23f8c0 2023-11-16T18:28:39.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.827+0000 7f7f3bfff700 1 --2- 172.21.15.2:0/1608029702 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ec7b23f8c0 0x55ec7b241cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.827+0000 7f7f1b7fe700 10 client.16494 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:39.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.827+0000 7f7f3bfff700 1 --2- 172.21.15.2:0/1608029702 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ec7b23f8c0 0x55ec7b241cd0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37ec4fc517219a9b server_cookie=f00b96685154bc12 in_seq=0 out_seq=0 2023-11-16T18:28:39.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f1b7fe700 1 -- 172.21.15.2:0/1608029702 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ec7b1390a0 con 0x55ec7b23f8c0 2023-11-16T18:28:39.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f1b7fe700 10 client.16494 handle_command_reply: tid=0 2023-11-16T18:28:39.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f4e56ab80 1 client.16494 shutdown 2023-11-16T18:28:39.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/1608029702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b22a8b0 msgr2=0x55ec7b22f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/1608029702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b22a8b0 0x55ec7b22f940 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7f34015000 tx=0x7f7f34003c30 comp rx=0 tx=0).stop 2023-11-16T18:28:39.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/1608029702 shutdown_connections 2023-11-16T18:28:39.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/1608029702 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ec7b23f8c0 0x55ec7b241cd0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/1608029702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 0x55ec7b232040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/1608029702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b22a8b0 0x55ec7b22f940 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f1b7fe700 0 client.16494 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:39.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/1608029702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b226470 0x55ec7b22f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/1608029702 >> 172.21.15.2:0/1608029702 conn(0x55ec7b21b360 msgr2=0x55ec7b21fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:39.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/1608029702 shutdown_connections 2023-11-16T18:28:39.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/1608029702 wait complete. 2023-11-16T18:28:39.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.828+0000 7f7f4e56ab80 20 client.16494 trim_cache size 0 max 16384 2023-11-16T18:28:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.846+0000 7f7f4e56ab80 1 Processor -- start 2023-11-16T18:28:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f4e56ab80 1 -- start start 2023-11-16T18:28:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f4e56ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 0x55ec7b2288b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f4e56ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b228df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f4e56ab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b22a8b0 0x55ec7b220560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f4e56ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec7b226bc0 con 0x55ec7b226470 2023-11-16T18:28:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f4e56ab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec7b22fcd0 con 0x55ec7b22a8b0 2023-11-16T18:28:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f4e56ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec7b21aee0 con 0x55ec7b225b60 2023-11-16T18:28:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f3b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b228df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f4097b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b22a8b0 0x55ec7b220560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f3bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 0x55ec7b2288b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f3b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b228df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52312/0 (socket says 172.21.15.2:52312) 2023-11-16T18:28:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f4097b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b22a8b0 0x55ec7b220560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42748/0 (socket says 172.21.15.2:42748) 2023-11-16T18:28:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f3b7fe700 1 -- 172.21.15.2:0/1170939806 learned_addr learned my addr 172.21.15.2:0/1170939806 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f3bfff700 1 -- 172.21.15.2:0/1170939806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b22a8b0 msgr2=0x55ec7b220560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f3bfff700 1 --2- 172.21.15.2:0/1170939806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b22a8b0 0x55ec7b220560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f3bfff700 1 -- 172.21.15.2:0/1170939806 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 msgr2=0x55ec7b228df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f3bfff700 1 --2- 172.21.15.2:0/1170939806 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b228df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f3bfff700 1 -- 172.21.15.2:0/1170939806 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec7b21cb00 con 0x55ec7b225b60 2023-11-16T18:28:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.847+0000 7f7f3bfff700 1 --2- 172.21.15.2:0/1170939806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 0x55ec7b2288b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0ca540 tx=0x7f7f2c01a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f6f9ec97486cdb5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.848+0000 7f7f22ffd700 1 -- 172.21.15.2:0/1170939806 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c0b82c0 con 0x55ec7b225b60 2023-11-16T18:28:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.848+0000 7f7f22ffd700 1 -- 172.21.15.2:0/1170939806 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f2c09def0 con 0x55ec7b225b60 2023-11-16T18:28:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.848+0000 7f7f22ffd700 1 -- 172.21.15.2:0/1170939806 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c09a120 con 0x55ec7b225b60 2023-11-16T18:28:39.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.848+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/1170939806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 msgr2=0x55ec7b2288b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.848+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/1170939806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 0x55ec7b2288b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0ca540 tx=0x7f7f2c01a040 comp rx=0 tx=0).stop 2023-11-16T18:28:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.848+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/1170939806 shutdown_connections 2023-11-16T18:28:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.848+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/1170939806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b225b60 0x55ec7b2288b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.848+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/1170939806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b22a8b0 0x55ec7b220560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.848+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/1170939806 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b228df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.848+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/1170939806 >> 172.21.15.2:0/1170939806 conn(0x55ec7b21b360 msgr2=0x55ec7b22cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.848+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/1170939806 shutdown_connections 2023-11-16T18:28:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.849+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/1170939806 wait complete. 2023-11-16T18:28:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.849+0000 7f7f4e56ab80 1 -- start start 2023-11-16T18:28:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.849+0000 7f7f4e56ab80 1 Processor -- start 2023-11-16T18:28:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.849+0000 7f7f4e56ab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b225b60 0x55ec7b2341c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.849+0000 7f7f4e56ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b234700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.849+0000 7f7f4e56ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b22a8b0 0x55ec7b234c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.849+0000 7f7f4e56ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec7b2927e0 con 0x55ec7b226470 2023-11-16T18:28:39.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.849+0000 7f7f4e56ab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec7b292950 con 0x55ec7b225b60 2023-11-16T18:28:39.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.849+0000 7f7f4e56ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec7b292b10 con 0x55ec7b22a8b0 2023-11-16T18:28:39.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.849+0000 7f7f4e56ab80 10 client.0 _ll_register_callbacks cb 0x7f7f34007de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:28:39.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.849+0000 7f7f3b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b234700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.849+0000 7f7f3bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b225b60 0x55ec7b2341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f3b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b234700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52316/0 (socket says 172.21.15.2:52316) 2023-11-16T18:28:39.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f3b7fe700 1 -- 172.21.15.2:0/3924669142 learned_addr learned my addr 172.21.15.2:0/3924669142 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:39.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f4097b700 1 --2- 172.21.15.2:0/3924669142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b22a8b0 0x55ec7b234c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f3b7fe700 1 -- 172.21.15.2:0/3924669142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b22a8b0 msgr2=0x55ec7b234c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f3b7fe700 1 --2- 172.21.15.2:0/3924669142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b22a8b0 0x55ec7b234c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f3b7fe700 1 -- 172.21.15.2:0/3924669142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b225b60 msgr2=0x55ec7b2341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f3b7fe700 1 --2- 172.21.15.2:0/3924669142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b225b60 0x55ec7b2341c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f3b7fe700 1 -- 172.21.15.2:0/3924669142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3c112340 con 0x55ec7b226470 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f3bfff700 1 --2- 172.21.15.2:0/3924669142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b225b60 0x55ec7b2341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f3b7fe700 1 --2- 172.21.15.2:0/3924669142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b234700 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7f30011d40 tx=0x7f7f300052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b82be0a5b240fa server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f1b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f1b7fe700 1 -- 172.21.15.2:0/3924669142 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f30003880 con 0x55ec7b226470 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/3924669142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7f30004020 con 0x55ec7b226470 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f1b7fe700 1 -- 172.21.15.2:0/3924669142 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f7f3002c050 con 0x55ec7b226470 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f1b7fe700 1 -- 172.21.15.2:0/3924669142 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f300056e0 con 0x55ec7b226470 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f4e56ab80 10 client.16500 fetch_fsmap learned FSMap version 29 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.850+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/3924669142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ec7afa4de0 con 0x55ec7b226470 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.851+0000 7f7f1b7fe700 1 -- 172.21.15.2:0/3924669142 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f30004020 con 0x55ec7b226470 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.851+0000 7f7f1b7fe700 1 -- 172.21.15.2:0/3924669142 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f7f30024030 con 0x55ec7b226470 2023-11-16T18:28:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.851+0000 7f7f4e56ab80 10 client.16500 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:28:39.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.851+0000 7f7f4e56ab80 10 client.16500 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:28:39.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.851+0000 7f7f4e56ab80 20 client.16500 populate_metadata read hostname 'smithi002' 2023-11-16T18:28:39.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.851+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/3924669142 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ec7b294210 0x55ec7b296620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:39.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.851+0000 7f7f4e56ab80 4 client.16500 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:28:39.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.851+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/3924669142 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ec7b145660 con 0x55ec7b294210 2023-11-16T18:28:39.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.851+0000 7f7f3bfff700 1 --2- 172.21.15.2:0/3924669142 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ec7b294210 0x55ec7b296620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:39.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.851+0000 7f7f1b7fe700 10 client.16500 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:39.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.851+0000 7f7f3bfff700 1 --2- 172.21.15.2:0/3924669142 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ec7b294210 0x55ec7b296620 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=811d05252c07bbad server_cookie=80e790c54206f7b6 in_seq=0 out_seq=0 2023-11-16T18:28:39.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f1b7fe700 1 -- 172.21.15.2:0/3924669142 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55ec7b145660 con 0x55ec7b294210 2023-11-16T18:28:39.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f1b7fe700 10 client.16500 handle_command_reply: tid=0 2023-11-16T18:28:39.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f4e56ab80 1 client.16500 shutdown 2023-11-16T18:28:39.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/3924669142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 msgr2=0x55ec7b234700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/3924669142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b234700 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7f30011d40 tx=0x7f7f300052e0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/3924669142 shutdown_connections 2023-11-16T18:28:39.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/3924669142 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ec7b294210 0x55ec7b296620 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/3924669142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec7b22a8b0 0x55ec7b234c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/3924669142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec7b225b60 0x55ec7b2341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f1b7fe700 0 client.16500 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:39.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f4e56ab80 1 --2- 172.21.15.2:0/3924669142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ec7b226470 0x55ec7b234700 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/3924669142 >> 172.21.15.2:0/3924669142 conn(0x55ec7b21b360 msgr2=0x55ec7b2251f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:39.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/3924669142 shutdown_connections 2023-11-16T18:28:39.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f4e56ab80 1 -- 172.21.15.2:0/3924669142 wait complete. 2023-11-16T18:28:39.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.852+0000 7f7f4e56ab80 20 client.16500 trim_cache size 0 max 16384 2023-11-16T18:28:39.860 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:28:39.860 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "scrub active (1 inodes in the stack)", 2023-11-16T18:28:39.860 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": { 2023-11-16T18:28:39.861 INFO:teuthology.orchestra.run.smithi002.stdout: "9607aa23-76ae-4198-9378-3740d8aff22c": { 2023-11-16T18:28:39.861 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/", 2023-11-16T18:28:39.861 INFO:teuthology.orchestra.run.smithi002.stdout: "tag": "9607aa23-76ae-4198-9378-3740d8aff22c", 2023-11-16T18:28:39.861 INFO:teuthology.orchestra.run.smithi002.stdout: "options": "recursive,force" 2023-11-16T18:28:39.861 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:28:39.861 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:28:39.861 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:28:39.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.853+0000 7f7f423de700 1 -- 172.21.15.2:0/299239620 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f7f240a12c0 msgr2=0x7f7f240a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.853+0000 7f7f423de700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f7f240a12c0 0x7f7f240a3770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c199c90 tx=0x7f7f30009260 comp rx=0 tx=0).stop 2023-11-16T18:28:39.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.853+0000 7f7f423de700 1 -- 172.21.15.2:0/299239620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f3c0746a0 msgr2=0x7f7f3c198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:39.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.853+0000 7f7f423de700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f3c0746a0 0x7f7f3c198680 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c00e9c0 tx=0x7f7f2c00ecd0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.853+0000 7f7f423de700 1 -- 172.21.15.2:0/299239620 shutdown_connections 2023-11-16T18:28:39.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.853+0000 7f7f423de700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f7f240a12c0 0x7f7f240a3770 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.853+0000 7f7f423de700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f3c10ce50 0x7f7f3c19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.853+0000 7f7f423de700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f3c0746a0 0x7f7f3c198680 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.853+0000 7f7f423de700 1 --2- 172.21.15.2:0/299239620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f3c074b00 0x7f7f3c198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:39.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.853+0000 7f7f423de700 1 -- 172.21.15.2:0/299239620 >> 172.21.15.2:0/299239620 conn(0x7f7f3c077500 msgr2=0x7f7f3c104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:39.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.853+0000 7f7f423de700 1 -- 172.21.15.2:0/299239620 shutdown_connections 2023-11-16T18:28:39.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:28:39.853+0000 7f7f423de700 1 -- 172.21.15.2:0/299239620 wait complete. 2023-11-16T18:28:39.867 INFO:tasks.cephfs.filesystem:scrub status for tag:9607aa23-76ae-4198-9378-3740d8aff22c - {'path': '/', 'tag': '9607aa23-76ae-4198-9378-3740d8aff22c', 'options': 'recursive,force'} 2023-11-16T18:28:42.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:42 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:42.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:42 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:42.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:42 smithi002 ceph-mon[98807]: pgmap v118: 129 pgs: 129 active+clean; 352 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 924 op/s 2023-11-16T18:28:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:42 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:42 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:42 smithi040 ceph-mon[100251]: pgmap v118: 129 pgs: 129 active+clean; 352 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 924 op/s 2023-11-16T18:28:42.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:42 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:42.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:42 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:42.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:42 smithi188 ceph-mon[100639]: pgmap v118: 129 pgs: 129 active+clean; 352 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 924 op/s 2023-11-16T18:28:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.121+0000 7f94b2525700 1 -- 172.21.15.2:0/1496110185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac108030 msgr2=0x7f94ac1084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.121+0000 7f94b2525700 1 --2- 172.21.15.2:0/1496110185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac108030 0x7f94ac1084a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f949c0098f0 tx=0x7f949c009c00 comp rx=0 tx=0).stop 2023-11-16T18:28:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.121+0000 7f94b2525700 1 -- 172.21.15.2:0/1496110185 shutdown_connections 2023-11-16T18:28:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.121+0000 7f94b2525700 1 --2- 172.21.15.2:0/1496110185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94ac1152f0 0x7f94ac1176d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.121+0000 7f94b2525700 1 --2- 172.21.15.2:0/1496110185 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94ac072840 0x7f94ac107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.121+0000 7f94b2525700 1 --2- 172.21.15.2:0/1496110185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac108030 0x7f94ac1084a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.121+0000 7f94b2525700 1 -- 172.21.15.2:0/1496110185 >> 172.21.15.2:0/1496110185 conn(0x7f94ac06c490 msgr2=0x7f94ac071690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.121+0000 7f94b2525700 1 -- 172.21.15.2:0/1496110185 shutdown_connections 2023-11-16T18:28:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.121+0000 7f94b2525700 1 -- 172.21.15.2:0/1496110185 wait complete. 2023-11-16T18:28:43.124 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.122+0000 7f94b2525700 1 Processor -- start 2023-11-16T18:28:43.124 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.122+0000 7f94b2525700 1 -- start start 2023-11-16T18:28:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94b2525700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac072840 0x7f94ac10aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94b2525700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94ac108030 0x7f94ac10b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94b2525700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94ac1152f0 0x7f94ac10b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94b2525700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac1b1610 con 0x7f94ac108030 2023-11-16T18:28:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94b2525700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac1b1780 con 0x7f94ac1152f0 2023-11-16T18:28:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94b2525700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac1b1940 con 0x7f94ac072840 2023-11-16T18:28:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94ab7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94ac108030 0x7f94ac10b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94ab7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94ac108030 0x7f94ac10b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52328/0 (socket says 172.21.15.2:52328) 2023-11-16T18:28:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94ab7fe700 1 -- 172.21.15.2:0/571702673 learned_addr learned my addr 172.21.15.2:0/571702673 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94b0ac2700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94ac1152f0 0x7f94ac10b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94ab7fe700 1 -- 172.21.15.2:0/571702673 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac072840 msgr2=0x7f94ac10aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94ab7fe700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac072840 0x7f94ac10aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.126 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94ab7fe700 1 -- 172.21.15.2:0/571702673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94ac1152f0 msgr2=0x7f94ac10b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:43.126 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94ab7fe700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94ac1152f0 0x7f94ac10b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.126 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94ab7fe700 1 -- 172.21.15.2:0/571702673 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a4009d00 con 0x7f94ac108030 2023-11-16T18:28:43.126 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.123+0000 7f94b0ac2700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94ac1152f0 0x7f94ac10b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:28:43.126 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.124+0000 7f94ab7fe700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94ac108030 0x7f94ac10b420 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f949c009230 tx=0x7f949c00b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.125+0000 7f94a97fa700 1 -- 172.21.15.2:0/571702673 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c01c070 con 0x7f94ac108030 2023-11-16T18:28:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.125+0000 7f94a97fa700 1 -- 172.21.15.2:0/571702673 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f949c0058e0 con 0x7f94ac108030 2023-11-16T18:28:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.125+0000 7f94b2525700 1 -- 172.21.15.2:0/571702673 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f949c009520 con 0x7f94ac108030 2023-11-16T18:28:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.125+0000 7f94a97fa700 1 -- 172.21.15.2:0/571702673 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c003970 con 0x7f94ac108030 2023-11-16T18:28:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.126+0000 7f94b2525700 1 -- 172.21.15.2:0/571702673 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac11ace0 con 0x7f94ac108030 2023-11-16T18:28:43.131 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.130+0000 7f94a97fa700 1 -- 172.21.15.2:0/571702673 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f949c020030 con 0x7f94ac108030 2023-11-16T18:28:43.131 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.131+0000 7f94be6b1b80 1 Processor -- start 2023-11-16T18:28:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.131+0000 7f94a97fa700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f94980a1340 0x7f94980a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.131+0000 7f94a97fa700 1 -- 172.21.15.2:0/571702673 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f949c0ce110 con 0x7f94ac108030 2023-11-16T18:28:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.132+0000 7f94be6b1b80 1 -- start start 2023-11-16T18:28:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.132+0000 7f94b0ac2700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f94980a1340 0x7f94980a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.132+0000 7f94be6b1b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5b646b0 0x55bba5d0e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.132+0000 7f94be6b1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5d0e720 0x55bba5d08560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.132+0000 7f94be6b1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0eb90 0x55bba5d08b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.132+0000 7f94be6b1b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bba5c0e000 con 0x55bba5b646b0 2023-11-16T18:28:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.132+0000 7f94be6b1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bba5c60840 con 0x55bba5d0eb90 2023-11-16T18:28:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.132+0000 7f94be6b1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bba5c59cd0 con 0x55bba5d0e720 2023-11-16T18:28:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.133+0000 7f94ab7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5d0e720 0x55bba5d08560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.133+0000 7f94ab7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5d0e720 0x55bba5d08560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43866/0 (socket says 172.21.15.2:43866) 2023-11-16T18:28:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.133+0000 7f94ab7fe700 1 -- 172.21.15.2:0/2373904066 learned_addr learned my addr 172.21.15.2:0/2373904066 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.133+0000 7f94abfff700 1 --2- 172.21.15.2:0/2373904066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5b646b0 0x55bba5d0e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.133+0000 7f94abfff700 1 -- 172.21.15.2:0/2373904066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5d0e720 msgr2=0x55bba5d08560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.133+0000 7f94abfff700 1 --2- 172.21.15.2:0/2373904066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5d0e720 0x55bba5d08560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.133+0000 7f94abfff700 1 -- 172.21.15.2:0/2373904066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0eb90 msgr2=0x55bba5d08b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.133+0000 7f94abfff700 1 --2- 172.21.15.2:0/2373904066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0eb90 0x55bba5d08b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.136 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.133+0000 7f94abfff700 1 -- 172.21.15.2:0/2373904066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bba5adb5d0 con 0x55bba5b646b0 2023-11-16T18:28:43.136 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.133+0000 7f94b0ac2700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f94980a1340 0x7f94980a37f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f94ac06a2a0 tx=0x7f94940103d0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:43.136 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.133+0000 7f94abfff700 1 --2- 172.21.15.2:0/2373904066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5b646b0 0x55bba5d0e1e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009fd0 tx=0x7f94a400e860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1aab64a56d5a66 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:43.136 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.134+0000 7f94a27fc700 1 -- 172.21.15.2:0/2373904066 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a400e050 con 0x55bba5b646b0 2023-11-16T18:28:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.134+0000 7f94a27fc700 1 -- 172.21.15.2:0/2373904066 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94a400f040 con 0x55bba5b646b0 2023-11-16T18:28:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.134+0000 7f94a27fc700 1 -- 172.21.15.2:0/2373904066 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a4007e10 con 0x55bba5b646b0 2023-11-16T18:28:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.135+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2373904066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5b646b0 msgr2=0x55bba5d0e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.135+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2373904066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5b646b0 0x55bba5d0e1e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009fd0 tx=0x7f94a400e860 comp rx=0 tx=0).stop 2023-11-16T18:28:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.135+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2373904066 shutdown_connections 2023-11-16T18:28:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.135+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2373904066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0eb90 0x55bba5d08b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.135+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2373904066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5b646b0 0x55bba5d0e1e0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.135+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2373904066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5d0e720 0x55bba5d08560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.135+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2373904066 >> 172.21.15.2:0/2373904066 conn(0x55bba5cfdf50 msgr2=0x55bba5d03270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.135+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2373904066 shutdown_connections 2023-11-16T18:28:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.135+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2373904066 wait complete. 2023-11-16T18:28:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.135+0000 7f94be6b1b80 1 -- start start 2023-11-16T18:28:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.136+0000 7f94be6b1b80 1 Processor -- start 2023-11-16T18:28:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.136+0000 7f94be6b1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5b646b0 0x55bba5d11d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.136+0000 7f94be6b1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5d0e720 0x55bba5d122a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.136+0000 7f94be6b1b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0eb90 0x55bba5d127e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.136+0000 7f94be6b1b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bba5bfb880 con 0x55bba5d0eb90 2023-11-16T18:28:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.136+0000 7f94be6b1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bba5bfb520 con 0x55bba5b646b0 2023-11-16T18:28:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.136+0000 7f94be6b1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bba5add8c0 con 0x55bba5d0e720 2023-11-16T18:28:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.136+0000 7f94be6b1b80 10 client.0 _ll_register_callbacks cb 0x55bba5c19720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:28:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.137+0000 7f94b0ac2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0eb90 0x55bba5d127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.138+0000 7f94ab7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5d0e720 0x55bba5d122a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.138+0000 7f94ab7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5d0e720 0x55bba5d122a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43874/0 (socket says 172.21.15.2:43874) 2023-11-16T18:28:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.138+0000 7f94ab7fe700 1 -- 172.21.15.2:0/2874129578 learned_addr learned my addr 172.21.15.2:0/2874129578 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.138+0000 7f94abfff700 1 --2- 172.21.15.2:0/2874129578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5b646b0 0x55bba5d11d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.139+0000 7f94abfff700 1 -- 172.21.15.2:0/2874129578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5d0e720 msgr2=0x55bba5d122a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.139+0000 7f94abfff700 1 --2- 172.21.15.2:0/2874129578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5d0e720 0x55bba5d122a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.139+0000 7f94abfff700 1 -- 172.21.15.2:0/2874129578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0eb90 msgr2=0x55bba5d127e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.139+0000 7f94abfff700 1 --2- 172.21.15.2:0/2874129578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0eb90 0x55bba5d127e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.139+0000 7f94abfff700 1 -- 172.21.15.2:0/2874129578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a400e260 con 0x55bba5b646b0 2023-11-16T18:28:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.139+0000 7f94abfff700 1 --2- 172.21.15.2:0/2874129578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5b646b0 0x55bba5d11d60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009a20 tx=0x7f94a4007840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=836b6af9f4cd2c9d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.139+0000 7f948b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:28:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.140+0000 7f948b7fe700 1 -- 172.21.15.2:0/2874129578 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a4004bb0 con 0x55bba5b646b0 2023-11-16T18:28:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.140+0000 7f948b7fe700 1 -- 172.21.15.2:0/2874129578 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f94a4004d10 con 0x55bba5b646b0 2023-11-16T18:28:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.140+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2874129578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bba5cfb710 con 0x55bba5b646b0 2023-11-16T18:28:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.141+0000 7f948b7fe700 1 -- 172.21.15.2:0/2874129578 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94a4003a00 con 0x55bba5b646b0 2023-11-16T18:28:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.141+0000 7f94be6b1b80 10 client.26051 fetch_fsmap learned FSMap version 29 2023-11-16T18:28:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.141+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2874129578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55bba5c20130 con 0x55bba5b646b0 2023-11-16T18:28:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.142+0000 7f948b7fe700 1 -- 172.21.15.2:0/2874129578 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bba5cfb710 con 0x55bba5b646b0 2023-11-16T18:28:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.142+0000 7f948b7fe700 1 -- 172.21.15.2:0/2874129578 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f94a4023070 con 0x55bba5b646b0 2023-11-16T18:28:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.142+0000 7f94be6b1b80 10 client.26051 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:28:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.142+0000 7f94be6b1b80 10 client.26051 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:28:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.142+0000 7f94be6b1b80 20 client.26051 populate_metadata read hostname 'smithi002' 2023-11-16T18:28:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.142+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2874129578 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bba5d21f80 0x55bba5d24390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.142+0000 7f94be6b1b80 4 client.26051 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:28:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.142+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2874129578 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bba5c1c040 con 0x55bba5d21f80 2023-11-16T18:28:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.143+0000 7f94ab7fe700 1 --2- 172.21.15.2:0/2874129578 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bba5d21f80 0x55bba5d24390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.143+0000 7f948b7fe700 10 client.26051 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.143+0000 7f94ab7fe700 1 --2- 172.21.15.2:0/2874129578 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bba5d21f80 0x55bba5d24390 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f90f9cc57bc03333 server_cookie=ea4a95a6e42779bc in_seq=0 out_seq=0 2023-11-16T18:28:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.149+0000 7f948b7fe700 1 -- 172.21.15.2:0/2874129578 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55bba5c1c040 con 0x55bba5d21f80 2023-11-16T18:28:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.149+0000 7f948b7fe700 10 client.26051 handle_command_reply: tid=0 2023-11-16T18:28:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.149+0000 7f94be6b1b80 1 client.26051 shutdown 2023-11-16T18:28:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.150+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2874129578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5b646b0 msgr2=0x55bba5d11d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.150+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2874129578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5b646b0 0x55bba5d11d60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009a20 tx=0x7f94a4007840 comp rx=0 tx=0).stop 2023-11-16T18:28:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.150+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2874129578 shutdown_connections 2023-11-16T18:28:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.150+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2874129578 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bba5d21f80 0x55bba5d24390 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.150+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2874129578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0eb90 0x55bba5d127e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.150+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2874129578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5b646b0 0x55bba5d11d60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.150+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2874129578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5d0e720 0x55bba5d122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.150+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2874129578 >> 172.21.15.2:0/2874129578 conn(0x55bba5cfdf50 msgr2=0x55bba5d027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.150+0000 7f948b7fe700 0 client.26051 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.151+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2874129578 shutdown_connections 2023-11-16T18:28:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.151+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2874129578 wait complete. 2023-11-16T18:28:43.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.152+0000 7f94be6b1b80 20 client.26051 trim_cache size 0 max 16384 2023-11-16T18:28:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.178+0000 7f94be6b1b80 1 Processor -- start 2023-11-16T18:28:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.178+0000 7f94be6b1b80 1 -- start start 2023-11-16T18:28:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.178+0000 7f94be6b1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5b646b0 0x55bba5d19090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.178+0000 7f94be6b1b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0e720 0x55bba5d0d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.178+0000 7f94be6b1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0eb90 0x55bba5d02f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.178+0000 7f94be6b1b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bba5cfffd0 con 0x55bba5d0e720 2023-11-16T18:28:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.178+0000 7f94be6b1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bba5d0c9f0 con 0x55bba5d0eb90 2023-11-16T18:28:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.178+0000 7f94be6b1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bba5d0ddd0 con 0x55bba5b646b0 2023-11-16T18:28:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94ab7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0e720 0x55bba5d0d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94ab7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0e720 0x55bba5d0d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52350/0 (socket says 172.21.15.2:52350) 2023-11-16T18:28:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94ab7fe700 1 -- 172.21.15.2:0/2512833456 learned_addr learned my addr 172.21.15.2:0/2512833456 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94abfff700 1 --2- 172.21.15.2:0/2512833456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5b646b0 0x55bba5d19090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94b0ac2700 1 --2- 172.21.15.2:0/2512833456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0eb90 0x55bba5d02f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94ab7fe700 1 -- 172.21.15.2:0/2512833456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5b646b0 msgr2=0x55bba5d19090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94ab7fe700 1 --2- 172.21.15.2:0/2512833456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5b646b0 0x55bba5d19090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94ab7fe700 1 -- 172.21.15.2:0/2512833456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0eb90 msgr2=0x55bba5d02f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94ab7fe700 1 --2- 172.21.15.2:0/2512833456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0eb90 0x55bba5d02f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94ab7fe700 1 -- 172.21.15.2:0/2512833456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bba5c1fe80 con 0x55bba5d0e720 2023-11-16T18:28:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94ab7fe700 1 --2- 172.21.15.2:0/2512833456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0e720 0x55bba5d0d510 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f949c0058b0 tx=0x7f949c0055e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77021d507b9e6d16 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:43.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94a2ffd700 1 -- 172.21.15.2:0/2512833456 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c0b7070 con 0x55bba5d0e720 2023-11-16T18:28:43.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94a2ffd700 1 -- 172.21.15.2:0/2512833456 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f949c099040 con 0x55bba5d0e720 2023-11-16T18:28:43.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.179+0000 7f94a2ffd700 1 -- 172.21.15.2:0/2512833456 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c005240 con 0x55bba5d0e720 2023-11-16T18:28:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.180+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2512833456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0e720 msgr2=0x55bba5d0d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.180+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2512833456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0e720 0x55bba5d0d510 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f949c0058b0 tx=0x7f949c0055e0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.181+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2512833456 shutdown_connections 2023-11-16T18:28:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.181+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2512833456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5b646b0 0x55bba5d19090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.181+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2512833456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0eb90 0x55bba5d02f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.181+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/2512833456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0e720 0x55bba5d0d510 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.181+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2512833456 >> 172.21.15.2:0/2512833456 conn(0x55bba5cfdf50 msgr2=0x55bba5d0ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.181+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2512833456 shutdown_connections 2023-11-16T18:28:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.181+0000 7f94be6b1b80 1 -- 172.21.15.2:0/2512833456 wait complete. 2023-11-16T18:28:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.181+0000 7f94be6b1b80 1 -- start start 2023-11-16T18:28:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.182+0000 7f94be6b1b80 1 Processor -- start 2023-11-16T18:28:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.182+0000 7f94be6b1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5b646b0 0x55bba5d19090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.182+0000 7f94be6b1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0e720 0x55bba5d14df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.182+0000 7f94be6b1b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0eb90 0x55bba5d15330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.182+0000 7f94be6b1b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bba5d159f0 con 0x55bba5d0eb90 2023-11-16T18:28:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.182+0000 7f94be6b1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bba5d15b60 con 0x55bba5d0e720 2023-11-16T18:28:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.182+0000 7f94be6b1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bba5d15ca0 con 0x55bba5b646b0 2023-11-16T18:28:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.182+0000 7f94be6b1b80 10 client.0 _ll_register_callbacks cb 0x7f949c09acd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:28:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.182+0000 7f94b0ac2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0eb90 0x55bba5d15330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.182+0000 7f94b0ac2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0eb90 0x55bba5d15330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52356/0 (socket says 172.21.15.2:52356) 2023-11-16T18:28:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.182+0000 7f94b0ac2700 1 -- 172.21.15.2:0/3981652862 learned_addr learned my addr 172.21.15.2:0/3981652862 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:28:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.183+0000 7f94ab7fe700 1 --2- 172.21.15.2:0/3981652862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0e720 0x55bba5d14df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.183+0000 7f94b0ac2700 1 -- 172.21.15.2:0/3981652862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5b646b0 msgr2=0x55bba5d19090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.183+0000 7f94b0ac2700 1 --2- 172.21.15.2:0/3981652862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5b646b0 0x55bba5d19090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.183+0000 7f94b0ac2700 1 -- 172.21.15.2:0/3981652862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0e720 msgr2=0x55bba5d14df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:28:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.183+0000 7f94b0ac2700 1 --2- 172.21.15.2:0/3981652862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0e720 0x55bba5d14df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.183+0000 7f94b0ac2700 1 -- 172.21.15.2:0/3981652862 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac11ace0 con 0x55bba5d0eb90 2023-11-16T18:28:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.183+0000 7f94b0ac2700 1 --2- 172.21.15.2:0/3981652862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0eb90 0x55bba5d15330 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9494011720 tx=0x7f94940047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90e8fae71bfa58c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:28:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.183+0000 7f948b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:28:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f948b7fe700 1 -- 172.21.15.2:0/3981652862 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9494020830 con 0x55bba5d0eb90 2023-11-16T18:28:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f948b7fe700 1 -- 172.21.15.2:0/3981652862 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f9494019070 con 0x55bba5d0eb90 2023-11-16T18:28:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f94be6b1b80 1 -- 172.21.15.2:0/3981652862 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f94a400f040 con 0x55bba5d0eb90 2023-11-16T18:28:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f948b7fe700 1 -- 172.21.15.2:0/3981652862 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9494020990 con 0x55bba5d0eb90 2023-11-16T18:28:43.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f94be6b1b80 10 client.16515 fetch_fsmap learned FSMap version 29 2023-11-16T18:28:43.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f94be6b1b80 1 -- 172.21.15.2:0/3981652862 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55bba5a70c00 con 0x55bba5d0eb90 2023-11-16T18:28:43.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f948b7fe700 1 -- 172.21.15.2:0/3981652862 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94a400f040 con 0x55bba5d0eb90 2023-11-16T18:28:43.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f948b7fe700 1 -- 172.21.15.2:0/3981652862 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f94940283b0 con 0x55bba5d0eb90 2023-11-16T18:28:43.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f94be6b1b80 10 client.16515 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:28:43.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f94be6b1b80 10 client.16515 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:28:43.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f94be6b1b80 20 client.16515 populate_metadata read hostname 'smithi002' 2023-11-16T18:28:43.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/3981652862 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bba5d874a0 0x55bba5d898b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:28:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f94be6b1b80 4 client.16515 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:28:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.184+0000 7f94be6b1b80 1 -- 172.21.15.2:0/3981652862 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bba5c28600 con 0x55bba5d874a0 2023-11-16T18:28:43.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.185+0000 7f94abfff700 1 --2- 172.21.15.2:0/3981652862 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bba5d874a0 0x55bba5d898b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:28:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.187+0000 7f948b7fe700 10 client.16515 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:43.187+0000 7f94abfff700 1 --2- 172.21.15.2:0/3981652862 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bba5d874a0 0x55bba5d898b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c073a824507c5c4d server_cookie=3fb5710dc34d9cb2 in_seq=0 out_seq=0 2023-11-16T18:28:44.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:44 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:44.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:44 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:44.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:44 smithi002 ceph-mon[98807]: pgmap v119: 129 pgs: 129 active+clean; 383 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.2 MiB/s wr, 1.06k op/s 2023-11-16T18:28:44.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:44 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:44.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:44 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:44.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:44 smithi040 ceph-mon[100251]: pgmap v119: 129 pgs: 129 active+clean; 383 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.2 MiB/s wr, 1.06k op/s 2023-11-16T18:28:44.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:44 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:44.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:44 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:44.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:44 smithi188 ceph-mon[100639]: pgmap v119: 129 pgs: 129 active+clean; 383 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.2 MiB/s wr, 1.06k op/s 2023-11-16T18:28:46.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:46 smithi002 ceph-mon[98807]: pgmap v120: 129 pgs: 129 active+clean; 383 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 612 op/s 2023-11-16T18:28:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:46 smithi040 ceph-mon[100251]: pgmap v120: 129 pgs: 129 active+clean; 383 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 612 op/s 2023-11-16T18:28:46.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:46 smithi188 ceph-mon[100639]: pgmap v120: 129 pgs: 129 active+clean; 383 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 612 op/s 2023-11-16T18:28:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.015+0000 7f948b7fe700 1 -- 172.21.15.2:0/3981652862 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bba5c28600 con 0x55bba5d874a0 2023-11-16T18:28:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.015+0000 7f948b7fe700 10 client.16515 handle_command_reply: tid=0 2023-11-16T18:28:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.015+0000 7f94be6b1b80 1 client.16515 shutdown 2023-11-16T18:28:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.016+0000 7f94be6b1b80 1 -- 172.21.15.2:0/3981652862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0eb90 msgr2=0x55bba5d15330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.016+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/3981652862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0eb90 0x55bba5d15330 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9494011720 tx=0x7f94940047c0 comp rx=0 tx=0).stop 2023-11-16T18:28:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.016+0000 7f94be6b1b80 1 -- 172.21.15.2:0/3981652862 shutdown_connections 2023-11-16T18:28:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.016+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/3981652862 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bba5d874a0 0x55bba5d898b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.016+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/3981652862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bba5b646b0 0x55bba5d19090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.016+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/3981652862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bba5d0eb90 0x55bba5d15330 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.016+0000 7f94be6b1b80 1 --2- 172.21.15.2:0/3981652862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bba5d0e720 0x55bba5d14df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.016+0000 7f94be6b1b80 1 -- 172.21.15.2:0/3981652862 >> 172.21.15.2:0/3981652862 conn(0x55bba5cfdf50 msgr2=0x55bba5d0d690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.016+0000 7f948b7fe700 0 client.16515 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:28:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.016+0000 7f94be6b1b80 1 -- 172.21.15.2:0/3981652862 shutdown_connections 2023-11-16T18:28:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.016+0000 7f94be6b1b80 1 -- 172.21.15.2:0/3981652862 wait complete. 2023-11-16T18:28:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.016+0000 7f94be6b1b80 20 client.16515 trim_cache size 0 max 16384 2023-11-16T18:28:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:28:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:28:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:28:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:28:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.017+0000 7f94b2525700 1 -- 172.21.15.2:0/571702673 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f94980a1340 msgr2=0x7f94980a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.017+0000 7f94b2525700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f94980a1340 0x7f94980a37f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f94ac06a2a0 tx=0x7f94940103d0 comp rx=0 tx=0).stop 2023-11-16T18:28:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.017+0000 7f94b2525700 1 -- 172.21.15.2:0/571702673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94ac108030 msgr2=0x7f94ac10b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:28:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.017+0000 7f94b2525700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94ac108030 0x7f94ac10b420 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f949c009230 tx=0x7f949c00b560 comp rx=0 tx=0).stop 2023-11-16T18:28:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.017+0000 7f94b2525700 1 -- 172.21.15.2:0/571702673 shutdown_connections 2023-11-16T18:28:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.017+0000 7f94b2525700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f94980a1340 0x7f94980a37f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.017+0000 7f94b2525700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac072840 0x7f94ac10aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.017+0000 7f94b2525700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94ac1152f0 0x7f94ac10b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.017+0000 7f94b2525700 1 --2- 172.21.15.2:0/571702673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94ac108030 0x7f94ac10b420 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:28:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.017+0000 7f94b2525700 1 -- 172.21.15.2:0/571702673 >> 172.21.15.2:0/571702673 conn(0x7f94ac06c490 msgr2=0x7f94ac071670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:28:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.017+0000 7f94b2525700 1 -- 172.21.15.2:0/571702673 shutdown_connections 2023-11-16T18:28:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:28:47.017+0000 7f94b2525700 1 -- 172.21.15.2:0/571702673 wait complete. 2023-11-16T18:28:47.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:47 smithi002 ceph-mon[98807]: pgmap v121: 129 pgs: 129 active+clean; 389 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.7 MiB/s wr, 652 op/s 2023-11-16T18:28:47.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:47 smithi040 ceph-mon[100251]: pgmap v121: 129 pgs: 129 active+clean; 389 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.7 MiB/s wr, 652 op/s 2023-11-16T18:28:47.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:47 smithi188 ceph-mon[100639]: pgmap v121: 129 pgs: 129 active+clean; 389 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.7 MiB/s wr, 652 op/s 2023-11-16T18:28:50.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:50 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:50.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:50 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:50.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:50 smithi002 ceph-mon[98807]: pgmap v122: 129 pgs: 129 active+clean; 396 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.2 MiB/s wr, 896 op/s 2023-11-16T18:28:50.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:50 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:50.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:50 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:50.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:50 smithi040 ceph-mon[100251]: pgmap v122: 129 pgs: 129 active+clean; 396 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.2 MiB/s wr, 896 op/s 2023-11-16T18:28:50.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:50 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:50.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:50 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:50.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:50 smithi188 ceph-mon[100639]: pgmap v122: 129 pgs: 129 active+clean; 396 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.2 MiB/s wr, 896 op/s 2023-11-16T18:28:52.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:52 smithi002 ceph-mon[98807]: pgmap v123: 129 pgs: 129 active+clean; 396 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.9 MiB/s wr, 457 op/s 2023-11-16T18:28:52.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:52 smithi040 ceph-mon[100251]: pgmap v123: 129 pgs: 129 active+clean; 396 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.9 MiB/s wr, 457 op/s 2023-11-16T18:28:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:52 smithi188 ceph-mon[100639]: pgmap v123: 129 pgs: 129 active+clean; 396 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.9 MiB/s wr, 457 op/s 2023-11-16T18:28:54.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:54 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:54.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:54 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:54.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:54 smithi002 ceph-mon[98807]: pgmap v124: 129 pgs: 129 active+clean; 425 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.3 MiB/s wr, 581 op/s 2023-11-16T18:28:54.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:54 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:54.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:54 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:54.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:54 smithi040 ceph-mon[100251]: pgmap v124: 129 pgs: 129 active+clean; 425 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.3 MiB/s wr, 581 op/s 2023-11-16T18:28:54.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:54 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:54.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:54 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:54.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:54 smithi188 ceph-mon[100639]: pgmap v124: 129 pgs: 129 active+clean; 425 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.3 MiB/s wr, 581 op/s 2023-11-16T18:28:56.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:56 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:28:56.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:56 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:56.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:56 smithi040 ceph-mon[100251]: pgmap v125: 129 pgs: 129 active+clean; 425 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.7 MiB/s wr, 441 op/s 2023-11-16T18:28:56.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:56 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:28:56.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:56 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:56.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:56 smithi188 ceph-mon[100639]: pgmap v125: 129 pgs: 129 active+clean; 425 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.7 MiB/s wr, 441 op/s 2023-11-16T18:28:56.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:56 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:28:56.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:56 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:28:56.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:56 smithi002 ceph-mon[98807]: pgmap v125: 129 pgs: 129 active+clean; 425 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.7 MiB/s wr, 441 op/s 2023-11-16T18:28:57.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:28:57 smithi188 ceph-mon[100639]: pgmap v126: 129 pgs: 129 active+clean; 425 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.7 MiB/s wr, 447 op/s 2023-11-16T18:28:57.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:28:57 smithi002 ceph-mon[98807]: pgmap v126: 129 pgs: 129 active+clean; 425 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.7 MiB/s wr, 447 op/s 2023-11-16T18:28:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:28:57 smithi040 ceph-mon[100251]: pgmap v126: 129 pgs: 129 active+clean; 425 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.7 MiB/s wr, 447 op/s 2023-11-16T18:29:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.150+0000 7f7488f07700 1 -- 172.21.15.2:0/3660693017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74840ffdd0 msgr2=0x7f7484103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.150+0000 7f7488f07700 1 --2- 172.21.15.2:0/3660693017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74840ffdd0 0x7f7484103d90 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f74740099c0 tx=0x7f7474009cd0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.150+0000 7f7488f07700 1 -- 172.21.15.2:0/3660693017 shutdown_connections 2023-11-16T18:29:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.150+0000 7f7488f07700 1 --2- 172.21.15.2:0/3660693017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74840ff430 0x7f74840ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.150+0000 7f7488f07700 1 --2- 172.21.15.2:0/3660693017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7484104470 0x7f748410c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.150+0000 7f7488f07700 1 --2- 172.21.15.2:0/3660693017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74840ffdd0 0x7f7484103d90 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.150+0000 7f7488f07700 1 -- 172.21.15.2:0/3660693017 >> 172.21.15.2:0/3660693017 conn(0x7f7484076370 msgr2=0x7f7484076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.150+0000 7f7488f07700 1 -- 172.21.15.2:0/3660693017 shutdown_connections 2023-11-16T18:29:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.150+0000 7f7488f07700 1 -- 172.21.15.2:0/3660693017 wait complete. 2023-11-16T18:29:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.151+0000 7f7488f07700 1 Processor -- start 2023-11-16T18:29:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.151+0000 7f7488f07700 1 -- start start 2023-11-16T18:29:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.151+0000 7f7488f07700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74840ff430 0x7f7484195310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.151+0000 7f7488f07700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74840ffdd0 0x7f7484195850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.151+0000 7f7488f07700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7484104470 0x7f7484199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.151+0000 7f7488f07700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748419a2f0 con 0x7f74840ffdd0 2023-11-16T18:29:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.151+0000 7f7488f07700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748419a460 con 0x7f7484104470 2023-11-16T18:29:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.151+0000 7f7488f07700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748419a620 con 0x7f74840ff430 2023-11-16T18:29:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f7481d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74840ffdd0 0x7f7484195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f7481d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74840ffdd0 0x7f7484195850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52370/0 (socket says 172.21.15.2:52370) 2023-11-16T18:29:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f7481d9b700 1 -- 172.21.15.2:0/645484803 learned_addr learned my addr 172.21.15.2:0/645484803 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f7482d9d700 1 --2- 172.21.15.2:0/645484803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7484104470 0x7f7484199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f7481d9b700 1 -- 172.21.15.2:0/645484803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74840ff430 msgr2=0x7f7484195310 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-16T18:29:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f7481d9b700 1 --2- 172.21.15.2:0/645484803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74840ff430 0x7f7484195310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f7481d9b700 1 -- 172.21.15.2:0/645484803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7484104470 msgr2=0x7f7484199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f7481d9b700 1 --2- 172.21.15.2:0/645484803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7484104470 0x7f7484199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f7481d9b700 1 -- 172.21.15.2:0/645484803 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74740095f0 con 0x7f74840ffdd0 2023-11-16T18:29:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f7481d9b700 1 --2- 172.21.15.2:0/645484803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74840ffdd0 0x7f7484195850 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f74740098e0 tx=0x7f747400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f747b7fe700 1 -- 172.21.15.2:0/645484803 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747401d070 con 0x7f74840ffdd0 2023-11-16T18:29:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f747b7fe700 1 -- 172.21.15.2:0/645484803 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f747400fb80 con 0x7f74840ffdd0 2023-11-16T18:29:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f7488f07700 1 -- 172.21.15.2:0/645484803 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748410f1f0 con 0x7f74840ffdd0 2023-11-16T18:29:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f747b7fe700 1 -- 172.21.15.2:0/645484803 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74740173f0 con 0x7f74840ffdd0 2023-11-16T18:29:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.152+0000 7f7488f07700 1 -- 172.21.15.2:0/645484803 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748410f060 con 0x7f74840ffdd0 2023-11-16T18:29:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.155+0000 7f747b7fe700 1 -- 172.21.15.2:0/645484803 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f7474022470 con 0x7f74840ffdd0 2023-11-16T18:29:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.155+0000 7f747b7fe700 1 --2- 172.21.15.2:0/645484803 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f74700a1340 0x7f74700a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.155+0000 7f748259c700 1 --2- 172.21.15.2:0/645484803 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f74700a1340 0x7f74700a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.155+0000 7f747b7fe700 1 -- 172.21.15.2:0/645484803 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f74740cdcb0 con 0x7f74840ffdd0 2023-11-16T18:29:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.156+0000 7f748259c700 1 --2- 172.21.15.2:0/645484803 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f74700a1340 0x7f74700a37f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f746c009af0 tx=0x7f746c009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.156+0000 7f7495093b80 1 Processor -- start 2023-11-16T18:29:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f7495093b80 1 -- start start 2023-11-16T18:29:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f7495093b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618cb31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f7495093b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3720 0x559618cad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f7495093b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3b90 0x559618cadb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f7495093b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559618bb3000 con 0x559618cb3b90 2023-11-16T18:29:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f7495093b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559618c05840 con 0x559618b096b0 2023-11-16T18:29:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f7495093b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x559618bfecd0 con 0x559618cb3720 2023-11-16T18:29:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f748259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618cb31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f748259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618cb31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42814/0 (socket says 172.21.15.2:42814) 2023-11-16T18:29:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f748259c700 1 -- 172.21.15.2:0/3115327036 learned_addr learned my addr 172.21.15.2:0/3115327036 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f7482d9d700 1 --2- 172.21.15.2:0/3115327036 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3b90 0x559618cadb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f7481d9b700 1 --2- 172.21.15.2:0/3115327036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3720 0x559618cad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f748259c700 1 -- 172.21.15.2:0/3115327036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3720 msgr2=0x559618cad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f748259c700 1 --2- 172.21.15.2:0/3115327036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3720 0x559618cad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f748259c700 1 -- 172.21.15.2:0/3115327036 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3b90 msgr2=0x559618cadb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f748259c700 1 --2- 172.21.15.2:0/3115327036 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3b90 0x559618cadb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.157+0000 7f748259c700 1 -- 172.21.15.2:0/3115327036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559618a805d0 con 0x559618b096b0 2023-11-16T18:29:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.158+0000 7f748259c700 1 --2- 172.21.15.2:0/3115327036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618cb31e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f746c00eb70 tx=0x7f746c011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d99c45a45ed2bec1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.158+0000 7f7478ff9700 1 -- 172.21.15.2:0/3115327036 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c010ac0 con 0x559618b096b0 2023-11-16T18:29:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.158+0000 7f7478ff9700 1 -- 172.21.15.2:0/3115327036 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f746c00ee80 con 0x559618b096b0 2023-11-16T18:29:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.158+0000 7f7478ff9700 1 -- 172.21.15.2:0/3115327036 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c0054c0 con 0x559618b096b0 2023-11-16T18:29:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.159+0000 7f7495093b80 1 -- 172.21.15.2:0/3115327036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 msgr2=0x559618cb31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.159+0000 7f7495093b80 1 --2- 172.21.15.2:0/3115327036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618cb31e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f746c00eb70 tx=0x7f746c011000 comp rx=0 tx=0).stop 2023-11-16T18:29:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.159+0000 7f7495093b80 1 -- 172.21.15.2:0/3115327036 shutdown_connections 2023-11-16T18:29:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.159+0000 7f7495093b80 1 --2- 172.21.15.2:0/3115327036 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3b90 0x559618cadb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.159+0000 7f7495093b80 1 --2- 172.21.15.2:0/3115327036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618cb31e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.159+0000 7f7495093b80 1 --2- 172.21.15.2:0/3115327036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3720 0x559618cad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.159+0000 7f7495093b80 1 -- 172.21.15.2:0/3115327036 >> 172.21.15.2:0/3115327036 conn(0x559618ca2f50 msgr2=0x559618ca8270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.159+0000 7f7495093b80 1 -- 172.21.15.2:0/3115327036 shutdown_connections 2023-11-16T18:29:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.159+0000 7f7495093b80 1 -- 172.21.15.2:0/3115327036 wait complete. 2023-11-16T18:29:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.159+0000 7f7495093b80 1 -- start start 2023-11-16T18:29:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.159+0000 7f7495093b80 1 Processor -- start 2023-11-16T18:29:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f7495093b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618b096b0 0x559618cb9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f7495093b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618cb3720 0x559618cb6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f7495093b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3b90 0x559618cb7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f7495093b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559618ba0880 con 0x559618cb3b90 2023-11-16T18:29:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f7495093b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559618ba0520 con 0x559618cb3720 2023-11-16T18:29:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f7495093b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x559618a828c0 con 0x559618b096b0 2023-11-16T18:29:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f7495093b80 10 client.0 _ll_register_callbacks cb 0x559618bbe720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:29:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f748259c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618b096b0 0x559618cb9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f748259c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618b096b0 0x559618cb9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43912/0 (socket says 172.21.15.2:43912) 2023-11-16T18:29:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f748259c700 1 -- 172.21.15.2:0/480230293 learned_addr learned my addr 172.21.15.2:0/480230293 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f7481d9b700 1 --2- 172.21.15.2:0/480230293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618cb3720 0x559618cb6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f7482d9d700 1 --2- 172.21.15.2:0/480230293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3b90 0x559618cb7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f748259c700 1 -- 172.21.15.2:0/480230293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618cb3720 msgr2=0x559618cb6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f748259c700 1 --2- 172.21.15.2:0/480230293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618cb3720 0x559618cb6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f748259c700 1 -- 172.21.15.2:0/480230293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3b90 msgr2=0x559618cb7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f748259c700 1 --2- 172.21.15.2:0/480230293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3b90 0x559618cb7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f748259c700 1 -- 172.21.15.2:0/480230293 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746c00e820 con 0x559618b096b0 2023-11-16T18:29:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f748259c700 1 --2- 172.21.15.2:0/480230293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618b096b0 0x559618cb9a90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f746c00eb40 tx=0x7f746c005390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcd9f5059f66bb3c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.160+0000 7f7469ffb700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:29:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7469ffb700 1 -- 172.21.15.2:0/480230293 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c02b040 con 0x559618b096b0 2023-11-16T18:29:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7469ffb700 1 -- 172.21.15.2:0/480230293 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f746c010e40 con 0x559618b096b0 2023-11-16T18:29:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7495093b80 1 -- 172.21.15.2:0/480230293 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559618ca0710 con 0x559618b096b0 2023-11-16T18:29:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7469ffb700 1 -- 172.21.15.2:0/480230293 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f746c004910 con 0x559618b096b0 2023-11-16T18:29:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7495093b80 10 client.25951 fetch_fsmap learned FSMap version 29 2023-11-16T18:29:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7495093b80 1 -- 172.21.15.2:0/480230293 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x559618bc5130 con 0x559618b096b0 2023-11-16T18:29:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7469ffb700 1 -- 172.21.15.2:0/480230293 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559618ca0710 con 0x559618b096b0 2023-11-16T18:29:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7469ffb700 1 -- 172.21.15.2:0/480230293 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f746c018070 con 0x559618b096b0 2023-11-16T18:29:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7495093b80 10 client.25951 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:29:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7495093b80 10 client.25951 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:29:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7495093b80 20 client.25951 populate_metadata read hostname 'smithi002' 2023-11-16T18:29:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7495093b80 1 --2- 172.21.15.2:0/480230293 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559618cc6e70 0x559618cc9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7495093b80 4 client.25951 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:29:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.161+0000 7f7495093b80 1 -- 172.21.15.2:0/480230293 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559618bc1040 con 0x559618cc6e70 2023-11-16T18:29:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.162+0000 7f7481d9b700 1 --2- 172.21.15.2:0/480230293 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559618cc6e70 0x559618cc9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.162+0000 7f7481d9b700 1 --2- 172.21.15.2:0/480230293 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559618cc6e70 0x559618cc9280 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ac3417aa6f44241 server_cookie=4a9f69efa54b80df in_seq=0 out_seq=0 2023-11-16T18:29:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.162+0000 7f7469ffb700 10 client.25951 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7469ffb700 1 -- 172.21.15.2:0/480230293 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559618bc1040 con 0x559618cc6e70 2023-11-16T18:29:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7469ffb700 10 client.25951 handle_command_reply: tid=0 2023-11-16T18:29:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7495093b80 1 client.25951 shutdown 2023-11-16T18:29:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7495093b80 1 -- 172.21.15.2:0/480230293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618b096b0 msgr2=0x559618cb9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7495093b80 1 --2- 172.21.15.2:0/480230293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618b096b0 0x559618cb9a90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f746c00eb40 tx=0x7f746c005390 comp rx=0 tx=0).stop 2023-11-16T18:29:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7495093b80 1 -- 172.21.15.2:0/480230293 shutdown_connections 2023-11-16T18:29:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7495093b80 1 --2- 172.21.15.2:0/480230293 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559618cc6e70 0x559618cc9280 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7495093b80 1 --2- 172.21.15.2:0/480230293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618b096b0 0x559618cb9a90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7495093b80 1 --2- 172.21.15.2:0/480230293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3b90 0x559618cb7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7495093b80 1 --2- 172.21.15.2:0/480230293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618cb3720 0x559618cb6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7469ffb700 0 client.25951 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7495093b80 1 -- 172.21.15.2:0/480230293 >> 172.21.15.2:0/480230293 conn(0x559618ca2f50 msgr2=0x559618ca77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7495093b80 1 -- 172.21.15.2:0/480230293 shutdown_connections 2023-11-16T18:29:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7495093b80 1 -- 172.21.15.2:0/480230293 wait complete. 2023-11-16T18:29:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.165+0000 7f7495093b80 20 client.25951 trim_cache size 0 max 16384 2023-11-16T18:29:00.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.181+0000 7f7495093b80 1 Processor -- start 2023-11-16T18:29:00.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.181+0000 7f7495093b80 1 -- start start 2023-11-16T18:29:00.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.181+0000 7f7495093b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618cbe090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.181+0000 7f7495093b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 0x559618cb9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.181+0000 7f7495093b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3b90 0x559618cb6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.181+0000 7f7495093b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559618ca2b70 con 0x559618cb3720 2023-11-16T18:29:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.181+0000 7f7495093b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559618cb1a50 con 0x559618b096b0 2023-11-16T18:29:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.181+0000 7f7495093b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x559618cbe5d0 con 0x559618cb3b90 2023-11-16T18:29:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.182+0000 7f7481d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 0x559618cb9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.182+0000 7f7481d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 0x559618cb9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52390/0 (socket says 172.21.15.2:52390) 2023-11-16T18:29:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.182+0000 7f7481d9b700 1 -- 172.21.15.2:0/1629659992 learned_addr learned my addr 172.21.15.2:0/1629659992 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.182+0000 7f748259c700 1 --2- 172.21.15.2:0/1629659992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618cbe090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.182+0000 7f7481d9b700 1 -- 172.21.15.2:0/1629659992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3b90 msgr2=0x559618cb6560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T18:29:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.182+0000 7f7481d9b700 1 --2- 172.21.15.2:0/1629659992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3b90 0x559618cb6560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.182+0000 7f7481d9b700 1 -- 172.21.15.2:0/1629659992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 msgr2=0x559618cbe090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.182+0000 7f7481d9b700 1 --2- 172.21.15.2:0/1629659992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618cbe090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.182+0000 7f7481d9b700 1 -- 172.21.15.2:0/1629659992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559618bc4e80 con 0x559618cb3720 2023-11-16T18:29:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.182+0000 7f7481d9b700 1 --2- 172.21.15.2:0/1629659992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 0x559618cb9630 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f7474000c00 tx=0x7f747401a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771d2e415d1ca5dc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.182+0000 7f74797fa700 1 -- 172.21.15.2:0/1629659992 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74740b2070 con 0x559618cb3720 2023-11-16T18:29:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.182+0000 7f74797fa700 1 -- 172.21.15.2:0/1629659992 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74740a2e60 con 0x559618cb3720 2023-11-16T18:29:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.182+0000 7f74797fa700 1 -- 172.21.15.2:0/1629659992 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474022720 con 0x559618cb3720 2023-11-16T18:29:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.183+0000 7f7495093b80 1 -- 172.21.15.2:0/1629659992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 msgr2=0x559618cb9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.183+0000 7f7495093b80 1 --2- 172.21.15.2:0/1629659992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 0x559618cb9630 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f7474000c00 tx=0x7f747401a040 comp rx=0 tx=0).stop 2023-11-16T18:29:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.183+0000 7f7495093b80 1 -- 172.21.15.2:0/1629659992 shutdown_connections 2023-11-16T18:29:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.183+0000 7f7495093b80 1 --2- 172.21.15.2:0/1629659992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3b90 0x559618cb6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.183+0000 7f7495093b80 1 --2- 172.21.15.2:0/1629659992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618cbe090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.183+0000 7f7495093b80 1 --2- 172.21.15.2:0/1629659992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 0x559618cb9630 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.183+0000 7f7495093b80 1 -- 172.21.15.2:0/1629659992 >> 172.21.15.2:0/1629659992 conn(0x559618ca2f50 msgr2=0x559618cb0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.183+0000 7f7495093b80 1 -- 172.21.15.2:0/1629659992 shutdown_connections 2023-11-16T18:29:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.183+0000 7f7495093b80 1 -- 172.21.15.2:0/1629659992 wait complete. 2023-11-16T18:29:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7495093b80 1 -- start start 2023-11-16T18:29:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7495093b80 1 Processor -- start 2023-11-16T18:29:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7495093b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618caec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7495093b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 0x559618caf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7495093b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3b90 0x559618caf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7495093b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559618cafce0 con 0x559618cb3720 2023-11-16T18:29:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7495093b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559618cafe50 con 0x559618b096b0 2023-11-16T18:29:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7495093b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x559618cb0010 con 0x559618cb3b90 2023-11-16T18:29:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7495093b80 10 client.0 _ll_register_callbacks cb 0x7f747c009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:29:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7481d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 0x559618caf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7481d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 0x559618caf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52398/0 (socket says 172.21.15.2:52398) 2023-11-16T18:29:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7481d9b700 1 -- 172.21.15.2:0/3795595864 learned_addr learned my addr 172.21.15.2:0/3795595864 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f748259c700 1 --2- 172.21.15.2:0/3795595864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618caec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7482d9d700 1 --2- 172.21.15.2:0/3795595864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3b90 0x559618caf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7481d9b700 1 -- 172.21.15.2:0/3795595864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3b90 msgr2=0x559618caf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7481d9b700 1 --2- 172.21.15.2:0/3795595864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3b90 0x559618caf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7481d9b700 1 -- 172.21.15.2:0/3795595864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 msgr2=0x559618caec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7481d9b700 1 --2- 172.21.15.2:0/3795595864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618caec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.184+0000 7f7481d9b700 1 -- 172.21.15.2:0/3795595864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748410f060 con 0x559618cb3720 2023-11-16T18:29:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7481d9b700 1 --2- 172.21.15.2:0/3795595864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 0x559618caf1a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f74740a2930 tx=0x7f74740cb070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b765bcd6930d22d0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7469ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:29:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7469ffb700 1 -- 172.21.15.2:0/3795595864 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74740b2030 con 0x559618cb3720 2023-11-16T18:29:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7469ffb700 1 -- 172.21.15.2:0/3795595864 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f74740b3070 con 0x559618cb3720 2023-11-16T18:29:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7495093b80 1 -- 172.21.15.2:0/3795595864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f746c00ee80 con 0x559618cb3720 2023-11-16T18:29:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7469ffb700 1 -- 172.21.15.2:0/3795595864 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74740b3230 con 0x559618cb3720 2023-11-16T18:29:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7495093b80 10 client.16533 fetch_fsmap learned FSMap version 29 2023-11-16T18:29:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7495093b80 1 -- 172.21.15.2:0/3795595864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x559618a15c00 con 0x559618cb3720 2023-11-16T18:29:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7469ffb700 1 -- 172.21.15.2:0/3795595864 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f746c00ee80 con 0x559618cb3720 2023-11-16T18:29:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7469ffb700 1 -- 172.21.15.2:0/3795595864 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f7474099f90 con 0x559618cb3720 2023-11-16T18:29:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7495093b80 10 client.16533 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:29:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7495093b80 10 client.16533 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:29:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7495093b80 20 client.16533 populate_metadata read hostname 'smithi002' 2023-11-16T18:29:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7495093b80 1 --2- 172.21.15.2:0/3795595864 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559618d1b4f0 0x559618d1d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7495093b80 4 client.16533 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:29:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.185+0000 7f7495093b80 1 -- 172.21.15.2:0/3795595864 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559618bcd600 con 0x559618d1b4f0 2023-11-16T18:29:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.186+0000 7f748259c700 1 --2- 172.21.15.2:0/3795595864 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559618d1b4f0 0x559618d1d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.189+0000 7f7469ffb700 10 client.16533 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.189+0000 7f748259c700 1 --2- 172.21.15.2:0/3795595864 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559618d1b4f0 0x559618d1d900 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13b5924a9c58bee7 server_cookie=605433d66edf28e9 in_seq=0 out_seq=0 2023-11-16T18:29:00.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:00 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:29:00.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:00 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:00.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:00 smithi002 ceph-mon[98807]: pgmap v127: 129 pgs: 129 active+clean; 451 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.4 MiB/s wr, 548 op/s 2023-11-16T18:29:00.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:00 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:29:00.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:00 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:00.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:00 smithi040 ceph-mon[100251]: pgmap v127: 129 pgs: 129 active+clean; 451 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.4 MiB/s wr, 548 op/s 2023-11-16T18:29:00.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:00 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:29:00.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:00 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:00.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:00 smithi188 ceph-mon[100639]: pgmap v127: 129 pgs: 129 active+clean; 451 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.4 MiB/s wr, 548 op/s 2023-11-16T18:29:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.924+0000 7f7469ffb700 1 -- 172.21.15.2:0/3795595864 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559618bcd600 con 0x559618d1b4f0 2023-11-16T18:29:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.924+0000 7f7469ffb700 10 client.16533 handle_command_reply: tid=0 2023-11-16T18:29:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.924+0000 7f7495093b80 1 client.16533 shutdown 2023-11-16T18:29:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.924+0000 7f7495093b80 1 -- 172.21.15.2:0/3795595864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 msgr2=0x559618caf1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.924+0000 7f7495093b80 1 --2- 172.21.15.2:0/3795595864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 0x559618caf1a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f74740a2930 tx=0x7f74740cb070 comp rx=0 tx=0).stop 2023-11-16T18:29:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.925+0000 7f7495093b80 1 -- 172.21.15.2:0/3795595864 shutdown_connections 2023-11-16T18:29:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.925+0000 7f7495093b80 1 --2- 172.21.15.2:0/3795595864 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x559618d1b4f0 0x559618d1d900 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.925+0000 7f7495093b80 1 --2- 172.21.15.2:0/3795595864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559618cb3b90 0x559618caf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.925+0000 7f7495093b80 1 --2- 172.21.15.2:0/3795595864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559618b096b0 0x559618caec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.925+0000 7f7495093b80 1 --2- 172.21.15.2:0/3795595864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559618cb3720 0x559618caf1a0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.925+0000 7f7469ffb700 0 client.16533 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.925+0000 7f7495093b80 1 -- 172.21.15.2:0/3795595864 >> 172.21.15.2:0/3795595864 conn(0x559618ca2f50 msgr2=0x559618cb1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.925+0000 7f7495093b80 1 -- 172.21.15.2:0/3795595864 shutdown_connections 2023-11-16T18:29:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.925+0000 7f7495093b80 1 -- 172.21.15.2:0/3795595864 wait complete. 2023-11-16T18:29:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.925+0000 7f7495093b80 20 client.16533 trim_cache size 0 max 16384 2023-11-16T18:29:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:29:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:29:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:29:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:29:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.926+0000 7f7488f07700 1 -- 172.21.15.2:0/645484803 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f74700a1340 msgr2=0x7f74700a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.926+0000 7f7488f07700 1 --2- 172.21.15.2:0/645484803 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f74700a1340 0x7f74700a37f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f746c009af0 tx=0x7f746c009260 comp rx=0 tx=0).stop 2023-11-16T18:29:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.926+0000 7f7488f07700 1 -- 172.21.15.2:0/645484803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74840ffdd0 msgr2=0x7f7484195850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.926+0000 7f7488f07700 1 --2- 172.21.15.2:0/645484803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74840ffdd0 0x7f7484195850 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f74740098e0 tx=0x7f747400f690 comp rx=0 tx=0).stop 2023-11-16T18:29:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.926+0000 7f7488f07700 1 -- 172.21.15.2:0/645484803 shutdown_connections 2023-11-16T18:29:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.926+0000 7f7488f07700 1 --2- 172.21.15.2:0/645484803 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f74700a1340 0x7f74700a37f0 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f746c009af0 tx=0x7f746c009260 comp rx=0 tx=0).stop 2023-11-16T18:29:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.926+0000 7f7488f07700 1 --2- 172.21.15.2:0/645484803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74840ff430 0x7f7484195310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.927+0000 7f7488f07700 1 --2- 172.21.15.2:0/645484803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7484104470 0x7f7484199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.927+0000 7f7488f07700 1 --2- 172.21.15.2:0/645484803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74840ffdd0 0x7f7484195850 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.927+0000 7f7488f07700 1 -- 172.21.15.2:0/645484803 >> 172.21.15.2:0/645484803 conn(0x7f7484076370 msgr2=0x7f74840fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.927+0000 7f7488f07700 1 -- 172.21.15.2:0/645484803 shutdown_connections 2023-11-16T18:29:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:00.927+0000 7f7488f07700 1 -- 172.21.15.2:0/645484803 wait complete. 2023-11-16T18:29:02.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:02 smithi002 ceph-mon[98807]: pgmap v128: 129 pgs: 129 active+clean; 451 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.6 MiB/s wr, 271 op/s 2023-11-16T18:29:02.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:02 smithi040 ceph-mon[100251]: pgmap v128: 129 pgs: 129 active+clean; 451 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.6 MiB/s wr, 271 op/s 2023-11-16T18:29:02.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:02 smithi188 ceph-mon[100639]: pgmap v128: 129 pgs: 129 active+clean; 451 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.6 MiB/s wr, 271 op/s 2023-11-16T18:29:04.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:04 smithi002 ceph-mon[98807]: pgmap v129: 129 pgs: 129 active+clean; 459 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.1 MiB/s wr, 515 op/s 2023-11-16T18:29:04.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:04 smithi040 ceph-mon[100251]: pgmap v129: 129 pgs: 129 active+clean; 459 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.1 MiB/s wr, 515 op/s 2023-11-16T18:29:04.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:04 smithi188 ceph-mon[100639]: pgmap v129: 129 pgs: 129 active+clean; 459 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.1 MiB/s wr, 515 op/s 2023-11-16T18:29:06.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:06 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:29:06.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:06 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:06.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:06 smithi040 ceph-mon[100251]: pgmap v130: 129 pgs: 129 active+clean; 459 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 391 op/s 2023-11-16T18:29:06.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:06 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:29:06.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:06 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:06.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:06 smithi188 ceph-mon[100639]: pgmap v130: 129 pgs: 129 active+clean; 459 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 391 op/s 2023-11-16T18:29:06.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:06 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:29:06.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:06 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:06.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:06 smithi002 ceph-mon[98807]: pgmap v130: 129 pgs: 129 active+clean; 459 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 391 op/s 2023-11-16T18:29:07.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:07 smithi002 ceph-mon[98807]: pgmap v131: 129 pgs: 129 active+clean; 461 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 400 op/s 2023-11-16T18:29:07.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:07 smithi040 ceph-mon[100251]: pgmap v131: 129 pgs: 129 active+clean; 461 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 400 op/s 2023-11-16T18:29:07.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:07 smithi188 ceph-mon[100639]: pgmap v131: 129 pgs: 129 active+clean; 461 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 400 op/s 2023-11-16T18:29:08.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:08 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:29:08.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:08 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:29:08.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:08 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:29:09.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:09 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:29:09.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:09 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:29:09.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:09 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:29:09.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:09 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:29:09.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:09 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:29:09.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:09 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:29:09.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:09 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:29:09.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:09 smithi002 ceph-mon[98807]: pgmap v132: 129 pgs: 129 active+clean; 482 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.6 MiB/s wr, 528 op/s 2023-11-16T18:29:09.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:09 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:29:09.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:09 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:29:09.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:09 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:29:09.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:09 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:29:09.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:09 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:29:09.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:09 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:29:09.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:09 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:29:09.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:09 smithi040 ceph-mon[100251]: pgmap v132: 129 pgs: 129 active+clean; 482 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.6 MiB/s wr, 528 op/s 2023-11-16T18:29:09.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:09 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:29:09.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:09 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:29:09.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:09 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:29:09.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:09 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:29:09.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:09 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:29:09.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:09 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:29:09.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:09 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:29:09.844 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:09 smithi188 ceph-mon[100639]: pgmap v132: 129 pgs: 129 active+clean; 482 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.6 MiB/s wr, 528 op/s 2023-11-16T18:29:09.869 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:29:10.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.054+0000 7fe326aa4700 1 -- 172.21.15.2:0/91632213 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe320100420 msgr2=0x7fe32010bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:10.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.054+0000 7fe326aa4700 1 --2- 172.21.15.2:0/91632213 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe320100420 0x7fe32010bfd0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe3140098f0 tx=0x7fe314009c00 comp rx=0 tx=0).stop 2023-11-16T18:29:10.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.054+0000 7fe326aa4700 1 -- 172.21.15.2:0/91632213 shutdown_connections 2023-11-16T18:29:10.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.054+0000 7fe326aa4700 1 --2- 172.21.15.2:0/91632213 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe320105250 0x7fe320105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.054+0000 7fe326aa4700 1 --2- 172.21.15.2:0/91632213 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe320100420 0x7fe32010bfd0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.054+0000 7fe326aa4700 1 --2- 172.21.15.2:0/91632213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3200ffa70 0x7fe3200ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.054+0000 7fe326aa4700 1 -- 172.21.15.2:0/91632213 >> 172.21.15.2:0/91632213 conn(0x7fe3200763a0 msgr2=0x7fe3200767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:10.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.055+0000 7fe326aa4700 1 -- 172.21.15.2:0/91632213 shutdown_connections 2023-11-16T18:29:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.056+0000 7fe326aa4700 1 -- 172.21.15.2:0/91632213 wait complete. 2023-11-16T18:29:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.057+0000 7fe326aa4700 1 Processor -- start 2023-11-16T18:29:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.057+0000 7fe326aa4700 1 -- start start 2023-11-16T18:29:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.057+0000 7fe326aa4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3200ffa70 0x7fe32006b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.057+0000 7fe326aa4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe320100420 0x7fe32006b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.057+0000 7fe326aa4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe320105250 0x7fe3201a8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.057+0000 7fe326aa4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3201a8c30 con 0x7fe320105250 2023-11-16T18:29:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.057+0000 7fe326aa4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3201a8da0 con 0x7fe3200ffa70 2023-11-16T18:29:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.057+0000 7fe326aa4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3201a8f60 con 0x7fe320100420 2023-11-16T18:29:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.058+0000 7fe324840700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3200ffa70 0x7fe32006b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.058+0000 7fe324840700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3200ffa70 0x7fe32006b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42848/0 (socket says 172.21.15.2:42848) 2023-11-16T18:29:10.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.058+0000 7fe324840700 1 -- 172.21.15.2:0/2157017891 learned_addr learned my addr 172.21.15.2:0/2157017891 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:10.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.058+0000 7fe325041700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe320105250 0x7fe3201a8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:10.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.058+0000 7fe31ffff700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe320100420 0x7fe32006b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:10.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.058+0000 7fe31ffff700 1 -- 172.21.15.2:0/2157017891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3200ffa70 msgr2=0x7fe32006b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:10.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.058+0000 7fe31ffff700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3200ffa70 0x7fe32006b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.058+0000 7fe31ffff700 1 -- 172.21.15.2:0/2157017891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe320105250 msgr2=0x7fe3201a8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:10.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.058+0000 7fe31ffff700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe320105250 0x7fe3201a8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.058+0000 7fe31ffff700 1 -- 172.21.15.2:0/2157017891 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe314009520 con 0x7fe320100420 2023-11-16T18:29:10.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.058+0000 7fe325041700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe320105250 0x7fe3201a8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:29:10.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.059+0000 7fe31ffff700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe320100420 0x7fe32006b570 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe3100078c0 tx=0x7fe310007bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:10.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.060+0000 7fe31dffb700 1 -- 172.21.15.2:0/2157017891 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310010070 con 0x7fe320100420 2023-11-16T18:29:10.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.060+0000 7fe31dffb700 1 -- 172.21.15.2:0/2157017891 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3100044e0 con 0x7fe320100420 2023-11-16T18:29:10.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.060+0000 7fe31dffb700 1 -- 172.21.15.2:0/2157017891 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3100038a0 con 0x7fe320100420 2023-11-16T18:29:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.061+0000 7fe326aa4700 1 -- 172.21.15.2:0/2157017891 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32010e1a0 con 0x7fe320100420 2023-11-16T18:29:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.062+0000 7fe326aa4700 1 -- 172.21.15.2:0/2157017891 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32010e010 con 0x7fe320100420 2023-11-16T18:29:10.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.069+0000 7fe332c30b80 1 Processor -- start 2023-11-16T18:29:10.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.070+0000 7fe31dffb700 1 -- 172.21.15.2:0/2157017891 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fe310003a00 con 0x7fe320100420 2023-11-16T18:29:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.071+0000 7fe31dffb700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe3080a1480 0x7fe3080a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.071+0000 7fe31dffb700 1 -- 172.21.15.2:0/2157017891 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe3100cc270 con 0x7fe320100420 2023-11-16T18:29:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.074+0000 7fe325041700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe3080a1480 0x7fe3080a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.076+0000 7fe332c30b80 1 -- start start 2023-11-16T18:29:10.079 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.078+0000 7fe325041700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe3080a1480 0x7fe3080a3930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe3140098c0 tx=0x7fe31401a040 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:10.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.079+0000 7fe332c30b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324cf370 0x564b324cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.079+0000 7fe332c30b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b324cfc80 0x564b324d3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.079+0000 7fe332c30b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 0x564b324d4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.079+0000 7fe332c30b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b323cc060 con 0x564b324d40c0 2023-11-16T18:29:10.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.079+0000 7fe332c30b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b3241e8a0 con 0x564b324cfc80 2023-11-16T18:29:10.081 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.079+0000 7fe332c30b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b32417d30 con 0x564b324cf370 2023-11-16T18:29:10.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.080+0000 7fe325041700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 0x564b324d4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:10.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.080+0000 7fe325041700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 0x564b324d4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52422/0 (socket says 172.21.15.2:52422) 2023-11-16T18:29:10.082 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.080+0000 7fe325041700 1 -- 172.21.15.2:0/4089419239 learned_addr learned my addr 172.21.15.2:0/4089419239 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.080+0000 7fe325041700 1 -- 172.21.15.2:0/4089419239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324cf370 msgr2=0x564b324cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.080+0000 7fe325041700 1 --2- 172.21.15.2:0/4089419239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324cf370 0x564b324cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.080+0000 7fe325041700 1 -- 172.21.15.2:0/4089419239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b324cfc80 msgr2=0x564b324d3b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T18:29:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.080+0000 7fe325041700 1 --2- 172.21.15.2:0/4089419239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b324cfc80 0x564b324d3b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.080+0000 7fe325041700 1 -- 172.21.15.2:0/4089419239 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b32299630 con 0x564b324d40c0 2023-11-16T18:29:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.081+0000 7fe325041700 1 --2- 172.21.15.2:0/4089419239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 0x564b324d4510 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe31400fae0 tx=0x7fe31401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f5b8a8ce86e476 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.081+0000 7fe306ffd700 1 -- 172.21.15.2:0/4089419239 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31401d040 con 0x564b324d40c0 2023-11-16T18:29:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.081+0000 7fe306ffd700 1 -- 172.21.15.2:0/4089419239 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe31400bca0 con 0x564b324d40c0 2023-11-16T18:29:10.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.081+0000 7fe306ffd700 1 -- 172.21.15.2:0/4089419239 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314003e10 con 0x564b324d40c0 2023-11-16T18:29:10.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.082+0000 7fe332c30b80 1 -- 172.21.15.2:0/4089419239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 msgr2=0x564b324d4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:10.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.082+0000 7fe332c30b80 1 --2- 172.21.15.2:0/4089419239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 0x564b324d4510 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe31400fae0 tx=0x7fe31401e000 comp rx=0 tx=0).stop 2023-11-16T18:29:10.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.085+0000 7fe332c30b80 1 -- 172.21.15.2:0/4089419239 shutdown_connections 2023-11-16T18:29:10.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.085+0000 7fe332c30b80 1 --2- 172.21.15.2:0/4089419239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324cf370 0x564b324cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.085+0000 7fe332c30b80 1 --2- 172.21.15.2:0/4089419239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 0x564b324d4510 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.085+0000 7fe332c30b80 1 --2- 172.21.15.2:0/4089419239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b324cfc80 0x564b324d3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.085+0000 7fe332c30b80 1 -- 172.21.15.2:0/4089419239 >> 172.21.15.2:0/4089419239 conn(0x564b324bc360 msgr2=0x564b324c1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:10.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.085+0000 7fe332c30b80 1 -- 172.21.15.2:0/4089419239 shutdown_connections 2023-11-16T18:29:10.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.085+0000 7fe332c30b80 1 -- 172.21.15.2:0/4089419239 wait complete. 2023-11-16T18:29:10.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.086+0000 7fe332c30b80 1 -- start start 2023-11-16T18:29:10.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.086+0000 7fe332c30b80 1 Processor -- start 2023-11-16T18:29:10.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.087+0000 7fe332c30b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324cf370 0x564b324d8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.087+0000 7fe332c30b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b324cfc80 0x564b324d90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.087+0000 7fe332c30b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324d40c0 0x564b324d9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.087+0000 7fe332c30b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b323b98e0 con 0x564b324cf370 2023-11-16T18:29:10.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.087+0000 7fe332c30b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b323b9580 con 0x564b324cfc80 2023-11-16T18:29:10.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.087+0000 7fe332c30b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b3229b920 con 0x564b324d40c0 2023-11-16T18:29:10.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.087+0000 7fe332c30b80 10 client.0 _ll_register_callbacks cb 0x564b323d7780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:29:10.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.087+0000 7fe325041700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324d40c0 0x564b324d9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:10.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.087+0000 7fe325041700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324d40c0 0x564b324d9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:43956/0 (socket says 172.21.15.2:43956) 2023-11-16T18:29:10.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.087+0000 7fe325041700 1 -- 172.21.15.2:0/1831448620 learned_addr learned my addr 172.21.15.2:0/1831448620 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:10.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.087+0000 7fe324840700 1 --2- 172.21.15.2:0/1831448620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324cf370 0x564b324d8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:10.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.088+0000 7fe31ffff700 1 --2- 172.21.15.2:0/1831448620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b324cfc80 0x564b324d90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:10.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.088+0000 7fe325041700 1 -- 172.21.15.2:0/1831448620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b324cfc80 msgr2=0x564b324d90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:10.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.088+0000 7fe325041700 1 --2- 172.21.15.2:0/1831448620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b324cfc80 0x564b324d90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.088+0000 7fe325041700 1 -- 172.21.15.2:0/1831448620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324cf370 msgr2=0x564b324d8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:10.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.088+0000 7fe325041700 1 --2- 172.21.15.2:0/1831448620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324cf370 0x564b324d8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.088+0000 7fe325041700 1 -- 172.21.15.2:0/1831448620 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32010c8d0 con 0x564b324d40c0 2023-11-16T18:29:10.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.088+0000 7fe325041700 1 --2- 172.21.15.2:0/1831448620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324d40c0 0x564b324d9630 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe314009c40 tx=0x7fe3140038b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfa9bed4e3bb8e12 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:10.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.088+0000 7fe2fffff700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:29:10.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.089+0000 7fe2fffff700 1 -- 172.21.15.2:0/1831448620 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31401d030 con 0x564b324d40c0 2023-11-16T18:29:10.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.089+0000 7fe2fffff700 1 -- 172.21.15.2:0/1831448620 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe314005040 con 0x564b324d40c0 2023-11-16T18:29:10.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.089+0000 7fe332c30b80 1 -- 172.21.15.2:0/1831448620 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564b324b9b20 con 0x564b324d40c0 2023-11-16T18:29:10.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.090+0000 7fe2fffff700 1 -- 172.21.15.2:0/1831448620 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe314005200 con 0x564b324d40c0 2023-11-16T18:29:10.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.090+0000 7fe332c30b80 10 client.25963 fetch_fsmap learned FSMap version 29 2023-11-16T18:29:10.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.090+0000 7fe332c30b80 1 -- 172.21.15.2:0/1831448620 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564b323de190 con 0x564b324d40c0 2023-11-16T18:29:10.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.091+0000 7fe2fffff700 1 -- 172.21.15.2:0/1831448620 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564b324b9b20 con 0x564b324d40c0 2023-11-16T18:29:10.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.091+0000 7fe2fffff700 1 -- 172.21.15.2:0/1831448620 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fe314037c30 con 0x564b324d40c0 2023-11-16T18:29:10.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.091+0000 7fe332c30b80 10 client.25963 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:29:10.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.091+0000 7fe332c30b80 10 client.25963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:29:10.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.091+0000 7fe332c30b80 20 client.25963 populate_metadata read hostname 'smithi002' 2023-11-16T18:29:10.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.092+0000 7fe332c30b80 1 --2- 172.21.15.2:0/1831448620 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564b324e8bc0 0x564b324eafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.092+0000 7fe332c30b80 4 client.25963 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:29:10.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.092+0000 7fe332c30b80 1 -- 172.21.15.2:0/1831448620 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564b323da0a0 con 0x564b324e8bc0 2023-11-16T18:29:10.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.092+0000 7fe324840700 1 --2- 172.21.15.2:0/1831448620 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564b324e8bc0 0x564b324eafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:10.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.095+0000 7fe324840700 1 --2- 172.21.15.2:0/1831448620 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564b324e8bc0 0x564b324eafa0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89473f5924053315 server_cookie=c2c249d7b7f5f70e in_seq=0 out_seq=0 2023-11-16T18:29:10.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.096+0000 7fe2fffff700 10 client.25963 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:10.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.096+0000 7fe2fffff700 1 -- 172.21.15.2:0/1831448620 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x564b323da0a0 con 0x564b324e8bc0 2023-11-16T18:29:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.096+0000 7fe2fffff700 10 client.25963 handle_command_reply: tid=0 2023-11-16T18:29:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.096+0000 7fe332c30b80 1 client.25963 shutdown 2023-11-16T18:29:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.097+0000 7fe332c30b80 1 -- 172.21.15.2:0/1831448620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324d40c0 msgr2=0x564b324d9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.097+0000 7fe332c30b80 1 --2- 172.21.15.2:0/1831448620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324d40c0 0x564b324d9630 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe314009c40 tx=0x7fe3140038b0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.098+0000 7fe332c30b80 1 -- 172.21.15.2:0/1831448620 shutdown_connections 2023-11-16T18:29:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.098+0000 7fe332c30b80 1 --2- 172.21.15.2:0/1831448620 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564b324e8bc0 0x564b324eafa0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.098+0000 7fe332c30b80 1 --2- 172.21.15.2:0/1831448620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324d40c0 0x564b324d9630 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.098+0000 7fe2fffff700 0 client.25963 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.098+0000 7fe332c30b80 1 --2- 172.21.15.2:0/1831448620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324cf370 0x564b324d8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.098+0000 7fe332c30b80 1 --2- 172.21.15.2:0/1831448620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b324cfc80 0x564b324d90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.098+0000 7fe332c30b80 1 -- 172.21.15.2:0/1831448620 >> 172.21.15.2:0/1831448620 conn(0x564b324bc360 msgr2=0x564b324d5570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:10.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.099+0000 7fe332c30b80 1 -- 172.21.15.2:0/1831448620 shutdown_connections 2023-11-16T18:29:10.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.100+0000 7fe332c30b80 1 -- 172.21.15.2:0/1831448620 wait complete. 2023-11-16T18:29:10.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.100+0000 7fe332c30b80 20 client.25963 trim_cache size 0 max 16384 2023-11-16T18:29:10.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.134+0000 7fe332c30b80 1 Processor -- start 2023-11-16T18:29:10.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.134+0000 7fe332c30b80 1 -- start start 2023-11-16T18:29:10.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.135+0000 7fe332c30b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324cf370 0x564b324e0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.135+0000 7fe332c30b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324cfc80 0x564b324d3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.135+0000 7fe332c30b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b324d40c0 0x564b324c1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.135+0000 7fe332c30b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b324e5b20 con 0x564b324cf370 2023-11-16T18:29:10.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.135+0000 7fe332c30b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b324c3200 con 0x564b324d40c0 2023-11-16T18:29:10.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.135+0000 7fe332c30b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b324c2830 con 0x564b324cfc80 2023-11-16T18:29:10.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.135+0000 7fe324840700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324cf370 0x564b324e0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:10.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.135+0000 7fe324840700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324cf370 0x564b324e0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52432/0 (socket says 172.21.15.2:52432) 2023-11-16T18:29:10.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.135+0000 7fe324840700 1 -- 172.21.15.2:0/2335470795 learned_addr learned my addr 172.21.15.2:0/2335470795 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:10.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.136+0000 7fe324840700 1 -- 172.21.15.2:0/2335470795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324cfc80 msgr2=0x564b324d3860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T18:29:10.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.136+0000 7fe324840700 1 --2- 172.21.15.2:0/2335470795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324cfc80 0x564b324d3860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.136+0000 7fe324840700 1 -- 172.21.15.2:0/2335470795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b324d40c0 msgr2=0x564b324c1560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T18:29:10.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.136+0000 7fe324840700 1 --2- 172.21.15.2:0/2335470795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b324d40c0 0x564b324c1560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.136+0000 7fe324840700 1 -- 172.21.15.2:0/2335470795 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b323ddee0 con 0x564b324cf370 2023-11-16T18:29:10.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.136+0000 7fe324840700 1 --2- 172.21.15.2:0/2335470795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324cf370 0x564b324e0150 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x564b324eb810 tx=0x7fe3180092b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d83515ffac4f9d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:10.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.136+0000 7fe3077fe700 1 -- 172.21.15.2:0/2335470795 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318004db0 con 0x564b324cf370 2023-11-16T18:29:10.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.136+0000 7fe3077fe700 1 -- 172.21.15.2:0/2335470795 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe31801f070 con 0x564b324cf370 2023-11-16T18:29:10.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.136+0000 7fe3077fe700 1 -- 172.21.15.2:0/2335470795 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318004510 con 0x564b324cf370 2023-11-16T18:29:10.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.138+0000 7fe332c30b80 1 -- 172.21.15.2:0/2335470795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324cf370 msgr2=0x564b324e0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:10.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.138+0000 7fe332c30b80 1 --2- 172.21.15.2:0/2335470795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324cf370 0x564b324e0150 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x564b324eb810 tx=0x7fe3180092b0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.138+0000 7fe332c30b80 1 -- 172.21.15.2:0/2335470795 shutdown_connections 2023-11-16T18:29:10.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.138+0000 7fe332c30b80 1 --2- 172.21.15.2:0/2335470795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b324d40c0 0x564b324c1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.138+0000 7fe332c30b80 1 --2- 172.21.15.2:0/2335470795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324cf370 0x564b324e0150 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.138+0000 7fe332c30b80 1 --2- 172.21.15.2:0/2335470795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324cfc80 0x564b324d3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.138+0000 7fe332c30b80 1 -- 172.21.15.2:0/2335470795 >> 172.21.15.2:0/2335470795 conn(0x564b324bc360 msgr2=0x564b324d3150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:10.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.139+0000 7fe332c30b80 1 -- 172.21.15.2:0/2335470795 shutdown_connections 2023-11-16T18:29:10.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.139+0000 7fe332c30b80 1 -- 172.21.15.2:0/2335470795 wait complete. 2023-11-16T18:29:10.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.140+0000 7fe332c30b80 1 -- start start 2023-11-16T18:29:10.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.140+0000 7fe332c30b80 1 Processor -- start 2023-11-16T18:29:10.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.140+0000 7fe332c30b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324cf370 0x564b324de330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.140+0000 7fe332c30b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 0x564b324de870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.140+0000 7fe332c30b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b325359b0 0x564b32535e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.140+0000 7fe332c30b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b32536360 con 0x564b324d40c0 2023-11-16T18:29:10.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.140+0000 7fe332c30b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b325364d0 con 0x564b325359b0 2023-11-16T18:29:10.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.140+0000 7fe332c30b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b32536610 con 0x564b324cf370 2023-11-16T18:29:10.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.140+0000 7fe332c30b80 10 client.0 _ll_register_callbacks cb 0x7fe31400f9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:29:10.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.141+0000 7fe31ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 0x564b324de870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:10.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.141+0000 7fe31ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 0x564b324de870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52436/0 (socket says 172.21.15.2:52436) 2023-11-16T18:29:10.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.141+0000 7fe31ffff700 1 -- 172.21.15.2:0/1697433764 learned_addr learned my addr 172.21.15.2:0/1697433764 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:10.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.141+0000 7fe325041700 1 --2- 172.21.15.2:0/1697433764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b325359b0 0x564b32535e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:10.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.141+0000 7fe31ffff700 1 -- 172.21.15.2:0/1697433764 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324cf370 msgr2=0x564b324de330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T18:29:10.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.141+0000 7fe31ffff700 1 --2- 172.21.15.2:0/1697433764 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324cf370 0x564b324de330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.163 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.141+0000 7fe31ffff700 1 -- 172.21.15.2:0/1697433764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b325359b0 msgr2=0x564b32535e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:10.164 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.141+0000 7fe31ffff700 1 --2- 172.21.15.2:0/1697433764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b325359b0 0x564b32535e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.164 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.141+0000 7fe31ffff700 1 -- 172.21.15.2:0/1697433764 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31800a930 con 0x564b324d40c0 2023-11-16T18:29:10.164 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.142+0000 7fe31ffff700 1 --2- 172.21.15.2:0/1697433764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 0x564b324de870 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe31000afd0 tx=0x7fe310098be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee728f078c5aad7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:10.164 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.142+0000 7fe2fffff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:29:10.164 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.142+0000 7fe2fffff700 1 -- 172.21.15.2:0/1697433764 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310098660 con 0x564b324d40c0 2023-11-16T18:29:10.164 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.142+0000 7fe2fffff700 1 -- 172.21.15.2:0/1697433764 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe310098f70 con 0x564b324d40c0 2023-11-16T18:29:10.166 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.142+0000 7fe332c30b80 1 -- 172.21.15.2:0/1697433764 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe31400bca0 con 0x564b324d40c0 2023-11-16T18:29:10.166 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.144+0000 7fe2fffff700 1 -- 172.21.15.2:0/1697433764 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe31009cfa0 con 0x564b324d40c0 2023-11-16T18:29:10.166 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.144+0000 7fe332c30b80 10 client.16551 fetch_fsmap learned FSMap version 29 2023-11-16T18:29:10.166 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.144+0000 7fe332c30b80 1 -- 172.21.15.2:0/1697433764 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564b3222ec60 con 0x564b324d40c0 2023-11-16T18:29:10.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.144+0000 7fe2fffff700 1 -- 172.21.15.2:0/1697433764 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe31400bca0 con 0x564b324d40c0 2023-11-16T18:29:10.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.144+0000 7fe2fffff700 1 -- 172.21.15.2:0/1697433764 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fe3100b8d20 con 0x564b324d40c0 2023-11-16T18:29:10.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.144+0000 7fe332c30b80 10 client.16551 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:29:10.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.144+0000 7fe332c30b80 10 client.16551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:29:10.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.144+0000 7fe332c30b80 20 client.16551 populate_metadata read hostname 'smithi002' 2023-11-16T18:29:10.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.144+0000 7fe332c30b80 1 --2- 172.21.15.2:0/1697433764 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564b324c4ce0 0x564b324c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:10.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.144+0000 7fe332c30b80 4 client.16551 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:29:10.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.144+0000 7fe332c30b80 1 -- 172.21.15.2:0/1697433764 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564b323e6660 con 0x564b324c4ce0 2023-11-16T18:29:10.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.145+0000 7fe324840700 1 --2- 172.21.15.2:0/1697433764 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564b324c4ce0 0x564b324c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:10.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.145+0000 7fe324840700 1 --2- 172.21.15.2:0/1697433764 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564b324c4ce0 0x564b324c70f0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5755bf381fa44e5e server_cookie=666d7e4ba348d5a2 in_seq=0 out_seq=0 2023-11-16T18:29:10.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.145+0000 7fe2fffff700 10 client.16551 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:10.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.146+0000 7fe2fffff700 1 -- 172.21.15.2:0/1697433764 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x564b323e6660 con 0x564b324c4ce0 2023-11-16T18:29:10.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.146+0000 7fe2fffff700 10 client.16551 handle_command_reply: tid=0 2023-11-16T18:29:10.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.146+0000 7fe332c30b80 1 client.16551 shutdown 2023-11-16T18:29:10.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.146+0000 7fe332c30b80 1 -- 172.21.15.2:0/1697433764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 msgr2=0x564b324de870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:10.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.146+0000 7fe332c30b80 1 --2- 172.21.15.2:0/1697433764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 0x564b324de870 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe31000afd0 tx=0x7fe310098be0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.146+0000 7fe332c30b80 1 -- 172.21.15.2:0/1697433764 shutdown_connections 2023-11-16T18:29:10.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.146+0000 7fe332c30b80 1 --2- 172.21.15.2:0/1697433764 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564b324c4ce0 0x564b324c70f0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.146+0000 7fe332c30b80 1 --2- 172.21.15.2:0/1697433764 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564b324cf370 0x564b324de330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.146+0000 7fe332c30b80 1 --2- 172.21.15.2:0/1697433764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b325359b0 0x564b32535e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.146+0000 7fe332c30b80 1 --2- 172.21.15.2:0/1697433764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564b324d40c0 0x564b324de870 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.146+0000 7fe332c30b80 1 -- 172.21.15.2:0/1697433764 >> 172.21.15.2:0/1697433764 conn(0x564b324bc360 msgr2=0x564b324d8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:10.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.146+0000 7fe2fffff700 0 client.16551 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:10.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.146+0000 7fe332c30b80 1 -- 172.21.15.2:0/1697433764 shutdown_connections 2023-11-16T18:29:10.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.147+0000 7fe332c30b80 1 -- 172.21.15.2:0/1697433764 wait complete. 2023-11-16T18:29:10.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.147+0000 7fe332c30b80 20 client.16551 trim_cache size 0 max 16384 2023-11-16T18:29:10.210 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:29:10.210 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "scrub active (1 inodes in the stack)", 2023-11-16T18:29:10.210 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": { 2023-11-16T18:29:10.248 INFO:teuthology.orchestra.run.smithi002.stdout: "9607aa23-76ae-4198-9378-3740d8aff22c": { 2023-11-16T18:29:10.248 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/", 2023-11-16T18:29:10.248 INFO:teuthology.orchestra.run.smithi002.stdout: "tag": "9607aa23-76ae-4198-9378-3740d8aff22c", 2023-11-16T18:29:10.248 INFO:teuthology.orchestra.run.smithi002.stdout: "options": "recursive,force" 2023-11-16T18:29:10.248 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:29:10.249 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:29:10.249 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:29:10.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.148+0000 7fe326aa4700 1 -- 172.21.15.2:0/2157017891 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe3080a1480 msgr2=0x7fe3080a3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:10.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.148+0000 7fe326aa4700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe3080a1480 0x7fe3080a3930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe3140098c0 tx=0x7fe31401a040 comp rx=0 tx=0).stop 2023-11-16T18:29:10.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.148+0000 7fe326aa4700 1 -- 172.21.15.2:0/2157017891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe320100420 msgr2=0x7fe32006b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:10.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.148+0000 7fe326aa4700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe320100420 0x7fe32006b570 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe3100078c0 tx=0x7fe310007bd0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.150+0000 7fe326aa4700 1 -- 172.21.15.2:0/2157017891 shutdown_connections 2023-11-16T18:29:10.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.150+0000 7fe326aa4700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe3080a1480 0x7fe3080a3930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.150+0000 7fe326aa4700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe320105250 0x7fe3201a8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.150+0000 7fe326aa4700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3200ffa70 0x7fe32006b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.150+0000 7fe326aa4700 1 --2- 172.21.15.2:0/2157017891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe320100420 0x7fe32006b570 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:10.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.150+0000 7fe326aa4700 1 -- 172.21.15.2:0/2157017891 >> 172.21.15.2:0/2157017891 conn(0x7fe3200763a0 msgr2=0x7fe3200fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:10.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.150+0000 7fe326aa4700 1 -- 172.21.15.2:0/2157017891 shutdown_connections 2023-11-16T18:29:10.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:10.150+0000 7fe326aa4700 1 -- 172.21.15.2:0/2157017891 wait complete. 2023-11-16T18:29:10.251 INFO:tasks.cephfs.filesystem:scrub status for tag:9607aa23-76ae-4198-9378-3740d8aff22c - {'path': '/', 'tag': '9607aa23-76ae-4198-9378-3740d8aff22c', 'options': 'recursive,force'} 2023-11-16T18:29:10.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:10 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:29:10.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:10 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:10.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:10 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:29:10.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:10 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:10.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:10 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:29:10.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:10 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:11.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:11 smithi040 ceph-mon[100251]: pgmap v133: 129 pgs: 129 active+clean; 482 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 387 op/s 2023-11-16T18:29:11.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:11 smithi188 ceph-mon[100639]: pgmap v133: 129 pgs: 129 active+clean; 482 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 387 op/s 2023-11-16T18:29:11.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:11 smithi002 ceph-mon[98807]: pgmap v133: 129 pgs: 129 active+clean; 482 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 387 op/s 2023-11-16T18:29:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.069+0000 7ff77fdd2700 1 -- 172.21.15.2:0/545148517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff77810a810 msgr2=0x7ff77810cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.069+0000 7ff77fdd2700 1 --2- 172.21.15.2:0/545148517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff77810a810 0x7ff77810cc30 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff77000b0a0 tx=0x7ff77000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.070+0000 7ff77fdd2700 1 -- 172.21.15.2:0/545148517 shutdown_connections 2023-11-16T18:29:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.070+0000 7ff77fdd2700 1 --2- 172.21.15.2:0/545148517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff77810a810 0x7ff77810cc30 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.070+0000 7ff77fdd2700 1 --2- 172.21.15.2:0/545148517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff778101440 0x7ff778101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.070+0000 7ff77fdd2700 1 --2- 172.21.15.2:0/545148517 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff778101d50 0x7ff77810a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.070+0000 7ff77fdd2700 1 -- 172.21.15.2:0/545148517 >> 172.21.15.2:0/545148517 conn(0x7ff7780763a0 msgr2=0x7ff7780767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.070+0000 7ff77fdd2700 1 -- 172.21.15.2:0/545148517 shutdown_connections 2023-11-16T18:29:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.070+0000 7ff77fdd2700 1 -- 172.21.15.2:0/545148517 wait complete. 2023-11-16T18:29:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.070+0000 7ff77fdd2700 1 Processor -- start 2023-11-16T18:29:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.070+0000 7ff77fdd2700 1 -- start start 2023-11-16T18:29:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.070+0000 7ff77fdd2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff778101440 0x7ff778198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77fdd2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff778101d50 0x7ff778198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77fdd2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff77810a810 0x7ff77819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77fdd2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77819d630 con 0x7ff778101d50 2023-11-16T18:29:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77fdd2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77819d7a0 con 0x7ff77810a810 2023-11-16T18:29:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77fdd2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77819d960 con 0x7ff778101440 2023-11-16T18:29:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77d36d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff778101d50 0x7ff778198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77d36d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff778101d50 0x7ff778198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52452/0 (socket says 172.21.15.2:52452) 2023-11-16T18:29:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77d36d700 1 -- 172.21.15.2:0/592439913 learned_addr learned my addr 172.21.15.2:0/592439913 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77e36f700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff77810a810 0x7ff77819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77db6e700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff778101440 0x7ff778198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77d36d700 1 -- 172.21.15.2:0/592439913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff778101440 msgr2=0x7ff778198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77d36d700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff778101440 0x7ff778198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77d36d700 1 -- 172.21.15.2:0/592439913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff77810a810 msgr2=0x7ff77819cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77d36d700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff77810a810 0x7ff77819cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77d36d700 1 -- 172.21.15.2:0/592439913 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff770009d00 con 0x7ff778101d50 2023-11-16T18:29:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77d36d700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff778101d50 0x7ff778198b90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff76800d980 tx=0x7ff76800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff76effd700 1 -- 172.21.15.2:0/592439913 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff768004220 con 0x7ff778101d50 2023-11-16T18:29:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77fdd2700 1 -- 172.21.15.2:0/592439913 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff778110260 con 0x7ff778101d50 2023-11-16T18:29:14.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff76effd700 1 -- 172.21.15.2:0/592439913 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff768007640 con 0x7ff778101d50 2023-11-16T18:29:14.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff76effd700 1 -- 172.21.15.2:0/592439913 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff768005610 con 0x7ff778101d50 2023-11-16T18:29:14.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.071+0000 7ff77fdd2700 1 -- 172.21.15.2:0/592439913 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7781100d0 con 0x7ff778101d50 2023-11-16T18:29:14.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.074+0000 7ff76effd700 1 -- 172.21.15.2:0/592439913 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7ff768004750 con 0x7ff778101d50 2023-11-16T18:29:14.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.074+0000 7ff76effd700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff7640a12c0 0x7ff7640a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.074+0000 7ff76effd700 1 -- 172.21.15.2:0/592439913 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff7680cce60 con 0x7ff778101d50 2023-11-16T18:29:14.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.075+0000 7ff77db6e700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff7640a12c0 0x7ff7640a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.075+0000 7ff77db6e700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff7640a12c0 0x7ff7640a3770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff774009bf0 tx=0x7ff774009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:14.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.075+0000 7ff78bf5eb80 1 Processor -- start 2023-11-16T18:29:14.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.075+0000 7ff78bf5eb80 1 -- start start 2023-11-16T18:29:14.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.075+0000 7ff78bf5eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 0x55b7c21191e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.075+0000 7ff78bf5eb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119720 0x55b7c2113560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff78bf5eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119b90 0x55b7c2113b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff78bf5eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c2019000 con 0x55b7c2119b90 2023-11-16T18:29:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff78bf5eb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c206b840 con 0x55b7c2119720 2023-11-16T18:29:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff78bf5eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c2064cd0 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff77e36f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119b90 0x55b7c2113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff77e36f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119b90 0x55b7c2113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52460/0 (socket says 172.21.15.2:52460) 2023-11-16T18:29:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff77d36d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119720 0x55b7c2113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff77e36f700 1 -- 172.21.15.2:0/2871895611 learned_addr learned my addr 172.21.15.2:0/2871895611 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff77db6e700 1 --2- 172.21.15.2:0/2871895611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 0x55b7c21191e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff77e36f700 1 -- 172.21.15.2:0/2871895611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 msgr2=0x55b7c21191e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff77e36f700 1 --2- 172.21.15.2:0/2871895611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 0x55b7c21191e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff77e36f700 1 -- 172.21.15.2:0/2871895611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119720 msgr2=0x55b7c2113560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff77e36f700 1 --2- 172.21.15.2:0/2871895611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119720 0x55b7c2113560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff77e36f700 1 -- 172.21.15.2:0/2871895611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7c1ee65d0 con 0x55b7c2119b90 2023-11-16T18:29:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff77d36d700 1 --2- 172.21.15.2:0/2871895611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119720 0x55b7c2113560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:29:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff77e36f700 1 --2- 172.21.15.2:0/2871895611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119b90 0x55b7c2113b00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff770009310 tx=0x7ff770009340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798cada09f855877 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.076+0000 7ff763fff700 1 -- 172.21.15.2:0/2871895611 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77000e040 con 0x55b7c2119b90 2023-11-16T18:29:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.077+0000 7ff763fff700 1 -- 172.21.15.2:0/2871895611 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff77000bcb0 con 0x55b7c2119b90 2023-11-16T18:29:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.077+0000 7ff763fff700 1 -- 172.21.15.2:0/2871895611 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77001c6a0 con 0x55b7c2119b90 2023-11-16T18:29:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.077+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/2871895611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119b90 msgr2=0x55b7c2113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.077+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/2871895611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119b90 0x55b7c2113b00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff770009310 tx=0x7ff770009340 comp rx=0 tx=0).stop 2023-11-16T18:29:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.077+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/2871895611 shutdown_connections 2023-11-16T18:29:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.077+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/2871895611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 0x55b7c21191e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.077+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/2871895611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119b90 0x55b7c2113b00 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.077+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/2871895611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119720 0x55b7c2113560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.077+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/2871895611 >> 172.21.15.2:0/2871895611 conn(0x55b7c2108f50 msgr2=0x55b7c210e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.077+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/2871895611 shutdown_connections 2023-11-16T18:29:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.078+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/2871895611 wait complete. 2023-11-16T18:29:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.078+0000 7ff78bf5eb80 1 -- start start 2023-11-16T18:29:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.078+0000 7ff78bf5eb80 1 Processor -- start 2023-11-16T18:29:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff78bf5eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 0x55b7c2122a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff78bf5eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 0x55b7c2122f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff78bf5eb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c211cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff78bf5eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c2006880 con 0x55b7c2119720 2023-11-16T18:29:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff78bf5eb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c2006520 con 0x55b7c2119b90 2023-11-16T18:29:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff78bf5eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c1ee88c0 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff78bf5eb80 10 client.0 _ll_register_callbacks cb 0x55b7c2024720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:29:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff77d36d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 0x55b7c2122f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff77db6e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 0x55b7c2122a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff77d36d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 0x55b7c2122f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52466/0 (socket says 172.21.15.2:52466) 2023-11-16T18:29:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff77d36d700 1 -- 172.21.15.2:0/2409246663 learned_addr learned my addr 172.21.15.2:0/2409246663 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff77e36f700 1 --2- 172.21.15.2:0/2409246663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c211cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff77d36d700 1 -- 172.21.15.2:0/2409246663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 msgr2=0x55b7c2122a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff77d36d700 1 --2- 172.21.15.2:0/2409246663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 0x55b7c2122a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff77d36d700 1 -- 172.21.15.2:0/2409246663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 msgr2=0x55b7c211cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff77d36d700 1 --2- 172.21.15.2:0/2409246663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c211cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff77d36d700 1 -- 172.21.15.2:0/2409246663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77810d8f0 con 0x55b7c2119720 2023-11-16T18:29:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff77d36d700 1 --2- 172.21.15.2:0/2409246663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 0x55b7c2122f90 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff7680960c0 tx=0x7ff768099cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3e4e85d631a462 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff7617fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:29:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff7617fa700 1 -- 172.21.15.2:0/2409246663 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7680a2010 con 0x55b7c2119720 2023-11-16T18:29:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff7617fa700 1 -- 172.21.15.2:0/2409246663 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff768099e20 con 0x55b7c2119720 2023-11-16T18:29:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.079+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/2409246663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b7c2106710 con 0x55b7c2119720 2023-11-16T18:29:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.080+0000 7ff7617fa700 1 -- 172.21.15.2:0/2409246663 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff768099fe0 con 0x55b7c2119720 2023-11-16T18:29:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.080+0000 7ff78bf5eb80 10 client.16563 fetch_fsmap learned FSMap version 29 2023-11-16T18:29:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.080+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/2409246663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55b7c202b130 con 0x55b7c2119720 2023-11-16T18:29:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.080+0000 7ff7617fa700 1 -- 172.21.15.2:0/2409246663 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff7680c8020 con 0x55b7c2119720 2023-11-16T18:29:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.080+0000 7ff7617fa700 1 -- 172.21.15.2:0/2409246663 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ff7680a1c10 con 0x55b7c2119720 2023-11-16T18:29:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.080+0000 7ff78bf5eb80 10 client.16563 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:29:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.080+0000 7ff78bf5eb80 10 client.16563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:29:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.080+0000 7ff78bf5eb80 20 client.16563 populate_metadata read hostname 'smithi002' 2023-11-16T18:29:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.080+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/2409246663 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55b7c212cf90 0x55b7c212f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.080+0000 7ff78bf5eb80 4 client.16563 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:29:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.080+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/2409246663 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b7c2027040 con 0x55b7c212cf90 2023-11-16T18:29:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.080+0000 7ff77db6e700 1 --2- 172.21.15.2:0/2409246663 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55b7c212cf90 0x55b7c212f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.081+0000 7ff7617fa700 10 client.16563 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.081+0000 7ff77db6e700 1 --2- 172.21.15.2:0/2409246663 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55b7c212cf90 0x55b7c212f3a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1de21e2958cc54cd server_cookie=6463d9759d25abaf in_seq=0 out_seq=0 2023-11-16T18:29:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.086+0000 7ff7617fa700 1 -- 172.21.15.2:0/2409246663 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b7c2027040 con 0x55b7c212cf90 2023-11-16T18:29:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.086+0000 7ff7617fa700 10 client.16563 handle_command_reply: tid=0 2023-11-16T18:29:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.086+0000 7ff78bf5eb80 1 client.16563 shutdown 2023-11-16T18:29:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.086+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/2409246663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 msgr2=0x55b7c2122f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.086+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/2409246663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 0x55b7c2122f90 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff7680960c0 tx=0x7ff768099cb0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.087+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/2409246663 shutdown_connections 2023-11-16T18:29:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.087+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/2409246663 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55b7c212cf90 0x55b7c212f3a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.087+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/2409246663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 0x55b7c2122a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.087+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/2409246663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c211cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.087+0000 7ff7617fa700 0 client.16563 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.087+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/2409246663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 0x55b7c2122f90 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.087+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/2409246663 >> 172.21.15.2:0/2409246663 conn(0x55b7c2108f50 msgr2=0x55b7c210d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.087+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/2409246663 shutdown_connections 2023-11-16T18:29:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.087+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/2409246663 wait complete. 2023-11-16T18:29:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.087+0000 7ff78bf5eb80 20 client.16563 trim_cache size 0 max 16384 2023-11-16T18:29:14.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff78bf5eb80 1 Processor -- start 2023-11-16T18:29:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff78bf5eb80 1 -- start start 2023-11-16T18:29:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff78bf5eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 0x55b7c211a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff78bf5eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 0x55b7c210df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff78bf5eb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c2113560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff78bf5eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c210fc50 con 0x55b7c2119720 2023-11-16T18:29:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff78bf5eb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c211b240 con 0x55b7c2119b90 2023-11-16T18:29:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff78bf5eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c2117990 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff77d36d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 0x55b7c210df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff77d36d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 0x55b7c210df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52474/0 (socket says 172.21.15.2:52474) 2023-11-16T18:29:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff77db6e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 0x55b7c211a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff77d36d700 1 -- 172.21.15.2:0/3145954302 learned_addr learned my addr 172.21.15.2:0/3145954302 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff77e36f700 1 --2- 172.21.15.2:0/3145954302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c2113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff77db6e700 1 -- 172.21.15.2:0/3145954302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 msgr2=0x55b7c2113560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff77db6e700 1 --2- 172.21.15.2:0/3145954302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c2113560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff77db6e700 1 -- 172.21.15.2:0/3145954302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 msgr2=0x55b7c210df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff77db6e700 1 --2- 172.21.15.2:0/3145954302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 0x55b7c210df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff77db6e700 1 -- 172.21.15.2:0/3145954302 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7c202ae80 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.103+0000 7ff77d36d700 1 --2- 172.21.15.2:0/3145954302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 0x55b7c210df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:29:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.104+0000 7ff77db6e700 1 --2- 172.21.15.2:0/3145954302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 0x55b7c211a6a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55b7c212fc50 tx=0x7ff774014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa523d5d7130686e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.104+0000 7ff76cff9700 1 -- 172.21.15.2:0/3145954302 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff774024040 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.104+0000 7ff76cff9700 1 -- 172.21.15.2:0/3145954302 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff774016070 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.104+0000 7ff76cff9700 1 -- 172.21.15.2:0/3145954302 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff774012d30 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.105+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/3145954302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 msgr2=0x55b7c211a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.105+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/3145954302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 0x55b7c211a6a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55b7c212fc50 tx=0x7ff774014000 comp rx=0 tx=0).stop 2023-11-16T18:29:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.105+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/3145954302 shutdown_connections 2023-11-16T18:29:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.105+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/3145954302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c1f6f6b0 0x55b7c211a6a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.105+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/3145954302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c2113560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.105+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/3145954302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c2119720 0x55b7c210df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.105+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/3145954302 >> 172.21.15.2:0/3145954302 conn(0x55b7c2108f50 msgr2=0x55b7c2119f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.105+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/3145954302 shutdown_connections 2023-11-16T18:29:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.105+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/3145954302 wait complete. 2023-11-16T18:29:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.105+0000 7ff78bf5eb80 1 -- start start 2023-11-16T18:29:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.105+0000 7ff78bf5eb80 1 Processor -- start 2023-11-16T18:29:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff78bf5eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c1f6f6b0 0x55b7c2122630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff78bf5eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c2119720 0x55b7c2122b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff78bf5eb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c2120b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff78bf5eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c21211a0 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff78bf5eb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c2121310 con 0x55b7c2119b90 2023-11-16T18:29:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff78bf5eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c21214d0 con 0x55b7c2119720 2023-11-16T18:29:14.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff78bf5eb80 10 client.0 _ll_register_callbacks cb 0x7ff76800f3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:29:14.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff77db6e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c1f6f6b0 0x55b7c2122630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff77d36d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c2119720 0x55b7c2122b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff77e36f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c2120b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff77e36f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c2120b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42920/0 (socket says 172.21.15.2:42920) 2023-11-16T18:29:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff77d36d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c2119720 0x55b7c2122b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:44010/0 (socket says 172.21.15.2:44010) 2023-11-16T18:29:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff77db6e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c1f6f6b0 0x55b7c2122630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52478/0 (socket says 172.21.15.2:52478) 2023-11-16T18:29:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff77e36f700 1 -- 172.21.15.2:0/4141890996 learned_addr learned my addr 172.21.15.2:0/4141890996 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff77db6e700 1 -- 172.21.15.2:0/4141890996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c2119720 msgr2=0x55b7c2122b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff77db6e700 1 --2- 172.21.15.2:0/4141890996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c2119720 0x55b7c2122b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff77db6e700 1 -- 172.21.15.2:0/4141890996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 msgr2=0x55b7c2120b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff77db6e700 1 --2- 172.21.15.2:0/4141890996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c2120b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff77db6e700 1 -- 172.21.15.2:0/4141890996 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b7c202b130 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.106+0000 7ff77e36f700 1 --2- 172.21.15.2:0/4141890996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c2120b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:29:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.107+0000 7ff77db6e700 1 --2- 172.21.15.2:0/4141890996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c1f6f6b0 0x55b7c2122630 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7ff774012010 tx=0x7ff7740128b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=973f8f213e9feadf server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.107+0000 7ff7617fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:29:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.108+0000 7ff7617fa700 1 -- 172.21.15.2:0/4141890996 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff774024050 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.108+0000 7ff7617fa700 1 -- 172.21.15.2:0/4141890996 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff774018070 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.108+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/4141890996 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff77000bcb0 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.109+0000 7ff7617fa700 1 -- 172.21.15.2:0/4141890996 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff774012bb0 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.109+0000 7ff78bf5eb80 10 client.16569 fetch_fsmap learned FSMap version 29 2023-11-16T18:29:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.109+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/4141890996 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7ff774029950 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.109+0000 7ff7617fa700 1 -- 172.21.15.2:0/4141890996 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff77000bcb0 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.109+0000 7ff7617fa700 1 -- 172.21.15.2:0/4141890996 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ff774030020 con 0x55b7c1f6f6b0 2023-11-16T18:29:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.109+0000 7ff78bf5eb80 10 client.16569 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:29:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.109+0000 7ff78bf5eb80 10 client.16569 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:29:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.109+0000 7ff78bf5eb80 20 client.16569 populate_metadata read hostname 'smithi002' 2023-11-16T18:29:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.109+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/4141890996 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55b7c2177e60 0x55b7c217a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.109+0000 7ff78bf5eb80 4 client.16569 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:29:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.109+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/4141890996 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b7c2033600 con 0x55b7c2177e60 2023-11-16T18:29:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.109+0000 7ff77d36d700 1 --2- 172.21.15.2:0/4141890996 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55b7c2177e60 0x55b7c217a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.110+0000 7ff7617fa700 10 client.16569 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:14.110+0000 7ff77d36d700 1 --2- 172.21.15.2:0/4141890996 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55b7c2177e60 0x55b7c217a270 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fc1c313f4d75aa9 server_cookie=109d409d458598b9 in_seq=0 out_seq=0 2023-11-16T18:29:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:14 smithi040 ceph-mon[100251]: pgmap v134: 129 pgs: 129 active+clean; 504 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.3 MiB/s wr, 566 op/s 2023-11-16T18:29:14.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:14 smithi188 ceph-mon[100639]: pgmap v134: 129 pgs: 129 active+clean; 504 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.3 MiB/s wr, 566 op/s 2023-11-16T18:29:14.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:14 smithi002 ceph-mon[98807]: pgmap v134: 129 pgs: 129 active+clean; 504 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.3 MiB/s wr, 566 op/s 2023-11-16T18:29:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.197+0000 7ff7617fa700 1 -- 172.21.15.2:0/4141890996 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b7c2033600 con 0x55b7c2177e60 2023-11-16T18:29:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.197+0000 7ff7617fa700 10 client.16569 handle_command_reply: tid=0 2023-11-16T18:29:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.197+0000 7ff78bf5eb80 1 client.16569 shutdown 2023-11-16T18:29:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.197+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/4141890996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c1f6f6b0 msgr2=0x55b7c2122630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.197+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/4141890996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c1f6f6b0 0x55b7c2122630 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7ff774012010 tx=0x7ff7740128b0 comp rx=0 tx=0).stop 2023-11-16T18:29:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.197+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/4141890996 shutdown_connections 2023-11-16T18:29:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.197+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/4141890996 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55b7c2177e60 0x55b7c217a270 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.197+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/4141890996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b7c2119b90 0x55b7c2120b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.197+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/4141890996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b7c1f6f6b0 0x55b7c2122630 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.197+0000 7ff78bf5eb80 1 --2- 172.21.15.2:0/4141890996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7c2119720 0x55b7c2122b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.197+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/4141890996 >> 172.21.15.2:0/4141890996 conn(0x55b7c2108f50 msgr2=0x55b7c211a820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.197+0000 7ff7617fa700 0 client.16569 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.198+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/4141890996 shutdown_connections 2023-11-16T18:29:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.198+0000 7ff78bf5eb80 1 -- 172.21.15.2:0/4141890996 wait complete. 2023-11-16T18:29:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.198+0000 7ff78bf5eb80 20 client.16569 trim_cache size 0 max 16384 2023-11-16T18:29:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:29:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:29:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:29:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:29:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.202+0000 7ff77fdd2700 1 -- 172.21.15.2:0/592439913 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff7640a12c0 msgr2=0x7ff7640a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.202+0000 7ff77fdd2700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff7640a12c0 0x7ff7640a3770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff774009bf0 tx=0x7ff774009260 comp rx=0 tx=0).stop 2023-11-16T18:29:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.202+0000 7ff77fdd2700 1 -- 172.21.15.2:0/592439913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff778101d50 msgr2=0x7ff778198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.202+0000 7ff77fdd2700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff778101d50 0x7ff778198b90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff76800d980 tx=0x7ff76800dc90 comp rx=0 tx=0).stop 2023-11-16T18:29:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.202+0000 7ff77fdd2700 1 -- 172.21.15.2:0/592439913 shutdown_connections 2023-11-16T18:29:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.202+0000 7ff77fdd2700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ff7640a12c0 0x7ff7640a3770 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:15.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.202+0000 7ff77fdd2700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff778101440 0x7ff778198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:15.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.202+0000 7ff77fdd2700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff77810a810 0x7ff77819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:15.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.202+0000 7ff77fdd2700 1 --2- 172.21.15.2:0/592439913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff778101d50 0x7ff778198b90 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:15.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.202+0000 7ff77fdd2700 1 -- 172.21.15.2:0/592439913 >> 172.21.15.2:0/592439913 conn(0x7ff7780763a0 msgr2=0x7ff7780ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:15.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.205+0000 7ff77fdd2700 1 -- 172.21.15.2:0/592439913 shutdown_connections 2023-11-16T18:29:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:15.205+0000 7ff77fdd2700 1 -- 172.21.15.2:0/592439913 wait complete. 2023-11-16T18:29:16.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:16 smithi040 ceph-mon[100251]: pgmap v135: 129 pgs: 129 active+clean; 504 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 322 op/s 2023-11-16T18:29:16.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:16 smithi188 ceph-mon[100639]: pgmap v135: 129 pgs: 129 active+clean; 504 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 322 op/s 2023-11-16T18:29:16.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:16 smithi002 ceph-mon[98807]: pgmap v135: 129 pgs: 129 active+clean; 504 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 322 op/s 2023-11-16T18:29:17.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:17 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:29:17.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:17 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:17 smithi002 ceph-mon[98807]: pgmap v136: 129 pgs: 129 active+clean; 504 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 356 op/s 2023-11-16T18:29:17.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:17 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:29:17.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:17 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:17.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:17 smithi040 ceph-mon[100251]: pgmap v136: 129 pgs: 129 active+clean; 504 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 356 op/s 2023-11-16T18:29:17.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:17 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:29:17.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:17 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:17.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:17 smithi188 ceph-mon[100639]: pgmap v136: 129 pgs: 129 active+clean; 504 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 356 op/s 2023-11-16T18:29:20.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:20 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:29:20.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:20 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:20.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:20 smithi040 ceph-mon[100251]: pgmap v137: 129 pgs: 129 active+clean; 521 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.1 MiB/s wr, 642 op/s 2023-11-16T18:29:20.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:20 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:29:20.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:20 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:20.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:20 smithi188 ceph-mon[100639]: pgmap v137: 129 pgs: 129 active+clean; 521 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.1 MiB/s wr, 642 op/s 2023-11-16T18:29:20.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:20 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:29:20.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:20 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:20.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:20 smithi002 ceph-mon[98807]: pgmap v137: 129 pgs: 129 active+clean; 521 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.1 MiB/s wr, 642 op/s 2023-11-16T18:29:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:22 smithi040 ceph-mon[100251]: pgmap v138: 129 pgs: 129 active+clean; 521 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 508 op/s 2023-11-16T18:29:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:22 smithi188 ceph-mon[100639]: pgmap v138: 129 pgs: 129 active+clean; 521 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 508 op/s 2023-11-16T18:29:22.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:22 smithi002 ceph-mon[98807]: pgmap v138: 129 pgs: 129 active+clean; 521 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 508 op/s 2023-11-16T18:29:24.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:24 smithi040 ceph-mon[100251]: pgmap v139: 129 pgs: 129 active+clean; 545 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.4 MiB/s wr, 679 op/s 2023-11-16T18:29:24.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:24 smithi188 ceph-mon[100639]: pgmap v139: 129 pgs: 129 active+clean; 545 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.4 MiB/s wr, 679 op/s 2023-11-16T18:29:24.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:24 smithi002 ceph-mon[98807]: pgmap v139: 129 pgs: 129 active+clean; 545 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.4 MiB/s wr, 679 op/s 2023-11-16T18:29:26.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:26 smithi040 ceph-mon[100251]: pgmap v140: 129 pgs: 129 active+clean; 545 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 499 op/s 2023-11-16T18:29:26.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:26 smithi188 ceph-mon[100639]: pgmap v140: 129 pgs: 129 active+clean; 545 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 499 op/s 2023-11-16T18:29:26.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:26 smithi002 ceph-mon[98807]: pgmap v140: 129 pgs: 129 active+clean; 545 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 499 op/s 2023-11-16T18:29:27.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:27 smithi002 ceph-mon[98807]: pgmap v141: 129 pgs: 129 active+clean; 550 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 513 op/s 2023-11-16T18:29:27.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:27 smithi040 ceph-mon[100251]: pgmap v141: 129 pgs: 129 active+clean; 550 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 513 op/s 2023-11-16T18:29:27.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:27 smithi188 ceph-mon[100639]: pgmap v141: 129 pgs: 129 active+clean; 550 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 513 op/s 2023-11-16T18:29:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.408+0000 7f17cf6e8700 1 -- 172.21.15.2:0/1555922266 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c8107af0 msgr2=0x7f17c810eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.408+0000 7f17cf6e8700 1 --2- 172.21.15.2:0/1555922266 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c8107af0 0x7f17c810eb50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f17c401a9a0 tx=0x7f17c401acb0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.408+0000 7f17cf6e8700 1 -- 172.21.15.2:0/1555922266 shutdown_connections 2023-11-16T18:29:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.408+0000 7f17cf6e8700 1 --2- 172.21.15.2:0/1555922266 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c8107af0 0x7f17c810eb50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.408+0000 7f17cf6e8700 1 --2- 172.21.15.2:0/1555922266 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17c8119a40 0x7f17c811be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.408+0000 7f17cf6e8700 1 --2- 172.21.15.2:0/1555922266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17c8107ec0 0x7f17c8108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.408+0000 7f17cf6e8700 1 -- 172.21.15.2:0/1555922266 >> 172.21.15.2:0/1555922266 conn(0x7f17c806c490 msgr2=0x7f17c8071690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.408+0000 7f17cf6e8700 1 -- 172.21.15.2:0/1555922266 shutdown_connections 2023-11-16T18:29:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.408+0000 7f17cf6e8700 1 -- 172.21.15.2:0/1555922266 wait complete. 2023-11-16T18:29:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.409+0000 7f17cf6e8700 1 Processor -- start 2023-11-16T18:29:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.409+0000 7f17cf6e8700 1 -- start start 2023-11-16T18:29:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.409+0000 7f17cf6e8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17c8107af0 0x7f17c8114e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.409+0000 7f17cf6e8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c8107ec0 0x7f17c81153c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.409+0000 7f17cf6e8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17c8119a40 0x7f17c81a26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.409+0000 7f17cf6e8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81a2cc0 con 0x7f17c8119a40 2023-11-16T18:29:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.409+0000 7f17cf6e8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81a2e30 con 0x7f17c8107af0 2023-11-16T18:29:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.409+0000 7f17cf6e8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81a2ff0 con 0x7f17c8107ec0 2023-11-16T18:29:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.410+0000 7f17cdc85700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17c8119a40 0x7f17c81a26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.410+0000 7f17cdc85700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17c8119a40 0x7f17c81a26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52494/0 (socket says 172.21.15.2:52494) 2023-11-16T18:29:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.410+0000 7f17cdc85700 1 -- 172.21.15.2:0/3921650866 learned_addr learned my addr 172.21.15.2:0/3921650866 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.410+0000 7f17ccc83700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c8107ec0 0x7f17c81153c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.410+0000 7f17cd484700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17c8107af0 0x7f17c8114e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.410+0000 7f17cdc85700 1 -- 172.21.15.2:0/3921650866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c8107ec0 msgr2=0x7f17c81153c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.410+0000 7f17cdc85700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c8107ec0 0x7f17c81153c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.410+0000 7f17cdc85700 1 -- 172.21.15.2:0/3921650866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17c8107af0 msgr2=0x7f17c8114e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.410+0000 7f17cdc85700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17c8107af0 0x7f17c8114e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.410+0000 7f17cdc85700 1 -- 172.21.15.2:0/3921650866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c401a600 con 0x7f17c8119a40 2023-11-16T18:29:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.410+0000 7f17cd484700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17c8107af0 0x7f17c8114e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:29:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.410+0000 7f17ccc83700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c8107ec0 0x7f17c81153c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:29:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.410+0000 7f17cdc85700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17c8119a40 0x7f17c81a26c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f17c000b970 tx=0x7f17c000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.411+0000 7f17be7fc700 1 -- 172.21.15.2:0/3921650866 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c000c9b0 con 0x7f17c8119a40 2023-11-16T18:29:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.411+0000 7f17cf6e8700 1 -- 172.21.15.2:0/3921650866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c8111ae0 con 0x7f17c8119a40 2023-11-16T18:29:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.411+0000 7f17cf6e8700 1 -- 172.21.15.2:0/3921650866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c8111950 con 0x7f17c8119a40 2023-11-16T18:29:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.411+0000 7f17be7fc700 1 -- 172.21.15.2:0/3921650866 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17c0014440 con 0x7f17c8119a40 2023-11-16T18:29:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.411+0000 7f17be7fc700 1 -- 172.21.15.2:0/3921650866 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c0007570 con 0x7f17c8119a40 2023-11-16T18:29:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.414+0000 7f17be7fc700 1 -- 172.21.15.2:0/3921650866 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f17c000cb10 con 0x7f17c8119a40 2023-11-16T18:29:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.415+0000 7f17be7fc700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f17b40a1400 0x7f17b40a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.416+0000 7f17be7fc700 1 -- 172.21.15.2:0/3921650866 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f17c00cc600 con 0x7f17c8119a40 2023-11-16T18:29:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.416+0000 7f17cd484700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f17b40a1400 0x7f17b40a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.416+0000 7f17db874b80 1 Processor -- start 2023-11-16T18:29:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.416+0000 7f17db874b80 1 -- start start 2023-11-16T18:29:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.416+0000 7f17db874b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e7635d6b0 0x557e765071e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.416+0000 7f17db874b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507720 0x557e76501560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.416+0000 7f17db874b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507b90 0x557e76501b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.416+0000 7f17db874b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e76407000 con 0x557e76507b90 2023-11-16T18:29:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.416+0000 7f17db874b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e76459840 con 0x557e7635d6b0 2023-11-16T18:29:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.416+0000 7f17db874b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e76452cd0 con 0x557e76507720 2023-11-16T18:29:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.417+0000 7f17cdc85700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507b90 0x557e76501b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.417+0000 7f17cdc85700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507b90 0x557e76501b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52502/0 (socket says 172.21.15.2:52502) 2023-11-16T18:29:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.417+0000 7f17cdc85700 1 -- 172.21.15.2:0/3537234360 learned_addr learned my addr 172.21.15.2:0/3537234360 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.417+0000 7f17cdc85700 1 -- 172.21.15.2:0/3537234360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507720 msgr2=0x557e76501560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.417+0000 7f17cdc85700 1 --2- 172.21.15.2:0/3537234360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507720 0x557e76501560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.417+0000 7f17cdc85700 1 -- 172.21.15.2:0/3537234360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e7635d6b0 msgr2=0x557e765071e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.417+0000 7f17cdc85700 1 --2- 172.21.15.2:0/3537234360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e7635d6b0 0x557e765071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.417+0000 7f17cdc85700 1 -- 172.21.15.2:0/3537234360 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e762d45d0 con 0x557e76507b90 2023-11-16T18:29:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.417+0000 7f17cdc85700 1 --2- 172.21.15.2:0/3537234360 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507b90 0x557e76501b00 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f17c00a1330 tx=0x7f17c0098df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617e1344324f09e7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.417+0000 7f17b37fe700 1 -- 172.21.15.2:0/3537234360 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c00cc070 con 0x557e76507b90 2023-11-16T18:29:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.418+0000 7f17b37fe700 1 -- 172.21.15.2:0/3537234360 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17c009d440 con 0x557e76507b90 2023-11-16T18:29:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.418+0000 7f17b37fe700 1 -- 172.21.15.2:0/3537234360 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c00cc1d0 con 0x557e76507b90 2023-11-16T18:29:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.419+0000 7f17db874b80 1 -- 172.21.15.2:0/3537234360 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507b90 msgr2=0x557e76501b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.419+0000 7f17db874b80 1 --2- 172.21.15.2:0/3537234360 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507b90 0x557e76501b00 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f17c00a1330 tx=0x7f17c0098df0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.419+0000 7f17db874b80 1 -- 172.21.15.2:0/3537234360 shutdown_connections 2023-11-16T18:29:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.419+0000 7f17db874b80 1 --2- 172.21.15.2:0/3537234360 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507b90 0x557e76501b00 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.419+0000 7f17db874b80 1 --2- 172.21.15.2:0/3537234360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e7635d6b0 0x557e765071e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.419+0000 7f17db874b80 1 --2- 172.21.15.2:0/3537234360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507720 0x557e76501560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.419+0000 7f17db874b80 1 -- 172.21.15.2:0/3537234360 >> 172.21.15.2:0/3537234360 conn(0x557e764f6f50 msgr2=0x557e764fc270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.422+0000 7f17cd484700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f17b40a1400 0x7f17b40a38b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f17c4004800 tx=0x7f17c4005fb0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.422+0000 7f17db874b80 1 -- 172.21.15.2:0/3537234360 shutdown_connections 2023-11-16T18:29:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.422+0000 7f17db874b80 1 -- 172.21.15.2:0/3537234360 wait complete. 2023-11-16T18:29:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.422+0000 7f17db874b80 1 -- start start 2023-11-16T18:29:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.422+0000 7f17db874b80 1 Processor -- start 2023-11-16T18:29:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.423+0000 7f17db874b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507720 0x557e7650aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.423+0000 7f17db874b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507b90 0x557e7650b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.423+0000 7f17db874b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e7650b930 0x557e76517160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.423+0000 7f17db874b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e763f4880 con 0x557e76507720 2023-11-16T18:29:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.423+0000 7f17db874b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e763f4520 con 0x557e7650b930 2023-11-16T18:29:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.423+0000 7f17db874b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e762d68c0 con 0x557e76507b90 2023-11-16T18:29:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.423+0000 7f17db874b80 10 client.0 _ll_register_callbacks cb 0x557e76412720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:29:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.424+0000 7f17cd484700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507720 0x557e7650aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.424+0000 7f17cd484700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507720 0x557e7650aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52508/0 (socket says 172.21.15.2:52508) 2023-11-16T18:29:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.424+0000 7f17cd484700 1 -- 172.21.15.2:0/2148184137 learned_addr learned my addr 172.21.15.2:0/2148184137 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.424+0000 7f17ccc83700 1 --2- 172.21.15.2:0/2148184137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507b90 0x557e7650b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.424+0000 7f17cdc85700 1 --2- 172.21.15.2:0/2148184137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e7650b930 0x557e76517160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.424+0000 7f17ccc83700 1 -- 172.21.15.2:0/2148184137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e7650b930 msgr2=0x557e76517160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.424+0000 7f17ccc83700 1 --2- 172.21.15.2:0/2148184137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e7650b930 0x557e76517160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.424+0000 7f17ccc83700 1 -- 172.21.15.2:0/2148184137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507720 msgr2=0x557e7650aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.424+0000 7f17ccc83700 1 --2- 172.21.15.2:0/2148184137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507720 0x557e7650aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.424+0000 7f17ccc83700 1 -- 172.21.15.2:0/2148184137 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c8111950 con 0x557e76507b90 2023-11-16T18:29:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.424+0000 7f17ccc83700 1 --2- 172.21.15.2:0/2148184137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507b90 0x557e7650b3f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f17c81165d0 tx=0x7f17b800d440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2034b6c948c6556f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.425+0000 7f17b0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:29:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.425+0000 7f17b0ff9700 1 -- 172.21.15.2:0/2148184137 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b800dd80 con 0x557e76507b90 2023-11-16T18:29:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.425+0000 7f17b0ff9700 1 -- 172.21.15.2:0/2148184137 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f17b8013480 con 0x557e76507b90 2023-11-16T18:29:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.425+0000 7f17db874b80 1 -- 172.21.15.2:0/2148184137 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557e764f4710 con 0x557e76507b90 2023-11-16T18:29:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.425+0000 7f17b0ff9700 1 -- 172.21.15.2:0/2148184137 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17b8013680 con 0x557e76507b90 2023-11-16T18:29:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.426+0000 7f17db874b80 10 client.25990 fetch_fsmap learned FSMap version 29 2023-11-16T18:29:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.426+0000 7f17db874b80 1 -- 172.21.15.2:0/2148184137 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557e76419130 con 0x557e76507b90 2023-11-16T18:29:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.430+0000 7f17b0ff9700 1 -- 172.21.15.2:0/2148184137 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557e764f4710 con 0x557e76507b90 2023-11-16T18:29:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.430+0000 7f17b0ff9700 1 -- 172.21.15.2:0/2148184137 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f17b8012920 con 0x557e76507b90 2023-11-16T18:29:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.430+0000 7f17db874b80 10 client.25990 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:29:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.430+0000 7f17db874b80 10 client.25990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:29:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.430+0000 7f17db874b80 20 client.25990 populate_metadata read hostname 'smithi002' 2023-11-16T18:29:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.430+0000 7f17db874b80 1 --2- 172.21.15.2:0/2148184137 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e7651b640 0x557e7651da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.430+0000 7f17db874b80 4 client.25990 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:29:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.430+0000 7f17db874b80 1 -- 172.21.15.2:0/2148184137 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557e76415040 con 0x557e7651b640 2023-11-16T18:29:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.430+0000 7f17cd484700 1 --2- 172.21.15.2:0/2148184137 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e7651b640 0x557e7651da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.431+0000 7f17cd484700 1 --2- 172.21.15.2:0/2148184137 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e7651b640 0x557e7651da50 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cea1327c37877e1 server_cookie=fa35b59ecf7aafc0 in_seq=0 out_seq=0 2023-11-16T18:29:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.434+0000 7f17b0ff9700 10 client.25990 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.436+0000 7f17b0ff9700 1 -- 172.21.15.2:0/2148184137 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557e76415040 con 0x557e7651b640 2023-11-16T18:29:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.436+0000 7f17b0ff9700 10 client.25990 handle_command_reply: tid=0 2023-11-16T18:29:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.436+0000 7f17db874b80 1 client.25990 shutdown 2023-11-16T18:29:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.437+0000 7f17db874b80 1 -- 172.21.15.2:0/2148184137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507b90 msgr2=0x557e7650b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.437+0000 7f17db874b80 1 --2- 172.21.15.2:0/2148184137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507b90 0x557e7650b3f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f17c81165d0 tx=0x7f17b800d440 comp rx=0 tx=0).stop 2023-11-16T18:29:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.437+0000 7f17db874b80 1 -- 172.21.15.2:0/2148184137 shutdown_connections 2023-11-16T18:29:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.438+0000 7f17db874b80 1 --2- 172.21.15.2:0/2148184137 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e7651b640 0x557e7651da50 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.438+0000 7f17db874b80 1 --2- 172.21.15.2:0/2148184137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e7650b930 0x557e76517160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.438+0000 7f17db874b80 1 --2- 172.21.15.2:0/2148184137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507720 0x557e7650aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.438+0000 7f17db874b80 1 --2- 172.21.15.2:0/2148184137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507b90 0x557e7650b3f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.438+0000 7f17db874b80 1 -- 172.21.15.2:0/2148184137 >> 172.21.15.2:0/2148184137 conn(0x557e764f6f50 msgr2=0x557e764fb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.438+0000 7f17b0ff9700 0 client.25990 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.438+0000 7f17db874b80 1 -- 172.21.15.2:0/2148184137 shutdown_connections 2023-11-16T18:29:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.438+0000 7f17db874b80 1 -- 172.21.15.2:0/2148184137 wait complete. 2023-11-16T18:29:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.438+0000 7f17db874b80 20 client.25990 trim_cache size 0 max 16384 2023-11-16T18:29:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.465+0000 7f17db874b80 1 Processor -- start 2023-11-16T18:29:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.465+0000 7f17db874b80 1 -- start start 2023-11-16T18:29:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.465+0000 7f17db874b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507720 0x557e76501560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.465+0000 7f17db874b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e76507b90 0x557e764fbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.465+0000 7f17db874b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e7650b930 0x557e765168d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.465+0000 7f17db874b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e764fc7e0 con 0x557e7650b930 2023-11-16T18:29:28.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.465+0000 7f17db874b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e764fd430 con 0x557e76507b90 2023-11-16T18:29:28.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.465+0000 7f17db874b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e76509430 con 0x557e76507720 2023-11-16T18:29:28.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.466+0000 7f17cdc85700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e7650b930 0x557e765168d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.466+0000 7f17cdc85700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e7650b930 0x557e765168d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52512/0 (socket says 172.21.15.2:52512) 2023-11-16T18:29:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.466+0000 7f17cdc85700 1 -- 172.21.15.2:0/3565866230 learned_addr learned my addr 172.21.15.2:0/3565866230 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.466+0000 7f17cd484700 1 --2- 172.21.15.2:0/3565866230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507720 0x557e76501560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.467+0000 7f17cdc85700 1 -- 172.21.15.2:0/3565866230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507720 msgr2=0x557e76501560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.467+0000 7f17cdc85700 1 --2- 172.21.15.2:0/3565866230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507720 0x557e76501560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.467+0000 7f17cdc85700 1 -- 172.21.15.2:0/3565866230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e76507b90 msgr2=0x557e764fbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.467+0000 7f17cdc85700 1 --2- 172.21.15.2:0/3565866230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e76507b90 0x557e764fbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.467+0000 7f17cdc85700 1 -- 172.21.15.2:0/3565866230 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e76418e80 con 0x557e7650b930 2023-11-16T18:29:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.468+0000 7f17cdc85700 1 --2- 172.21.15.2:0/3565866230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e7650b930 0x557e765168d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f17c00993a0 tx=0x7f17c0084040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8e4d7626c72dc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.469+0000 7f17b3fff700 1 -- 172.21.15.2:0/3565866230 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c00cc040 con 0x557e7650b930 2023-11-16T18:29:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.469+0000 7f17b3fff700 1 -- 172.21.15.2:0/3565866230 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17c00a1860 con 0x557e7650b930 2023-11-16T18:29:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.469+0000 7f17b3fff700 1 -- 172.21.15.2:0/3565866230 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c00cc400 con 0x557e7650b930 2023-11-16T18:29:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.470+0000 7f17db874b80 1 -- 172.21.15.2:0/3565866230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e7650b930 msgr2=0x557e765168d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.470+0000 7f17db874b80 1 --2- 172.21.15.2:0/3565866230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e7650b930 0x557e765168d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f17c00993a0 tx=0x7f17c0084040 comp rx=0 tx=0).stop 2023-11-16T18:29:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.471+0000 7f17db874b80 1 -- 172.21.15.2:0/3565866230 shutdown_connections 2023-11-16T18:29:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.471+0000 7f17db874b80 1 --2- 172.21.15.2:0/3565866230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e76507720 0x557e76501560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.471+0000 7f17db874b80 1 --2- 172.21.15.2:0/3565866230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e7650b930 0x557e765168d0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.471+0000 7f17db874b80 1 --2- 172.21.15.2:0/3565866230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e76507b90 0x557e764fbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.471+0000 7f17db874b80 1 -- 172.21.15.2:0/3565866230 >> 172.21.15.2:0/3565866230 conn(0x557e764f6f50 msgr2=0x557e76517820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.471+0000 7f17db874b80 1 -- 172.21.15.2:0/3565866230 shutdown_connections 2023-11-16T18:29:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.471+0000 7f17db874b80 1 -- 172.21.15.2:0/3565866230 wait complete. 2023-11-16T18:29:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.472+0000 7f17db874b80 1 -- start start 2023-11-16T18:29:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.472+0000 7f17db874b80 1 Processor -- start 2023-11-16T18:29:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.474+0000 7f17db874b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e76507720 0x557e76512130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.474+0000 7f17db874b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507b90 0x557e76510110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.474+0000 7f17db874b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e7650b930 0x557e76510650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.474+0000 7f17db874b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e76510b90 con 0x557e76507b90 2023-11-16T18:29:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.474+0000 7f17db874b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e76510d00 con 0x557e76507720 2023-11-16T18:29:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.474+0000 7f17db874b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e76510ec0 con 0x557e7650b930 2023-11-16T18:29:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.474+0000 7f17db874b80 10 client.0 _ll_register_callbacks cb 0x7f17b8007820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:29:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.474+0000 7f17cd484700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e76507720 0x557e76512130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.474+0000 7f17cd484700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e76507720 0x557e76512130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:42956/0 (socket says 172.21.15.2:42956) 2023-11-16T18:29:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.474+0000 7f17cd484700 1 -- 172.21.15.2:0/3173996301 learned_addr learned my addr 172.21.15.2:0/3173996301 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.474+0000 7f17ccc83700 1 --2- 172.21.15.2:0/3173996301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507b90 0x557e76510110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.475+0000 7f17ccc83700 1 -- 172.21.15.2:0/3173996301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e7650b930 msgr2=0x557e76510650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T18:29:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.475+0000 7f17ccc83700 1 --2- 172.21.15.2:0/3173996301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e7650b930 0x557e76510650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.475+0000 7f17ccc83700 1 -- 172.21.15.2:0/3173996301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e76507720 msgr2=0x557e76512130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.475+0000 7f17ccc83700 1 --2- 172.21.15.2:0/3173996301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e76507720 0x557e76512130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.475+0000 7f17ccc83700 1 -- 172.21.15.2:0/3173996301 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557e762d45d0 con 0x557e76507b90 2023-11-16T18:29:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.475+0000 7f17ccc83700 1 --2- 172.21.15.2:0/3173996301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507b90 0x557e76510110 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f17b800d840 tx=0x7f17b800df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6038694e2e33057d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.475+0000 7f17b0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:29:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.476+0000 7f17b0ff9700 1 -- 172.21.15.2:0/3173996301 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8006fd0 con 0x557e76507b90 2023-11-16T18:29:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.476+0000 7f17b0ff9700 1 -- 172.21.15.2:0/3173996301 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f17b8012ea0 con 0x557e76507b90 2023-11-16T18:29:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.476+0000 7f17db874b80 1 -- 172.21.15.2:0/3173996301 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f17c009d440 con 0x557e76507b90 2023-11-16T18:29:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.477+0000 7f17b0ff9700 1 -- 172.21.15.2:0/3173996301 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17b801c870 con 0x557e76507b90 2023-11-16T18:29:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.477+0000 7f17db874b80 10 client.16584 fetch_fsmap learned FSMap version 29 2023-11-16T18:29:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.477+0000 7f17db874b80 1 -- 172.21.15.2:0/3173996301 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557e76269c00 con 0x557e76507b90 2023-11-16T18:29:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.478+0000 7f17b0ff9700 1 -- 172.21.15.2:0/3173996301 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17c009d440 con 0x557e76507b90 2023-11-16T18:29:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.478+0000 7f17b0ff9700 1 -- 172.21.15.2:0/3173996301 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f17b8013480 con 0x557e76507b90 2023-11-16T18:29:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.479+0000 7f17db874b80 10 client.16584 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:29:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.479+0000 7f17db874b80 10 client.16584 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:29:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.479+0000 7f17db874b80 20 client.16584 populate_metadata read hostname 'smithi002' 2023-11-16T18:29:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.479+0000 7f17db874b80 1 --2- 172.21.15.2:0/3173996301 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e7656f8d0 0x557e76571ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.479+0000 7f17db874b80 4 client.16584 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:29:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.479+0000 7f17db874b80 1 -- 172.21.15.2:0/3173996301 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557e76421600 con 0x557e7656f8d0 2023-11-16T18:29:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.482+0000 7f17cd484700 1 --2- 172.21.15.2:0/3173996301 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e7656f8d0 0x557e76571ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.489+0000 7f17cd484700 1 --2- 172.21.15.2:0/3173996301 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e7656f8d0 0x557e76571ce0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce05c61c820f3d8e server_cookie=db0e639baf251048 in_seq=0 out_seq=0 2023-11-16T18:29:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:28.490+0000 7f17b0ff9700 10 client.16584 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:28.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:28 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:29:28.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:28 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:28.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:28 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:29:28.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:28 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:28.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:28 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:29:28.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:28 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:29.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:29 smithi188 ceph-mon[100639]: pgmap v142: 129 pgs: 129 active+clean; 567 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.4 MiB/s wr, 613 op/s 2023-11-16T18:29:29.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:29 smithi002 ceph-mon[98807]: pgmap v142: 129 pgs: 129 active+clean; 567 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.4 MiB/s wr, 613 op/s 2023-11-16T18:29:29.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:29 smithi040 ceph-mon[100251]: pgmap v142: 129 pgs: 129 active+clean; 567 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.4 MiB/s wr, 613 op/s 2023-11-16T18:29:32.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:32 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:29:32.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:32 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:32.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:32 smithi040 ceph-mon[100251]: pgmap v143: 129 pgs: 129 active+clean; 567 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 318 op/s 2023-11-16T18:29:32.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:32 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:29:32.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:32 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:32.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:32 smithi188 ceph-mon[100639]: pgmap v143: 129 pgs: 129 active+clean; 567 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 318 op/s 2023-11-16T18:29:32.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:32 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:29:32.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:32 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:32.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:32 smithi002 ceph-mon[98807]: pgmap v143: 129 pgs: 129 active+clean; 567 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 318 op/s 2023-11-16T18:29:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.070+0000 7f17b0ff9700 1 -- 172.21.15.2:0/3173996301 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557e76421600 con 0x557e7656f8d0 2023-11-16T18:29:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.070+0000 7f17b0ff9700 10 client.16584 handle_command_reply: tid=0 2023-11-16T18:29:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.070+0000 7f17db874b80 1 client.16584 shutdown 2023-11-16T18:29:33.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.070+0000 7f17db874b80 1 -- 172.21.15.2:0/3173996301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507b90 msgr2=0x557e76510110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:33.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.070+0000 7f17db874b80 1 --2- 172.21.15.2:0/3173996301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507b90 0x557e76510110 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f17b800d840 tx=0x7f17b800df90 comp rx=0 tx=0).stop 2023-11-16T18:29:33.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.070+0000 7f17db874b80 1 -- 172.21.15.2:0/3173996301 shutdown_connections 2023-11-16T18:29:33.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.070+0000 7f17db874b80 1 --2- 172.21.15.2:0/3173996301 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e7656f8d0 0x557e76571ce0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:33.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.070+0000 7f17db874b80 1 --2- 172.21.15.2:0/3173996301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e7650b930 0x557e76510650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.070+0000 7f17b0ff9700 0 client.16584 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.070+0000 7f17db874b80 1 --2- 172.21.15.2:0/3173996301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e76507720 0x557e76512130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.070+0000 7f17db874b80 1 --2- 172.21.15.2:0/3173996301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e76507b90 0x557e76510110 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.070+0000 7f17db874b80 1 -- 172.21.15.2:0/3173996301 >> 172.21.15.2:0/3173996301 conn(0x557e764f6f50 msgr2=0x557e765168d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.071+0000 7f17db874b80 1 -- 172.21.15.2:0/3173996301 shutdown_connections 2023-11-16T18:29:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.071+0000 7f17db874b80 1 -- 172.21.15.2:0/3173996301 wait complete. 2023-11-16T18:29:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.071+0000 7f17db874b80 20 client.16584 trim_cache size 0 max 16384 2023-11-16T18:29:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:29:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:29:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:29:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:29:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.072+0000 7f17cf6e8700 1 -- 172.21.15.2:0/3921650866 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f17b40a1400 msgr2=0x7f17b40a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.072+0000 7f17cf6e8700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f17b40a1400 0x7f17b40a38b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f17c4004800 tx=0x7f17c4005fb0 comp rx=0 tx=0).stop 2023-11-16T18:29:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.072+0000 7f17cf6e8700 1 -- 172.21.15.2:0/3921650866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17c8119a40 msgr2=0x7f17c81a26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.072+0000 7f17cf6e8700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17c8119a40 0x7f17c81a26c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f17c000b970 tx=0x7f17c000bd30 comp rx=0 tx=0).stop 2023-11-16T18:29:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.074+0000 7f17cf6e8700 1 -- 172.21.15.2:0/3921650866 shutdown_connections 2023-11-16T18:29:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.074+0000 7f17cf6e8700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f17b40a1400 0x7f17b40a38b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.074+0000 7f17cf6e8700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17c8119a40 0x7f17c81a26c0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.074+0000 7f17cf6e8700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17c8107af0 0x7f17c8114e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.074+0000 7f17cf6e8700 1 --2- 172.21.15.2:0/3921650866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17c8107ec0 0x7f17c81153c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.074+0000 7f17cf6e8700 1 -- 172.21.15.2:0/3921650866 >> 172.21.15.2:0/3921650866 conn(0x7f17c806c490 msgr2=0x7f17c811bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.074+0000 7f17cf6e8700 1 -- 172.21.15.2:0/3921650866 shutdown_connections 2023-11-16T18:29:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:33.074+0000 7f17cf6e8700 1 -- 172.21.15.2:0/3921650866 wait complete. 2023-11-16T18:29:34.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:34 smithi040 ceph-mon[100251]: pgmap v144: 129 pgs: 129 active+clean; 589 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.8 MiB/s wr, 734 op/s 2023-11-16T18:29:34.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:34 smithi188 ceph-mon[100639]: pgmap v144: 129 pgs: 129 active+clean; 589 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.8 MiB/s wr, 734 op/s 2023-11-16T18:29:34.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:34 smithi002 ceph-mon[98807]: pgmap v144: 129 pgs: 129 active+clean; 589 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.8 MiB/s wr, 734 op/s 2023-11-16T18:29:36.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:36 smithi040 ceph-mon[100251]: pgmap v145: 129 pgs: 129 active+clean; 589 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 563 op/s 2023-11-16T18:29:36.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:36 smithi188 ceph-mon[100639]: pgmap v145: 129 pgs: 129 active+clean; 589 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 563 op/s 2023-11-16T18:29:36.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:36 smithi002 ceph-mon[98807]: pgmap v145: 129 pgs: 129 active+clean; 589 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 563 op/s 2023-11-16T18:29:37.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:37 smithi002 ceph-mon[98807]: pgmap v146: 129 pgs: 129 active+clean; 589 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 563 op/s 2023-11-16T18:29:37.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:37 smithi040 ceph-mon[100251]: pgmap v146: 129 pgs: 129 active+clean; 589 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 563 op/s 2023-11-16T18:29:37.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:37 smithi188 ceph-mon[100639]: pgmap v146: 129 pgs: 129 active+clean; 589 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 563 op/s 2023-11-16T18:29:38.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:38 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:29:38.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:38 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:38.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:38 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:29:38.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:38 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:38.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:38 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:29:38.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:38 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:39.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:39 smithi188 ceph-mon[100639]: pgmap v147: 129 pgs: 129 active+clean; 588 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 5.0 MiB/s wr, 821 op/s 2023-11-16T18:29:39.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:39 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:29:39.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:39 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:39.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:39 smithi002 ceph-mon[98807]: pgmap v147: 129 pgs: 129 active+clean; 588 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 5.0 MiB/s wr, 821 op/s 2023-11-16T18:29:39.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:39 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:29:39.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:39 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:39.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:39 smithi040 ceph-mon[100251]: pgmap v147: 129 pgs: 129 active+clean; 588 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 5.0 MiB/s wr, 821 op/s 2023-11-16T18:29:39.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:39 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:29:39.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:39 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:40.252 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:29:40.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.455+0000 7fcbf1f1d700 1 -- 172.21.15.2:0/3750292710 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbec0ff3f0 msgr2=0x7fcbec104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:40.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.455+0000 7fcbf1f1d700 1 --2- 172.21.15.2:0/3750292710 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbec0ff3f0 0x7fcbec104ec0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00099c0 tx=0x7fcbe0009cd0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.455+0000 7fcbf1f1d700 1 -- 172.21.15.2:0/3750292710 shutdown_connections 2023-11-16T18:29:40.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.455+0000 7fcbf1f1d700 1 --2- 172.21.15.2:0/3750292710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbec1055f0 0x7fcbec10d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.455+0000 7fcbf1f1d700 1 --2- 172.21.15.2:0/3750292710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbec0fea50 0x7fcbec0fee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.455+0000 7fcbf1f1d700 1 --2- 172.21.15.2:0/3750292710 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbec0ff3f0 0x7fcbec104ec0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.455+0000 7fcbf1f1d700 1 -- 172.21.15.2:0/3750292710 >> 172.21.15.2:0/3750292710 conn(0x7fcbec075880 msgr2=0x7fcbec075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:40.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.455+0000 7fcbf1f1d700 1 -- 172.21.15.2:0/3750292710 shutdown_connections 2023-11-16T18:29:40.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.455+0000 7fcbf1f1d700 1 -- 172.21.15.2:0/3750292710 wait complete. 2023-11-16T18:29:40.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.456+0000 7fcbf1f1d700 1 Processor -- start 2023-11-16T18:29:40.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.456+0000 7fcbf1f1d700 1 -- start start 2023-11-16T18:29:40.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.456+0000 7fcbf1f1d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbec0fea50 0x7fcbec1962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.456+0000 7fcbf1f1d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbec0ff3f0 0x7fcbec196830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.456+0000 7fcbf1f1d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbec1055f0 0x7fcbec19ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.456+0000 7fcbf1f1d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec19b2d0 con 0x7fcbec1055f0 2023-11-16T18:29:40.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.456+0000 7fcbf1f1d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec19b440 con 0x7fcbec0ff3f0 2023-11-16T18:29:40.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.456+0000 7fcbf1f1d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec19b600 con 0x7fcbec0fea50 2023-11-16T18:29:40.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.456+0000 7fcbebfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbec1055f0 0x7fcbec19ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.456+0000 7fcbebfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbec1055f0 0x7fcbec19ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52534/0 (socket says 172.21.15.2:52534) 2023-11-16T18:29:40.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.456+0000 7fcbebfff700 1 -- 172.21.15.2:0/712018808 learned_addr learned my addr 172.21.15.2:0/712018808 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:40.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.456+0000 7fcbeaffd700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbec0ff3f0 0x7fcbec196830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.456+0000 7fcbeb7fe700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbec0fea50 0x7fcbec1962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.457+0000 7fcbebfff700 1 -- 172.21.15.2:0/712018808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbec0fea50 msgr2=0x7fcbec1962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:40.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.457+0000 7fcbebfff700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbec0fea50 0x7fcbec1962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.457+0000 7fcbebfff700 1 -- 172.21.15.2:0/712018808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbec0ff3f0 msgr2=0x7fcbec196830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:40.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.457+0000 7fcbebfff700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbec0ff3f0 0x7fcbec196830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.457+0000 7fcbebfff700 1 -- 172.21.15.2:0/712018808 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbdc009520 con 0x7fcbec1055f0 2023-11-16T18:29:40.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.457+0000 7fcbeaffd700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbec0ff3f0 0x7fcbec196830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:29:40.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.457+0000 7fcbebfff700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbec1055f0 0x7fcbec19ac90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400ef30 tx=0x7fcbe400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:40.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.457+0000 7fcbe8ff9700 1 -- 172.21.15.2:0/712018808 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4018070 con 0x7fcbec1055f0 2023-11-16T18:29:40.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.457+0000 7fcbe8ff9700 1 -- 172.21.15.2:0/712018808 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcbe400f040 con 0x7fcbec1055f0 2023-11-16T18:29:40.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.457+0000 7fcbf1f1d700 1 -- 172.21.15.2:0/712018808 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe00095f0 con 0x7fcbec1055f0 2023-11-16T18:29:40.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.457+0000 7fcbe8ff9700 1 -- 172.21.15.2:0/712018808 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe400cd00 con 0x7fcbec1055f0 2023-11-16T18:29:40.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.457+0000 7fcbf1f1d700 1 -- 172.21.15.2:0/712018808 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbec1102d0 con 0x7fcbec1055f0 2023-11-16T18:29:40.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.459+0000 7fcbe8ff9700 1 -- 172.21.15.2:0/712018808 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fcbe4003a00 con 0x7fcbec1055f0 2023-11-16T18:29:40.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.460+0000 7fcbe8ff9700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fcbd40a1340 0x7fcbd40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.460+0000 7fcbeb7fe700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fcbd40a1340 0x7fcbd40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.460+0000 7fcbe8ff9700 1 -- 172.21.15.2:0/712018808 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fcbe40ccb90 con 0x7fcbec1055f0 2023-11-16T18:29:40.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.460+0000 7fcbeb7fe700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fcbd40a1340 0x7fcbd40a37f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009dd0 tx=0x7fcbdc009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:40.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.461+0000 7fcbfe0a9b80 1 Processor -- start 2023-11-16T18:29:40.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.461+0000 7fcbfe0a9b80 1 -- start start 2023-11-16T18:29:40.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.461+0000 7fcbfe0a9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560091010b60 0x560091010f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.461+0000 7fcbfe0a9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x560091015370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.461+0000 7fcbfe0a9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600910158b0 0x560091015d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.461+0000 7fcbfe0a9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x560090f16060 con 0x560091011470 2023-11-16T18:29:40.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.461+0000 7fcbfe0a9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560090f688a0 con 0x5600910158b0 2023-11-16T18:29:40.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.461+0000 7fcbfe0a9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560090f61d30 con 0x560091010b60 2023-11-16T18:29:40.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.461+0000 7fcbeaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x560091015370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.461+0000 7fcbebfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600910158b0 0x560091015d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.461+0000 7fcbeaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x560091015370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52540/0 (socket says 172.21.15.2:52540) 2023-11-16T18:29:40.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.461+0000 7fcbeaffd700 1 -- 172.21.15.2:0/1928408291 learned_addr learned my addr 172.21.15.2:0/1928408291 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:40.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.461+0000 7fcbeb7fe700 1 --2- 172.21.15.2:0/1928408291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560091010b60 0x560091010f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.462+0000 7fcbebfff700 1 -- 172.21.15.2:0/1928408291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560091010b60 msgr2=0x560091010f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.462+0000 7fcbebfff700 1 --2- 172.21.15.2:0/1928408291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560091010b60 0x560091010f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.462+0000 7fcbebfff700 1 -- 172.21.15.2:0/1928408291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 msgr2=0x560091015370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.462+0000 7fcbebfff700 1 --2- 172.21.15.2:0/1928408291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x560091015370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.462+0000 7fcbebfff700 1 -- 172.21.15.2:0/1928408291 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560090de3630 con 0x5600910158b0 2023-11-16T18:29:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.462+0000 7fcbebfff700 1 --2- 172.21.15.2:0/1928408291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600910158b0 0x560091015d00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40c9f10 tx=0x7fcbe4099460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37e1b89aa8b76a6d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.462+0000 7fcbd1ffb700 1 -- 172.21.15.2:0/1928408291 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe400ce60 con 0x5600910158b0 2023-11-16T18:29:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.462+0000 7fcbd1ffb700 1 -- 172.21.15.2:0/1928408291 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcbe4099920 con 0x5600910158b0 2023-11-16T18:29:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.462+0000 7fcbd1ffb700 1 -- 172.21.15.2:0/1928408291 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4099de0 con 0x5600910158b0 2023-11-16T18:29:40.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.463+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/1928408291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600910158b0 msgr2=0x560091015d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:40.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.463+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/1928408291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600910158b0 0x560091015d00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40c9f10 tx=0x7fcbe4099460 comp rx=0 tx=0).stop 2023-11-16T18:29:40.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.463+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/1928408291 shutdown_connections 2023-11-16T18:29:40.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.463+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/1928408291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560091010b60 0x560091010f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.463+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/1928408291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600910158b0 0x560091015d00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.463+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/1928408291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x560091015370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.463+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/1928408291 >> 172.21.15.2:0/1928408291 conn(0x560091006360 msgr2=0x56009100bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:40.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.463+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/1928408291 shutdown_connections 2023-11-16T18:29:40.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.463+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/1928408291 wait complete. 2023-11-16T18:29:40.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.463+0000 7fcbfe0a9b80 1 -- start start 2023-11-16T18:29:40.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.463+0000 7fcbfe0a9b80 1 Processor -- start 2023-11-16T18:29:40.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbfe0a9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560091010b60 0x56009101d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbfe0a9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x56009101a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbfe0a9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600910158b0 0x56009101aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbfe0a9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x560090f038e0 con 0x560091011470 2023-11-16T18:29:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbfe0a9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560090f03580 con 0x5600910158b0 2023-11-16T18:29:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbfe0a9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560090de5920 con 0x560091010b60 2023-11-16T18:29:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbfe0a9b80 10 client.0 _ll_register_callbacks cb 0x560090f21780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:29:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbeaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x56009101a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbeb7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560091010b60 0x56009101d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbeb7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560091010b60 0x56009101d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:44074/0 (socket says 172.21.15.2:44074) 2023-11-16T18:29:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbebfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600910158b0 0x56009101aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbeaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x56009101a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52548/0 (socket says 172.21.15.2:52548) 2023-11-16T18:29:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbeb7fe700 1 -- 172.21.15.2:0/1525891430 learned_addr learned my addr 172.21.15.2:0/1525891430 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbeb7fe700 1 -- 172.21.15.2:0/1525891430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600910158b0 msgr2=0x56009101aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbeb7fe700 1 --2- 172.21.15.2:0/1525891430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600910158b0 0x56009101aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbeb7fe700 1 -- 172.21.15.2:0/1525891430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 msgr2=0x56009101a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbeb7fe700 1 --2- 172.21.15.2:0/1525891430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x56009101a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbeb7fe700 1 -- 172.21.15.2:0/1525891430 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbec1102d0 con 0x560091010b60 2023-11-16T18:29:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbebfff700 1 --2- 172.21.15.2:0/1525891430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600910158b0 0x56009101aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:29:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbeaffd700 1 --2- 172.21.15.2:0/1525891430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x56009101a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:29:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbeb7fe700 1 --2- 172.21.15.2:0/1525891430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560091010b60 0x56009101d070 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00edf0 tx=0x7fcbdc0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fada19781e17d9c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.464+0000 7fcbcaffd700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:29:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbcaffd700 1 -- 172.21.15.2:0/1525891430 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc022070 con 0x560091010b60 2023-11-16T18:29:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbcaffd700 1 -- 172.21.15.2:0/1525891430 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fcbdc00ee20 con 0x560091010b60 2023-11-16T18:29:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/1525891430 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560091003b20 con 0x560091010b60 2023-11-16T18:29:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbcaffd700 1 -- 172.21.15.2:0/1525891430 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbdc01dd10 con 0x560091010b60 2023-11-16T18:29:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbfe0a9b80 10 client.26005 fetch_fsmap learned FSMap version 29 2023-11-16T18:29:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/1525891430 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x560090f28190 con 0x560091010b60 2023-11-16T18:29:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbcaffd700 1 -- 172.21.15.2:0/1525891430 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560091003b20 con 0x560091010b60 2023-11-16T18:29:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbcaffd700 1 -- 172.21.15.2:0/1525891430 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fcbdc01c400 con 0x560091010b60 2023-11-16T18:29:40.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbfe0a9b80 10 client.26005 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:29:40.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbfe0a9b80 10 client.26005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:29:40.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbfe0a9b80 20 client.26005 populate_metadata read hostname 'smithi002' 2023-11-16T18:29:40.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/1525891430 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56009102a820 0x56009102cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbfe0a9b80 4 client.26005 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:29:40.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.465+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/1525891430 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560090f240a0 con 0x56009102a820 2023-11-16T18:29:40.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.466+0000 7fcbeaffd700 1 --2- 172.21.15.2:0/1525891430 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56009102a820 0x56009102cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.466+0000 7fcbeaffd700 1 --2- 172.21.15.2:0/1525891430 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56009102a820 0x56009102cc30 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a7b4a905ecd34bd server_cookie=d7bec08d79af7f31 in_seq=0 out_seq=0 2023-11-16T18:29:40.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.466+0000 7fcbcaffd700 10 client.26005 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:40.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbcaffd700 1 -- 172.21.15.2:0/1525891430 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560090f240a0 con 0x56009102a820 2023-11-16T18:29:40.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbcaffd700 10 client.26005 handle_command_reply: tid=0 2023-11-16T18:29:40.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbfe0a9b80 1 client.26005 shutdown 2023-11-16T18:29:40.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/1525891430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560091010b60 msgr2=0x56009101d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:40.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/1525891430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560091010b60 0x56009101d070 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00edf0 tx=0x7fcbdc0047c0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/1525891430 shutdown_connections 2023-11-16T18:29:40.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/1525891430 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56009102a820 0x56009102cc30 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/1525891430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560091010b60 0x56009101d070 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbcaffd700 0 client.26005 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:40.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/1525891430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600910158b0 0x56009101aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/1525891430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x56009101a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/1525891430 >> 172.21.15.2:0/1525891430 conn(0x560091006360 msgr2=0x560091016d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:40.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/1525891430 shutdown_connections 2023-11-16T18:29:40.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/1525891430 wait complete. 2023-11-16T18:29:40.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.477+0000 7fcbfe0a9b80 20 client.26005 trim_cache size 0 max 16384 2023-11-16T18:29:40.497 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.494+0000 7fcbfe0a9b80 1 Processor -- start 2023-11-16T18:29:40.497 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.494+0000 7fcbfe0a9b80 1 -- start start 2023-11-16T18:29:40.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.494+0000 7fcbfe0a9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560091010b60 0x56009101cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.494+0000 7fcbfe0a9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x56009100b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.494+0000 7fcbfe0a9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600910158b0 0x56009100baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.494+0000 7fcbfe0a9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56009100c7d0 con 0x560091011470 2023-11-16T18:29:40.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.494+0000 7fcbfe0a9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560091005ee0 con 0x560091010b60 2023-11-16T18:29:40.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.494+0000 7fcbfe0a9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560091011900 con 0x5600910158b0 2023-11-16T18:29:40.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.494+0000 7fcbeaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x56009100b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.494+0000 7fcbeaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x56009100b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52554/0 (socket says 172.21.15.2:52554) 2023-11-16T18:29:40.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.494+0000 7fcbeb7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560091010b60 0x56009101cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.494+0000 7fcbeaffd700 1 -- 172.21.15.2:0/45774653 learned_addr learned my addr 172.21.15.2:0/45774653 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:40.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.495+0000 7fcbebfff700 1 --2- 172.21.15.2:0/45774653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600910158b0 0x56009100baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.495+0000 7fcbeaffd700 1 -- 172.21.15.2:0/45774653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600910158b0 msgr2=0x56009100baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:40.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.495+0000 7fcbeaffd700 1 --2- 172.21.15.2:0/45774653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600910158b0 0x56009100baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.495+0000 7fcbeaffd700 1 -- 172.21.15.2:0/45774653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560091010b60 msgr2=0x56009101cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:40.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.495+0000 7fcbeaffd700 1 --2- 172.21.15.2:0/45774653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560091010b60 0x56009101cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.495+0000 7fcbeaffd700 1 -- 172.21.15.2:0/45774653 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600910186d0 con 0x560091011470 2023-11-16T18:29:40.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.495+0000 7fcbeb7fe700 1 --2- 172.21.15.2:0/45774653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560091010b60 0x56009101cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:29:40.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.495+0000 7fcbeaffd700 1 --2- 172.21.15.2:0/45774653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x56009100b560 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00099c0 tx=0x7fcbe0005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbbb87d1fab92a95 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:40.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.495+0000 7fcbd27fc700 1 -- 172.21.15.2:0/45774653 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe002be10 con 0x560091011470 2023-11-16T18:29:40.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.495+0000 7fcbd27fc700 1 -- 172.21.15.2:0/45774653 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcbe001e040 con 0x560091011470 2023-11-16T18:29:40.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.495+0000 7fcbd27fc700 1 -- 172.21.15.2:0/45774653 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe000fd80 con 0x560091011470 2023-11-16T18:29:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.496+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/45774653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 msgr2=0x56009100b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.496+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/45774653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x56009100b560 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00099c0 tx=0x7fcbe0005fb0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.496+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/45774653 shutdown_connections 2023-11-16T18:29:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.496+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/45774653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600910158b0 0x56009100baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.496+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/45774653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560091010b60 0x56009101cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.496+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/45774653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091011470 0x56009100b560 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.496+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/45774653 >> 172.21.15.2:0/45774653 conn(0x560091006360 msgr2=0x560091013970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.496+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/45774653 shutdown_connections 2023-11-16T18:29:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.496+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/45774653 wait complete. 2023-11-16T18:29:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.496+0000 7fcbfe0a9b80 1 -- start start 2023-11-16T18:29:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbfe0a9b80 1 Processor -- start 2023-11-16T18:29:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbfe0a9b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091010b60 0x5600910212a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbfe0a9b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560091011470 0x5600910217e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbfe0a9b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600910158b0 0x56009101f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbfe0a9b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56009101fe40 con 0x560091010b60 2023-11-16T18:29:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbfe0a9b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56009101ff80 con 0x560091011470 2023-11-16T18:29:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbfe0a9b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560091020140 con 0x5600910158b0 2023-11-16T18:29:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbfe0a9b80 10 client.0 _ll_register_callbacks cb 0x560090f21780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:29:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbeb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091010b60 0x5600910212a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbeaffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560091011470 0x5600910217e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbeb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091010b60 0x5600910212a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52558/0 (socket says 172.21.15.2:52558) 2023-11-16T18:29:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbeb7fe700 1 -- 172.21.15.2:0/4020900336 learned_addr learned my addr 172.21.15.2:0/4020900336 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbebfff700 1 --2- 172.21.15.2:0/4020900336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600910158b0 0x56009101f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbeaffd700 1 -- 172.21.15.2:0/4020900336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600910158b0 msgr2=0x56009101f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:40.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbeaffd700 1 --2- 172.21.15.2:0/4020900336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600910158b0 0x56009101f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbeaffd700 1 -- 172.21.15.2:0/4020900336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091010b60 msgr2=0x5600910212a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:40.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbeaffd700 1 --2- 172.21.15.2:0/4020900336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091010b60 0x5600910212a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.497+0000 7fcbeaffd700 1 -- 172.21.15.2:0/4020900336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560090f28190 con 0x560091011470 2023-11-16T18:29:40.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbeaffd700 1 --2- 172.21.15.2:0/4020900336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560091011470 0x5600910217e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x5600910146a0 tx=0x7fcbe0009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee558438a764e870 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:40.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbcaffd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:29:40.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbcaffd700 1 -- 172.21.15.2:0/4020900336 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe000fe50 con 0x560091011470 2023-11-16T18:29:40.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbcaffd700 1 -- 172.21.15.2:0/4020900336 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fcbe002b100 con 0x560091011470 2023-11-16T18:29:40.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/4020900336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbe4099920 con 0x560091011470 2023-11-16T18:29:40.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbcaffd700 1 -- 172.21.15.2:0/4020900336 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbe002b2c0 con 0x560091011470 2023-11-16T18:29:40.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbfe0a9b80 10 client.26129 fetch_fsmap learned FSMap version 29 2023-11-16T18:29:40.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/4020900336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7fcbec10d960 con 0x560091011470 2023-11-16T18:29:40.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbcaffd700 1 -- 172.21.15.2:0/4020900336 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcbe4099920 con 0x560091011470 2023-11-16T18:29:40.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbcaffd700 1 -- 172.21.15.2:0/4020900336 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fcbe00099f0 con 0x560091011470 2023-11-16T18:29:40.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbfe0a9b80 10 client.26129 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:29:40.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbfe0a9b80 10 client.26129 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:29:40.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbfe0a9b80 20 client.26129 populate_metadata read hostname 'smithi002' 2023-11-16T18:29:40.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/4020900336 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56009107f3e0 0x5600910817c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:40.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbfe0a9b80 4 client.26129 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:29:40.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.498+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/4020900336 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560090f30660 con 0x56009107f3e0 2023-11-16T18:29:40.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.499+0000 7fcbeb7fe700 1 --2- 172.21.15.2:0/4020900336 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56009107f3e0 0x5600910817c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:40.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.499+0000 7fcbcaffd700 10 client.26129 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:40.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.499+0000 7fcbeb7fe700 1 --2- 172.21.15.2:0/4020900336 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56009107f3e0 0x5600910817c0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2c3edc2cf4ff5de server_cookie=45d1d0a88c6dd86c in_seq=0 out_seq=0 2023-11-16T18:29:40.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.509+0000 7fcbcaffd700 1 -- 172.21.15.2:0/4020900336 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x560090f30660 con 0x56009107f3e0 2023-11-16T18:29:40.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.509+0000 7fcbcaffd700 10 client.26129 handle_command_reply: tid=0 2023-11-16T18:29:40.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.509+0000 7fcbfe0a9b80 1 client.26129 shutdown 2023-11-16T18:29:40.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/4020900336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560091011470 msgr2=0x5600910217e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:40.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/4020900336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560091011470 0x5600910217e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x5600910146a0 tx=0x7fcbe0009cd0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/4020900336 shutdown_connections 2023-11-16T18:29:40.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/4020900336 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x56009107f3e0 0x5600910817c0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/4020900336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600910158b0 0x56009101f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/4020900336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x560091010b60 0x5600910212a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbfe0a9b80 1 --2- 172.21.15.2:0/4020900336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560091011470 0x5600910217e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbcaffd700 0 client.26129 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:40.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/4020900336 >> 172.21.15.2:0/4020900336 conn(0x560091006360 msgr2=0x5600910148c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:40.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/4020900336 shutdown_connections 2023-11-16T18:29:40.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbfe0a9b80 1 -- 172.21.15.2:0/4020900336 wait complete. 2023-11-16T18:29:40.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbfe0a9b80 20 client.26129 trim_cache size 0 max 16384 2023-11-16T18:29:40.514 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:29:40.514 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "scrub active (1 inodes in the stack)", 2023-11-16T18:29:40.514 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": { 2023-11-16T18:29:40.514 INFO:teuthology.orchestra.run.smithi002.stdout: "9607aa23-76ae-4198-9378-3740d8aff22c": { 2023-11-16T18:29:40.514 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/", 2023-11-16T18:29:40.514 INFO:teuthology.orchestra.run.smithi002.stdout: "tag": "9607aa23-76ae-4198-9378-3740d8aff22c", 2023-11-16T18:29:40.514 INFO:teuthology.orchestra.run.smithi002.stdout: "options": "recursive,force" 2023-11-16T18:29:40.514 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:29:40.514 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:29:40.514 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:29:40.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbf1f1d700 1 -- 172.21.15.2:0/712018808 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fcbd40a1340 msgr2=0x7fcbd40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:40.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbf1f1d700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fcbd40a1340 0x7fcbd40a37f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009dd0 tx=0x7fcbdc009260 comp rx=0 tx=0).stop 2023-11-16T18:29:40.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbf1f1d700 1 -- 172.21.15.2:0/712018808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbec1055f0 msgr2=0x7fcbec19ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:40.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbf1f1d700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbec1055f0 0x7fcbec19ac90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400ef30 tx=0x7fcbe400c3a0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbf1f1d700 1 -- 172.21.15.2:0/712018808 shutdown_connections 2023-11-16T18:29:40.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbf1f1d700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fcbd40a1340 0x7fcbd40a37f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbf1f1d700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbec0fea50 0x7fcbec1962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbf1f1d700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbec1055f0 0x7fcbec19ac90 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbf1f1d700 1 --2- 172.21.15.2:0/712018808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbec0ff3f0 0x7fcbec196830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:40.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbf1f1d700 1 -- 172.21.15.2:0/712018808 >> 172.21.15.2:0/712018808 conn(0x7fcbec075880 msgr2=0x7fcbec0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:40.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.510+0000 7fcbf1f1d700 1 -- 172.21.15.2:0/712018808 shutdown_connections 2023-11-16T18:29:40.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:29:40.511+0000 7fcbf1f1d700 1 -- 172.21.15.2:0/712018808 wait complete. 2023-11-16T18:29:40.525 INFO:tasks.cephfs.filesystem:scrub status for tag:9607aa23-76ae-4198-9378-3740d8aff22c - {'path': '/', 'tag': '9607aa23-76ae-4198-9378-3740d8aff22c', 'options': 'recursive,force'} 2023-11-16T18:29:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:42 smithi040 ceph-mon[100251]: pgmap v148: 129 pgs: 129 active+clean; 588 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 687 op/s 2023-11-16T18:29:42.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:42 smithi188 ceph-mon[100639]: pgmap v148: 129 pgs: 129 active+clean; 588 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 687 op/s 2023-11-16T18:29:42.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:42 smithi002 ceph-mon[98807]: pgmap v148: 129 pgs: 129 active+clean; 588 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 687 op/s 2023-11-16T18:29:44.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:44 smithi040 ceph-mon[100251]: pgmap v149: 129 pgs: 129 active+clean; 601 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.0 MiB/s wr, 924 op/s 2023-11-16T18:29:44.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:44 smithi188 ceph-mon[100639]: pgmap v149: 129 pgs: 129 active+clean; 601 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.0 MiB/s wr, 924 op/s 2023-11-16T18:29:44.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:44 smithi002 ceph-mon[98807]: pgmap v149: 129 pgs: 129 active+clean; 601 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.0 MiB/s wr, 924 op/s 2023-11-16T18:29:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.196+0000 7fa00d593700 1 -- 172.21.15.2:0/2342678366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa00810cc20 msgr2=0x7fa00810f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.196+0000 7fa00d593700 1 --2- 172.21.15.2:0/2342678366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa00810cc20 0x7fa00810f000 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fa00000b0a0 tx=0x7fa00000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.196+0000 7fa00d593700 1 -- 172.21.15.2:0/2342678366 shutdown_connections 2023-11-16T18:29:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.196+0000 7fa00d593700 1 --2- 172.21.15.2:0/2342678366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0080ff070 0x7fa0080ff440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.196+0000 7fa00d593700 1 --2- 172.21.15.2:0/2342678366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa00810cc20 0x7fa00810f000 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.196+0000 7fa00d593700 1 --2- 172.21.15.2:0/2342678366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0080ff980 0x7fa00810c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.196+0000 7fa00d593700 1 -- 172.21.15.2:0/2342678366 >> 172.21.15.2:0/2342678366 conn(0x7fa008076370 msgr2=0x7fa008076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.196+0000 7fa00d593700 1 -- 172.21.15.2:0/2342678366 shutdown_connections 2023-11-16T18:29:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.196+0000 7fa00d593700 1 -- 172.21.15.2:0/2342678366 wait complete. 2023-11-16T18:29:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.196+0000 7fa00d593700 1 Processor -- start 2023-11-16T18:29:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa00d593700 1 -- start start 2023-11-16T18:29:46.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa00d593700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0080ff070 0x7fa008198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa00d593700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0080ff980 0x7fa008198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa00d593700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa00810cc20 0x7fa00819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa00d593700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00819d630 con 0x7fa0080ff070 2023-11-16T18:29:46.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa00d593700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00819d7a0 con 0x7fa0080ff980 2023-11-16T18:29:46.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa00d593700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00819d960 con 0x7fa00810cc20 2023-11-16T18:29:46.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa006ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0080ff070 0x7fa008198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa0077fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa00810cc20 0x7fa00819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa0067fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0080ff980 0x7fa008198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa006ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0080ff070 0x7fa008198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52572/0 (socket says 172.21.15.2:52572) 2023-11-16T18:29:46.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa006ffd700 1 -- 172.21.15.2:0/3737337379 learned_addr learned my addr 172.21.15.2:0/3737337379 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:46.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa0077fe700 1 -- 172.21.15.2:0/3737337379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0080ff980 msgr2=0x7fa008198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:46.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa0077fe700 1 --2- 172.21.15.2:0/3737337379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0080ff980 0x7fa008198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa0077fe700 1 -- 172.21.15.2:0/3737337379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0080ff070 msgr2=0x7fa008198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:46.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa0077fe700 1 --2- 172.21.15.2:0/3737337379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0080ff070 0x7fa008198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa0077fe700 1 -- 172.21.15.2:0/3737337379 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa000009d00 con 0x7fa00810cc20 2023-11-16T18:29:46.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa006ffd700 1 --2- 172.21.15.2:0/3737337379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0080ff070 0x7fa008198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:29:46.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.197+0000 7fa0077fe700 1 --2- 172.21.15.2:0/3737337379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa00810cc20 0x7fa00819cff0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa00000b070 tx=0x7fa0000078e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:46.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.198+0000 7f9feffff700 1 -- 172.21.15.2:0/3737337379 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00000e040 con 0x7fa00810cc20 2023-11-16T18:29:46.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.198+0000 7f9feffff700 1 -- 172.21.15.2:0/3737337379 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa00001d470 con 0x7fa00810cc20 2023-11-16T18:29:46.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.198+0000 7fa00d593700 1 -- 172.21.15.2:0/3737337379 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0081124f0 con 0x7fa00810cc20 2023-11-16T18:29:46.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.198+0000 7f9feffff700 1 -- 172.21.15.2:0/3737337379 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0000052a0 con 0x7fa00810cc20 2023-11-16T18:29:46.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.198+0000 7fa00d593700 1 -- 172.21.15.2:0/3737337379 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa008112360 con 0x7fa00810cc20 2023-11-16T18:29:46.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.201+0000 7f9feffff700 1 -- 172.21.15.2:0/3737337379 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fa00001d890 con 0x7fa00810cc20 2023-11-16T18:29:46.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.201+0000 7f9feffff700 1 --2- 172.21.15.2:0/3737337379 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f9ff00a12c0 0x7f9ff00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.201+0000 7fa006ffd700 1 --2- 172.21.15.2:0/3737337379 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f9ff00a12c0 0x7f9ff00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.201+0000 7f9feffff700 1 -- 172.21.15.2:0/3737337379 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa0000ce310 con 0x7fa00810cc20 2023-11-16T18:29:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa01971fb80 1 Processor -- start 2023-11-16T18:29:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa006ffd700 1 --2- 172.21.15.2:0/3737337379 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f9ff00a12c0 0x7f9ff00a3770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8011550 tx=0x7f9ff8009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa01971fb80 1 -- start start 2023-11-16T18:29:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa01971fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c341e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa01971fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c2e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa01971fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 0x557e63c2eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa01971fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e63b34000 con 0x557e63a8a6b0 2023-11-16T18:29:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa01971fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e63b86840 con 0x557e63c34720 2023-11-16T18:29:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa01971fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e63b7fcd0 con 0x557e63c34b90 2023-11-16T18:29:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa006ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa0067fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c2e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa0067fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c2e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43020/0 (socket says 172.21.15.2:43020) 2023-11-16T18:29:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa006ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c341e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52580/0 (socket says 172.21.15.2:52580) 2023-11-16T18:29:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa0067fc700 1 -- 172.21.15.2:0/3987362879 learned_addr learned my addr 172.21.15.2:0/3987362879 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.202+0000 7fa0077fe700 1 --2- 172.21.15.2:0/3987362879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 0x557e63c2eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.203+0000 7fa0067fc700 1 -- 172.21.15.2:0/3987362879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 msgr2=0x557e63c2eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.203+0000 7fa0067fc700 1 --2- 172.21.15.2:0/3987362879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 0x557e63c2eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.203+0000 7fa0067fc700 1 -- 172.21.15.2:0/3987362879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 msgr2=0x557e63c341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.203+0000 7fa0067fc700 1 --2- 172.21.15.2:0/3987362879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c341e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.203+0000 7fa0067fc700 1 -- 172.21.15.2:0/3987362879 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e63a015d0 con 0x557e63c34720 2023-11-16T18:29:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.203+0000 7fa006ffd700 1 --2- 172.21.15.2:0/3987362879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:29:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.203+0000 7fa0067fc700 1 --2- 172.21.15.2:0/3987362879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c2e560 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009ca0 tx=0x7f9ffc011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33ad23df096833c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.203+0000 7f9fed7fa700 1 -- 172.21.15.2:0/3987362879 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc011d80 con 0x557e63c34720 2023-11-16T18:29:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.203+0000 7f9fed7fa700 1 -- 172.21.15.2:0/3987362879 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ffc004cf0 con 0x557e63c34720 2023-11-16T18:29:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.203+0000 7f9fed7fa700 1 -- 172.21.15.2:0/3987362879 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc019690 con 0x557e63c34720 2023-11-16T18:29:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 -- 172.21.15.2:0/3987362879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 msgr2=0x557e63c2e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 --2- 172.21.15.2:0/3987362879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c2e560 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009ca0 tx=0x7f9ffc011440 comp rx=0 tx=0).stop 2023-11-16T18:29:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 -- 172.21.15.2:0/3987362879 shutdown_connections 2023-11-16T18:29:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 --2- 172.21.15.2:0/3987362879 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 0x557e63c2eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 --2- 172.21.15.2:0/3987362879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 --2- 172.21.15.2:0/3987362879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c2e560 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 -- 172.21.15.2:0/3987362879 >> 172.21.15.2:0/3987362879 conn(0x557e63c23f50 msgr2=0x557e63c29270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 -- 172.21.15.2:0/3987362879 shutdown_connections 2023-11-16T18:29:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 -- 172.21.15.2:0/3987362879 wait complete. 2023-11-16T18:29:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 -- start start 2023-11-16T18:29:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 Processor -- start 2023-11-16T18:29:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c3ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c37d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 0x557e63c38460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.204+0000 7fa01971fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e63b21880 con 0x557e63a8a6b0 2023-11-16T18:29:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa01971fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e63b21520 con 0x557e63c34720 2023-11-16T18:29:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa01971fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e63a038c0 con 0x557e63c34b90 2023-11-16T18:29:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa01971fb80 10 client.0 _ll_register_callbacks cb 0x557e63b3f720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:29:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa006ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c3ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa0067fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c37d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa006ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c3ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52586/0 (socket says 172.21.15.2:52586) 2023-11-16T18:29:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa006ffd700 1 -- 172.21.15.2:0/1778216164 learned_addr learned my addr 172.21.15.2:0/1778216164 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa0077fe700 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 0x557e63c38460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa0067fc700 1 -- 172.21.15.2:0/1778216164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 msgr2=0x557e63c38460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa0067fc700 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 0x557e63c38460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa0067fc700 1 -- 172.21.15.2:0/1778216164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 msgr2=0x557e63c3ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa0067fc700 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c3ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa0067fc700 1 -- 172.21.15.2:0/1778216164 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ffc009900 con 0x557e63c34720 2023-11-16T18:29:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa0077fe700 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 0x557e63c38460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:29:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa006ffd700 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c3ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:29:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa0067fc700 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c37d20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc000c00 tx=0x7f9ffc005090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96814420a73cfa77 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7f9feaff5700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:29:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7f9feaff5700 1 -- 172.21.15.2:0/1778216164 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc011e50 con 0x557e63c34720 2023-11-16T18:29:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7f9feaff5700 1 -- 172.21.15.2:0/1778216164 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f9ffc004750 con 0x557e63c34720 2023-11-16T18:29:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.205+0000 7fa01971fb80 1 -- 172.21.15.2:0/1778216164 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557e63c21710 con 0x557e63c34720 2023-11-16T18:29:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.206+0000 7f9feaff5700 1 -- 172.21.15.2:0/1778216164 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ffc004930 con 0x557e63c34720 2023-11-16T18:29:46.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.206+0000 7fa01971fb80 10 client.26141 fetch_fsmap learned FSMap version 29 2023-11-16T18:29:46.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.206+0000 7fa01971fb80 1 -- 172.21.15.2:0/1778216164 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557e63b46130 con 0x557e63c34720 2023-11-16T18:29:46.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.206+0000 7f9feaff5700 1 -- 172.21.15.2:0/1778216164 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557e63c21710 con 0x557e63c34720 2023-11-16T18:29:46.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.206+0000 7f9feaff5700 1 -- 172.21.15.2:0/1778216164 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f9ffc021040 con 0x557e63c34720 2023-11-16T18:29:46.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.206+0000 7fa01971fb80 10 client.26141 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:29:46.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.206+0000 7fa01971fb80 10 client.26141 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:29:46.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.206+0000 7fa01971fb80 20 client.26141 populate_metadata read hostname 'smithi002' 2023-11-16T18:29:46.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.206+0000 7fa01971fb80 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e63c47ed0 0x557e63c4a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.206+0000 7fa01971fb80 4 client.26141 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:29:46.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.206+0000 7fa01971fb80 1 -- 172.21.15.2:0/1778216164 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557e63b42040 con 0x557e63c47ed0 2023-11-16T18:29:46.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.206+0000 7fa006ffd700 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e63c47ed0 0x557e63c4a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.207+0000 7f9feaff5700 10 client.26141 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:46.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.207+0000 7fa006ffd700 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e63c47ed0 0x557e63c4a2e0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29da4bfecdfde14b server_cookie=b5af7b41d0cf2ea2 in_seq=0 out_seq=0 2023-11-16T18:29:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.207+0000 7f9feaff5700 1 -- 172.21.15.2:0/1778216164 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557e63b42040 con 0x557e63c47ed0 2023-11-16T18:29:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.207+0000 7f9feaff5700 10 client.26141 handle_command_reply: tid=0 2023-11-16T18:29:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.207+0000 7fa01971fb80 1 client.26141 shutdown 2023-11-16T18:29:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.208+0000 7fa01971fb80 1 -- 172.21.15.2:0/1778216164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 msgr2=0x557e63c37d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.208+0000 7fa01971fb80 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c37d20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc000c00 tx=0x7f9ffc005090 comp rx=0 tx=0).stop 2023-11-16T18:29:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.208+0000 7fa01971fb80 1 -- 172.21.15.2:0/1778216164 shutdown_connections 2023-11-16T18:29:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.208+0000 7fa01971fb80 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e63c47ed0 0x557e63c4a2e0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.208+0000 7fa01971fb80 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 0x557e63c38460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.208+0000 7fa01971fb80 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c3ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.208+0000 7fa01971fb80 1 --2- 172.21.15.2:0/1778216164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c37d20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.208+0000 7f9feaff5700 0 client.26141 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.208+0000 7fa01971fb80 1 -- 172.21.15.2:0/1778216164 >> 172.21.15.2:0/1778216164 conn(0x557e63c23f50 msgr2=0x557e63c287e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.208+0000 7fa01971fb80 1 -- 172.21.15.2:0/1778216164 shutdown_connections 2023-11-16T18:29:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.208+0000 7fa01971fb80 1 -- 172.21.15.2:0/1778216164 wait complete. 2023-11-16T18:29:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.208+0000 7fa01971fb80 20 client.26141 trim_cache size 0 max 16384 2023-11-16T18:29:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.224+0000 7fa01971fb80 1 Processor -- start 2023-11-16T18:29:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa01971fb80 1 -- start start 2023-11-16T18:29:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa01971fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c2e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa01971fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c442d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa01971fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 0x557e63c28f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa01971fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e63c2ef90 con 0x557e63a8a6b0 2023-11-16T18:29:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa01971fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e63c391b0 con 0x557e63c34720 2023-11-16T18:29:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa01971fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e63c21570 con 0x557e63c34b90 2023-11-16T18:29:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa006ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c2e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa006ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c2e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52594/0 (socket says 172.21.15.2:52594) 2023-11-16T18:29:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa006ffd700 1 -- 172.21.15.2:0/2046898822 learned_addr learned my addr 172.21.15.2:0/2046898822 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa0067fc700 1 --2- 172.21.15.2:0/2046898822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa0077fe700 1 --2- 172.21.15.2:0/2046898822 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 0x557e63c28f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa006ffd700 1 -- 172.21.15.2:0/2046898822 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 msgr2=0x557e63c28f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa006ffd700 1 --2- 172.21.15.2:0/2046898822 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 0x557e63c28f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa006ffd700 1 -- 172.21.15.2:0/2046898822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 msgr2=0x557e63c442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa006ffd700 1 --2- 172.21.15.2:0/2046898822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c442d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa006ffd700 1 -- 172.21.15.2:0/2046898822 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e63b45e80 con 0x557e63a8a6b0 2023-11-16T18:29:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.225+0000 7fa006ffd700 1 --2- 172.21.15.2:0/2046898822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c2e560 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8013010 tx=0x7f9ff8013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a69f0fdd8f306c3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.226+0000 7f9fedffb700 1 -- 172.21.15.2:0/2046898822 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8022040 con 0x557e63a8a6b0 2023-11-16T18:29:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.226+0000 7f9fedffb700 1 -- 172.21.15.2:0/2046898822 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ff8023040 con 0x557e63a8a6b0 2023-11-16T18:29:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.226+0000 7f9fedffb700 1 -- 172.21.15.2:0/2046898822 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff80293f0 con 0x557e63a8a6b0 2023-11-16T18:29:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.227+0000 7fa01971fb80 1 -- 172.21.15.2:0/2046898822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 msgr2=0x557e63c2e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.227+0000 7fa01971fb80 1 --2- 172.21.15.2:0/2046898822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c2e560 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8013010 tx=0x7f9ff8013040 comp rx=0 tx=0).stop 2023-11-16T18:29:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.227+0000 7fa01971fb80 1 -- 172.21.15.2:0/2046898822 shutdown_connections 2023-11-16T18:29:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.227+0000 7fa01971fb80 1 --2- 172.21.15.2:0/2046898822 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63c34b90 0x557e63c28f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.227+0000 7fa01971fb80 1 --2- 172.21.15.2:0/2046898822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63a8a6b0 0x557e63c2e560 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.227+0000 7fa01971fb80 1 --2- 172.21.15.2:0/2046898822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.227+0000 7fa01971fb80 1 -- 172.21.15.2:0/2046898822 >> 172.21.15.2:0/2046898822 conn(0x557e63c23f50 msgr2=0x557e63c2aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:46.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.227+0000 7fa01971fb80 1 -- 172.21.15.2:0/2046898822 shutdown_connections 2023-11-16T18:29:46.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.227+0000 7fa01971fb80 1 -- 172.21.15.2:0/2046898822 wait complete. 2023-11-16T18:29:46.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.227+0000 7fa01971fb80 1 -- start start 2023-11-16T18:29:46.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.227+0000 7fa01971fb80 1 Processor -- start 2023-11-16T18:29:46.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa01971fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63a8a6b0 0x557e63c3b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa01971fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c3bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa01971fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63c34b90 0x557e63c3c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa01971fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e63c9ae20 con 0x557e63c34b90 2023-11-16T18:29:46.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa01971fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e63c9af90 con 0x557e63c34720 2023-11-16T18:29:46.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa01971fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e63c9b150 con 0x557e63a8a6b0 2023-11-16T18:29:46.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa01971fb80 10 client.0 _ll_register_callbacks cb 0x7f9ff8009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:29:46.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa0077fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63c34b90 0x557e63c3c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa0077fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63c34b90 0x557e63c3c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52604/0 (socket says 172.21.15.2:52604) 2023-11-16T18:29:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa0077fe700 1 -- 172.21.15.2:0/686055446 learned_addr learned my addr 172.21.15.2:0/686055446 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:29:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa0067fc700 1 --2- 172.21.15.2:0/686055446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c3bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa0077fe700 1 -- 172.21.15.2:0/686055446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63a8a6b0 msgr2=0x557e63c3b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa0077fe700 1 --2- 172.21.15.2:0/686055446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63a8a6b0 0x557e63c3b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa0077fe700 1 -- 172.21.15.2:0/686055446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 msgr2=0x557e63c3bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:29:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa0077fe700 1 --2- 172.21.15.2:0/686055446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c3bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa0077fe700 1 -- 172.21.15.2:0/686055446 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff800ed40 con 0x557e63c34b90 2023-11-16T18:29:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa0077fe700 1 --2- 172.21.15.2:0/686055446 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63c34b90 0x557e63c3c0c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa0000cbcf0 tx=0x7fa00009ab30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5e09728807c41a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:29:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7f9feaff5700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:29:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7f9feaff5700 1 -- 172.21.15.2:0/686055446 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00009b560 con 0x557e63c34b90 2023-11-16T18:29:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7f9feaff5700 1 -- 172.21.15.2:0/686055446 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa0000cbe40 con 0x557e63c34b90 2023-11-16T18:29:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.228+0000 7fa01971fb80 1 -- 172.21.15.2:0/686055446 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9ffc004cf0 con 0x557e63c34b90 2023-11-16T18:29:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7f9feaff5700 1 -- 172.21.15.2:0/686055446 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0000a34c0 con 0x557e63c34b90 2023-11-16T18:29:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7fa01971fb80 10 client.16620 fetch_fsmap learned FSMap version 29 2023-11-16T18:29:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7fa01971fb80 1 -- 172.21.15.2:0/686055446 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557e63996c00 con 0x557e63c34b90 2023-11-16T18:29:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7f9feaff5700 1 -- 172.21.15.2:0/686055446 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ffc004cf0 con 0x557e63c34b90 2023-11-16T18:29:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7f9feaff5700 1 -- 172.21.15.2:0/686055446 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa0000a3030 con 0x557e63c34b90 2023-11-16T18:29:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7fa01971fb80 10 client.16620 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:29:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7fa01971fb80 10 client.16620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:29:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7fa01971fb80 20 client.16620 populate_metadata read hostname 'smithi002' 2023-11-16T18:29:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7fa01971fb80 1 --2- 172.21.15.2:0/686055446 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e63c9c6c0 0x557e63c9ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:29:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7fa01971fb80 4 client.16620 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:29:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7fa01971fb80 1 -- 172.21.15.2:0/686055446 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557e63b4e600 con 0x557e63c9c6c0 2023-11-16T18:29:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7fa006ffd700 1 --2- 172.21.15.2:0/686055446 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e63c9c6c0 0x557e63c9ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:29:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7f9feaff5700 10 client.16620 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:46.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:46.229+0000 7fa006ffd700 1 --2- 172.21.15.2:0/686055446 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e63c9c6c0 0x557e63c9ead0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d0a7b49c220ee05 server_cookie=50e4d902fd349d71 in_seq=0 out_seq=0 2023-11-16T18:29:46.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:46 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:29:46.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:46 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:46.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:46 smithi040 ceph-mon[100251]: pgmap v150: 129 pgs: 129 active+clean; 601 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 508 op/s 2023-11-16T18:29:46.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:46 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:29:46.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:46 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:46.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:46 smithi188 ceph-mon[100639]: pgmap v150: 129 pgs: 129 active+clean; 601 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 508 op/s 2023-11-16T18:29:46.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:46 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:29:46.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:46 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:46.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:46 smithi002 ceph-mon[98807]: pgmap v150: 129 pgs: 129 active+clean; 601 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 508 op/s 2023-11-16T18:29:47.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7f9feaff5700 1 -- 172.21.15.2:0/686055446 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557e63b4e600 con 0x557e63c9c6c0 2023-11-16T18:29:47.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7f9feaff5700 10 client.16620 handle_command_reply: tid=0 2023-11-16T18:29:47.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7fa01971fb80 1 client.16620 shutdown 2023-11-16T18:29:47.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7fa01971fb80 1 -- 172.21.15.2:0/686055446 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63c34b90 msgr2=0x557e63c3c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:47.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7fa01971fb80 1 --2- 172.21.15.2:0/686055446 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63c34b90 0x557e63c3c0c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa0000cbcf0 tx=0x7fa00009ab30 comp rx=0 tx=0).stop 2023-11-16T18:29:47.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7fa01971fb80 1 -- 172.21.15.2:0/686055446 shutdown_connections 2023-11-16T18:29:47.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7fa01971fb80 1 --2- 172.21.15.2:0/686055446 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557e63c9c6c0 0x557e63c9ead0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:47.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7fa01971fb80 1 --2- 172.21.15.2:0/686055446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557e63a8a6b0 0x557e63c3b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:47.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7fa01971fb80 1 --2- 172.21.15.2:0/686055446 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557e63c34b90 0x557e63c3c0c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:47.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7fa01971fb80 1 --2- 172.21.15.2:0/686055446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e63c34720 0x557e63c3bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:47.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7fa01971fb80 1 -- 172.21.15.2:0/686055446 >> 172.21.15.2:0/686055446 conn(0x557e63c23f50 msgr2=0x557e63c375c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:47.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7f9feaff5700 0 client.16620 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:29:47.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7fa01971fb80 1 -- 172.21.15.2:0/686055446 shutdown_connections 2023-11-16T18:29:47.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7fa01971fb80 1 -- 172.21.15.2:0/686055446 wait complete. 2023-11-16T18:29:47.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.307+0000 7fa01971fb80 20 client.16620 trim_cache size 0 max 16384 2023-11-16T18:29:47.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:29:47.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:29:47.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:29:47.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:29:47.312 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.308+0000 7fa00d593700 1 -- 172.21.15.2:0/3737337379 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f9ff00a12c0 msgr2=0x7f9ff00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:47.312 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.308+0000 7fa00d593700 1 --2- 172.21.15.2:0/3737337379 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f9ff00a12c0 0x7f9ff00a3770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8011550 tx=0x7f9ff8009260 comp rx=0 tx=0).stop 2023-11-16T18:29:47.312 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.308+0000 7fa00d593700 1 -- 172.21.15.2:0/3737337379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa00810cc20 msgr2=0x7fa00819cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:29:47.312 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.308+0000 7fa00d593700 1 --2- 172.21.15.2:0/3737337379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa00810cc20 0x7fa00819cff0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa00000b070 tx=0x7fa0000078e0 comp rx=0 tx=0).stop 2023-11-16T18:29:47.312 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.308+0000 7fa00d593700 1 -- 172.21.15.2:0/3737337379 shutdown_connections 2023-11-16T18:29:47.312 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.308+0000 7fa00d593700 1 --2- 172.21.15.2:0/3737337379 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f9ff00a12c0 0x7f9ff00a3770 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:47.312 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.308+0000 7fa00d593700 1 --2- 172.21.15.2:0/3737337379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa00810cc20 0x7fa00819cff0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:47.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.308+0000 7fa00d593700 1 --2- 172.21.15.2:0/3737337379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0080ff070 0x7fa008198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:47.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.308+0000 7fa00d593700 1 --2- 172.21.15.2:0/3737337379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0080ff980 0x7fa008198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:29:47.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.308+0000 7fa00d593700 1 -- 172.21.15.2:0/3737337379 >> 172.21.15.2:0/3737337379 conn(0x7fa008076370 msgr2=0x7fa0080fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:29:47.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.309+0000 7fa00d593700 1 -- 172.21.15.2:0/3737337379 shutdown_connections 2023-11-16T18:29:47.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:29:47.309+0000 7fa00d593700 1 -- 172.21.15.2:0/3737337379 wait complete. 2023-11-16T18:29:47.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:47 smithi002 ceph-mon[98807]: pgmap v151: 129 pgs: 129 active+clean; 601 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 508 op/s 2023-11-16T18:29:47.738 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:47 smithi040 ceph-mon[100251]: pgmap v151: 129 pgs: 129 active+clean; 601 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 508 op/s 2023-11-16T18:29:47.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:47 smithi188 ceph-mon[100639]: pgmap v151: 129 pgs: 129 active+clean; 601 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 508 op/s 2023-11-16T18:29:50.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:50 smithi040 ceph-mon[100251]: pgmap v152: 129 pgs: 129 active+clean; 613 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 839 op/s 2023-11-16T18:29:50.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:50 smithi188 ceph-mon[100639]: pgmap v152: 129 pgs: 129 active+clean; 613 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 839 op/s 2023-11-16T18:29:50.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:50 smithi002 ceph-mon[98807]: pgmap v152: 129 pgs: 129 active+clean; 613 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 839 op/s 2023-11-16T18:29:52.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:52 smithi040 ceph-mon[100251]: pgmap v153: 129 pgs: 129 active+clean; 613 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 567 op/s 2023-11-16T18:29:52.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:52 smithi188 ceph-mon[100639]: pgmap v153: 129 pgs: 129 active+clean; 613 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 567 op/s 2023-11-16T18:29:52.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:52 smithi002 ceph-mon[98807]: pgmap v153: 129 pgs: 129 active+clean; 613 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 567 op/s 2023-11-16T18:29:53.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:53 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:29:53.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:53 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:53.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:53 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:29:53.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:53 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:53.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:53 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:29:53.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:53 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:53.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:53 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:29:53.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:53 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:53.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:53 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:29:53.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:53 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:53.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:53 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:29:53.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:53 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:54.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:54 smithi040 ceph-mon[100251]: pgmap v154: 129 pgs: 129 active+clean; 615 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 756 op/s 2023-11-16T18:29:54.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:54 smithi188 ceph-mon[100639]: pgmap v154: 129 pgs: 129 active+clean; 615 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 756 op/s 2023-11-16T18:29:54.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:54 smithi002 ceph-mon[98807]: pgmap v154: 129 pgs: 129 active+clean; 615 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 756 op/s 2023-11-16T18:29:56.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:56 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:29:56.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:56 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:56.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:56 smithi040 ceph-mon[100251]: pgmap v155: 129 pgs: 129 active+clean; 615 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 519 op/s 2023-11-16T18:29:56.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:56 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:29:56.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:56 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:56.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:56 smithi188 ceph-mon[100639]: pgmap v155: 129 pgs: 129 active+clean; 615 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 519 op/s 2023-11-16T18:29:56.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:56 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:29:56.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:56 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:29:56.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:56 smithi002 ceph-mon[98807]: pgmap v155: 129 pgs: 129 active+clean; 615 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 519 op/s 2023-11-16T18:29:57.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:29:57 smithi002 ceph-mon[98807]: pgmap v156: 129 pgs: 129 active+clean; 615 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 519 op/s 2023-11-16T18:29:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:29:57 smithi040 ceph-mon[100251]: pgmap v156: 129 pgs: 129 active+clean; 615 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 519 op/s 2023-11-16T18:29:57.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:29:57 smithi188 ceph-mon[100639]: pgmap v156: 129 pgs: 129 active+clean; 615 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 519 op/s 2023-11-16T18:30:00.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-a[98783]: 2023-11-16T18:29:59.999+0000 7f30d8773700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2023-11-16T18:30:00.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-a[98783]: 2023-11-16T18:29:59.999+0000 7f30d8773700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2023-11-16T18:30:00.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-a[98783]: 2023-11-16T18:29:59.999+0000 7f30d8773700 -1 log_channel(cluster) log [ERR] : mds.h(mds.2): Metadata damage detected 2023-11-16T18:30:00.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:00 smithi002 ceph-mon[98807]: pgmap v157: 129 pgs: 129 active+clean; 628 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 665 op/s 2023-11-16T18:30:00.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:00 smithi002 ceph-mon[98807]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2023-11-16T18:30:00.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:00 smithi002 ceph-mon[98807]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2023-11-16T18:30:00.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:00 smithi002 ceph-mon[98807]: mds.h(mds.2): Metadata damage detected 2023-11-16T18:30:00.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.465+0000 7f6d507e3700 1 -- 172.21.15.2:0/3835604077 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d480ff270 msgr2=0x7f6d4810b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:00.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.465+0000 7f6d507e3700 1 --2- 172.21.15.2:0/3835604077 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d480ff270 0x7f6d4810b700 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6d4001c0d0 tx=0x7f6d4001c3e0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.465+0000 7f6d507e3700 1 -- 172.21.15.2:0/3835604077 shutdown_connections 2023-11-16T18:30:00.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.465+0000 7f6d507e3700 1 --2- 172.21.15.2:0/3835604077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d481048d0 0x7f6d48104ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.465+0000 7f6d507e3700 1 --2- 172.21.15.2:0/3835604077 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d480ff270 0x7f6d4810b700 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.465+0000 7f6d507e3700 1 --2- 172.21.15.2:0/3835604077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d480fe8c0 0x7f6d480fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.465+0000 7f6d507e3700 1 -- 172.21.15.2:0/3835604077 >> 172.21.15.2:0/3835604077 conn(0x7f6d48075810 msgr2=0x7f6d48075c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:00.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.465+0000 7f6d507e3700 1 -- 172.21.15.2:0/3835604077 shutdown_connections 2023-11-16T18:30:00.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.465+0000 7f6d507e3700 1 -- 172.21.15.2:0/3835604077 wait complete. 2023-11-16T18:30:00.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.466+0000 7f6d507e3700 1 Processor -- start 2023-11-16T18:30:00.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.467+0000 7f6d507e3700 1 -- start start 2023-11-16T18:30:00.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.467+0000 7f6d507e3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d480fe8c0 0x7f6d48108860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.467+0000 7f6d4e57f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d480fe8c0 0x7f6d48108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.467+0000 7f6d4e57f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d480fe8c0 0x7f6d48108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:44144/0 (socket says 172.21.15.2:44144) 2023-11-16T18:30:00.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.468+0000 7f6d507e3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d480ff270 0x7f6d48108da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.468+0000 7f6d507e3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d481048d0 0x7f6d481092e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.468+0000 7f6d507e3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d481bc000 con 0x7f6d480ff270 2023-11-16T18:30:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.468+0000 7f6d507e3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d481bc140 con 0x7f6d481048d0 2023-11-16T18:30:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.468+0000 7f6d507e3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d481bc2d0 con 0x7f6d480fe8c0 2023-11-16T18:30:00.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.468+0000 7f6d4e57f700 1 -- 172.21.15.2:0/316436164 learned_addr learned my addr 172.21.15.2:0/316436164 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:00.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.469+0000 7f6d4dd7e700 1 --2- 172.21.15.2:0/316436164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d480ff270 0x7f6d48108da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.469+0000 7f6d4e57f700 1 -- 172.21.15.2:0/316436164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d481048d0 msgr2=0x7f6d481092e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T18:30:00.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.469+0000 7f6d4e57f700 1 --2- 172.21.15.2:0/316436164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d481048d0 0x7f6d481092e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.469+0000 7f6d4e57f700 1 -- 172.21.15.2:0/316436164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d480ff270 msgr2=0x7f6d48108da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:00.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.469+0000 7f6d4e57f700 1 --2- 172.21.15.2:0/316436164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d480ff270 0x7f6d48108da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.469+0000 7f6d4e57f700 1 -- 172.21.15.2:0/316436164 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4001ad10 con 0x7f6d480fe8c0 2023-11-16T18:30:00.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.471+0000 7f6d4e57f700 1 --2- 172.21.15.2:0/316436164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d480fe8c0 0x7f6d48108860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400e9a0 tx=0x7f6d4400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.471+0000 7f6d3f7fe700 1 -- 172.21.15.2:0/316436164 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d44009790 con 0x7f6d480fe8c0 2023-11-16T18:30:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.472+0000 7f6d3f7fe700 1 -- 172.21.15.2:0/316436164 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d440044e0 con 0x7f6d480fe8c0 2023-11-16T18:30:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.472+0000 7f6d507e3700 1 -- 172.21.15.2:0/316436164 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4806ccd0 con 0x7f6d480fe8c0 2023-11-16T18:30:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.472+0000 7f6d3f7fe700 1 -- 172.21.15.2:0/316436164 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4400ce80 con 0x7f6d480fe8c0 2023-11-16T18:30:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.472+0000 7f6d507e3700 1 -- 172.21.15.2:0/316436164 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4806cb40 con 0x7f6d480fe8c0 2023-11-16T18:30:00.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.478+0000 7f6d3f7fe700 1 -- 172.21.15.2:0/316436164 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f6d4401b440 con 0x7f6d480fe8c0 2023-11-16T18:30:00.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.479+0000 7f6d3f7fe700 1 --2- 172.21.15.2:0/316436164 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6d340a13c0 0x7f6d340a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.479+0000 7f6d3f7fe700 1 -- 172.21.15.2:0/316436164 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6d440d1170 con 0x7f6d480fe8c0 2023-11-16T18:30:00.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.479+0000 7f6d4dd7e700 1 --2- 172.21.15.2:0/316436164 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6d340a13c0 0x7f6d340a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.479+0000 7f6d5c96fb80 1 Processor -- start 2023-11-16T18:30:00.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:00 smithi040 ceph-mon[100251]: pgmap v157: 129 pgs: 129 active+clean; 628 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 665 op/s 2023-11-16T18:30:00.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:00 smithi040 ceph-mon[100251]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2023-11-16T18:30:00.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:00 smithi040 ceph-mon[100251]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2023-11-16T18:30:00.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:00 smithi040 ceph-mon[100251]: mds.h(mds.2): Metadata damage detected 2023-11-16T18:30:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.480+0000 7f6d5c96fb80 1 -- start start 2023-11-16T18:30:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.480+0000 7f6d4dd7e700 1 --2- 172.21.15.2:0/316436164 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6d340a13c0 0x7f6d340a3870 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6d4810a050 tx=0x7f6d38009190 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.480+0000 7f6d5c96fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56418614a6b0 0x5641862fc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.480+0000 7f6d5c96fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5641862fcef0 0x5641862ee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.480+0000 7f6d5c96fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 0x5641862eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.480+0000 7f6d5c96fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641861f4000 con 0x5641862fcef0 2023-11-16T18:30:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.480+0000 7f6d5c96fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564186246840 con 0x5641862fd360 2023-11-16T18:30:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.480+0000 7f6d5c96fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56418623fcd0 con 0x56418614a6b0 2023-11-16T18:30:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.485+0000 7f6d4ed80700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 0x5641862eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.485+0000 7f6d4dd7e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5641862fcef0 0x5641862ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.485+0000 7f6d4dd7e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5641862fcef0 0x5641862ee560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52622/0 (socket says 172.21.15.2:52622) 2023-11-16T18:30:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.485+0000 7f6d4dd7e700 1 -- 172.21.15.2:0/1125626919 learned_addr learned my addr 172.21.15.2:0/1125626919 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.485+0000 7f6d4dd7e700 1 -- 172.21.15.2:0/1125626919 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56418614a6b0 msgr2=0x5641862fc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.485+0000 7f6d4dd7e700 1 --2- 172.21.15.2:0/1125626919 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56418614a6b0 0x5641862fc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.485+0000 7f6d4dd7e700 1 -- 172.21.15.2:0/1125626919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 msgr2=0x5641862eeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.485+0000 7f6d4dd7e700 1 --2- 172.21.15.2:0/1125626919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 0x5641862eeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.485+0000 7f6d4dd7e700 1 -- 172.21.15.2:0/1125626919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641860c15d0 con 0x5641862fcef0 2023-11-16T18:30:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.486+0000 7f6d4dd7e700 1 --2- 172.21.15.2:0/1125626919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5641862fcef0 0x5641862ee560 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009fd0 tx=0x7f6d38011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5532682f6c1f8a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.486+0000 7f6d3cff9700 1 -- 172.21.15.2:0/1125626919 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38015480 con 0x5641862fcef0 2023-11-16T18:30:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.486+0000 7f6d3cff9700 1 -- 172.21.15.2:0/1125626919 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d38015aa0 con 0x5641862fcef0 2023-11-16T18:30:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.486+0000 7f6d3cff9700 1 -- 172.21.15.2:0/1125626919 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38013a40 con 0x5641862fcef0 2023-11-16T18:30:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.487+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/1125626919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5641862fcef0 msgr2=0x5641862ee560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.487+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/1125626919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5641862fcef0 0x5641862ee560 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009fd0 tx=0x7f6d38011000 comp rx=0 tx=0).stop 2023-11-16T18:30:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.487+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/1125626919 shutdown_connections 2023-11-16T18:30:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.487+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/1125626919 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56418614a6b0 0x5641862fc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.487+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/1125626919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 0x5641862eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.487+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/1125626919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5641862fcef0 0x5641862ee560 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.487+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/1125626919 >> 172.21.15.2:0/1125626919 conn(0x5641862e3f50 msgr2=0x5641862e9410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.488+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/1125626919 shutdown_connections 2023-11-16T18:30:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.488+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/1125626919 wait complete. 2023-11-16T18:30:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.488+0000 7f6d5c96fb80 1 -- start start 2023-11-16T18:30:00.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.488+0000 7f6d5c96fb80 1 Processor -- start 2023-11-16T18:30:00.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.488+0000 7f6d5c96fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 0x5641863032d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d5c96fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fcef0 0x564186300550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d5c96fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fd360 0x564186300c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d5c96fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641861e1880 con 0x56418614a6b0 2023-11-16T18:30:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d5c96fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641861e1520 con 0x5641862fcef0 2023-11-16T18:30:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d5c96fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641860c38c0 con 0x5641862fd360 2023-11-16T18:30:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d5c96fb80 10 client.0 _ll_register_callbacks cb 0x5641861ff720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:30:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d4ed80700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fd360 0x564186300c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d4dd7e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fcef0 0x564186300550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d4ed80700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fd360 0x564186300c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:44158/0 (socket says 172.21.15.2:44158) 2023-11-16T18:30:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d4dd7e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fcef0 0x564186300550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43068/0 (socket says 172.21.15.2:43068) 2023-11-16T18:30:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d4ed80700 1 -- 172.21.15.2:0/2772802820 learned_addr learned my addr 172.21.15.2:0/2772802820 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d4ed80700 1 -- 172.21.15.2:0/2772802820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fcef0 msgr2=0x564186300550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d4ed80700 1 --2- 172.21.15.2:0/2772802820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fcef0 0x564186300550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d4ed80700 1 -- 172.21.15.2:0/2772802820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 msgr2=0x5641863032d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d4ed80700 1 --2- 172.21.15.2:0/2772802820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 0x5641863032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d4ed80700 1 -- 172.21.15.2:0/2772802820 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3800e690 con 0x5641862fd360 2023-11-16T18:30:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d4ed80700 1 --2- 172.21.15.2:0/2772802820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fd360 0x564186300c90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6d4001de60 tx=0x7f6d4001df40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4064d434b5bd7d53 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.489+0000 7f6d31ffb700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:30:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.490+0000 7f6d31ffb700 1 -- 172.21.15.2:0/2772802820 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4001f040 con 0x5641862fd360 2023-11-16T18:30:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.490+0000 7f6d31ffb700 1 -- 172.21.15.2:0/2772802820 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6d4000d030 con 0x5641862fd360 2023-11-16T18:30:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.490+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/2772802820 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5641862e1710 con 0x5641862fd360 2023-11-16T18:30:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.490+0000 7f6d31ffb700 1 -- 172.21.15.2:0/2772802820 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d400074a0 con 0x5641862fd360 2023-11-16T18:30:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.490+0000 7f6d5c96fb80 10 client.26032 fetch_fsmap learned FSMap version 29 2023-11-16T18:30:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.490+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/2772802820 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564186206130 con 0x5641862fd360 2023-11-16T18:30:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.490+0000 7f6d31ffb700 1 -- 172.21.15.2:0/2772802820 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5641862e1710 con 0x5641862fd360 2023-11-16T18:30:00.650 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:00 smithi188 ceph-mon[100639]: pgmap v157: 129 pgs: 129 active+clean; 628 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 665 op/s 2023-11-16T18:30:00.650 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:00 smithi188 ceph-mon[100639]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2023-11-16T18:30:00.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:00 smithi188 ceph-mon[100639]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2023-11-16T18:30:00.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:00 smithi188 ceph-mon[100639]: mds.h(mds.2): Metadata damage detected 2023-11-16T18:30:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.490+0000 7f6d31ffb700 1 -- 172.21.15.2:0/2772802820 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f6d4002a070 con 0x5641862fd360 2023-11-16T18:30:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.491+0000 7f6d5c96fb80 10 client.26032 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:30:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.491+0000 7f6d5c96fb80 10 client.26032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:30:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.491+0000 7f6d5c96fb80 20 client.26032 populate_metadata read hostname 'smithi002' 2023-11-16T18:30:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.491+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/2772802820 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564186310980 0x564186312d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.491+0000 7f6d5c96fb80 4 client.26032 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:30:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.491+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/2772802820 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564186202040 con 0x564186310980 2023-11-16T18:30:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.492+0000 7f6d4e57f700 1 --2- 172.21.15.2:0/2772802820 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564186310980 0x564186312d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.493+0000 7f6d4e57f700 1 --2- 172.21.15.2:0/2772802820 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564186310980 0x564186312d90 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94ac87a5c34b8c08 server_cookie=137e086ba0bdc738 in_seq=0 out_seq=0 2023-11-16T18:30:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.493+0000 7f6d31ffb700 10 client.26032 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.495+0000 7f6d31ffb700 1 -- 172.21.15.2:0/2772802820 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x564186202040 con 0x564186310980 2023-11-16T18:30:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.495+0000 7f6d31ffb700 10 client.26032 handle_command_reply: tid=0 2023-11-16T18:30:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.496+0000 7f6d5c96fb80 1 client.26032 shutdown 2023-11-16T18:30:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.496+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/2772802820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fd360 msgr2=0x564186300c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.496+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/2772802820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fd360 0x564186300c90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6d4001de60 tx=0x7f6d4001df40 comp rx=0 tx=0).stop 2023-11-16T18:30:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.496+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/2772802820 shutdown_connections 2023-11-16T18:30:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.496+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/2772802820 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564186310980 0x564186312d90 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.496+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/2772802820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fd360 0x564186300c90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.496+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/2772802820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 0x5641863032d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.496+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/2772802820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fcef0 0x564186300550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.496+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/2772802820 >> 172.21.15.2:0/2772802820 conn(0x5641862e3f50 msgr2=0x5641862e87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.496+0000 7f6d31ffb700 0 client.26032 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.498+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/2772802820 shutdown_connections 2023-11-16T18:30:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.499+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/2772802820 wait complete. 2023-11-16T18:30:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.505+0000 7f6d5c96fb80 20 client.26032 trim_cache size 0 max 16384 2023-11-16T18:30:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.530+0000 7f6d5c96fb80 1 Processor -- start 2023-11-16T18:30:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.530+0000 7f6d5c96fb80 1 -- start start 2023-11-16T18:30:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d5c96fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 0x5641863078f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d5c96fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fcef0 0x564186312990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d5c96fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 0x5641862fa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d5c96fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641862e1250 con 0x56418614a6b0 2023-11-16T18:30:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d5c96fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564186301b30 con 0x5641862fd360 2023-11-16T18:30:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d5c96fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641862eb050 con 0x5641862fcef0 2023-11-16T18:30:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d4dd7e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fcef0 0x564186312990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d4ed80700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 0x5641862fa7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d4dd7e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fcef0 0x564186312990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:44168/0 (socket says 172.21.15.2:44168) 2023-11-16T18:30:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d4ed80700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 0x5641862fa7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43074/0 (socket says 172.21.15.2:43074) 2023-11-16T18:30:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d4dd7e700 1 -- 172.21.15.2:0/3984262683 learned_addr learned my addr 172.21.15.2:0/3984262683 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d4e57f700 1 --2- 172.21.15.2:0/3984262683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 0x5641863078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d4dd7e700 1 -- 172.21.15.2:0/3984262683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 msgr2=0x5641862fa7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d4dd7e700 1 --2- 172.21.15.2:0/3984262683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 0x5641862fa7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d4dd7e700 1 -- 172.21.15.2:0/3984262683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 msgr2=0x5641863078f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d4dd7e700 1 --2- 172.21.15.2:0/3984262683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 0x5641863078f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.531+0000 7f6d4dd7e700 1 -- 172.21.15.2:0/3984262683 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564186205e80 con 0x5641862fcef0 2023-11-16T18:30:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.532+0000 7f6d4dd7e700 1 --2- 172.21.15.2:0/3984262683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fcef0 0x564186312990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009270 tx=0x7f6d38006e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2b0ab42aeb7e0ea server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.532+0000 7f6d3d7fa700 1 -- 172.21.15.2:0/3984262683 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38015680 con 0x5641862fcef0 2023-11-16T18:30:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.532+0000 7f6d3d7fa700 1 -- 172.21.15.2:0/3984262683 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d3802c070 con 0x5641862fcef0 2023-11-16T18:30:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.532+0000 7f6d3d7fa700 1 -- 172.21.15.2:0/3984262683 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d380326f0 con 0x5641862fcef0 2023-11-16T18:30:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.533+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/3984262683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fcef0 msgr2=0x564186312990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.533+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/3984262683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fcef0 0x564186312990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009270 tx=0x7f6d38006e50 comp rx=0 tx=0).stop 2023-11-16T18:30:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.533+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/3984262683 shutdown_connections 2023-11-16T18:30:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.533+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/3984262683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 0x5641862fa7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.533+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/3984262683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 0x5641863078f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.533+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/3984262683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fcef0 0x564186312990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.533+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/3984262683 >> 172.21.15.2:0/3984262683 conn(0x5641862e3f50 msgr2=0x56418630c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.533+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/3984262683 shutdown_connections 2023-11-16T18:30:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.533+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/3984262683 wait complete. 2023-11-16T18:30:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.534+0000 7f6d5c96fb80 1 -- start start 2023-11-16T18:30:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.534+0000 7f6d5c96fb80 1 Processor -- start 2023-11-16T18:30:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.535+0000 7f6d5c96fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 0x5641863038d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.535+0000 7f6d5c96fb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fcef0 0x564186303e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.535+0000 7f6d5c96fb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 0x564186304350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.535+0000 7f6d5c96fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x564186307ef0 con 0x56418614a6b0 2023-11-16T18:30:00.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.535+0000 7f6d5c96fb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641863637f0 con 0x5641862fd360 2023-11-16T18:30:00.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.535+0000 7f6d5c96fb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x564186363960 con 0x5641862fcef0 2023-11-16T18:30:00.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.535+0000 7f6d5c96fb80 10 client.0 _ll_register_callbacks cb 0x7f6d44005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:30:00.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.536+0000 7f6d4e57f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 0x5641863038d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.536+0000 7f6d4e57f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 0x5641863038d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52640/0 (socket says 172.21.15.2:52640) 2023-11-16T18:30:00.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.536+0000 7f6d4e57f700 1 -- 172.21.15.2:0/66672000 learned_addr learned my addr 172.21.15.2:0/66672000 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:00.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.536+0000 7f6d4ed80700 1 --2- 172.21.15.2:0/66672000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 0x564186304350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.536+0000 7f6d4dd7e700 1 --2- 172.21.15.2:0/66672000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fcef0 0x564186303e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.536+0000 7f6d4e57f700 1 -- 172.21.15.2:0/66672000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fcef0 msgr2=0x564186303e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:00.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.536+0000 7f6d4e57f700 1 --2- 172.21.15.2:0/66672000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fcef0 0x564186303e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.536+0000 7f6d4e57f700 1 -- 172.21.15.2:0/66672000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 msgr2=0x564186304350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:00.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.536+0000 7f6d4e57f700 1 --2- 172.21.15.2:0/66672000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 0x564186304350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.536+0000 7f6d4e57f700 1 -- 172.21.15.2:0/66672000 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5641860c15d0 con 0x56418614a6b0 2023-11-16T18:30:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.536+0000 7f6d4e57f700 1 --2- 172.21.15.2:0/66672000 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 0x5641863038d0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400c5b0 tx=0x7f6d4409de00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f281708856b28d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.536+0000 7f6d31ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:30:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.537+0000 7f6d31ffb700 1 -- 172.21.15.2:0/66672000 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4409dfe0 con 0x56418614a6b0 2023-11-16T18:30:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.537+0000 7f6d31ffb700 1 -- 172.21.15.2:0/66672000 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6d440a2010 con 0x56418614a6b0 2023-11-16T18:30:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.537+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/66672000 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d38015aa0 con 0x56418614a6b0 2023-11-16T18:30:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.537+0000 7f6d31ffb700 1 -- 172.21.15.2:0/66672000 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d440a6430 con 0x56418614a6b0 2023-11-16T18:30:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.537+0000 7f6d5c96fb80 10 client.16635 fetch_fsmap learned FSMap version 29 2023-11-16T18:30:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.537+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/66672000 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564186056c00 con 0x56418614a6b0 2023-11-16T18:30:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.538+0000 7f6d31ffb700 1 -- 172.21.15.2:0/66672000 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d38015aa0 con 0x56418614a6b0 2023-11-16T18:30:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.538+0000 7f6d31ffb700 1 -- 172.21.15.2:0/66672000 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f6d4409d5e0 con 0x56418614a6b0 2023-11-16T18:30:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.538+0000 7f6d5c96fb80 10 client.16635 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:30:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.538+0000 7f6d5c96fb80 10 client.16635 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:30:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.538+0000 7f6d5c96fb80 20 client.16635 populate_metadata read hostname 'smithi002' 2023-11-16T18:30:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.538+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/66672000 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564186365090 0x5641863674a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.538+0000 7f6d5c96fb80 4 client.16635 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:30:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.538+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/66672000 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56418620e600 con 0x564186365090 2023-11-16T18:30:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.540+0000 7f6d4dd7e700 1 --2- 172.21.15.2:0/66672000 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564186365090 0x5641863674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.541+0000 7f6d4dd7e700 1 --2- 172.21.15.2:0/66672000 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564186365090 0x5641863674a0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e0d998c05825d21 server_cookie=5514017fd48e4cde in_seq=0 out_seq=0 2023-11-16T18:30:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:00.541+0000 7f6d31ffb700 10 client.16635 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:01.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:01 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:01.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:01 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d31ffb700 1 -- 172.21.15.2:0/66672000 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56418620e600 con 0x564186365090 2023-11-16T18:30:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d31ffb700 10 client.16635 handle_command_reply: tid=0 2023-11-16T18:30:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d5c96fb80 1 client.16635 shutdown 2023-11-16T18:30:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/66672000 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 msgr2=0x5641863038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/66672000 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 0x5641863038d0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400c5b0 tx=0x7f6d4409de00 comp rx=0 tx=0).stop 2023-11-16T18:30:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/66672000 shutdown_connections 2023-11-16T18:30:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/66672000 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x564186365090 0x5641863674a0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/66672000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641862fd360 0x564186304350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d31ffb700 0 client.16635 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/66672000 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56418614a6b0 0x5641863038d0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d5c96fb80 1 --2- 172.21.15.2:0/66672000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5641862fcef0 0x564186303e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/66672000 >> 172.21.15.2:0/66672000 conn(0x5641862e3f50 msgr2=0x5641862fa7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/66672000 shutdown_connections 2023-11-16T18:30:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.508+0000 7f6d5c96fb80 1 -- 172.21.15.2:0/66672000 wait complete. 2023-11-16T18:30:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.509+0000 7f6d5c96fb80 20 client.16635 trim_cache size 0 max 16384 2023-11-16T18:30:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:30:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:30:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:30:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:30:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.509+0000 7f6d507e3700 1 -- 172.21.15.2:0/316436164 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6d340a13c0 msgr2=0x7f6d340a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.509+0000 7f6d507e3700 1 --2- 172.21.15.2:0/316436164 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6d340a13c0 0x7f6d340a3870 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6d4810a050 tx=0x7f6d38009190 comp rx=0 tx=0).stop 2023-11-16T18:30:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.509+0000 7f6d507e3700 1 -- 172.21.15.2:0/316436164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d480fe8c0 msgr2=0x7f6d48108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.509+0000 7f6d507e3700 1 --2- 172.21.15.2:0/316436164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d480fe8c0 0x7f6d48108860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400e9a0 tx=0x7f6d4400ecb0 comp rx=0 tx=0).stop 2023-11-16T18:30:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.509+0000 7f6d507e3700 1 -- 172.21.15.2:0/316436164 shutdown_connections 2023-11-16T18:30:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.509+0000 7f6d507e3700 1 --2- 172.21.15.2:0/316436164 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6d340a13c0 0x7f6d340a3870 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.509+0000 7f6d507e3700 1 --2- 172.21.15.2:0/316436164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d480fe8c0 0x7f6d48108860 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.509+0000 7f6d507e3700 1 --2- 172.21.15.2:0/316436164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d481048d0 0x7f6d481092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.509+0000 7f6d507e3700 1 --2- 172.21.15.2:0/316436164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d480ff270 0x7f6d48108da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.509+0000 7f6d507e3700 1 -- 172.21.15.2:0/316436164 >> 172.21.15.2:0/316436164 conn(0x7f6d48075810 msgr2=0x7f6d480fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.509+0000 7f6d507e3700 1 -- 172.21.15.2:0/316436164 shutdown_connections 2023-11-16T18:30:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:01.509+0000 7f6d507e3700 1 -- 172.21.15.2:0/316436164 wait complete. 2023-11-16T18:30:01.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:01 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:01.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:01 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:01.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:01 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:01.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:01 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:02.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:02 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:02.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:02 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:02.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:02 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:02.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:02 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:02.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:02 smithi040 ceph-mon[100251]: pgmap v158: 129 pgs: 129 active+clean; 628 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 335 op/s 2023-11-16T18:30:02.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:02 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:02.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:02 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:02.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:02 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:02.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:02 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:02.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:02 smithi188 ceph-mon[100639]: pgmap v158: 129 pgs: 129 active+clean; 628 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 335 op/s 2023-11-16T18:30:02.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:02 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:02.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:02 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:02.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:02 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:02.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:02 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:02.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:02 smithi002 ceph-mon[98807]: pgmap v158: 129 pgs: 129 active+clean; 628 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 335 op/s 2023-11-16T18:30:04.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:04 smithi040 ceph-mon[100251]: pgmap v159: 129 pgs: 129 active+clean; 628 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 691 op/s 2023-11-16T18:30:04.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:04 smithi188 ceph-mon[100639]: pgmap v159: 129 pgs: 129 active+clean; 628 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 691 op/s 2023-11-16T18:30:04.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:04 smithi002 ceph-mon[98807]: pgmap v159: 129 pgs: 129 active+clean; 628 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 691 op/s 2023-11-16T18:30:06.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:06 smithi040 ceph-mon[100251]: pgmap v160: 129 pgs: 129 active+clean; 628 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 502 op/s 2023-11-16T18:30:06.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:06 smithi188 ceph-mon[100639]: pgmap v160: 129 pgs: 129 active+clean; 628 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 502 op/s 2023-11-16T18:30:06.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:06 smithi002 ceph-mon[98807]: pgmap v160: 129 pgs: 129 active+clean; 628 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 502 op/s 2023-11-16T18:30:07.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:07 smithi002 ceph-mon[98807]: pgmap v161: 129 pgs: 129 active+clean; 628 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 502 op/s 2023-11-16T18:30:07.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:07 smithi040 ceph-mon[100251]: pgmap v161: 129 pgs: 129 active+clean; 628 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 502 op/s 2023-11-16T18:30:07.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:07 smithi188 ceph-mon[100639]: pgmap v161: 129 pgs: 129 active+clean; 628 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 502 op/s 2023-11-16T18:30:08.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:08 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:08.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:08 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:08.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:08 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:08.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:08 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:08.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:08 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:08.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:08 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:09.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:09 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:09.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:09 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:09.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:09 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:30:09.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:09 smithi002 ceph-mon[98807]: pgmap v162: 129 pgs: 129 active+clean; 643 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.5 MiB/s wr, 666 op/s 2023-11-16T18:30:09.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:09 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:09.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:09 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:09.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:09 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:30:09.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:09 smithi040 ceph-mon[100251]: pgmap v162: 129 pgs: 129 active+clean; 643 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.5 MiB/s wr, 666 op/s 2023-11-16T18:30:09.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:09 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:09.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:09 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:09.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:09 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:30:09.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:09 smithi188 ceph-mon[100639]: pgmap v162: 129 pgs: 129 active+clean; 643 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.5 MiB/s wr, 666 op/s 2023-11-16T18:30:10.526 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:30:10.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:10 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:30:10.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:10 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:30:10.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:10 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:30:10.586 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:10 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:30:10.586 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:10 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:30:10.586 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:10 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:30:10.586 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:10 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:30:10.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.671+0000 7f3bdb06a700 1 -- 172.21.15.2:0/960980654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bd4105250 msgr2=0x7f3bd4105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:10.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.671+0000 7f3bdb06a700 1 --2- 172.21.15.2:0/960980654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bd4105250 0x7f3bd4105620 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009990 tx=0x7f3bc8009ca0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.671+0000 7f3bdb06a700 1 -- 172.21.15.2:0/960980654 shutdown_connections 2023-11-16T18:30:10.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.671+0000 7f3bdb06a700 1 --2- 172.21.15.2:0/960980654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd4100420 0x7f3bd410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.671+0000 7f3bdb06a700 1 --2- 172.21.15.2:0/960980654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bd4105250 0x7f3bd4105620 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.671+0000 7f3bdb06a700 1 --2- 172.21.15.2:0/960980654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bd40ffa70 0x7f3bd40ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.671+0000 7f3bdb06a700 1 -- 172.21.15.2:0/960980654 >> 172.21.15.2:0/960980654 conn(0x7f3bd40763a0 msgr2=0x7f3bd40767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.671+0000 7f3bdb06a700 1 -- 172.21.15.2:0/960980654 shutdown_connections 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.671+0000 7f3bdb06a700 1 -- 172.21.15.2:0/960980654 wait complete. 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bdb06a700 1 Processor -- start 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bdb06a700 1 -- start start 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bdb06a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd40ffa70 0x7f3bd4197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bdb06a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bd4100420 0x7f3bd4197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bdb06a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bd4105250 0x7f3bd41a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bdb06a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd4197f90 con 0x7f3bd4100420 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bdb06a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd41a26d0 con 0x7f3bd40ffa70 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bdb06a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd41a2890 con 0x7f3bd4105250 2023-11-16T18:30:10.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bd3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bd4100420 0x7f3bd4197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bd3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bd4100420 0x7f3bd4197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52654/0 (socket says 172.21.15.2:52654) 2023-11-16T18:30:10.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bd3fff700 1 -- 172.21.15.2:0/2384542731 learned_addr learned my addr 172.21.15.2:0/2384542731 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:10.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bd8e06700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd40ffa70 0x7f3bd4197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bd9607700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bd4105250 0x7f3bd41a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bd3fff700 1 -- 172.21.15.2:0/2384542731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bd4105250 msgr2=0x7f3bd41a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:10.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.672+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bd4105250 0x7f3bd41a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.673+0000 7f3bd3fff700 1 -- 172.21.15.2:0/2384542731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd40ffa70 msgr2=0x7f3bd4197110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:10.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.673+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd40ffa70 0x7f3bd4197110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.673+0000 7f3bd3fff700 1 -- 172.21.15.2:0/2384542731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc80095f0 con 0x7f3bd4100420 2023-11-16T18:30:10.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.673+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bd4100420 0x7f3bd4197650 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3bc000e9a0 tx=0x7f3bc000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:10.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.673+0000 7f3bd1ffb700 1 -- 172.21.15.2:0/2384542731 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0009790 con 0x7f3bd4100420 2023-11-16T18:30:10.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.673+0000 7f3bd1ffb700 1 -- 172.21.15.2:0/2384542731 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3bc000cbd0 con 0x7f3bd4100420 2023-11-16T18:30:10.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.673+0000 7f3bdb06a700 1 -- 172.21.15.2:0/2384542731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd410f1b0 con 0x7f3bd4100420 2023-11-16T18:30:10.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.673+0000 7f3bd1ffb700 1 -- 172.21.15.2:0/2384542731 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0018680 con 0x7f3bd4100420 2023-11-16T18:30:10.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.673+0000 7f3bdb06a700 1 -- 172.21.15.2:0/2384542731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd410f020 con 0x7f3bd4100420 2023-11-16T18:30:10.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.675+0000 7f3bd1ffb700 1 -- 172.21.15.2:0/2384542731 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f3bc000cd40 con 0x7f3bd4100420 2023-11-16T18:30:10.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.676+0000 7f3bd1ffb700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f3bc40a12c0 0x7f3bc40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.676+0000 7f3bd8e06700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f3bc40a12c0 0x7f3bc40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.676+0000 7f3bd1ffb700 1 -- 172.21.15.2:0/2384542731 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f3bc0014070 con 0x7f3bd4100420 2023-11-16T18:30:10.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.676+0000 7f3bd8e06700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f3bc40a12c0 0x7f3bc40a3770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800b2e0 tx=0x7f3bc8005fd0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:10.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3be71f6b80 1 Processor -- start 2023-11-16T18:30:10.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3be71f6b80 1 -- start start 2023-11-16T18:30:10.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3be71f6b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da3126f70 0x555da3127340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3be71f6b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3127880 0x555da312b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3be71f6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da312bcc0 0x555da312c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3be71f6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555da3028060 con 0x555da312bcc0 2023-11-16T18:30:10.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3be71f6b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555da307a8a0 con 0x555da3127880 2023-11-16T18:30:10.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3be71f6b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555da3073d30 con 0x555da3126f70 2023-11-16T18:30:10.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3bd9607700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da312bcc0 0x555da312c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3bd3fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3127880 0x555da312b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3bd3fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3127880 0x555da312b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43100/0 (socket says 172.21.15.2:43100) 2023-11-16T18:30:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3bd9607700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da312bcc0 0x555da312c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52664/0 (socket says 172.21.15.2:52664) 2023-11-16T18:30:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3bd3fff700 1 -- 172.21.15.2:0/2558601487 learned_addr learned my addr 172.21.15.2:0/2558601487 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.677+0000 7f3bd8e06700 1 --2- 172.21.15.2:0/2558601487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da3126f70 0x555da3127340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.678+0000 7f3bd3fff700 1 -- 172.21.15.2:0/2558601487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da3126f70 msgr2=0x555da3127340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.678+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/2558601487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da3126f70 0x555da3127340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.678+0000 7f3bd3fff700 1 -- 172.21.15.2:0/2558601487 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da312bcc0 msgr2=0x555da312c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.678+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/2558601487 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da312bcc0 0x555da312c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.678+0000 7f3bd3fff700 1 -- 172.21.15.2:0/2558601487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555da2ef5630 con 0x555da3127880 2023-11-16T18:30:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.678+0000 7f3bd9607700 1 --2- 172.21.15.2:0/2558601487 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da312bcc0 0x555da312c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:30:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.678+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/2558601487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3127880 0x555da312b780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00a1f60 tx=0x7f3bc0099a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=922cc194a80153e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.678+0000 7f3bbeffd700 1 -- 172.21.15.2:0/2558601487 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc009e070 con 0x555da3127880 2023-11-16T18:30:10.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.678+0000 7f3bbeffd700 1 -- 172.21.15.2:0/2558601487 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3bc00a2490 con 0x555da3127880 2023-11-16T18:30:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.678+0000 7f3bbeffd700 1 -- 172.21.15.2:0/2558601487 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00a2000 con 0x555da3127880 2023-11-16T18:30:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.679+0000 7f3be71f6b80 1 -- 172.21.15.2:0/2558601487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3127880 msgr2=0x555da312b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.679+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/2558601487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3127880 0x555da312b780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00a1f60 tx=0x7f3bc0099a20 comp rx=0 tx=0).stop 2023-11-16T18:30:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.679+0000 7f3be71f6b80 1 -- 172.21.15.2:0/2558601487 shutdown_connections 2023-11-16T18:30:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.679+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/2558601487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da3126f70 0x555da3127340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.679+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/2558601487 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da312bcc0 0x555da312c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.679+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/2558601487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3127880 0x555da312b780 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.679+0000 7f3be71f6b80 1 -- 172.21.15.2:0/2558601487 >> 172.21.15.2:0/2558601487 conn(0x555da3118360 msgr2=0x555da311db40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.679+0000 7f3be71f6b80 1 -- 172.21.15.2:0/2558601487 shutdown_connections 2023-11-16T18:30:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.679+0000 7f3be71f6b80 1 -- 172.21.15.2:0/2558601487 wait complete. 2023-11-16T18:30:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.679+0000 7f3be71f6b80 1 -- start start 2023-11-16T18:30:10.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.679+0000 7f3be71f6b80 1 Processor -- start 2023-11-16T18:30:10.687 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3be71f6b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3126f70 0x555da3133510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.687 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3be71f6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 0x555da3130790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.687 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3be71f6b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da312bcc0 0x555da3130ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.687 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3be71f6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555da30158e0 con 0x555da3127880 2023-11-16T18:30:10.687 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3be71f6b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555da3015580 con 0x555da3126f70 2023-11-16T18:30:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3be71f6b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555da2ef7920 con 0x555da312bcc0 2023-11-16T18:30:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3be71f6b80 10 client.0 _ll_register_callbacks cb 0x555da3033780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:30:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3bd3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 0x555da3130790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3bd8e06700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3126f70 0x555da3133510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3bd3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 0x555da3130790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52668/0 (socket says 172.21.15.2:52668) 2023-11-16T18:30:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3bd3fff700 1 -- 172.21.15.2:0/3096919409 learned_addr learned my addr 172.21.15.2:0/3096919409 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3bd9607700 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da312bcc0 0x555da3130ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3bd3fff700 1 -- 172.21.15.2:0/3096919409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da312bcc0 msgr2=0x555da3130ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da312bcc0 0x555da3130ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3bd3fff700 1 -- 172.21.15.2:0/3096919409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3126f70 msgr2=0x555da3133510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3126f70 0x555da3133510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3bd3fff700 1 -- 172.21.15.2:0/3096919409 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd410f020 con 0x555da3127880 2023-11-16T18:30:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3bd9607700 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da312bcc0 0x555da3130ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:30:10.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.680+0000 7f3bd8e06700 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3126f70 0x555da3133510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:30:10.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 0x555da3130790 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00ca770 tx=0x7f3bc0099740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6a0e5cf2027e20 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:10.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3bb3fff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:30:10.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3bb3fff700 1 -- 172.21.15.2:0/3096919409 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00a2210 con 0x555da3127880 2023-11-16T18:30:10.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3bb3fff700 1 -- 172.21.15.2:0/3096919409 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f3bc0095e30 con 0x555da3127880 2023-11-16T18:30:10.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3be71f6b80 1 -- 172.21.15.2:0/3096919409 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555da3115b20 con 0x555da3127880 2023-11-16T18:30:10.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3bb3fff700 1 -- 172.21.15.2:0/3096919409 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3bc0095ff0 con 0x555da3127880 2023-11-16T18:30:10.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3be71f6b80 10 client.16647 fetch_fsmap learned FSMap version 29 2023-11-16T18:30:10.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3be71f6b80 1 -- 172.21.15.2:0/3096919409 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x555da303a190 con 0x555da3127880 2023-11-16T18:30:10.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3bb3fff700 1 -- 172.21.15.2:0/3096919409 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555da3115b20 con 0x555da3127880 2023-11-16T18:30:10.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3bb3fff700 1 -- 172.21.15.2:0/3096919409 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f3bc00b2bb0 con 0x555da3127880 2023-11-16T18:30:10.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3be71f6b80 10 client.16647 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:30:10.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3be71f6b80 10 client.16647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:30:10.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3be71f6b80 20 client.16647 populate_metadata read hostname 'smithi002' 2023-11-16T18:30:10.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555da3140930 0x555da3142d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3be71f6b80 4 client.16647 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:30:10.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.681+0000 7f3be71f6b80 1 -- 172.21.15.2:0/3096919409 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555da30360a0 con 0x555da3140930 2023-11-16T18:30:10.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.682+0000 7f3bd8e06700 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555da3140930 0x555da3142d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.682+0000 7f3bd8e06700 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555da3140930 0x555da3142d40 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4f651bd7b0af3cd server_cookie=632fb5ce15d4a588 in_seq=0 out_seq=0 2023-11-16T18:30:10.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.682+0000 7f3bb3fff700 10 client.16647 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:10.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3bb3fff700 1 -- 172.21.15.2:0/3096919409 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555da30360a0 con 0x555da3140930 2023-11-16T18:30:10.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3bb3fff700 10 client.16647 handle_command_reply: tid=0 2023-11-16T18:30:10.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3be71f6b80 1 client.16647 shutdown 2023-11-16T18:30:10.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3be71f6b80 1 -- 172.21.15.2:0/3096919409 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 msgr2=0x555da3130790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:10.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 0x555da3130790 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00ca770 tx=0x7f3bc0099740 comp rx=0 tx=0).stop 2023-11-16T18:30:10.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3be71f6b80 1 -- 172.21.15.2:0/3096919409 shutdown_connections 2023-11-16T18:30:10.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555da3140930 0x555da3142d40 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da312bcc0 0x555da3130ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3126f70 0x555da3133510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3bb3fff700 0 client.16647 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:10.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/3096919409 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 0x555da3130790 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3be71f6b80 1 -- 172.21.15.2:0/3096919409 >> 172.21.15.2:0/3096919409 conn(0x555da3118360 msgr2=0x555da312d170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:10.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3be71f6b80 1 -- 172.21.15.2:0/3096919409 shutdown_connections 2023-11-16T18:30:10.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3be71f6b80 1 -- 172.21.15.2:0/3096919409 wait complete. 2023-11-16T18:30:10.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.683+0000 7f3be71f6b80 20 client.16647 trim_cache size 0 max 16384 2023-11-16T18:30:10.703 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3be71f6b80 1 Processor -- start 2023-11-16T18:30:10.703 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3be71f6b80 1 -- start start 2023-11-16T18:30:10.703 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3be71f6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3126f70 0x555da312ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.703 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3be71f6b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3127880 0x555da31330b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.703 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3be71f6b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da312bcc0 0x555da312e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.703 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3be71f6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555da3117ee0 con 0x555da3126f70 2023-11-16T18:30:10.703 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3be71f6b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555da312ed10 con 0x555da3127880 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3be71f6b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555da312a8f0 con 0x555da312bcc0 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3bd8e06700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3126f70 0x555da312ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3bd8e06700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3126f70 0x555da312ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52674/0 (socket says 172.21.15.2:52674) 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3bd8e06700 1 -- 172.21.15.2:0/2777649128 learned_addr learned my addr 172.21.15.2:0/2777649128 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/2777649128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3127880 0x555da31330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3bd9607700 1 --2- 172.21.15.2:0/2777649128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da312bcc0 0x555da312e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3bd8e06700 1 -- 172.21.15.2:0/2777649128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da312bcc0 msgr2=0x555da312e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3bd8e06700 1 --2- 172.21.15.2:0/2777649128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da312bcc0 0x555da312e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3bd8e06700 1 -- 172.21.15.2:0/2777649128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3127880 msgr2=0x555da31330b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3bd8e06700 1 --2- 172.21.15.2:0/2777649128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3127880 0x555da31330b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3bd8e06700 1 -- 172.21.15.2:0/2777649128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555da3039ee0 con 0x555da3126f70 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.701+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/2777649128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3127880 0x555da31330b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.702+0000 7f3bd8e06700 1 --2- 172.21.15.2:0/2777649128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3126f70 0x555da312ff70 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009990 tx=0x7f3bc8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dbeef060d83349c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.702+0000 7f3bbf7fe700 1 -- 172.21.15.2:0/2777649128 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc802d040 con 0x555da3126f70 2023-11-16T18:30:10.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.702+0000 7f3bbf7fe700 1 -- 172.21.15.2:0/2777649128 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3bc802e040 con 0x555da3126f70 2023-11-16T18:30:10.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.702+0000 7f3bbf7fe700 1 -- 172.21.15.2:0/2777649128 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8034320 con 0x555da3126f70 2023-11-16T18:30:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 -- 172.21.15.2:0/2777649128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3126f70 msgr2=0x555da312ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/2777649128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3126f70 0x555da312ff70 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009990 tx=0x7f3bc8020000 comp rx=0 tx=0).stop 2023-11-16T18:30:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 -- 172.21.15.2:0/2777649128 shutdown_connections 2023-11-16T18:30:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/2777649128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da312bcc0 0x555da312e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/2777649128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3126f70 0x555da312ff70 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/2777649128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da3127880 0x555da31330b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 -- 172.21.15.2:0/2777649128 >> 172.21.15.2:0/2777649128 conn(0x555da3118360 msgr2=0x555da3129de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 -- 172.21.15.2:0/2777649128 shutdown_connections 2023-11-16T18:30:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 -- 172.21.15.2:0/2777649128 wait complete. 2023-11-16T18:30:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 -- start start 2023-11-16T18:30:10.709 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 Processor -- start 2023-11-16T18:30:10.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da3126f70 0x555da31377e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 0x555da3137d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.703+0000 7f3be71f6b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da312bcc0 0x555da3135da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3be71f6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555da31364b0 con 0x555da3127880 2023-11-16T18:30:10.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3be71f6b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555da3136620 con 0x555da312bcc0 2023-11-16T18:30:10.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3be71f6b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555da31367e0 con 0x555da3126f70 2023-11-16T18:30:10.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3be71f6b80 10 client.0 _ll_register_callbacks cb 0x7f3bcc009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:30:10.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bd3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 0x555da3137d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bd3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 0x555da3137d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52682/0 (socket says 172.21.15.2:52682) 2023-11-16T18:30:10.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bd3fff700 1 -- 172.21.15.2:0/1564911246 learned_addr learned my addr 172.21.15.2:0/1564911246 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:10.728 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bd9607700 1 --2- 172.21.15.2:0/1564911246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da312bcc0 0x555da3135da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bd3fff700 1 -- 172.21.15.2:0/1564911246 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da3126f70 msgr2=0x555da31377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/1564911246 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da3126f70 0x555da31377e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bd3fff700 1 -- 172.21.15.2:0/1564911246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da312bcc0 msgr2=0x555da3135da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/1564911246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da312bcc0 0x555da3135da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bd3fff700 1 -- 172.21.15.2:0/1564911246 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555da303a190 con 0x555da3127880 2023-11-16T18:30:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bd9607700 1 --2- 172.21.15.2:0/1564911246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da312bcc0 0x555da3135da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:30:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bd3fff700 1 --2- 172.21.15.2:0/1564911246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 0x555da3137d20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0096540 tx=0x7f3bc009a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24fb2787b5d46ce6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bb3fff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:30:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bb3fff700 1 -- 172.21.15.2:0/1564911246 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc009b2b0 con 0x555da3127880 2023-11-16T18:30:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bb3fff700 1 -- 172.21.15.2:0/1564911246 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f3bc00a2210 con 0x555da3127880 2023-11-16T18:30:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3be71f6b80 1 -- 172.21.15.2:0/1564911246 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3bc00a2490 con 0x555da3127880 2023-11-16T18:30:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3bb3fff700 1 -- 172.21.15.2:0/1564911246 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3bc0096380 con 0x555da3127880 2023-11-16T18:30:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3be71f6b80 10 client.16653 fetch_fsmap learned FSMap version 29 2023-11-16T18:30:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.704+0000 7f3be71f6b80 1 -- 172.21.15.2:0/1564911246 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f3bd410c8d0 con 0x555da3127880 2023-11-16T18:30:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.705+0000 7f3bb3fff700 1 -- 172.21.15.2:0/1564911246 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3bc00a2490 con 0x555da3127880 2023-11-16T18:30:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.705+0000 7f3bb3fff700 1 -- 172.21.15.2:0/1564911246 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f3bc00b2be0 con 0x555da3127880 2023-11-16T18:30:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.705+0000 7f3be71f6b80 10 client.16653 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:30:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.705+0000 7f3be71f6b80 10 client.16653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:30:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.705+0000 7f3be71f6b80 20 client.16653 populate_metadata read hostname 'smithi002' 2023-11-16T18:30:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.705+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/1564911246 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555da31954c0 0x555da31978d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.705+0000 7f3be71f6b80 4 client.16653 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:30:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.705+0000 7f3be71f6b80 1 -- 172.21.15.2:0/1564911246 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555da3042660 con 0x555da31954c0 2023-11-16T18:30:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.705+0000 7f3bd8e06700 1 --2- 172.21.15.2:0/1564911246 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555da31954c0 0x555da31978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.705+0000 7f3bb3fff700 10 client.16653 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.705+0000 7f3bd8e06700 1 --2- 172.21.15.2:0/1564911246 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555da31954c0 0x555da31978d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7de54eac3b8cccbf server_cookie=1496d03f2028c5fb in_seq=0 out_seq=0 2023-11-16T18:30:10.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3bb3fff700 1 -- 172.21.15.2:0/1564911246 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x555da3042660 con 0x555da31954c0 2023-11-16T18:30:10.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3bb3fff700 10 client.16653 handle_command_reply: tid=0 2023-11-16T18:30:10.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3be71f6b80 1 client.16653 shutdown 2023-11-16T18:30:10.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3be71f6b80 1 -- 172.21.15.2:0/1564911246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 msgr2=0x555da3137d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:10.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/1564911246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 0x555da3137d20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0096540 tx=0x7f3bc009a0d0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3be71f6b80 1 -- 172.21.15.2:0/1564911246 shutdown_connections 2023-11-16T18:30:10.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/1564911246 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555da31954c0 0x555da31978d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/1564911246 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555da3126f70 0x555da31377e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/1564911246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555da312bcc0 0x555da3135da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3be71f6b80 1 --2- 172.21.15.2:0/1564911246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555da3127880 0x555da3137d20 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3bb3fff700 0 client.16653 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:10.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3be71f6b80 1 -- 172.21.15.2:0/1564911246 >> 172.21.15.2:0/1564911246 conn(0x555da3118360 msgr2=0x555da312e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:10.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3be71f6b80 1 -- 172.21.15.2:0/1564911246 shutdown_connections 2023-11-16T18:30:10.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3be71f6b80 1 -- 172.21.15.2:0/1564911246 wait complete. 2023-11-16T18:30:10.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.709+0000 7f3be71f6b80 20 client.16653 trim_cache size 0 max 16384 2023-11-16T18:30:10.780 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:30:10.780 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "scrub active (1 inodes in the stack)", 2023-11-16T18:30:10.780 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": { 2023-11-16T18:30:10.780 INFO:teuthology.orchestra.run.smithi002.stdout: "9607aa23-76ae-4198-9378-3740d8aff22c": { 2023-11-16T18:30:10.781 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/", 2023-11-16T18:30:10.781 INFO:teuthology.orchestra.run.smithi002.stdout: "tag": "9607aa23-76ae-4198-9378-3740d8aff22c", 2023-11-16T18:30:10.781 INFO:teuthology.orchestra.run.smithi002.stdout: "options": "recursive,force" 2023-11-16T18:30:10.781 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:30:10.781 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:30:10.781 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:30:10.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.710+0000 7f3bdb06a700 1 -- 172.21.15.2:0/2384542731 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f3bc40a12c0 msgr2=0x7f3bc40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:10.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.710+0000 7f3bdb06a700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f3bc40a12c0 0x7f3bc40a3770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800b2e0 tx=0x7f3bc8005fd0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.710+0000 7f3bdb06a700 1 -- 172.21.15.2:0/2384542731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bd4100420 msgr2=0x7f3bd4197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:10.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.710+0000 7f3bdb06a700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bd4100420 0x7f3bd4197650 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3bc000e9a0 tx=0x7f3bc000ecb0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.710+0000 7f3bdb06a700 1 -- 172.21.15.2:0/2384542731 shutdown_connections 2023-11-16T18:30:10.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.710+0000 7f3bdb06a700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f3bc40a12c0 0x7f3bc40a3770 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.710+0000 7f3bdb06a700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bd4105250 0x7f3bd41a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.710+0000 7f3bdb06a700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd40ffa70 0x7f3bd4197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.710+0000 7f3bdb06a700 1 --2- 172.21.15.2:0/2384542731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bd4100420 0x7f3bd4197650 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:10.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.710+0000 7f3bdb06a700 1 -- 172.21.15.2:0/2384542731 >> 172.21.15.2:0/2384542731 conn(0x7f3bd40763a0 msgr2=0x7f3bd40fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:10.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.710+0000 7f3bdb06a700 1 -- 172.21.15.2:0/2384542731 shutdown_connections 2023-11-16T18:30:10.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:10.710+0000 7f3bdb06a700 1 -- 172.21.15.2:0/2384542731 wait complete. 2023-11-16T18:30:10.883 INFO:tasks.cephfs.filesystem:scrub status for tag:9607aa23-76ae-4198-9378-3740d8aff22c - {'path': '/', 'tag': '9607aa23-76ae-4198-9378-3740d8aff22c', 'options': 'recursive,force'} 2023-11-16T18:30:10.883 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:10 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:30:10.884 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:10 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:30:10.884 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:10 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:30:10.884 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:10 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:30:10.884 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:10 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:30:10.884 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:10 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:30:10.884 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:10 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:30:10.884 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:10 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:30:10.884 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:10 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:30:10.884 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:10 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:30:10.884 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:10 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:30:10.884 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:10 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:30:10.884 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:10 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:30:10.885 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:10 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:30:11.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:11 smithi002 ceph-mon[98807]: pgmap v163: 129 pgs: 129 active+clean; 643 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 520 op/s 2023-11-16T18:30:11.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:11 smithi040 ceph-mon[100251]: pgmap v163: 129 pgs: 129 active+clean; 643 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 520 op/s 2023-11-16T18:30:11.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:11 smithi188 ceph-mon[100639]: pgmap v163: 129 pgs: 129 active+clean; 643 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 520 op/s 2023-11-16T18:30:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:14 smithi040 ceph-mon[100251]: pgmap v164: 129 pgs: 129 active+clean; 653 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 628 op/s 2023-11-16T18:30:14.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:14 smithi188 ceph-mon[100639]: pgmap v164: 129 pgs: 129 active+clean; 653 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 628 op/s 2023-11-16T18:30:14.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:14 smithi002 ceph-mon[98807]: pgmap v164: 129 pgs: 129 active+clean; 653 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 628 op/s 2023-11-16T18:30:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.626+0000 7f9042cb5700 1 -- 172.21.15.2:0/3849492570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f903c0ff090 msgr2=0x7f903c0ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.626+0000 7f9042cb5700 1 --2- 172.21.15.2:0/3849492570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f903c0ff090 0x7f903c0ff460 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9030009990 tx=0x7f9030009ca0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.626+0000 7f9042cb5700 1 -- 172.21.15.2:0/3849492570 shutdown_connections 2023-11-16T18:30:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.626+0000 7f9042cb5700 1 --2- 172.21.15.2:0/3849492570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f903c10cc40 0x7f903c10f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.626+0000 7f9042cb5700 1 --2- 172.21.15.2:0/3849492570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f903c0ff090 0x7f903c0ff460 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.626+0000 7f9042cb5700 1 --2- 172.21.15.2:0/3849492570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f903c0ff9a0 0x7f903c10c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.626+0000 7f9042cb5700 1 -- 172.21.15.2:0/3849492570 >> 172.21.15.2:0/3849492570 conn(0x7f903c076370 msgr2=0x7f903c076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.626+0000 7f9042cb5700 1 -- 172.21.15.2:0/3849492570 shutdown_connections 2023-11-16T18:30:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.626+0000 7f9042cb5700 1 -- 172.21.15.2:0/3849492570 wait complete. 2023-11-16T18:30:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.627+0000 7f9042cb5700 1 Processor -- start 2023-11-16T18:30:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.627+0000 7f9042cb5700 1 -- start start 2023-11-16T18:30:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.627+0000 7f9042cb5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f903c0ff090 0x7f903c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.627+0000 7f9042cb5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f903c0ff9a0 0x7f903c198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.627+0000 7f9042cb5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f903c10cc40 0x7f903c19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.627+0000 7f9042cb5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c19d5a0 con 0x7f903c0ff090 2023-11-16T18:30:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.627+0000 7f9042cb5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c19d710 con 0x7f903c10cc40 2023-11-16T18:30:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.627+0000 7f9042cb5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c19d8d0 con 0x7f903c0ff9a0 2023-11-16T18:30:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.627+0000 7f9040a51700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f903c0ff090 0x7f903c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.627+0000 7f9040a51700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f903c0ff090 0x7f903c1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52694/0 (socket says 172.21.15.2:52694) 2023-11-16T18:30:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.627+0000 7f9040a51700 1 -- 172.21.15.2:0/2857394405 learned_addr learned my addr 172.21.15.2:0/2857394405 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.627+0000 7f9041252700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f903c10cc40 0x7f903c19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.627+0000 7f903bfff700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f903c0ff9a0 0x7f903c198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.628+0000 7f9040a51700 1 -- 172.21.15.2:0/2857394405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f903c0ff9a0 msgr2=0x7f903c198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.628+0000 7f9040a51700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f903c0ff9a0 0x7f903c198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.628+0000 7f9040a51700 1 -- 172.21.15.2:0/2857394405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f903c10cc40 msgr2=0x7f903c19cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.628+0000 7f9040a51700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f903c10cc40 0x7f903c19cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.628+0000 7f9040a51700 1 -- 172.21.15.2:0/2857394405 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90300095f0 con 0x7f903c0ff090 2023-11-16T18:30:14.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.628+0000 7f9040a51700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f903c0ff090 0x7f903c1985c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9030000c00 tx=0x7f9030004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:14.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.628+0000 7f9039ffb700 1 -- 172.21.15.2:0/2857394405 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903001d070 con 0x7f903c0ff090 2023-11-16T18:30:14.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.628+0000 7f9039ffb700 1 -- 172.21.15.2:0/2857394405 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9030021410 con 0x7f903c0ff090 2023-11-16T18:30:14.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.628+0000 7f9042cb5700 1 -- 172.21.15.2:0/2857394405 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903c112530 con 0x7f903c0ff090 2023-11-16T18:30:14.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.628+0000 7f9039ffb700 1 -- 172.21.15.2:0/2857394405 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9030017650 con 0x7f903c0ff090 2023-11-16T18:30:14.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.628+0000 7f9042cb5700 1 -- 172.21.15.2:0/2857394405 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903c1123a0 con 0x7f903c0ff090 2023-11-16T18:30:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.630+0000 7f9039ffb700 1 -- 172.21.15.2:0/2857394405 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f9030021580 con 0x7f903c0ff090 2023-11-16T18:30:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.631+0000 7f9039ffb700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f902c0a12c0 0x7f902c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.631+0000 7f903bfff700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f902c0a12c0 0x7f902c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.631+0000 7f9039ffb700 1 -- 172.21.15.2:0/2857394405 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f90300ce330 con 0x7f903c0ff090 2023-11-16T18:30:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.632+0000 7f903bfff700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f902c0a12c0 0x7f902c0a3770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f903c199bd0 tx=0x7f9028009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.632+0000 7f904ee41b80 1 Processor -- start 2023-11-16T18:30:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.632+0000 7f904ee41b80 1 -- start start 2023-11-16T18:30:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.632+0000 7f904ee41b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 0x56065019b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.632+0000 7f904ee41b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019b720 0x560650195560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.632+0000 7f904ee41b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56065019bb90 0x560650195b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.632+0000 7f904ee41b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56065009b000 con 0x56065019b720 2023-11-16T18:30:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.632+0000 7f904ee41b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606500ed840 con 0x56064fff16b0 2023-11-16T18:30:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.632+0000 7f904ee41b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606500e6cd0 con 0x56065019bb90 2023-11-16T18:30:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.632+0000 7f903bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019b720 0x560650195560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.632+0000 7f9040a51700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 0x56065019b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.632+0000 7f9040a51700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 0x56065019b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43140/0 (socket says 172.21.15.2:43140) 2023-11-16T18:30:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.632+0000 7f9040a51700 1 -- 172.21.15.2:0/738224739 learned_addr learned my addr 172.21.15.2:0/738224739 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.633+0000 7f9041252700 1 --2- 172.21.15.2:0/738224739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56065019bb90 0x560650195b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.633+0000 7f903bfff700 1 -- 172.21.15.2:0/738224739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56065019bb90 msgr2=0x560650195b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.633+0000 7f903bfff700 1 --2- 172.21.15.2:0/738224739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56065019bb90 0x560650195b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.633+0000 7f903bfff700 1 -- 172.21.15.2:0/738224739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 msgr2=0x56065019b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.633+0000 7f903bfff700 1 --2- 172.21.15.2:0/738224739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 0x56065019b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.633+0000 7f903bfff700 1 -- 172.21.15.2:0/738224739 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56064ff685d0 con 0x56065019b720 2023-11-16T18:30:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.633+0000 7f9040a51700 1 --2- 172.21.15.2:0/738224739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 0x56065019b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:30:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.633+0000 7f903bfff700 1 --2- 172.21.15.2:0/738224739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019b720 0x560650195560 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f9028011c90 tx=0x7f9028014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814de4083c36527d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.633+0000 7f9026ffd700 1 -- 172.21.15.2:0/738224739 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9028004230 con 0x56065019b720 2023-11-16T18:30:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.633+0000 7f9026ffd700 1 -- 172.21.15.2:0/738224739 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9028021070 con 0x56065019b720 2023-11-16T18:30:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.633+0000 7f9026ffd700 1 -- 172.21.15.2:0/738224739 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9028003bf0 con 0x56065019b720 2023-11-16T18:30:14.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.634+0000 7f904ee41b80 1 -- 172.21.15.2:0/738224739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019b720 msgr2=0x560650195560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:14.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.634+0000 7f904ee41b80 1 --2- 172.21.15.2:0/738224739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019b720 0x560650195560 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f9028011c90 tx=0x7f9028014000 comp rx=0 tx=0).stop 2023-11-16T18:30:14.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.634+0000 7f904ee41b80 1 -- 172.21.15.2:0/738224739 shutdown_connections 2023-11-16T18:30:14.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.634+0000 7f904ee41b80 1 --2- 172.21.15.2:0/738224739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56065019bb90 0x560650195b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.634+0000 7f904ee41b80 1 --2- 172.21.15.2:0/738224739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 0x56065019b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.634+0000 7f904ee41b80 1 --2- 172.21.15.2:0/738224739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019b720 0x560650195560 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.634+0000 7f904ee41b80 1 -- 172.21.15.2:0/738224739 >> 172.21.15.2:0/738224739 conn(0x56065018af50 msgr2=0x560650190270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.634+0000 7f904ee41b80 1 -- 172.21.15.2:0/738224739 shutdown_connections 2023-11-16T18:30:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.634+0000 7f904ee41b80 1 -- 172.21.15.2:0/738224739 wait complete. 2023-11-16T18:30:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.634+0000 7f904ee41b80 1 -- start start 2023-11-16T18:30:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.634+0000 7f904ee41b80 1 Processor -- start 2023-11-16T18:30:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f904ee41b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56064fff16b0 0x5606501a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f904ee41b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56065019b720 0x56065019ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f904ee41b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 0x56065019f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f904ee41b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x560650088880 con 0x56065019bb90 2023-11-16T18:30:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f904ee41b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560650088520 con 0x56065019b720 2023-11-16T18:30:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f904ee41b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56064ff6a8c0 con 0x56064fff16b0 2023-11-16T18:30:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f904ee41b80 10 client.0 _ll_register_callbacks cb 0x5606500a6720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:30:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f9041252700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 0x56065019f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f9040a51700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56064fff16b0 0x5606501a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f9041252700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 0x56065019f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52712/0 (socket says 172.21.15.2:52712) 2023-11-16T18:30:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f903bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56065019b720 0x56065019ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f9041252700 1 -- 172.21.15.2:0/543804427 learned_addr learned my addr 172.21.15.2:0/543804427 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f9040a51700 1 -- 172.21.15.2:0/543804427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56065019b720 msgr2=0x56065019ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f9040a51700 1 --2- 172.21.15.2:0/543804427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56065019b720 0x56065019ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f9040a51700 1 -- 172.21.15.2:0/543804427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 msgr2=0x56065019f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f9040a51700 1 --2- 172.21.15.2:0/543804427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 0x56065019f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f9040a51700 1 -- 172.21.15.2:0/543804427 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9028017070 con 0x56064fff16b0 2023-11-16T18:30:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.635+0000 7f9041252700 1 --2- 172.21.15.2:0/543804427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 0x56065019f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:30:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f9040a51700 1 --2- 172.21.15.2:0/543804427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56064fff16b0 0x5606501a1af0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9030097620 tx=0x7f903009aa80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce45881fb4c4dbad server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f901bfff700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:30:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f901bfff700 1 -- 172.21.15.2:0/543804427 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90300b2070 con 0x56064fff16b0 2023-11-16T18:30:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f901bfff700 1 -- 172.21.15.2:0/543804427 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f903009b120 con 0x56064fff16b0 2023-11-16T18:30:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f904ee41b80 1 -- 172.21.15.2:0/543804427 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560650188710 con 0x56064fff16b0 2023-11-16T18:30:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f901bfff700 1 -- 172.21.15.2:0/543804427 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f903009f150 con 0x56064fff16b0 2023-11-16T18:30:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f904ee41b80 10 client.26065 fetch_fsmap learned FSMap version 29 2023-11-16T18:30:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f904ee41b80 1 -- 172.21.15.2:0/543804427 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5606500ad130 con 0x56064fff16b0 2023-11-16T18:30:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f901bfff700 1 -- 172.21.15.2:0/543804427 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90300c9080 con 0x56064fff16b0 2023-11-16T18:30:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f901bfff700 1 -- 172.21.15.2:0/543804427 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f90300a3050 con 0x56064fff16b0 2023-11-16T18:30:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f904ee41b80 10 client.26065 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:30:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f904ee41b80 10 client.26065 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:30:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f904ee41b80 20 client.26065 populate_metadata read hostname 'smithi002' 2023-11-16T18:30:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f904ee41b80 1 --2- 172.21.15.2:0/543804427 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5606501aed10 0x5606501b1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f904ee41b80 4 client.26065 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:30:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.636+0000 7f904ee41b80 1 -- 172.21.15.2:0/543804427 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5606500a9040 con 0x5606501aed10 2023-11-16T18:30:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.637+0000 7f903bfff700 1 --2- 172.21.15.2:0/543804427 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5606501aed10 0x5606501b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.637+0000 7f903bfff700 1 --2- 172.21.15.2:0/543804427 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5606501aed10 0x5606501b1120 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fa0ea855c87735e server_cookie=84b4576a24ded64e in_seq=0 out_seq=0 2023-11-16T18:30:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.637+0000 7f901bfff700 10 client.26065 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f901bfff700 1 -- 172.21.15.2:0/543804427 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5606500a9040 con 0x5606501aed10 2023-11-16T18:30:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f901bfff700 10 client.26065 handle_command_reply: tid=0 2023-11-16T18:30:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f904ee41b80 1 client.26065 shutdown 2023-11-16T18:30:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f904ee41b80 1 -- 172.21.15.2:0/543804427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56064fff16b0 msgr2=0x5606501a1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f904ee41b80 1 --2- 172.21.15.2:0/543804427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56064fff16b0 0x5606501a1af0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9030097620 tx=0x7f903009aa80 comp rx=0 tx=0).stop 2023-11-16T18:30:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f904ee41b80 1 -- 172.21.15.2:0/543804427 shutdown_connections 2023-11-16T18:30:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f904ee41b80 1 --2- 172.21.15.2:0/543804427 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5606501aed10 0x5606501b1120 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f904ee41b80 1 --2- 172.21.15.2:0/543804427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56064fff16b0 0x5606501a1af0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f901bfff700 0 client.26065 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f904ee41b80 1 --2- 172.21.15.2:0/543804427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 0x56065019f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f904ee41b80 1 --2- 172.21.15.2:0/543804427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56065019b720 0x56065019ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f904ee41b80 1 -- 172.21.15.2:0/543804427 >> 172.21.15.2:0/543804427 conn(0x56065018af50 msgr2=0x56065018f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f904ee41b80 1 -- 172.21.15.2:0/543804427 shutdown_connections 2023-11-16T18:30:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f904ee41b80 1 -- 172.21.15.2:0/543804427 wait complete. 2023-11-16T18:30:14.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.645+0000 7f904ee41b80 20 client.26065 trim_cache size 0 max 16384 2023-11-16T18:30:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f904ee41b80 1 Processor -- start 2023-11-16T18:30:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f904ee41b80 1 -- start start 2023-11-16T18:30:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f904ee41b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56064fff16b0 0x5606501a6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f904ee41b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56065019b720 0x56065019a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f904ee41b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 0x5606501a1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f904ee41b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56065018abf0 con 0x56065019bb90 2023-11-16T18:30:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f904ee41b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56065019d390 con 0x56065019b720 2023-11-16T18:30:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f904ee41b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560650199ab0 con 0x56064fff16b0 2023-11-16T18:30:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f9041252700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 0x5606501a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f903bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56065019b720 0x56065019a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f9041252700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 0x5606501a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52720/0 (socket says 172.21.15.2:52720) 2023-11-16T18:30:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f903bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56065019b720 0x56065019a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43156/0 (socket says 172.21.15.2:43156) 2023-11-16T18:30:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f9041252700 1 -- 172.21.15.2:0/2295003374 learned_addr learned my addr 172.21.15.2:0/2295003374 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.660+0000 7f9040a51700 1 --2- 172.21.15.2:0/2295003374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56064fff16b0 0x5606501a6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.661+0000 7f9041252700 1 -- 172.21.15.2:0/2295003374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56064fff16b0 msgr2=0x5606501a6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.661+0000 7f9041252700 1 --2- 172.21.15.2:0/2295003374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56064fff16b0 0x5606501a6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.661+0000 7f9041252700 1 -- 172.21.15.2:0/2295003374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56065019b720 msgr2=0x56065019a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.661+0000 7f9041252700 1 --2- 172.21.15.2:0/2295003374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56065019b720 0x56065019a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.661+0000 7f9041252700 1 -- 172.21.15.2:0/2295003374 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56065018c6f0 con 0x56065019bb90 2023-11-16T18:30:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.661+0000 7f9041252700 1 --2- 172.21.15.2:0/2295003374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 0x5606501a1690 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9034011cc0 tx=0x7f9034009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78250a6a2f0b45c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.661+0000 7f90277fe700 1 -- 172.21.15.2:0/2295003374 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903400e040 con 0x56065019bb90 2023-11-16T18:30:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.661+0000 7f90277fe700 1 -- 172.21.15.2:0/2295003374 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f903400f040 con 0x56065019bb90 2023-11-16T18:30:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.661+0000 7f90277fe700 1 -- 172.21.15.2:0/2295003374 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9034004420 con 0x56065019bb90 2023-11-16T18:30:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.662+0000 7f904ee41b80 1 -- 172.21.15.2:0/2295003374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 msgr2=0x5606501a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.662+0000 7f904ee41b80 1 --2- 172.21.15.2:0/2295003374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 0x5606501a1690 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9034011cc0 tx=0x7f9034009920 comp rx=0 tx=0).stop 2023-11-16T18:30:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.662+0000 7f904ee41b80 1 -- 172.21.15.2:0/2295003374 shutdown_connections 2023-11-16T18:30:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.662+0000 7f904ee41b80 1 --2- 172.21.15.2:0/2295003374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56064fff16b0 0x5606501a6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.662+0000 7f904ee41b80 1 --2- 172.21.15.2:0/2295003374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019bb90 0x5606501a1690 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.662+0000 7f904ee41b80 1 --2- 172.21.15.2:0/2295003374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56065019b720 0x56065019a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.662+0000 7f904ee41b80 1 -- 172.21.15.2:0/2295003374 >> 172.21.15.2:0/2295003374 conn(0x56065018af50 msgr2=0x560650199ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.662+0000 7f904ee41b80 1 -- 172.21.15.2:0/2295003374 shutdown_connections 2023-11-16T18:30:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.662+0000 7f904ee41b80 1 -- 172.21.15.2:0/2295003374 wait complete. 2023-11-16T18:30:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.662+0000 7f904ee41b80 1 -- start start 2023-11-16T18:30:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.662+0000 7f904ee41b80 1 Processor -- start 2023-11-16T18:30:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f904ee41b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 0x5606501a1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f904ee41b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019b720 0x5606501a1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f904ee41b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56065019bb90 0x5606501a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f904ee41b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x560650201c60 con 0x56065019b720 2023-11-16T18:30:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f904ee41b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560650201dd0 con 0x56064fff16b0 2023-11-16T18:30:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f904ee41b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x560650201f90 con 0x56065019bb90 2023-11-16T18:30:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f904ee41b80 10 client.0 _ll_register_callbacks cb 0x7f903009af60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:30:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f903bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019b720 0x5606501a1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f9040a51700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 0x5606501a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f9040a51700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 0x5606501a1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43160/0 (socket says 172.21.15.2:43160) 2023-11-16T18:30:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f903bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019b720 0x5606501a1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52724/0 (socket says 172.21.15.2:52724) 2023-11-16T18:30:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f9040a51700 1 -- 172.21.15.2:0/631156442 learned_addr learned my addr 172.21.15.2:0/631156442 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f9041252700 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56065019bb90 0x5606501a22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f9041252700 1 -- 172.21.15.2:0/631156442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 msgr2=0x5606501a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f9041252700 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 0x5606501a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f9041252700 1 -- 172.21.15.2:0/631156442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019b720 msgr2=0x5606501a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f9041252700 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019b720 0x5606501a1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f9041252700 1 -- 172.21.15.2:0/631156442 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90340119a0 con 0x56065019bb90 2023-11-16T18:30:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f9040a51700 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 0x5606501a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:30:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f903bfff700 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019b720 0x5606501a1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:30:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f9041252700 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56065019bb90 0x5606501a22d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f903400b010 tx=0x7f9034004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecad808032ec69e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.663+0000 7f901bfff700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:30:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f901bfff700 1 -- 172.21.15.2:0/631156442 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903400e070 con 0x56065019bb90 2023-11-16T18:30:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f904ee41b80 1 -- 172.21.15.2:0/631156442 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9028021070 con 0x56065019bb90 2023-11-16T18:30:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f901bfff700 1 -- 172.21.15.2:0/631156442 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f9034003c00 con 0x56065019bb90 2023-11-16T18:30:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f901bfff700 1 -- 172.21.15.2:0/631156442 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9034003dc0 con 0x56065019bb90 2023-11-16T18:30:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f904ee41b80 10 client.26071 fetch_fsmap learned FSMap version 29 2023-11-16T18:30:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f904ee41b80 1 -- 172.21.15.2:0/631156442 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56064ff14d80 con 0x56065019bb90 2023-11-16T18:30:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f901bfff700 1 -- 172.21.15.2:0/631156442 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9028021070 con 0x56065019bb90 2023-11-16T18:30:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f901bfff700 1 -- 172.21.15.2:0/631156442 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f9034007520 con 0x56065019bb90 2023-11-16T18:30:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f904ee41b80 10 client.26071 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:30:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f904ee41b80 10 client.26071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:30:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f904ee41b80 20 client.26071 populate_metadata read hostname 'smithi002' 2023-11-16T18:30:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f904ee41b80 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5606502036d0 0x560650205ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f904ee41b80 4 client.26071 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:30:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f904ee41b80 1 -- 172.21.15.2:0/631156442 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5606500b5600 con 0x5606502036d0 2023-11-16T18:30:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.664+0000 7f9040a51700 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5606502036d0 0x560650205ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.665+0000 7f901bfff700 10 client.26071 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:14.665+0000 7f9040a51700 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5606502036d0 0x560650205ae0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9f570aa799d306f server_cookie=1b27dc6bbf5158ba in_seq=0 out_seq=0 2023-11-16T18:30:15.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:15 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:15.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:15 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:15.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:15 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:15.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:15 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:15.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:15 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:15.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:15 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:15 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:15.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:15 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:15.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:15 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:16.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:16 smithi040 ceph-mon[100251]: pgmap v165: 129 pgs: 129 active+clean; 653 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 273 op/s 2023-11-16T18:30:16.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:16 smithi188 ceph-mon[100639]: pgmap v165: 129 pgs: 129 active+clean; 653 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 273 op/s 2023-11-16T18:30:16.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:16 smithi002 ceph-mon[98807]: pgmap v165: 129 pgs: 129 active+clean; 653 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 273 op/s 2023-11-16T18:30:17.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:17 smithi002 ceph-mon[98807]: pgmap v166: 129 pgs: 129 active+clean; 653 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 273 op/s 2023-11-16T18:30:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:17 smithi040 ceph-mon[100251]: pgmap v166: 129 pgs: 129 active+clean; 653 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 273 op/s 2023-11-16T18:30:17.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:17 smithi188 ceph-mon[100639]: pgmap v166: 129 pgs: 129 active+clean; 653 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 273 op/s 2023-11-16T18:30:18.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:18 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:18.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:18 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:18.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:18 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:18.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:18 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:18.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:18 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:18.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:18 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f901bfff700 1 -- 172.21.15.2:0/631156442 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5606500b5600 con 0x5606502036d0 2023-11-16T18:30:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f901bfff700 10 client.26071 handle_command_reply: tid=0 2023-11-16T18:30:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f904ee41b80 1 client.26071 shutdown 2023-11-16T18:30:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f904ee41b80 1 -- 172.21.15.2:0/631156442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56065019bb90 msgr2=0x5606501a22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f904ee41b80 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56065019bb90 0x5606501a22d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f903400b010 tx=0x7f9034004ad0 comp rx=0 tx=0).stop 2023-11-16T18:30:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f904ee41b80 1 -- 172.21.15.2:0/631156442 shutdown_connections 2023-11-16T18:30:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f904ee41b80 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5606502036d0 0x560650205ae0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f904ee41b80 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56065019bb90 0x5606501a22d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f901bfff700 0 client.26071 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f904ee41b80 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56064fff16b0 0x5606501a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f904ee41b80 1 --2- 172.21.15.2:0/631156442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56065019b720 0x5606501a1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f904ee41b80 1 -- 172.21.15.2:0/631156442 >> 172.21.15.2:0/631156442 conn(0x56065018af50 msgr2=0x56065019bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f904ee41b80 1 -- 172.21.15.2:0/631156442 shutdown_connections 2023-11-16T18:30:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f904ee41b80 1 -- 172.21.15.2:0/631156442 wait complete. 2023-11-16T18:30:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.462+0000 7f904ee41b80 20 client.26071 trim_cache size 0 max 16384 2023-11-16T18:30:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:30:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:30:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:30:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:30:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.463+0000 7f9042cb5700 1 -- 172.21.15.2:0/2857394405 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f902c0a12c0 msgr2=0x7f902c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.463+0000 7f9042cb5700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f902c0a12c0 0x7f902c0a3770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f903c199bd0 tx=0x7f9028009260 comp rx=0 tx=0).stop 2023-11-16T18:30:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.463+0000 7f9042cb5700 1 -- 172.21.15.2:0/2857394405 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f903c0ff090 msgr2=0x7f903c1985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.463+0000 7f9042cb5700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f903c0ff090 0x7f903c1985c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9030000c00 tx=0x7f9030004790 comp rx=0 tx=0).stop 2023-11-16T18:30:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.463+0000 7f9042cb5700 1 -- 172.21.15.2:0/2857394405 shutdown_connections 2023-11-16T18:30:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.463+0000 7f9042cb5700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f902c0a12c0 0x7f902c0a3770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.463+0000 7f9042cb5700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f903c10cc40 0x7f903c19cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.463+0000 7f9042cb5700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f903c0ff090 0x7f903c1985c0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.463+0000 7f9042cb5700 1 --2- 172.21.15.2:0/2857394405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f903c0ff9a0 0x7f903c198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.463+0000 7f9042cb5700 1 -- 172.21.15.2:0/2857394405 >> 172.21.15.2:0/2857394405 conn(0x7f903c076370 msgr2=0x7f903c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.463+0000 7f9042cb5700 1 -- 172.21.15.2:0/2857394405 shutdown_connections 2023-11-16T18:30:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:19.463+0000 7f9042cb5700 1 -- 172.21.15.2:0/2857394405 wait complete. 2023-11-16T18:30:19.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:19 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:19.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:19 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:19.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:19 smithi002 ceph-mon[98807]: pgmap v167: 129 pgs: 129 active+clean; 662 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 398 op/s 2023-11-16T18:30:19.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:19 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:19.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:19 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:19.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:19 smithi040 ceph-mon[100251]: pgmap v167: 129 pgs: 129 active+clean; 662 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 398 op/s 2023-11-16T18:30:19.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:19 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:19.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:19 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:19.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:19 smithi188 ceph-mon[100639]: pgmap v167: 129 pgs: 129 active+clean; 662 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 398 op/s 2023-11-16T18:30:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:22 smithi040 ceph-mon[100251]: pgmap v168: 129 pgs: 129 active+clean; 662 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 233 op/s 2023-11-16T18:30:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:22 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:22 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:22.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:22.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:22.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:22 smithi188 ceph-mon[100639]: pgmap v168: 129 pgs: 129 active+clean; 662 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 233 op/s 2023-11-16T18:30:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:22 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:22 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:22.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:22.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:22.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:22 smithi002 ceph-mon[98807]: pgmap v168: 129 pgs: 129 active+clean; 662 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 233 op/s 2023-11-16T18:30:22.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:22 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:22.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:22 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:24.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:24 smithi040 ceph-mon[100251]: pgmap v169: 129 pgs: 129 active+clean; 665 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 540 op/s 2023-11-16T18:30:24.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:24 smithi188 ceph-mon[100639]: pgmap v169: 129 pgs: 129 active+clean; 665 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 540 op/s 2023-11-16T18:30:24.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:24 smithi002 ceph-mon[98807]: pgmap v169: 129 pgs: 129 active+clean; 665 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 540 op/s 2023-11-16T18:30:25.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:25 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:25.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:25 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:25.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:25 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:25.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:25 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:25.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:25 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:25.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:25 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:26.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:26 smithi040 ceph-mon[100251]: pgmap v170: 129 pgs: 129 active+clean; 665 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 431 op/s 2023-11-16T18:30:26.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:26 smithi188 ceph-mon[100639]: pgmap v170: 129 pgs: 129 active+clean; 665 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 431 op/s 2023-11-16T18:30:26.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:26 smithi002 ceph-mon[98807]: pgmap v170: 129 pgs: 129 active+clean; 665 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 431 op/s 2023-11-16T18:30:27.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:27 smithi002 ceph-mon[98807]: pgmap v171: 129 pgs: 129 active+clean; 665 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 431 op/s 2023-11-16T18:30:27.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:27 smithi040 ceph-mon[100251]: pgmap v171: 129 pgs: 129 active+clean; 665 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 431 op/s 2023-11-16T18:30:27.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:27 smithi188 ceph-mon[100639]: pgmap v171: 129 pgs: 129 active+clean; 665 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 431 op/s 2023-11-16T18:30:28.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:28 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:28.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:28 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:28.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:28 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:28.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:28 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:28.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:28 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:28.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:28 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:29.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:29 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:29.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:29 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:29.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:29 smithi002 ceph-mon[98807]: pgmap v172: 129 pgs: 129 active+clean; 673 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 522 op/s 2023-11-16T18:30:29.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:29 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:29.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:29 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:29.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:29 smithi040 ceph-mon[100251]: pgmap v172: 129 pgs: 129 active+clean; 673 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 522 op/s 2023-11-16T18:30:29.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:29 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:29.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:29 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:29.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:29 smithi188 ceph-mon[100639]: pgmap v172: 129 pgs: 129 active+clean; 673 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 522 op/s 2023-11-16T18:30:30.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:30 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:30.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:30 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:30.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:30 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:30.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:30 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:30.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:30 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:30.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:30 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:31.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:31 smithi002 ceph-mon[98807]: pgmap v173: 129 pgs: 129 active+clean; 673 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 396 op/s 2023-11-16T18:30:31.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:31 smithi040 ceph-mon[100251]: pgmap v173: 129 pgs: 129 active+clean; 673 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 396 op/s 2023-11-16T18:30:31.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:31 smithi188 ceph-mon[100639]: pgmap v173: 129 pgs: 129 active+clean; 673 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 396 op/s 2023-11-16T18:30:32.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.575+0000 7fe16a1db700 1 -- 172.21.15.2:0/1117292649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1640ff390 msgr2=0x7fe1640ff760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:32.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.575+0000 7fe16a1db700 1 --2- 172.21.15.2:0/1117292649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1640ff390 0x7fe1640ff760 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe1540099c0 tx=0x7fe154009cd0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.575+0000 7fe16a1db700 1 -- 172.21.15.2:0/1117292649 shutdown_connections 2023-11-16T18:30:32.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.575+0000 7fe16a1db700 1 --2- 172.21.15.2:0/1117292649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1641043d0 0x7fe16410bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.575+0000 7fe16a1db700 1 --2- 172.21.15.2:0/1117292649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1640ff390 0x7fe1640ff760 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.575+0000 7fe16a1db700 1 --2- 172.21.15.2:0/1117292649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe1640ffd30 0x7fe164103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.575+0000 7fe16a1db700 1 -- 172.21.15.2:0/1117292649 >> 172.21.15.2:0/1117292649 conn(0x7fe164076310 msgr2=0x7fe164076710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:32.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.575+0000 7fe16a1db700 1 -- 172.21.15.2:0/1117292649 shutdown_connections 2023-11-16T18:30:32.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.575+0000 7fe16a1db700 1 -- 172.21.15.2:0/1117292649 wait complete. 2023-11-16T18:30:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.575+0000 7fe16a1db700 1 Processor -- start 2023-11-16T18:30:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.576+0000 7fe16a1db700 1 -- start start 2023-11-16T18:30:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.576+0000 7fe16a1db700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1640ff390 0x7fe1641971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.576+0000 7fe16a1db700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1640ffd30 0x7fe164197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.576+0000 7fe16a1db700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe1641043d0 0x7fe1641a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.576+0000 7fe16a1db700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe164198010 con 0x7fe1640ffd30 2023-11-16T18:30:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.576+0000 7fe16a1db700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1641a2780 con 0x7fe1640ff390 2023-11-16T18:30:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.576+0000 7fe16a1db700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1641a28f0 con 0x7fe1641043d0 2023-11-16T18:30:32.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.576+0000 7fe162ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1640ffd30 0x7fe164197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.576+0000 7fe162ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1640ffd30 0x7fe164197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52738/0 (socket says 172.21.15.2:52738) 2023-11-16T18:30:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.576+0000 7fe162ffd700 1 -- 172.21.15.2:0/501092267 learned_addr learned my addr 172.21.15.2:0/501092267 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.576+0000 7fe163fff700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe1641043d0 0x7fe1641a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.576+0000 7fe1637fe700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1640ff390 0x7fe1641971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.576+0000 7fe162ffd700 1 -- 172.21.15.2:0/501092267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe1641043d0 msgr2=0x7fe1641a2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.577+0000 7fe162ffd700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe1641043d0 0x7fe1641a2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.577+0000 7fe162ffd700 1 -- 172.21.15.2:0/501092267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1640ff390 msgr2=0x7fe1641971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.577+0000 7fe162ffd700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1640ff390 0x7fe1641971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.577+0000 7fe162ffd700 1 -- 172.21.15.2:0/501092267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1540095f0 con 0x7fe1640ffd30 2023-11-16T18:30:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.577+0000 7fe162ffd700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1640ffd30 0x7fe164197700 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe15800e940 tx=0x7fe15800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.577+0000 7fe160ff9700 1 -- 172.21.15.2:0/501092267 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158009730 con 0x7fe1640ffd30 2023-11-16T18:30:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.577+0000 7fe160ff9700 1 -- 172.21.15.2:0/501092267 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe158010430 con 0x7fe1640ffd30 2023-11-16T18:30:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.577+0000 7fe16a1db700 1 -- 172.21.15.2:0/501092267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16410f190 con 0x7fe1640ffd30 2023-11-16T18:30:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.577+0000 7fe160ff9700 1 -- 172.21.15.2:0/501092267 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15800cea0 con 0x7fe1640ffd30 2023-11-16T18:30:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.577+0000 7fe16a1db700 1 -- 172.21.15.2:0/501092267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16410f000 con 0x7fe1640ffd30 2023-11-16T18:30:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.579+0000 7fe160ff9700 1 -- 172.21.15.2:0/501092267 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fe158010810 con 0x7fe1640ffd30 2023-11-16T18:30:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.580+0000 7fe160ff9700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe14c0a12c0 0x7fe14c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.580+0000 7fe1637fe700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe14c0a12c0 0x7fe14c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.580+0000 7fe160ff9700 1 -- 172.21.15.2:0/501092267 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe158014070 con 0x7fe1640ffd30 2023-11-16T18:30:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.580+0000 7fe1637fe700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe14c0a12c0 0x7fe14c0a3770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe154009990 tx=0x7fe154005fb0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.580+0000 7fe176367b80 1 Processor -- start 2023-11-16T18:30:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.580+0000 7fe176367b80 1 -- start start 2023-11-16T18:30:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe176367b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72fdc56b0 0x55d72ff6f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe176367b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6f720 0x55d72ff69560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe176367b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6fb90 0x55d72ff69b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe176367b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72fe6f000 con 0x55d72ff6fb90 2023-11-16T18:30:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe176367b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72fec1840 con 0x55d72ff6f720 2023-11-16T18:30:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe176367b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72febacd0 con 0x55d72fdc56b0 2023-11-16T18:30:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe163fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6fb90 0x55d72ff69b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe162ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6f720 0x55d72ff69560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe163fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6fb90 0x55d72ff69b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52748/0 (socket says 172.21.15.2:52748) 2023-11-16T18:30:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe163fff700 1 -- 172.21.15.2:0/2841766186 learned_addr learned my addr 172.21.15.2:0/2841766186 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe1637fe700 1 --2- 172.21.15.2:0/2841766186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72fdc56b0 0x55d72ff6f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe162ffd700 1 -- 172.21.15.2:0/2841766186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72fdc56b0 msgr2=0x55d72ff6f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe162ffd700 1 --2- 172.21.15.2:0/2841766186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72fdc56b0 0x55d72ff6f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe162ffd700 1 -- 172.21.15.2:0/2841766186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6fb90 msgr2=0x55d72ff69b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe162ffd700 1 --2- 172.21.15.2:0/2841766186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6fb90 0x55d72ff69b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe162ffd700 1 -- 172.21.15.2:0/2841766186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d72fd3c5d0 con 0x55d72ff6f720 2023-11-16T18:30:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe163fff700 1 --2- 172.21.15.2:0/2841766186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6fb90 0x55d72ff69b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:30:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.581+0000 7fe162ffd700 1 --2- 172.21.15.2:0/2841766186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6f720 0x55d72ff69560 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe1580ca610 tx=0x7fe1580998f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39b78cf09c2ecf2b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.582+0000 7fe149ffb700 1 -- 172.21.15.2:0/2841766186 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1580a23e0 con 0x55d72ff6f720 2023-11-16T18:30:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.582+0000 7fe149ffb700 1 -- 172.21.15.2:0/2841766186 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe158098330 con 0x55d72ff6f720 2023-11-16T18:30:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.582+0000 7fe149ffb700 1 -- 172.21.15.2:0/2841766186 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1580a1f50 con 0x55d72ff6f720 2023-11-16T18:30:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.582+0000 7fe176367b80 1 -- 172.21.15.2:0/2841766186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6f720 msgr2=0x55d72ff69560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.582+0000 7fe176367b80 1 --2- 172.21.15.2:0/2841766186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6f720 0x55d72ff69560 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe1580ca610 tx=0x7fe1580998f0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.582+0000 7fe176367b80 1 -- 172.21.15.2:0/2841766186 shutdown_connections 2023-11-16T18:30:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.582+0000 7fe176367b80 1 --2- 172.21.15.2:0/2841766186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72fdc56b0 0x55d72ff6f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.582+0000 7fe176367b80 1 --2- 172.21.15.2:0/2841766186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6fb90 0x55d72ff69b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.582+0000 7fe176367b80 1 --2- 172.21.15.2:0/2841766186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6f720 0x55d72ff69560 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.582+0000 7fe176367b80 1 -- 172.21.15.2:0/2841766186 >> 172.21.15.2:0/2841766186 conn(0x55d72ff5ef50 msgr2=0x55d72ff64270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.582+0000 7fe176367b80 1 -- 172.21.15.2:0/2841766186 shutdown_connections 2023-11-16T18:30:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.582+0000 7fe176367b80 1 -- 172.21.15.2:0/2841766186 wait complete. 2023-11-16T18:30:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.583+0000 7fe176367b80 1 -- start start 2023-11-16T18:30:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.583+0000 7fe176367b80 1 Processor -- start 2023-11-16T18:30:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.583+0000 7fe176367b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72fdc56b0 0x55d72ff78aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.583+0000 7fe176367b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72ff6f720 0x55d72ff78fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.583+0000 7fe176367b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6fb90 0x55d72ff72d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.583+0000 7fe176367b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72fe5c880 con 0x55d72ff6fb90 2023-11-16T18:30:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.583+0000 7fe176367b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72fe5c520 con 0x55d72fdc56b0 2023-11-16T18:30:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.583+0000 7fe176367b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72fd3e8c0 con 0x55d72ff6f720 2023-11-16T18:30:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.583+0000 7fe176367b80 10 client.0 _ll_register_callbacks cb 0x55d72fe7a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:30:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.583+0000 7fe1637fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72fdc56b0 0x55d72ff78aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe163fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6fb90 0x55d72ff72d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe1637fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72fdc56b0 0x55d72ff78aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43188/0 (socket says 172.21.15.2:43188) 2023-11-16T18:30:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe1637fe700 1 -- 172.21.15.2:0/138018946 learned_addr learned my addr 172.21.15.2:0/138018946 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe162ffd700 1 --2- 172.21.15.2:0/138018946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72ff6f720 0x55d72ff78fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe1637fe700 1 -- 172.21.15.2:0/138018946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72ff6f720 msgr2=0x55d72ff78fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe1637fe700 1 --2- 172.21.15.2:0/138018946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72ff6f720 0x55d72ff78fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe1637fe700 1 -- 172.21.15.2:0/138018946 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6fb90 msgr2=0x55d72ff72d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe1637fe700 1 --2- 172.21.15.2:0/138018946 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6fb90 0x55d72ff72d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe1637fe700 1 -- 172.21.15.2:0/138018946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16410f000 con 0x55d72fdc56b0 2023-11-16T18:30:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe1637fe700 1 --2- 172.21.15.2:0/138018946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72fdc56b0 0x55d72ff78aa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe1540099c0 tx=0x7fe154009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=279ae25e349c3518 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe142ffd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:30:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe142ffd700 1 -- 172.21.15.2:0/138018946 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15402c070 con 0x55d72fdc56b0 2023-11-16T18:30:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe176367b80 1 -- 172.21.15.2:0/138018946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d72ff5c710 con 0x55d72fdc56b0 2023-11-16T18:30:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe142ffd700 1 -- 172.21.15.2:0/138018946 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe154004260 con 0x55d72fdc56b0 2023-11-16T18:30:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe142ffd700 1 -- 172.21.15.2:0/138018946 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe15400bec0 con 0x55d72fdc56b0 2023-11-16T18:30:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe176367b80 10 client.26204 fetch_fsmap learned FSMap version 29 2023-11-16T18:30:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.584+0000 7fe176367b80 1 -- 172.21.15.2:0/138018946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d72fe81130 con 0x55d72fdc56b0 2023-11-16T18:30:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.585+0000 7fe142ffd700 1 -- 172.21.15.2:0/138018946 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d72ff5c710 con 0x55d72fdc56b0 2023-11-16T18:30:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.585+0000 7fe142ffd700 1 -- 172.21.15.2:0/138018946 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fe15400f460 con 0x55d72fdc56b0 2023-11-16T18:30:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.585+0000 7fe176367b80 10 client.26204 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:30:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.585+0000 7fe176367b80 10 client.26204 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:30:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.585+0000 7fe176367b80 20 client.26204 populate_metadata read hostname 'smithi002' 2023-11-16T18:30:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.585+0000 7fe176367b80 1 --2- 172.21.15.2:0/138018946 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55d72ff82fa0 0x55d72ff853b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.585+0000 7fe176367b80 4 client.26204 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:30:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.585+0000 7fe176367b80 1 -- 172.21.15.2:0/138018946 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d72fe7d040 con 0x55d72ff82fa0 2023-11-16T18:30:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.585+0000 7fe162ffd700 1 --2- 172.21.15.2:0/138018946 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55d72ff82fa0 0x55d72ff853b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.585+0000 7fe142ffd700 10 client.26204 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.585+0000 7fe162ffd700 1 --2- 172.21.15.2:0/138018946 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55d72ff82fa0 0x55d72ff853b0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abe38d2ea1085872 server_cookie=e2f71aeb8e380585 in_seq=0 out_seq=0 2023-11-16T18:30:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe142ffd700 1 -- 172.21.15.2:0/138018946 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d72fe7d040 con 0x55d72ff82fa0 2023-11-16T18:30:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe142ffd700 10 client.26204 handle_command_reply: tid=0 2023-11-16T18:30:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe176367b80 1 client.26204 shutdown 2023-11-16T18:30:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe176367b80 1 -- 172.21.15.2:0/138018946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72fdc56b0 msgr2=0x55d72ff78aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe176367b80 1 --2- 172.21.15.2:0/138018946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72fdc56b0 0x55d72ff78aa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe1540099c0 tx=0x7fe154009310 comp rx=0 tx=0).stop 2023-11-16T18:30:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe176367b80 1 -- 172.21.15.2:0/138018946 shutdown_connections 2023-11-16T18:30:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe176367b80 1 --2- 172.21.15.2:0/138018946 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55d72ff82fa0 0x55d72ff853b0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe176367b80 1 --2- 172.21.15.2:0/138018946 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6fb90 0x55d72ff72d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe176367b80 1 --2- 172.21.15.2:0/138018946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72fdc56b0 0x55d72ff78aa0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe176367b80 1 --2- 172.21.15.2:0/138018946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72ff6f720 0x55d72ff78fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe142ffd700 0 client.26204 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe176367b80 1 -- 172.21.15.2:0/138018946 >> 172.21.15.2:0/138018946 conn(0x55d72ff5ef50 msgr2=0x55d72ff637e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe176367b80 1 -- 172.21.15.2:0/138018946 shutdown_connections 2023-11-16T18:30:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe176367b80 1 -- 172.21.15.2:0/138018946 wait complete. 2023-11-16T18:30:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.599+0000 7fe176367b80 20 client.26204 trim_cache size 0 max 16384 2023-11-16T18:30:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe176367b80 1 Processor -- start 2023-11-16T18:30:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe176367b80 1 -- start start 2023-11-16T18:30:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe176367b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72fdc56b0 0x55d72ff63f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe176367b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6f720 0x55d72ff724f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe176367b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6fb90 0x55d72ff72a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe176367b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72ff69560 con 0x55d72ff6f720 2023-11-16T18:30:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe176367b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72ff6edc0 con 0x55d72ff6fb90 2023-11-16T18:30:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe176367b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72ff6d730 con 0x55d72fdc56b0 2023-11-16T18:30:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe162ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6f720 0x55d72ff724f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe162ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6f720 0x55d72ff724f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52760/0 (socket says 172.21.15.2:52760) 2023-11-16T18:30:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe162ffd700 1 -- 172.21.15.2:0/2970661988 learned_addr learned my addr 172.21.15.2:0/2970661988 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe1637fe700 1 --2- 172.21.15.2:0/2970661988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72fdc56b0 0x55d72ff63f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe163fff700 1 --2- 172.21.15.2:0/2970661988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6fb90 0x55d72ff72a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe162ffd700 1 -- 172.21.15.2:0/2970661988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72fdc56b0 msgr2=0x55d72ff63f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe162ffd700 1 --2- 172.21.15.2:0/2970661988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72fdc56b0 0x55d72ff63f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.614+0000 7fe162ffd700 1 -- 172.21.15.2:0/2970661988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6fb90 msgr2=0x55d72ff72a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.615+0000 7fe162ffd700 1 --2- 172.21.15.2:0/2970661988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6fb90 0x55d72ff72a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.615+0000 7fe162ffd700 1 -- 172.21.15.2:0/2970661988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d72fe80e80 con 0x55d72ff6f720 2023-11-16T18:30:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.615+0000 7fe162ffd700 1 --2- 172.21.15.2:0/2970661988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6f720 0x55d72ff724f0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fe158099f20 tx=0x7fe15809e140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec25c43596827c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.615+0000 7fe14a7fc700 1 -- 172.21.15.2:0/2970661988 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15809a3f0 con 0x55d72ff6f720 2023-11-16T18:30:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.615+0000 7fe14a7fc700 1 -- 172.21.15.2:0/2970661988 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe158010620 con 0x55d72ff6f720 2023-11-16T18:30:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.615+0000 7fe14a7fc700 1 -- 172.21.15.2:0/2970661988 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158094ba0 con 0x55d72ff6f720 2023-11-16T18:30:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.615+0000 7fe176367b80 1 -- 172.21.15.2:0/2970661988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6f720 msgr2=0x55d72ff724f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.615+0000 7fe176367b80 1 --2- 172.21.15.2:0/2970661988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6f720 0x55d72ff724f0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fe158099f20 tx=0x7fe15809e140 comp rx=0 tx=0).stop 2023-11-16T18:30:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 -- 172.21.15.2:0/2970661988 shutdown_connections 2023-11-16T18:30:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 --2- 172.21.15.2:0/2970661988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72fdc56b0 0x55d72ff63f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 --2- 172.21.15.2:0/2970661988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6fb90 0x55d72ff72a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 --2- 172.21.15.2:0/2970661988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72ff6f720 0x55d72ff724f0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 -- 172.21.15.2:0/2970661988 >> 172.21.15.2:0/2970661988 conn(0x55d72ff5ef50 msgr2=0x55d72ff736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 -- 172.21.15.2:0/2970661988 shutdown_connections 2023-11-16T18:30:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 -- 172.21.15.2:0/2970661988 wait complete. 2023-11-16T18:30:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 -- start start 2023-11-16T18:30:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 Processor -- start 2023-11-16T18:30:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72fdc56b0 0x55d72ff76730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6f720 0x55d72ff76c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72ff6fb90 0x55d72ffd5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72ff77400 con 0x55d72fdc56b0 2023-11-16T18:30:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72ff77570 con 0x55d72ff6f720 2023-11-16T18:30:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72ffd6360 con 0x55d72ff6fb90 2023-11-16T18:30:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.616+0000 7fe176367b80 10 client.0 _ll_register_callbacks cb 0x7fe154003e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:30:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe1637fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72fdc56b0 0x55d72ff76730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe1637fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72fdc56b0 0x55d72ff76730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52764/0 (socket says 172.21.15.2:52764) 2023-11-16T18:30:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe1637fe700 1 -- 172.21.15.2:0/41284757 learned_addr learned my addr 172.21.15.2:0/41284757 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe162ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6f720 0x55d72ff76c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe163fff700 1 --2- 172.21.15.2:0/41284757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72ff6fb90 0x55d72ffd5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe1637fe700 1 -- 172.21.15.2:0/41284757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72ff6fb90 msgr2=0x55d72ffd5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe1637fe700 1 --2- 172.21.15.2:0/41284757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72ff6fb90 0x55d72ffd5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe1637fe700 1 -- 172.21.15.2:0/41284757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6f720 msgr2=0x55d72ff76c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe1637fe700 1 --2- 172.21.15.2:0/41284757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6f720 0x55d72ff76c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe1637fe700 1 -- 172.21.15.2:0/41284757 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d72fd3c5d0 con 0x55d72fdc56b0 2023-11-16T18:30:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe163fff700 1 --2- 172.21.15.2:0/41284757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72ff6fb90 0x55d72ffd5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:30:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe1637fe700 1 --2- 172.21.15.2:0/41284757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72fdc56b0 0x55d72ff76730 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fe154005a60 tx=0x7fe154003920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00a89aa67811e9d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe142ffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:30:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe142ffd700 1 -- 172.21.15.2:0/41284757 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15402c040 con 0x55d72fdc56b0 2023-11-16T18:30:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe142ffd700 1 -- 172.21.15.2:0/41284757 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe15400f5a0 con 0x55d72fdc56b0 2023-11-16T18:30:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe176367b80 1 -- 172.21.15.2:0/41284757 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe158098330 con 0x55d72fdc56b0 2023-11-16T18:30:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe142ffd700 1 -- 172.21.15.2:0/41284757 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe154004470 con 0x55d72fdc56b0 2023-11-16T18:30:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe176367b80 10 client.16689 fetch_fsmap learned FSMap version 29 2023-11-16T18:30:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.617+0000 7fe176367b80 1 -- 172.21.15.2:0/41284757 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d72fcd1c00 con 0x55d72fdc56b0 2023-11-16T18:30:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.618+0000 7fe142ffd700 1 -- 172.21.15.2:0/41284757 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe158098330 con 0x55d72fdc56b0 2023-11-16T18:30:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.618+0000 7fe142ffd700 1 -- 172.21.15.2:0/41284757 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fe15403e430 con 0x55d72fdc56b0 2023-11-16T18:30:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.618+0000 7fe176367b80 10 client.16689 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:30:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.618+0000 7fe176367b80 10 client.16689 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:30:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.618+0000 7fe176367b80 20 client.16689 populate_metadata read hostname 'smithi002' 2023-11-16T18:30:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.618+0000 7fe176367b80 1 --2- 172.21.15.2:0/41284757 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55d72ffd7660 0x55d72ffd9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.618+0000 7fe176367b80 4 client.16689 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:30:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.618+0000 7fe176367b80 1 -- 172.21.15.2:0/41284757 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d72fe89600 con 0x55d72ffd7660 2023-11-16T18:30:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.618+0000 7fe162ffd700 1 --2- 172.21.15.2:0/41284757 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55d72ffd7660 0x55d72ffd9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.618+0000 7fe142ffd700 10 client.16689 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:32.618+0000 7fe162ffd700 1 --2- 172.21.15.2:0/41284757 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55d72ffd7660 0x55d72ffd9a70 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f46b019768dbeab server_cookie=b755c63ee3405475 in_seq=0 out_seq=0 2023-11-16T18:30:34.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:34 smithi040 ceph-mon[100251]: pgmap v174: 129 pgs: 129 active+clean; 684 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 519 op/s 2023-11-16T18:30:34.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:34 smithi188 ceph-mon[100639]: pgmap v174: 129 pgs: 129 active+clean; 684 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 519 op/s 2023-11-16T18:30:34.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:34 smithi002 ceph-mon[98807]: pgmap v174: 129 pgs: 129 active+clean; 684 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 519 op/s 2023-11-16T18:30:36.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:36 smithi040 ceph-mon[100251]: pgmap v175: 129 pgs: 129 active+clean; 684 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 213 op/s 2023-11-16T18:30:36.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:36 smithi188 ceph-mon[100639]: pgmap v175: 129 pgs: 129 active+clean; 684 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 213 op/s 2023-11-16T18:30:36.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:36 smithi002 ceph-mon[98807]: pgmap v175: 129 pgs: 129 active+clean; 684 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 213 op/s 2023-11-16T18:30:37.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:37 smithi002 ceph-mon[98807]: pgmap v176: 129 pgs: 129 active+clean; 684 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 213 op/s 2023-11-16T18:30:37.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:37 smithi040 ceph-mon[100251]: pgmap v176: 129 pgs: 129 active+clean; 684 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 213 op/s 2023-11-16T18:30:37.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:37 smithi188 ceph-mon[100639]: pgmap v176: 129 pgs: 129 active+clean; 684 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 213 op/s 2023-11-16T18:30:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.320+0000 7fe142ffd700 1 -- 172.21.15.2:0/41284757 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d72fe89600 con 0x55d72ffd7660 2023-11-16T18:30:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.320+0000 7fe142ffd700 10 client.16689 handle_command_reply: tid=0 2023-11-16T18:30:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.320+0000 7fe176367b80 1 client.16689 shutdown 2023-11-16T18:30:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.320+0000 7fe176367b80 1 -- 172.21.15.2:0/41284757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72fdc56b0 msgr2=0x55d72ff76730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.320+0000 7fe176367b80 1 --2- 172.21.15.2:0/41284757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72fdc56b0 0x55d72ff76730 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fe154005a60 tx=0x7fe154003920 comp rx=0 tx=0).stop 2023-11-16T18:30:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.320+0000 7fe176367b80 1 -- 172.21.15.2:0/41284757 shutdown_connections 2023-11-16T18:30:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.320+0000 7fe176367b80 1 --2- 172.21.15.2:0/41284757 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55d72ffd7660 0x55d72ffd9a70 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.320+0000 7fe176367b80 1 --2- 172.21.15.2:0/41284757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d72ff6fb90 0x55d72ffd5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.320+0000 7fe176367b80 1 --2- 172.21.15.2:0/41284757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d72fdc56b0 0x55d72ff76730 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.320+0000 7fe142ffd700 0 client.16689 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.320+0000 7fe176367b80 1 --2- 172.21.15.2:0/41284757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72ff6f720 0x55d72ff76c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.320+0000 7fe176367b80 1 -- 172.21.15.2:0/41284757 >> 172.21.15.2:0/41284757 conn(0x55d72ff5ef50 msgr2=0x55d72ff6ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.321+0000 7fe176367b80 1 -- 172.21.15.2:0/41284757 shutdown_connections 2023-11-16T18:30:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.321+0000 7fe176367b80 1 -- 172.21.15.2:0/41284757 wait complete. 2023-11-16T18:30:38.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.321+0000 7fe176367b80 20 client.16689 trim_cache size 0 max 16384 2023-11-16T18:30:38.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:30:38.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:30:38.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:30:38.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:30:38.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.321+0000 7fe16a1db700 1 -- 172.21.15.2:0/501092267 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe14c0a12c0 msgr2=0x7fe14c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:38.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.321+0000 7fe16a1db700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe14c0a12c0 0x7fe14c0a3770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe154009990 tx=0x7fe154005fb0 comp rx=0 tx=0).stop 2023-11-16T18:30:38.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.321+0000 7fe16a1db700 1 -- 172.21.15.2:0/501092267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1640ffd30 msgr2=0x7fe164197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:38.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.322+0000 7fe16a1db700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1640ffd30 0x7fe164197700 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe15800e940 tx=0x7fe15800ec50 comp rx=0 tx=0).stop 2023-11-16T18:30:38.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.322+0000 7fe16a1db700 1 -- 172.21.15.2:0/501092267 shutdown_connections 2023-11-16T18:30:38.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.322+0000 7fe16a1db700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fe14c0a12c0 0x7fe14c0a3770 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:38.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.322+0000 7fe16a1db700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe1641043d0 0x7fe1641a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:38.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.322+0000 7fe16a1db700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1640ff390 0x7fe1641971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:38.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.322+0000 7fe16a1db700 1 --2- 172.21.15.2:0/501092267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1640ffd30 0x7fe164197700 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:38.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.322+0000 7fe16a1db700 1 -- 172.21.15.2:0/501092267 >> 172.21.15.2:0/501092267 conn(0x7fe164076310 msgr2=0x7fe164102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:38.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.322+0000 7fe16a1db700 1 -- 172.21.15.2:0/501092267 shutdown_connections 2023-11-16T18:30:38.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:38.322+0000 7fe16a1db700 1 -- 172.21.15.2:0/501092267 wait complete. 2023-11-16T18:30:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:40 smithi040 ceph-mon[100251]: pgmap v177: 129 pgs: 129 active+clean; 694 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 494 op/s 2023-11-16T18:30:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:40 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:40 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:40.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:40 smithi188 ceph-mon[100639]: pgmap v177: 129 pgs: 129 active+clean; 694 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 494 op/s 2023-11-16T18:30:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:40 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:40 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:40.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:40 smithi002 ceph-mon[98807]: pgmap v177: 129 pgs: 129 active+clean; 694 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 494 op/s 2023-11-16T18:30:40.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:40 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:40.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:40 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:40.887 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:30:41.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.023+0000 7faa48167700 1 -- 172.21.15.2:0/3621750208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa40100420 msgr2=0x7faa4010bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:41.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.023+0000 7faa48167700 1 --2- 172.21.15.2:0/3621750208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa40100420 0x7faa4010bfd0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7faa3800b0a0 tx=0x7faa3800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.023+0000 7faa48167700 1 -- 172.21.15.2:0/3621750208 shutdown_connections 2023-11-16T18:30:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.023+0000 7faa48167700 1 --2- 172.21.15.2:0/3621750208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa40100420 0x7faa4010bfd0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.023+0000 7faa48167700 1 --2- 172.21.15.2:0/3621750208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa40105250 0x7faa40105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.023+0000 7faa48167700 1 --2- 172.21.15.2:0/3621750208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa400ffa70 0x7faa400ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.023+0000 7faa48167700 1 -- 172.21.15.2:0/3621750208 >> 172.21.15.2:0/3621750208 conn(0x7faa400763a0 msgr2=0x7faa400767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.023+0000 7faa48167700 1 -- 172.21.15.2:0/3621750208 shutdown_connections 2023-11-16T18:30:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.023+0000 7faa48167700 1 -- 172.21.15.2:0/3621750208 wait complete. 2023-11-16T18:30:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.023+0000 7faa48167700 1 Processor -- start 2023-11-16T18:30:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa48167700 1 -- start start 2023-11-16T18:30:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa48167700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa400ffa70 0x7faa4010c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa48167700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa40100420 0x7faa4010cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa48167700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa40105250 0x7faa401a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa48167700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4010d5d0 con 0x7faa400ffa70 2023-11-16T18:30:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa48167700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa401a4b20 con 0x7faa40105250 2023-11-16T18:30:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa48167700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa401a4c60 con 0x7faa40100420 2023-11-16T18:30:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa45f03700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa400ffa70 0x7faa4010c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa45f03700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa400ffa70 0x7faa4010c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52778/0 (socket says 172.21.15.2:52778) 2023-11-16T18:30:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa45f03700 1 -- 172.21.15.2:0/3824509529 learned_addr learned my addr 172.21.15.2:0/3824509529 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa45702700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa40100420 0x7faa4010cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa46704700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa40105250 0x7faa401a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa45f03700 1 -- 172.21.15.2:0/3824509529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa40100420 msgr2=0x7faa4010cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa45f03700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa40100420 0x7faa4010cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa45f03700 1 -- 172.21.15.2:0/3824509529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa40105250 msgr2=0x7faa401a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:41.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa45f03700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa40105250 0x7faa401a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa45f03700 1 -- 172.21.15.2:0/3824509529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa38009d00 con 0x7faa400ffa70 2023-11-16T18:30:41.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa45f03700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa400ffa70 0x7faa4010c860 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00e9a0 tx=0x7faa3c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:41.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa36ffd700 1 -- 172.21.15.2:0/3824509529 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c009790 con 0x7faa400ffa70 2023-11-16T18:30:41.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa36ffd700 1 -- 172.21.15.2:0/3824509529 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa3c010430 con 0x7faa400ffa70 2023-11-16T18:30:41.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa36ffd700 1 -- 172.21.15.2:0/3824509529 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c018680 con 0x7faa400ffa70 2023-11-16T18:30:41.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.024+0000 7faa48167700 1 -- 172.21.15.2:0/3824509529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa40110370 con 0x7faa400ffa70 2023-11-16T18:30:41.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.025+0000 7faa48167700 1 -- 172.21.15.2:0/3824509529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa401101e0 con 0x7faa400ffa70 2023-11-16T18:30:41.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.027+0000 7faa36ffd700 1 -- 172.21.15.2:0/3824509529 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7faa3c0105a0 con 0x7faa400ffa70 2023-11-16T18:30:41.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.027+0000 7faa36ffd700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7faa2c0a12c0 0x7faa2c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.027+0000 7faa36ffd700 1 -- 172.21.15.2:0/3824509529 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7faa3c014070 con 0x7faa400ffa70 2023-11-16T18:30:41.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.028+0000 7faa45702700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7faa2c0a12c0 0x7faa2c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.028+0000 7faa45702700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7faa2c0a12c0 0x7faa2c0a3770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faa4010e050 tx=0x7faa3000b410 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:41.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.028+0000 7faa542f3b80 1 Processor -- start 2023-11-16T18:30:41.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.028+0000 7faa542f3b80 1 -- start start 2023-11-16T18:30:41.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa542f3b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940558b60 0x55e940558f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa542f3b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 0x55e94055d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa542f3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e94055d8b0 0x55e94055dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa542f3b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94045e060 con 0x55e940559470 2023-11-16T18:30:41.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa542f3b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9404b08a0 con 0x55e940558b60 2023-11-16T18:30:41.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa542f3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9404a9d30 con 0x55e94055d8b0 2023-11-16T18:30:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa45702700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 0x55e94055d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa45702700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 0x55e94055d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52788/0 (socket says 172.21.15.2:52788) 2023-11-16T18:30:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa45702700 1 -- 172.21.15.2:0/811780376 learned_addr learned my addr 172.21.15.2:0/811780376 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa45f03700 1 --2- 172.21.15.2:0/811780376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940558b60 0x55e940558f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa45702700 1 -- 172.21.15.2:0/811780376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e94055d8b0 msgr2=0x55e94055dd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T18:30:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa45702700 1 --2- 172.21.15.2:0/811780376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e94055d8b0 0x55e94055dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa45702700 1 -- 172.21.15.2:0/811780376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940558b60 msgr2=0x55e940558f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa45702700 1 --2- 172.21.15.2:0/811780376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940558b60 0x55e940558f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa45702700 1 -- 172.21.15.2:0/811780376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e94032b630 con 0x55e940559470 2023-11-16T18:30:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.029+0000 7faa45702700 1 --2- 172.21.15.2:0/811780376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 0x55e94055d370 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7faa30010f70 tx=0x7faa30013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bca45a82a1ed15f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.030+0000 7faa2bfff700 1 -- 172.21.15.2:0/811780376 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa30011db0 con 0x55e940559470 2023-11-16T18:30:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.030+0000 7faa2bfff700 1 -- 172.21.15.2:0/811780376 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa30015070 con 0x55e940559470 2023-11-16T18:30:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.030+0000 7faa2bfff700 1 -- 172.21.15.2:0/811780376 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3001c610 con 0x55e940559470 2023-11-16T18:30:41.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.031+0000 7faa542f3b80 1 -- 172.21.15.2:0/811780376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 msgr2=0x55e94055d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:41.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.031+0000 7faa542f3b80 1 --2- 172.21.15.2:0/811780376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 0x55e94055d370 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7faa30010f70 tx=0x7faa30013000 comp rx=0 tx=0).stop 2023-11-16T18:30:41.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.031+0000 7faa542f3b80 1 -- 172.21.15.2:0/811780376 shutdown_connections 2023-11-16T18:30:41.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.031+0000 7faa542f3b80 1 --2- 172.21.15.2:0/811780376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e94055d8b0 0x55e94055dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.031+0000 7faa542f3b80 1 --2- 172.21.15.2:0/811780376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940558b60 0x55e940558f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.031+0000 7faa542f3b80 1 --2- 172.21.15.2:0/811780376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 0x55e94055d370 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.031+0000 7faa542f3b80 1 -- 172.21.15.2:0/811780376 >> 172.21.15.2:0/811780376 conn(0x55e94054e360 msgr2=0x55e940553b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:41.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.031+0000 7faa542f3b80 1 -- 172.21.15.2:0/811780376 shutdown_connections 2023-11-16T18:30:41.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.031+0000 7faa542f3b80 1 -- 172.21.15.2:0/811780376 wait complete. 2023-11-16T18:30:41.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.031+0000 7faa542f3b80 1 -- start start 2023-11-16T18:30:41.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.032+0000 7faa542f3b80 1 Processor -- start 2023-11-16T18:30:41.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.032+0000 7faa542f3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e940558b60 0x55e940565050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.032+0000 7faa542f3b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940559470 0x55e940562260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.032+0000 7faa542f3b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e94055d8b0 0x55e940562870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.032+0000 7faa542f3b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94044b8e0 con 0x55e94055d8b0 2023-11-16T18:30:41.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.032+0000 7faa542f3b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94044b580 con 0x55e940559470 2023-11-16T18:30:41.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.032+0000 7faa542f3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94032d920 con 0x55e940558b60 2023-11-16T18:30:41.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.032+0000 7faa542f3b80 10 client.0 _ll_register_callbacks cb 0x55e940469780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:30:41.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.032+0000 7faa46704700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e94055d8b0 0x55e940562870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.032+0000 7faa46704700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e94055d8b0 0x55e940562870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52794/0 (socket says 172.21.15.2:52794) 2023-11-16T18:30:41.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.032+0000 7faa46704700 1 -- 172.21.15.2:0/54660516 learned_addr learned my addr 172.21.15.2:0/54660516 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.032+0000 7faa45f03700 1 --2- 172.21.15.2:0/54660516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e940558b60 0x55e940565050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.032+0000 7faa45702700 1 --2- 172.21.15.2:0/54660516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940559470 0x55e940562260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa46704700 1 -- 172.21.15.2:0/54660516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e940558b60 msgr2=0x55e940565050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa46704700 1 --2- 172.21.15.2:0/54660516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e940558b60 0x55e940565050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa46704700 1 -- 172.21.15.2:0/54660516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940559470 msgr2=0x55e940562260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa46704700 1 --2- 172.21.15.2:0/54660516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940559470 0x55e940562260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa46704700 1 -- 172.21.15.2:0/54660516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa30010c20 con 0x55e94055d8b0 2023-11-16T18:30:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa46704700 1 --2- 172.21.15.2:0/54660516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e94055d8b0 0x55e940562870 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7faa3800be20 tx=0x7faa3800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa7860aa00524c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa297fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:30:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa297fa700 1 -- 172.21.15.2:0/54660516 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3800e040 con 0x55e94055d8b0 2023-11-16T18:30:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa542f3b80 1 -- 172.21.15.2:0/54660516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e94054bb20 con 0x55e94055d8b0 2023-11-16T18:30:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa297fa700 1 -- 172.21.15.2:0/54660516 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7faa38003e10 con 0x55e94055d8b0 2023-11-16T18:30:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa297fa700 1 -- 172.21.15.2:0/54660516 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa380074e0 con 0x55e94055d8b0 2023-11-16T18:30:41.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa542f3b80 10 client.16701 fetch_fsmap learned FSMap version 29 2023-11-16T18:30:41.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa542f3b80 1 -- 172.21.15.2:0/54660516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e940470190 con 0x55e94055d8b0 2023-11-16T18:30:41.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.033+0000 7faa297fa700 1 -- 172.21.15.2:0/54660516 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e94054bb20 con 0x55e94055d8b0 2023-11-16T18:30:41.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.034+0000 7faa297fa700 1 -- 172.21.15.2:0/54660516 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7faa38019030 con 0x55e94055d8b0 2023-11-16T18:30:41.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.034+0000 7faa542f3b80 10 client.16701 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:30:41.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.034+0000 7faa542f3b80 10 client.16701 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:30:41.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.034+0000 7faa542f3b80 20 client.16701 populate_metadata read hostname 'smithi002' 2023-11-16T18:30:41.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.034+0000 7faa542f3b80 1 --2- 172.21.15.2:0/54660516 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55e9405728f0 0x55e940574d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.034+0000 7faa542f3b80 4 client.16701 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:30:41.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.034+0000 7faa542f3b80 1 -- 172.21.15.2:0/54660516 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e94046c0a0 con 0x55e9405728f0 2023-11-16T18:30:41.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.034+0000 7faa45f03700 1 --2- 172.21.15.2:0/54660516 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55e9405728f0 0x55e940574d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.034+0000 7faa297fa700 10 client.16701 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.034+0000 7faa45f03700 1 --2- 172.21.15.2:0/54660516 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55e9405728f0 0x55e940574d00 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=365542555597a83f server_cookie=2d567bb05c4bc453 in_seq=0 out_seq=0 2023-11-16T18:30:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa297fa700 1 -- 172.21.15.2:0/54660516 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e94046c0a0 con 0x55e9405728f0 2023-11-16T18:30:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa297fa700 10 client.16701 handle_command_reply: tid=0 2023-11-16T18:30:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa542f3b80 1 client.16701 shutdown 2023-11-16T18:30:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa542f3b80 1 -- 172.21.15.2:0/54660516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e94055d8b0 msgr2=0x55e940562870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa542f3b80 1 --2- 172.21.15.2:0/54660516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e94055d8b0 0x55e940562870 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7faa3800be20 tx=0x7faa3800be50 comp rx=0 tx=0).stop 2023-11-16T18:30:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa542f3b80 1 -- 172.21.15.2:0/54660516 shutdown_connections 2023-11-16T18:30:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa542f3b80 1 --2- 172.21.15.2:0/54660516 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55e9405728f0 0x55e940574d00 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa542f3b80 1 --2- 172.21.15.2:0/54660516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e940558b60 0x55e940565050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa542f3b80 1 --2- 172.21.15.2:0/54660516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e94055d8b0 0x55e940562870 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa542f3b80 1 --2- 172.21.15.2:0/54660516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940559470 0x55e940562260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa297fa700 0 client.16701 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:41.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa542f3b80 1 -- 172.21.15.2:0/54660516 >> 172.21.15.2:0/54660516 conn(0x55e94054e360 msgr2=0x55e94055ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:41.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa542f3b80 1 -- 172.21.15.2:0/54660516 shutdown_connections 2023-11-16T18:30:41.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa542f3b80 1 -- 172.21.15.2:0/54660516 wait complete. 2023-11-16T18:30:41.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.038+0000 7faa542f3b80 20 client.16701 trim_cache size 0 max 16384 2023-11-16T18:30:41.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.056+0000 7faa542f3b80 1 Processor -- start 2023-11-16T18:30:41.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.056+0000 7faa542f3b80 1 -- start start 2023-11-16T18:30:41.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.056+0000 7faa542f3b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940558b60 0x55e940561a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.056+0000 7faa542f3b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 0x55e940564b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.056+0000 7faa542f3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e94055d8b0 0x55e94055b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.056+0000 7faa542f3b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e940559bc0 con 0x55e940559470 2023-11-16T18:30:41.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.056+0000 7faa542f3b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9405549a0 con 0x55e940558b60 2023-11-16T18:30:41.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.056+0000 7faa542f3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9405533a0 con 0x55e94055d8b0 2023-11-16T18:30:41.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa45702700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 0x55e940564b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa45702700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 0x55e940564b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52800/0 (socket says 172.21.15.2:52800) 2023-11-16T18:30:41.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa45702700 1 -- 172.21.15.2:0/2231899512 learned_addr learned my addr 172.21.15.2:0/2231899512 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:41.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa45f03700 1 --2- 172.21.15.2:0/2231899512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940558b60 0x55e940561a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa46704700 1 --2- 172.21.15.2:0/2231899512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e94055d8b0 0x55e94055b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa45702700 1 -- 172.21.15.2:0/2231899512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e94055d8b0 msgr2=0x55e94055b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:41.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa45702700 1 --2- 172.21.15.2:0/2231899512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e94055d8b0 0x55e94055b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa45702700 1 -- 172.21.15.2:0/2231899512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940558b60 msgr2=0x55e940561a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:41.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa45702700 1 --2- 172.21.15.2:0/2231899512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940558b60 0x55e940561a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa45702700 1 -- 172.21.15.2:0/2231899512 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e940552820 con 0x55e940559470 2023-11-16T18:30:41.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa45702700 1 --2- 172.21.15.2:0/2231899512 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 0x55e940564b80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7faa30011a00 tx=0x7faa3000bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad06787f538ad207 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:41.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa34ff9700 1 -- 172.21.15.2:0/2231899512 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3002b040 con 0x55e940559470 2023-11-16T18:30:41.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa34ff9700 1 -- 172.21.15.2:0/2231899512 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa30011db0 con 0x55e940559470 2023-11-16T18:30:41.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.057+0000 7faa34ff9700 1 -- 172.21.15.2:0/2231899512 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3002f910 con 0x55e940559470 2023-11-16T18:30:41.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.058+0000 7faa542f3b80 1 -- 172.21.15.2:0/2231899512 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 msgr2=0x55e940564b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:41.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.058+0000 7faa542f3b80 1 --2- 172.21.15.2:0/2231899512 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 0x55e940564b80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7faa30011a00 tx=0x7faa3000bf70 comp rx=0 tx=0).stop 2023-11-16T18:30:41.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.058+0000 7faa542f3b80 1 -- 172.21.15.2:0/2231899512 shutdown_connections 2023-11-16T18:30:41.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.058+0000 7faa542f3b80 1 --2- 172.21.15.2:0/2231899512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e94055d8b0 0x55e94055b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.058+0000 7faa542f3b80 1 --2- 172.21.15.2:0/2231899512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e940558b60 0x55e940561a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.058+0000 7faa542f3b80 1 --2- 172.21.15.2:0/2231899512 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940559470 0x55e940564b80 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.058+0000 7faa542f3b80 1 -- 172.21.15.2:0/2231899512 >> 172.21.15.2:0/2231899512 conn(0x55e94054e360 msgr2=0x55e940552c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:41.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.058+0000 7faa542f3b80 1 -- 172.21.15.2:0/2231899512 shutdown_connections 2023-11-16T18:30:41.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.058+0000 7faa542f3b80 1 -- 172.21.15.2:0/2231899512 wait complete. 2023-11-16T18:30:41.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.058+0000 7faa542f3b80 1 -- start start 2023-11-16T18:30:41.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa542f3b80 1 Processor -- start 2023-11-16T18:30:41.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa542f3b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940558b60 0x55e9405676d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa542f3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e940559470 0x55e940567c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa542f3b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e94055d8b0 0x55e9405c57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa542f3b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9405c5cf0 con 0x55e940558b60 2023-11-16T18:30:41.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa542f3b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9405c5e30 con 0x55e94055d8b0 2023-11-16T18:30:41.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa542f3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9405c5f70 con 0x55e940559470 2023-11-16T18:30:41.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa542f3b80 10 client.0 _ll_register_callbacks cb 0x7faa38009bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:30:41.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa45f03700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940558b60 0x55e9405676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa45f03700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940558b60 0x55e9405676d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52804/0 (socket says 172.21.15.2:52804) 2023-11-16T18:30:41.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa45702700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e940559470 0x55e940567c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa46704700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e94055d8b0 0x55e9405c57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa45f03700 1 -- 172.21.15.2:0/2921623125 learned_addr learned my addr 172.21.15.2:0/2921623125 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:41.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa45f03700 1 -- 172.21.15.2:0/2921623125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e940559470 msgr2=0x55e940567c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:41.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa45f03700 1 --2- 172.21.15.2:0/2921623125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e940559470 0x55e940567c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa45f03700 1 -- 172.21.15.2:0/2921623125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e94055d8b0 msgr2=0x55e9405c57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:41.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa45f03700 1 --2- 172.21.15.2:0/2921623125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e94055d8b0 0x55e9405c57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.059+0000 7faa45f03700 1 -- 172.21.15.2:0/2921623125 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e94032b630 con 0x55e940558b60 2023-11-16T18:30:41.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa45f03700 1 --2- 172.21.15.2:0/2921623125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940558b60 0x55e9405676d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00c2d0 tx=0x7faa3c095b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb82409afab95f0b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:41.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa297fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:30:41.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa297fa700 1 -- 172.21.15.2:0/2921623125 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c010880 con 0x55e940558b60 2023-11-16T18:30:41.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa297fa700 1 -- 172.21.15.2:0/2921623125 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7faa3c09a180 con 0x55e940558b60 2023-11-16T18:30:41.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa542f3b80 1 -- 172.21.15.2:0/2921623125 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faa30015070 con 0x55e940558b60 2023-11-16T18:30:41.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa297fa700 1 -- 172.21.15.2:0/2921623125 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa3c0109e0 con 0x55e940558b60 2023-11-16T18:30:41.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa542f3b80 10 client.16707 fetch_fsmap learned FSMap version 29 2023-11-16T18:30:41.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa542f3b80 1 -- 172.21.15.2:0/2921623125 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e9402d7de0 con 0x55e940558b60 2023-11-16T18:30:41.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa297fa700 1 -- 172.21.15.2:0/2921623125 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa3c063070 con 0x55e940558b60 2023-11-16T18:30:41.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa297fa700 1 -- 172.21.15.2:0/2921623125 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7faa3c0ca080 con 0x55e940558b60 2023-11-16T18:30:41.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa542f3b80 10 client.16707 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:30:41.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa542f3b80 10 client.16707 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:30:41.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa542f3b80 20 client.16707 populate_metadata read hostname 'smithi002' 2023-11-16T18:30:41.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa542f3b80 1 --2- 172.21.15.2:0/2921623125 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55e9405c76a0 0x55e9405c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:41.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa542f3b80 4 client.16707 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:30:41.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa542f3b80 1 -- 172.21.15.2:0/2921623125 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e940478660 con 0x55e9405c76a0 2023-11-16T18:30:41.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.060+0000 7faa45702700 1 --2- 172.21.15.2:0/2921623125 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55e9405c76a0 0x55e9405c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:41.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.061+0000 7faa297fa700 10 client.16707 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:41.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.061+0000 7faa45702700 1 --2- 172.21.15.2:0/2921623125 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55e9405c76a0 0x55e9405c9ab0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4aa5d0216d70a15a server_cookie=278e8717de54bf2a in_seq=0 out_seq=0 2023-11-16T18:30:41.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa297fa700 1 -- 172.21.15.2:0/2921623125 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55e940478660 con 0x55e9405c76a0 2023-11-16T18:30:41.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa297fa700 10 client.16707 handle_command_reply: tid=0 2023-11-16T18:30:41.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa542f3b80 1 client.16707 shutdown 2023-11-16T18:30:41.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa542f3b80 1 -- 172.21.15.2:0/2921623125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940558b60 msgr2=0x55e9405676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:41.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa542f3b80 1 --2- 172.21.15.2:0/2921623125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940558b60 0x55e9405676d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00c2d0 tx=0x7faa3c095b40 comp rx=0 tx=0).stop 2023-11-16T18:30:41.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa542f3b80 1 -- 172.21.15.2:0/2921623125 shutdown_connections 2023-11-16T18:30:41.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa542f3b80 1 --2- 172.21.15.2:0/2921623125 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55e9405c76a0 0x55e9405c9ab0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa542f3b80 1 --2- 172.21.15.2:0/2921623125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e94055d8b0 0x55e9405c57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa542f3b80 1 --2- 172.21.15.2:0/2921623125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e940558b60 0x55e9405676d0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa542f3b80 1 --2- 172.21.15.2:0/2921623125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e940559470 0x55e940567c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa542f3b80 1 -- 172.21.15.2:0/2921623125 >> 172.21.15.2:0/2921623125 conn(0x55e94054e360 msgr2=0x55e94055b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:41.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa297fa700 0 client.16707 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:41.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa542f3b80 1 -- 172.21.15.2:0/2921623125 shutdown_connections 2023-11-16T18:30:41.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa542f3b80 1 -- 172.21.15.2:0/2921623125 wait complete. 2023-11-16T18:30:41.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa542f3b80 20 client.16707 trim_cache size 0 max 16384 2023-11-16T18:30:41.102 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:30:41.102 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-16T18:30:41.102 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": { 2023-11-16T18:30:41.102 INFO:teuthology.orchestra.run.smithi002.stdout: "9607aa23-76ae-4198-9378-3740d8aff22c": { 2023-11-16T18:30:41.102 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/", 2023-11-16T18:30:41.102 INFO:teuthology.orchestra.run.smithi002.stdout: "tag": "9607aa23-76ae-4198-9378-3740d8aff22c", 2023-11-16T18:30:41.102 INFO:teuthology.orchestra.run.smithi002.stdout: "options": "recursive,force" 2023-11-16T18:30:41.102 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:30:41.103 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:30:41.103 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:30:41.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa48167700 1 -- 172.21.15.2:0/3824509529 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7faa2c0a12c0 msgr2=0x7faa2c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:41.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa48167700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7faa2c0a12c0 0x7faa2c0a3770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faa4010e050 tx=0x7faa3000b410 comp rx=0 tx=0).stop 2023-11-16T18:30:41.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa48167700 1 -- 172.21.15.2:0/3824509529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa400ffa70 msgr2=0x7faa4010c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:41.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa48167700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa400ffa70 0x7faa4010c860 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00e9a0 tx=0x7faa3c00ecb0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa48167700 1 -- 172.21.15.2:0/3824509529 shutdown_connections 2023-11-16T18:30:41.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa48167700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7faa2c0a12c0 0x7faa2c0a3770 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa48167700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa40105250 0x7faa401a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa48167700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa400ffa70 0x7faa4010c860 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa48167700 1 --2- 172.21.15.2:0/3824509529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa40100420 0x7faa4010cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:41.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa48167700 1 -- 172.21.15.2:0/3824509529 >> 172.21.15.2:0/3824509529 conn(0x7faa400763a0 msgr2=0x7faa400fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:41.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa48167700 1 -- 172.21.15.2:0/3824509529 shutdown_connections 2023-11-16T18:30:41.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:30:41.083+0000 7faa48167700 1 -- 172.21.15.2:0/3824509529 wait complete. 2023-11-16T18:30:41.105 INFO:tasks.cephfs.filesystem:scrub status for tag:9607aa23-76ae-4198-9378-3740d8aff22c - {'path': '/', 'tag': '9607aa23-76ae-4198-9378-3740d8aff22c', 'options': 'recursive,force'} 2023-11-16T18:30:42.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:42 smithi040 ceph-mon[100251]: pgmap v178: 129 pgs: 129 active+clean; 694 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 404 op/s 2023-11-16T18:30:42.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:42 smithi188 ceph-mon[100639]: pgmap v178: 129 pgs: 129 active+clean; 694 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 404 op/s 2023-11-16T18:30:42.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:42 smithi002 ceph-mon[98807]: pgmap v178: 129 pgs: 129 active+clean; 694 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 404 op/s 2023-11-16T18:30:44.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:44 smithi040 ceph-mon[100251]: pgmap v179: 129 pgs: 129 active+clean; 717 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.8 MiB/s wr, 556 op/s 2023-11-16T18:30:44.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:44 smithi188 ceph-mon[100639]: pgmap v179: 129 pgs: 129 active+clean; 717 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.8 MiB/s wr, 556 op/s 2023-11-16T18:30:44.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:44 smithi002 ceph-mon[98807]: pgmap v179: 129 pgs: 129 active+clean; 717 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.8 MiB/s wr, 556 op/s 2023-11-16T18:30:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:46 smithi040 ceph-mon[100251]: pgmap v180: 129 pgs: 129 active+clean; 717 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.0 MiB/s wr, 433 op/s 2023-11-16T18:30:46.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:46 smithi188 ceph-mon[100639]: pgmap v180: 129 pgs: 129 active+clean; 717 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.0 MiB/s wr, 433 op/s 2023-11-16T18:30:46.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:46 smithi002 ceph-mon[98807]: pgmap v180: 129 pgs: 129 active+clean; 717 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.0 MiB/s wr, 433 op/s 2023-11-16T18:30:47.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:47 smithi002 ceph-mon[98807]: pgmap v181: 129 pgs: 129 active+clean; 717 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.0 MiB/s wr, 433 op/s 2023-11-16T18:30:47.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:47 smithi040 ceph-mon[100251]: pgmap v181: 129 pgs: 129 active+clean; 717 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.0 MiB/s wr, 433 op/s 2023-11-16T18:30:47.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:47 smithi188 ceph-mon[100639]: pgmap v181: 129 pgs: 129 active+clean; 717 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.0 MiB/s wr, 433 op/s 2023-11-16T18:30:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:48 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:48 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:48.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:48 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:48.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:48 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:48.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:48 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:48.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:48 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:49.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:49 smithi002 ceph-mon[98807]: pgmap v182: 129 pgs: 129 active+clean; 738 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.8 MiB/s wr, 549 op/s 2023-11-16T18:30:49.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:49 smithi040 ceph-mon[100251]: pgmap v182: 129 pgs: 129 active+clean; 738 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.8 MiB/s wr, 549 op/s 2023-11-16T18:30:49.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:49 smithi188 ceph-mon[100639]: pgmap v182: 129 pgs: 129 active+clean; 738 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.8 MiB/s wr, 549 op/s 2023-11-16T18:30:50.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:50 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:50.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:50 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:50.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:50 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:50.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:50 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:50.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:50 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:50.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:50 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.440+0000 7f3695b25700 1 -- 172.21.15.2:0/3985214400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3690100560 msgr2=0x7f3690100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.440+0000 7f3695b25700 1 --2- 172.21.15.2:0/3985214400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3690100560 0x7f3690100930 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f36800099c0 tx=0x7f3680009cd0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.440+0000 7f3695b25700 1 -- 172.21.15.2:0/3985214400 shutdown_connections 2023-11-16T18:30:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.440+0000 7f3695b25700 1 --2- 172.21.15.2:0/3985214400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36901055a0 0x7f369010d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.440+0000 7f3695b25700 1 --2- 172.21.15.2:0/3985214400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3690100560 0x7f3690100930 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.440+0000 7f3695b25700 1 --2- 172.21.15.2:0/3985214400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3690100f00 0x7f3690104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.440+0000 7f3695b25700 1 -- 172.21.15.2:0/3985214400 >> 172.21.15.2:0/3985214400 conn(0x7f36900781e0 msgr2=0x7f36900785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.440+0000 7f3695b25700 1 -- 172.21.15.2:0/3985214400 shutdown_connections 2023-11-16T18:30:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.440+0000 7f3695b25700 1 -- 172.21.15.2:0/3985214400 wait complete. 2023-11-16T18:30:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.440+0000 7f3695b25700 1 Processor -- start 2023-11-16T18:30:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.440+0000 7f3695b25700 1 -- start start 2023-11-16T18:30:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f3695b25700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3690100560 0x7f3690072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f3695b25700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3690100f00 0x7f3690073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f3695b25700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36901055a0 0x7f36901a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f3695b25700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3690073b60 con 0x7f36901055a0 2023-11-16T18:30:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f3695b25700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36901a4b70 con 0x7f3690100f00 2023-11-16T18:30:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f3695b25700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36901a4cb0 con 0x7f3690100560 2023-11-16T18:30:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f368ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36901055a0 0x7f36901a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f368f7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3690100560 0x7f3690072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f368ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36901055a0 0x7f36901a4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52822/0 (socket says 172.21.15.2:52822) 2023-11-16T18:30:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f368ffff700 1 -- 172.21.15.2:0/2994035010 learned_addr learned my addr 172.21.15.2:0/2994035010 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f3686ffd700 1 --2- 172.21.15.2:0/2994035010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3690100f00 0x7f3690073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f368ffff700 1 -- 172.21.15.2:0/2994035010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3690100560 msgr2=0x7f3690072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f368ffff700 1 --2- 172.21.15.2:0/2994035010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3690100560 0x7f3690072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f368ffff700 1 -- 172.21.15.2:0/2994035010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3690100f00 msgr2=0x7f3690073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f368ffff700 1 --2- 172.21.15.2:0/2994035010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3690100f00 0x7f3690073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f368ffff700 1 -- 172.21.15.2:0/2994035010 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36800095f0 con 0x7f36901055a0 2023-11-16T18:30:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.441+0000 7f368ffff700 1 --2- 172.21.15.2:0/2994035010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36901055a0 0x7f36901a4630 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f368800ef30 tx=0x7f368800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.442+0000 7f368d7fa700 1 -- 172.21.15.2:0/2994035010 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688018070 con 0x7f36901055a0 2023-11-16T18:30:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.442+0000 7f368d7fa700 1 -- 172.21.15.2:0/2994035010 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f368800f040 con 0x7f36901055a0 2023-11-16T18:30:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.442+0000 7f3695b25700 1 -- 172.21.15.2:0/2994035010 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36901103c0 con 0x7f36901055a0 2023-11-16T18:30:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.442+0000 7f368d7fa700 1 -- 172.21.15.2:0/2994035010 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368800cd00 con 0x7f36901055a0 2023-11-16T18:30:51.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.442+0000 7f3695b25700 1 -- 172.21.15.2:0/2994035010 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3690110230 con 0x7f36901055a0 2023-11-16T18:30:51.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.444+0000 7f368d7fa700 1 -- 172.21.15.2:0/2994035010 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f3688003a00 con 0x7f36901055a0 2023-11-16T18:30:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.444+0000 7f368d7fa700 1 --2- 172.21.15.2:0/2994035010 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f367c0a12c0 0x7f367c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.445+0000 7f368d7fa700 1 -- 172.21.15.2:0/2994035010 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f36880ccbe0 con 0x7f36901055a0 2023-11-16T18:30:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.445+0000 7f368f7fe700 1 --2- 172.21.15.2:0/2994035010 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f367c0a12c0 0x7f367c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.445+0000 7f368f7fe700 1 --2- 172.21.15.2:0/2994035010 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f367c0a12c0 0x7f367c0a3770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3680009fd0 tx=0x7f3680003680 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.445+0000 7f36a1cb1b80 1 Processor -- start 2023-11-16T18:30:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.445+0000 7f36a1cb1b80 1 -- start start 2023-11-16T18:30:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f36a1cb1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba2576b0 0x5628ba4011e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f36a1cb1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401720 0x5628ba3fb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f36a1cb1b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba401b90 0x5628ba3fbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f36a1cb1b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628ba301000 con 0x5628ba401b90 2023-11-16T18:30:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f36a1cb1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628ba353840 con 0x5628ba401720 2023-11-16T18:30:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f36a1cb1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628ba34ccd0 con 0x5628ba2576b0 2023-11-16T18:30:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f368ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba401b90 0x5628ba3fbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f368f7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba2576b0 0x5628ba4011e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f368ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba401b90 0x5628ba3fbb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52830/0 (socket says 172.21.15.2:52830) 2023-11-16T18:30:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f368ffff700 1 -- 172.21.15.2:0/1972282157 learned_addr learned my addr 172.21.15.2:0/1972282157 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f3686ffd700 1 --2- 172.21.15.2:0/1972282157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401720 0x5628ba3fb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f368f7fe700 1 -- 172.21.15.2:0/1972282157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401720 msgr2=0x5628ba3fb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f368f7fe700 1 --2- 172.21.15.2:0/1972282157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401720 0x5628ba3fb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f368f7fe700 1 -- 172.21.15.2:0/1972282157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba401b90 msgr2=0x5628ba3fbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f368f7fe700 1 --2- 172.21.15.2:0/1972282157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba401b90 0x5628ba3fbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f368f7fe700 1 -- 172.21.15.2:0/1972282157 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628ba1ce5d0 con 0x5628ba2576b0 2023-11-16T18:30:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f368f7fe700 1 --2- 172.21.15.2:0/1972282157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba2576b0 0x5628ba4011e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f368000b580 tx=0x7f3680005050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c13055b893d9877c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.446+0000 7f3685ffb700 1 -- 172.21.15.2:0/1972282157 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368002a070 con 0x5628ba2576b0 2023-11-16T18:30:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.447+0000 7f3685ffb700 1 -- 172.21.15.2:0/1972282157 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f368001e040 con 0x5628ba2576b0 2023-11-16T18:30:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.447+0000 7f3685ffb700 1 -- 172.21.15.2:0/1972282157 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368002f8c0 con 0x5628ba2576b0 2023-11-16T18:30:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.447+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/1972282157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba2576b0 msgr2=0x5628ba4011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.447+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/1972282157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba2576b0 0x5628ba4011e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f368000b580 tx=0x7f3680005050 comp rx=0 tx=0).stop 2023-11-16T18:30:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.447+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/1972282157 shutdown_connections 2023-11-16T18:30:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.447+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/1972282157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba2576b0 0x5628ba4011e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.447+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/1972282157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba401b90 0x5628ba3fbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.447+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/1972282157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401720 0x5628ba3fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.447+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/1972282157 >> 172.21.15.2:0/1972282157 conn(0x5628ba3f0f50 msgr2=0x5628ba3f6270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.447+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/1972282157 shutdown_connections 2023-11-16T18:30:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.447+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/1972282157 wait complete. 2023-11-16T18:30:51.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f36a1cb1b80 1 -- start start 2023-11-16T18:30:51.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f36a1cb1b80 1 Processor -- start 2023-11-16T18:30:51.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f36a1cb1b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 0x5628ba40aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f36a1cb1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401720 0x5628ba40afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f36a1cb1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401b90 0x5628ba404d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f36a1cb1b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628ba2ee880 con 0x5628ba2576b0 2023-11-16T18:30:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f36a1cb1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628ba2ee520 con 0x5628ba401720 2023-11-16T18:30:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f36a1cb1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628ba1d08c0 con 0x5628ba401b90 2023-11-16T18:30:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f36a1cb1b80 10 client.0 _ll_register_callbacks cb 0x5628ba30c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:30:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f368f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 0x5628ba40aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f368f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 0x5628ba40aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52832/0 (socket says 172.21.15.2:52832) 2023-11-16T18:30:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f368f7fe700 1 -- 172.21.15.2:0/2026998632 learned_addr learned my addr 172.21.15.2:0/2026998632 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f368ffff700 1 --2- 172.21.15.2:0/2026998632 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401b90 0x5628ba404d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.448+0000 7f3686ffd700 1 --2- 172.21.15.2:0/2026998632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401720 0x5628ba40afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f368f7fe700 1 -- 172.21.15.2:0/2026998632 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401b90 msgr2=0x5628ba404d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f368f7fe700 1 --2- 172.21.15.2:0/2026998632 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401b90 0x5628ba404d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f368f7fe700 1 -- 172.21.15.2:0/2026998632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401720 msgr2=0x5628ba40afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f368f7fe700 1 --2- 172.21.15.2:0/2026998632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401720 0x5628ba40afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f368f7fe700 1 -- 172.21.15.2:0/2026998632 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369010d9b0 con 0x5628ba2576b0 2023-11-16T18:30:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f368f7fe700 1 --2- 172.21.15.2:0/2026998632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 0x5628ba40aa70 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f3680006010 tx=0x7f3680009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d84464278321fb7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f366effd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:30:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f366effd700 1 -- 172.21.15.2:0/2026998632 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368002a040 con 0x5628ba2576b0 2023-11-16T18:30:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f366effd700 1 -- 172.21.15.2:0/2026998632 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f368000b790 con 0x5628ba2576b0 2023-11-16T18:30:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/2026998632 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5628ba3ee710 con 0x5628ba2576b0 2023-11-16T18:30:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f366effd700 1 -- 172.21.15.2:0/2026998632 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f368000b950 con 0x5628ba2576b0 2023-11-16T18:30:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f36a1cb1b80 10 client.16719 fetch_fsmap learned FSMap version 29 2023-11-16T18:30:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/2026998632 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5628ba313130 con 0x5628ba2576b0 2023-11-16T18:30:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.449+0000 7f366effd700 1 -- 172.21.15.2:0/2026998632 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5628ba3ee710 con 0x5628ba2576b0 2023-11-16T18:30:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.450+0000 7f366effd700 1 -- 172.21.15.2:0/2026998632 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f3680023c20 con 0x5628ba2576b0 2023-11-16T18:30:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.450+0000 7f36a1cb1b80 10 client.16719 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:30:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.450+0000 7f36a1cb1b80 10 client.16719 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:30:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.450+0000 7f36a1cb1b80 20 client.16719 populate_metadata read hostname 'smithi002' 2023-11-16T18:30:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.450+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/2026998632 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5628ba414b30 0x5628ba416f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.450+0000 7f36a1cb1b80 4 client.16719 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:30:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.450+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/2026998632 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5628ba30f040 con 0x5628ba414b30 2023-11-16T18:30:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.450+0000 7f3686ffd700 1 --2- 172.21.15.2:0/2026998632 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5628ba414b30 0x5628ba416f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.450+0000 7f3686ffd700 1 --2- 172.21.15.2:0/2026998632 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5628ba414b30 0x5628ba416f40 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5c51440e458ec06 server_cookie=2afc2245726c27fb in_seq=0 out_seq=0 2023-11-16T18:30:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.450+0000 7f366effd700 10 client.16719 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:51.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f366effd700 1 -- 172.21.15.2:0/2026998632 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5628ba30f040 con 0x5628ba414b30 2023-11-16T18:30:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f366effd700 10 client.16719 handle_command_reply: tid=0 2023-11-16T18:30:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f36a1cb1b80 1 client.16719 shutdown 2023-11-16T18:30:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/2026998632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 msgr2=0x5628ba40aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/2026998632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 0x5628ba40aa70 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f3680006010 tx=0x7f3680009290 comp rx=0 tx=0).stop 2023-11-16T18:30:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/2026998632 shutdown_connections 2023-11-16T18:30:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/2026998632 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5628ba414b30 0x5628ba416f40 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/2026998632 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401b90 0x5628ba404d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f366effd700 0 client.16719 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/2026998632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 0x5628ba40aa70 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/2026998632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401720 0x5628ba40afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/2026998632 >> 172.21.15.2:0/2026998632 conn(0x5628ba3f0f50 msgr2=0x5628ba3f57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/2026998632 shutdown_connections 2023-11-16T18:30:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/2026998632 wait complete. 2023-11-16T18:30:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.457+0000 7f36a1cb1b80 20 client.16719 trim_cache size 0 max 16384 2023-11-16T18:30:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.473+0000 7f36a1cb1b80 1 Processor -- start 2023-11-16T18:30:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.473+0000 7f36a1cb1b80 1 -- start start 2023-11-16T18:30:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f36a1cb1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba2576b0 0x5628ba404cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f36a1cb1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401720 0x5628ba3f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f36a1cb1b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba401b90 0x5628ba3fdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f36a1cb1b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628ba3f6d60 con 0x5628ba401b90 2023-11-16T18:30:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f36a1cb1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628ba3f8030 con 0x5628ba2576b0 2023-11-16T18:30:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f36a1cb1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628ba400df0 con 0x5628ba401720 2023-11-16T18:30:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f368ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba401b90 0x5628ba3fdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f3686ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401720 0x5628ba3f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f368f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba2576b0 0x5628ba404cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f368ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba401b90 0x5628ba3fdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52844/0 (socket says 172.21.15.2:52844) 2023-11-16T18:30:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f368ffff700 1 -- 172.21.15.2:0/2334683570 learned_addr learned my addr 172.21.15.2:0/2334683570 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f3686ffd700 1 -- 172.21.15.2:0/2334683570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba2576b0 msgr2=0x5628ba404cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f3686ffd700 1 --2- 172.21.15.2:0/2334683570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba2576b0 0x5628ba404cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f3686ffd700 1 -- 172.21.15.2:0/2334683570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba401b90 msgr2=0x5628ba3fdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f3686ffd700 1 --2- 172.21.15.2:0/2334683570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba401b90 0x5628ba3fdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f3686ffd700 1 -- 172.21.15.2:0/2334683570 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628ba312e80 con 0x5628ba401720 2023-11-16T18:30:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f368f7fe700 1 --2- 172.21.15.2:0/2334683570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba2576b0 0x5628ba404cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:30:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.474+0000 7f3686ffd700 1 --2- 172.21.15.2:0/2334683570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401720 0x5628ba3f5f60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x5628ba417810 tx=0x7f3678009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ab325d0a424a785 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.475+0000 7f36867fc700 1 -- 172.21.15.2:0/2334683570 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367801b040 con 0x5628ba401720 2023-11-16T18:30:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.475+0000 7f36867fc700 1 -- 172.21.15.2:0/2334683570 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3678022070 con 0x5628ba401720 2023-11-16T18:30:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.475+0000 7f36867fc700 1 -- 172.21.15.2:0/2334683570 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3678015bf0 con 0x5628ba401720 2023-11-16T18:30:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.475+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/2334683570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401720 msgr2=0x5628ba3f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.475+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/2334683570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401720 0x5628ba3f5f60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x5628ba417810 tx=0x7f3678009190 comp rx=0 tx=0).stop 2023-11-16T18:30:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.475+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/2334683570 shutdown_connections 2023-11-16T18:30:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.475+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/2334683570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba401b90 0x5628ba3fdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.475+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/2334683570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba2576b0 0x5628ba404cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.475+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/2334683570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401720 0x5628ba3f5f60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.475+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/2334683570 >> 172.21.15.2:0/2334683570 conn(0x5628ba3f0f50 msgr2=0x5628ba4045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.475+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/2334683570 shutdown_connections 2023-11-16T18:30:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.475+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/2334683570 wait complete. 2023-11-16T18:30:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f36a1cb1b80 1 -- start start 2023-11-16T18:30:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f36a1cb1b80 1 Processor -- start 2023-11-16T18:30:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f36a1cb1b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 0x5628ba4087c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f36a1cb1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401720 0x5628ba408d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f36a1cb1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401b90 0x5628ba4679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f36a1cb1b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628ba409610 con 0x5628ba2576b0 2023-11-16T18:30:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f36a1cb1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628ba467ef0 con 0x5628ba401b90 2023-11-16T18:30:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f36a1cb1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628ba468030 con 0x5628ba401720 2023-11-16T18:30:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f36a1cb1b80 10 client.0 _ll_register_callbacks cb 0x7f3688009370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:30:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f368f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 0x5628ba4087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f368f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 0x5628ba4087c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52846/0 (socket says 172.21.15.2:52846) 2023-11-16T18:30:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f368f7fe700 1 -- 172.21.15.2:0/4008244825 learned_addr learned my addr 172.21.15.2:0/4008244825 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:30:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f368ffff700 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401b90 0x5628ba4679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.476+0000 7f3686ffd700 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401720 0x5628ba408d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f368f7fe700 1 -- 172.21.15.2:0/4008244825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401720 msgr2=0x5628ba408d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f368f7fe700 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401720 0x5628ba408d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f368f7fe700 1 -- 172.21.15.2:0/4008244825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401b90 msgr2=0x5628ba4679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:30:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f368f7fe700 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401b90 0x5628ba4679b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f368f7fe700 1 -- 172.21.15.2:0/4008244825 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628ba313130 con 0x5628ba2576b0 2023-11-16T18:30:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f368ffff700 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401b90 0x5628ba4679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:30:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f3686ffd700 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401720 0x5628ba408d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:30:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f368f7fe700 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 0x5628ba4087c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f3680006010 tx=0x7f36800395b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf537c3706b51ffd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:30:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f366effd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:30:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f366effd700 1 -- 172.21.15.2:0/4008244825 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368002a040 con 0x5628ba2576b0 2023-11-16T18:30:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f366effd700 1 -- 172.21.15.2:0/4008244825 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f3680010680 con 0x5628ba2576b0 2023-11-16T18:30:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/4008244825 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f368001e040 con 0x5628ba2576b0 2023-11-16T18:30:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f366effd700 1 -- 172.21.15.2:0/4008244825 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3680010af0 con 0x5628ba2576b0 2023-11-16T18:30:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f36a1cb1b80 10 client.16725 fetch_fsmap learned FSMap version 29 2023-11-16T18:30:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/4008244825 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f3678009960 con 0x5628ba2576b0 2023-11-16T18:30:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.477+0000 7f366effd700 1 -- 172.21.15.2:0/4008244825 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f368001e040 con 0x5628ba2576b0 2023-11-16T18:30:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.478+0000 7f366effd700 1 -- 172.21.15.2:0/4008244825 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f3680039720 con 0x5628ba2576b0 2023-11-16T18:30:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.478+0000 7f36a1cb1b80 10 client.16725 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:30:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.478+0000 7f36a1cb1b80 10 client.16725 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:30:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.478+0000 7f36a1cb1b80 20 client.16725 populate_metadata read hostname 'smithi002' 2023-11-16T18:30:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.478+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5628ba469280 0x5628ba46b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:30:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.478+0000 7f36a1cb1b80 4 client.16725 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:30:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.478+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/4008244825 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5628ba31b600 con 0x5628ba469280 2023-11-16T18:30:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.478+0000 7f3686ffd700 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5628ba469280 0x5628ba46b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:30:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.478+0000 7f366effd700 10 client.16725 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:51.478+0000 7f3686ffd700 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5628ba469280 0x5628ba46b660 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fe6aa0941b3fcc server_cookie=e7a0b3f5308e238c in_seq=0 out_seq=0 2023-11-16T18:30:51.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:51 smithi188 ceph-mon[100639]: pgmap v183: 129 pgs: 129 active+clean; 738 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 268 op/s 2023-11-16T18:30:51.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:51 smithi002 ceph-mon[98807]: pgmap v183: 129 pgs: 129 active+clean; 738 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 268 op/s 2023-11-16T18:30:51.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:51 smithi040 ceph-mon[100251]: pgmap v183: 129 pgs: 129 active+clean; 738 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 268 op/s 2023-11-16T18:30:54.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:54 smithi040 ceph-mon[100251]: pgmap v184: 129 pgs: 129 active+clean; 747 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.8 MiB/s wr, 443 op/s 2023-11-16T18:30:59.310 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:54 smithi002 ceph-mon[98807]: pgmap v184: 129 pgs: 129 active+clean; 747 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.8 MiB/s wr, 443 op/s 2023-11-16T18:30:59.311 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:55 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:59.311 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:55 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:59.311 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:56 smithi040 ceph-mon[100251]: scrub summary: active paths [/] 2023-11-16T18:30:59.312 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:56 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:59.312 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:56 smithi040 ceph-mon[100251]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:59.312 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:56 smithi040 ceph-mon[100251]: pgmap v185: 129 pgs: 129 active+clean; 747 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 291 op/s 2023-11-16T18:30:59.312 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:30:57 smithi040 ceph-mon[100251]: pgmap v186: 129 pgs: 129 active+clean; 747 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 291 op/s 2023-11-16T18:30:59.312 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:54 smithi188 ceph-mon[100639]: pgmap v184: 129 pgs: 129 active+clean; 747 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.8 MiB/s wr, 443 op/s 2023-11-16T18:30:59.312 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:55 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:59.312 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:55 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:59.312 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:56 smithi188 ceph-mon[100639]: scrub summary: active paths [/] 2023-11-16T18:30:59.312 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:56 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:59.312 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:56 smithi188 ceph-mon[100639]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:59.312 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:56 smithi188 ceph-mon[100639]: pgmap v185: 129 pgs: 129 active+clean; 747 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 291 op/s 2023-11-16T18:30:59.313 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:30:57 smithi188 ceph-mon[100639]: pgmap v186: 129 pgs: 129 active+clean; 747 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 291 op/s 2023-11-16T18:30:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.911+0000 7f366effd700 1 -- 172.21.15.2:0/4008244825 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5628ba31b600 con 0x5628ba469280 2023-11-16T18:30:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.911+0000 7f366effd700 10 client.16725 handle_command_reply: tid=0 2023-11-16T18:30:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.911+0000 7f36a1cb1b80 1 client.16725 shutdown 2023-11-16T18:30:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.912+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/4008244825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 msgr2=0x5628ba4087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.912+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 0x5628ba4087c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f3680006010 tx=0x7f36800395b0 comp rx=0 tx=0).stop 2023-11-16T18:30:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.912+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/4008244825 shutdown_connections 2023-11-16T18:30:59.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.912+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5628ba469280 0x5628ba46b660 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:59.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.912+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628ba401b90 0x5628ba4679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:59.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.912+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5628ba2576b0 0x5628ba4087c0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.912+0000 7f36a1cb1b80 1 --2- 172.21.15.2:0/4008244825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628ba401720 0x5628ba408d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.912+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/4008244825 >> 172.21.15.2:0/4008244825 conn(0x5628ba3f0f50 msgr2=0x5628ba3ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.912+0000 7f366effd700 0 client.16725 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:30:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.912+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/4008244825 shutdown_connections 2023-11-16T18:30:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.912+0000 7f36a1cb1b80 1 -- 172.21.15.2:0/4008244825 wait complete. 2023-11-16T18:30:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.913+0000 7f36a1cb1b80 20 client.16725 trim_cache size 0 max 16384 2023-11-16T18:30:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:30:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:30:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:30:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:30:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.916+0000 7f3695b25700 1 -- 172.21.15.2:0/2994035010 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f367c0a12c0 msgr2=0x7f367c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.916+0000 7f3695b25700 1 --2- 172.21.15.2:0/2994035010 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f367c0a12c0 0x7f367c0a3770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3680009fd0 tx=0x7f3680003680 comp rx=0 tx=0).stop 2023-11-16T18:30:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.916+0000 7f3695b25700 1 -- 172.21.15.2:0/2994035010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36901055a0 msgr2=0x7f36901a4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:30:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.916+0000 7f3695b25700 1 --2- 172.21.15.2:0/2994035010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36901055a0 0x7f36901a4630 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f368800ef30 tx=0x7f368800c3a0 comp rx=0 tx=0).stop 2023-11-16T18:30:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.916+0000 7f3695b25700 1 -- 172.21.15.2:0/2994035010 shutdown_connections 2023-11-16T18:30:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.916+0000 7f3695b25700 1 --2- 172.21.15.2:0/2994035010 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f367c0a12c0 0x7f367c0a3770 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.916+0000 7f3695b25700 1 --2- 172.21.15.2:0/2994035010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3690100560 0x7f3690072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.916+0000 7f3695b25700 1 --2- 172.21.15.2:0/2994035010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36901055a0 0x7f36901a4630 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.916+0000 7f3695b25700 1 --2- 172.21.15.2:0/2994035010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3690100f00 0x7f3690073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:30:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.916+0000 7f3695b25700 1 -- 172.21.15.2:0/2994035010 >> 172.21.15.2:0/2994035010 conn(0x7f36900781e0 msgr2=0x7f3690103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:30:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.916+0000 7f3695b25700 1 -- 172.21.15.2:0/2994035010 shutdown_connections 2023-11-16T18:30:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:30:54.916+0000 7f3695b25700 1 -- 172.21.15.2:0/2994035010 wait complete. 2023-11-16T18:30:59.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:55 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:59.473 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:55 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:59.473 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:56 smithi002 ceph-mon[98807]: scrub summary: active paths [/] 2023-11-16T18:30:59.473 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:56 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:59.473 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:56 smithi002 ceph-mon[98807]: scrub summary: idle+waiting paths [/] 2023-11-16T18:30:59.473 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:56 smithi002 ceph-mon[98807]: pgmap v185: 129 pgs: 129 active+clean; 747 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 291 op/s 2023-11-16T18:30:59.473 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:30:57 smithi002 ceph-mon[98807]: pgmap v186: 129 pgs: 129 active+clean; 747 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 291 op/s 2023-11-16T18:31:00.487 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:00 smithi040 ceph-mon[100251]: pgmap v187: 129 pgs: 129 active+clean; 700 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.7 MiB/s wr, 677 op/s 2023-11-16T18:31:00.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:00 smithi188 ceph-mon[100639]: pgmap v187: 129 pgs: 129 active+clean; 700 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.7 MiB/s wr, 677 op/s 2023-11-16T18:31:00.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:00 smithi002 ceph-mon[98807]: pgmap v187: 129 pgs: 129 active+clean; 700 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.7 MiB/s wr, 677 op/s 2023-11-16T18:31:02.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:02 smithi040 ceph-mon[100251]: pgmap v188: 129 pgs: 129 active+clean; 700 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 561 op/s 2023-11-16T18:31:02.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:02 smithi188 ceph-mon[100639]: pgmap v188: 129 pgs: 129 active+clean; 700 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 561 op/s 2023-11-16T18:31:02.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:02 smithi002 ceph-mon[98807]: pgmap v188: 129 pgs: 129 active+clean; 700 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 561 op/s 2023-11-16T18:31:04.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:04 smithi040 ceph-mon[100251]: pgmap v189: 129 pgs: 129 active+clean; 725 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.4 MiB/s wr, 747 op/s 2023-11-16T18:31:04.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:04 smithi188 ceph-mon[100639]: pgmap v189: 129 pgs: 129 active+clean; 725 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.4 MiB/s wr, 747 op/s 2023-11-16T18:31:04.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:04 smithi002 ceph-mon[98807]: pgmap v189: 129 pgs: 129 active+clean; 725 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.4 MiB/s wr, 747 op/s 2023-11-16T18:31:06.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:06 smithi040 ceph-mon[100251]: pgmap v190: 129 pgs: 129 active+clean; 725 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 571 op/s 2023-11-16T18:31:06.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:06 smithi188 ceph-mon[100639]: pgmap v190: 129 pgs: 129 active+clean; 725 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 571 op/s 2023-11-16T18:31:06.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:06 smithi002 ceph-mon[98807]: pgmap v190: 129 pgs: 129 active+clean; 725 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 571 op/s 2023-11-16T18:31:07.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:07 smithi040 ceph-mon[100251]: scrub summary: idle 2023-11-16T18:31:07.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:07 smithi040 ceph-mon[100251]: scrub summary: idle 2023-11-16T18:31:07.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:07 smithi040 ceph-mon[100251]: scrub summary: idle 2023-11-16T18:31:07.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:07 smithi188 ceph-mon[100639]: scrub summary: idle 2023-11-16T18:31:07.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:07 smithi188 ceph-mon[100639]: scrub summary: idle 2023-11-16T18:31:07.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:07 smithi188 ceph-mon[100639]: scrub summary: idle 2023-11-16T18:31:07.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:07 smithi002 ceph-mon[98807]: scrub summary: idle 2023-11-16T18:31:07.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:07 smithi002 ceph-mon[98807]: scrub summary: idle 2023-11-16T18:31:07.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:07 smithi002 ceph-mon[98807]: scrub summary: idle 2023-11-16T18:31:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.044+0000 7f434c85a700 1 -- 172.21.15.2:0/2559954572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4348104440 msgr2=0x7f434810c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.044+0000 7f434c85a700 1 --2- 172.21.15.2:0/2559954572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4348104440 0x7f434810c010 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f434000b0d0 tx=0x7f434000b3e0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.044+0000 7f434c85a700 1 -- 172.21.15.2:0/2559954572 shutdown_connections 2023-11-16T18:31:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.044+0000 7f434c85a700 1 --2- 172.21.15.2:0/2559954572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4348104440 0x7f434810c010 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.044+0000 7f434c85a700 1 --2- 172.21.15.2:0/2559954572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43480ff400 0x7f43480ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.044+0000 7f434c85a700 1 --2- 172.21.15.2:0/2559954572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43480ffda0 0x7f4348103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.044+0000 7f434c85a700 1 -- 172.21.15.2:0/2559954572 >> 172.21.15.2:0/2559954572 conn(0x7f4348075fa0 msgr2=0x7f43480763a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.044+0000 7f434c85a700 1 -- 172.21.15.2:0/2559954572 shutdown_connections 2023-11-16T18:31:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.044+0000 7f434c85a700 1 -- 172.21.15.2:0/2559954572 wait complete. 2023-11-16T18:31:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.045+0000 7f434c85a700 1 Processor -- start 2023-11-16T18:31:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.045+0000 7f434c85a700 1 -- start start 2023-11-16T18:31:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f434c85a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43480ff400 0x7f43481950e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f434c85a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43480ffda0 0x7f4348195620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f434c85a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4348104440 0x7f434819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f434c85a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434819a670 con 0x7f4348104440 2023-11-16T18:31:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f434c85a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434819a7e0 con 0x7f43480ffda0 2023-11-16T18:31:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f434c85a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434819a9a0 con 0x7f43480ff400 2023-11-16T18:31:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f4345d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43480ffda0 0x7f4348195620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f4346d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4348104440 0x7f434819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f4345d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43480ffda0 0x7f4348195620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43300/0 (socket says 172.21.15.2:43300) 2023-11-16T18:31:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f4345d9b700 1 -- 172.21.15.2:0/2439253258 learned_addr learned my addr 172.21.15.2:0/2439253258 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f4346d9d700 1 -- 172.21.15.2:0/2439253258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43480ff400 msgr2=0x7f43481950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f4346d9d700 1 --2- 172.21.15.2:0/2439253258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43480ff400 0x7f43481950e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f4346d9d700 1 -- 172.21.15.2:0/2439253258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43480ffda0 msgr2=0x7f4348195620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f4346d9d700 1 --2- 172.21.15.2:0/2439253258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43480ffda0 0x7f4348195620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f4346d9d700 1 -- 172.21.15.2:0/2439253258 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4340009d00 con 0x7f4348104440 2023-11-16T18:31:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f4345d9b700 1 --2- 172.21.15.2:0/2439253258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43480ffda0 0x7f4348195620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.046+0000 7f4346d9d700 1 --2- 172.21.15.2:0/2439253258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4348104440 0x7f434819a130 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f434000b770 tx=0x7f4340007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.047+0000 7f433f7fe700 1 -- 172.21.15.2:0/2439253258 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434000e040 con 0x7f4348104440 2023-11-16T18:31:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.047+0000 7f433f7fe700 1 -- 172.21.15.2:0/2439253258 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f434001d930 con 0x7f4348104440 2023-11-16T18:31:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.047+0000 7f434c85a700 1 -- 172.21.15.2:0/2439253258 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434810f210 con 0x7f4348104440 2023-11-16T18:31:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.047+0000 7f433f7fe700 1 -- 172.21.15.2:0/2439253258 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434001b920 con 0x7f4348104440 2023-11-16T18:31:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.047+0000 7f434c85a700 1 -- 172.21.15.2:0/2439253258 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434810f080 con 0x7f4348104440 2023-11-16T18:31:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.049+0000 7f433f7fe700 1 -- 172.21.15.2:0/2439253258 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f434001daa0 con 0x7f4348104440 2023-11-16T18:31:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.049+0000 7f433f7fe700 1 --2- 172.21.15.2:0/2439253258 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f43340a12c0 0x7f43340a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.050+0000 7f433f7fe700 1 -- 172.21.15.2:0/2439253258 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f43400ce400 con 0x7f4348104440 2023-11-16T18:31:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.050+0000 7f434659c700 1 --2- 172.21.15.2:0/2439253258 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f43340a12c0 0x7f43340a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.050+0000 7f434659c700 1 --2- 172.21.15.2:0/2439253258 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f43340a12c0 0x7f43340a3770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4338005950 tx=0x7f43380058e0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.050+0000 7f43589e6b80 1 Processor -- start 2023-11-16T18:31:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.050+0000 7f43589e6b80 1 -- start start 2023-11-16T18:31:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f43589e6b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963aaf6b0 0x555963c591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f43589e6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 0x555963c53560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f43589e6b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59b90 0x555963c53b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f43589e6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555963b59000 con 0x555963c59720 2023-11-16T18:31:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f43589e6b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555963bab840 con 0x555963c59b90 2023-11-16T18:31:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f43589e6b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555963ba4cd0 con 0x555963aaf6b0 2023-11-16T18:31:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f4345d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 0x555963c53560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f4345d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 0x555963c53560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52870/0 (socket says 172.21.15.2:52870) 2023-11-16T18:31:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f4345d9b700 1 -- 172.21.15.2:0/849129961 learned_addr learned my addr 172.21.15.2:0/849129961 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f434659c700 1 --2- 172.21.15.2:0/849129961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963aaf6b0 0x555963c591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f4346d9d700 1 --2- 172.21.15.2:0/849129961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59b90 0x555963c53b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f4345d9b700 1 -- 172.21.15.2:0/849129961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963aaf6b0 msgr2=0x555963c591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f4345d9b700 1 --2- 172.21.15.2:0/849129961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963aaf6b0 0x555963c591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f4345d9b700 1 -- 172.21.15.2:0/849129961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59b90 msgr2=0x555963c53b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f4345d9b700 1 --2- 172.21.15.2:0/849129961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59b90 0x555963c53b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f4345d9b700 1 -- 172.21.15.2:0/849129961 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555963a265d0 con 0x555963c59720 2023-11-16T18:31:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f4346d9d700 1 --2- 172.21.15.2:0/849129961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59b90 0x555963c53b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:31:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f4345d9b700 1 --2- 172.21.15.2:0/849129961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 0x555963c53560 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f4330010450 tx=0x7f4330010760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6734f013ec2953e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f433cff9700 1 -- 172.21.15.2:0/849129961 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330004020 con 0x555963c59720 2023-11-16T18:31:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f433cff9700 1 -- 172.21.15.2:0/849129961 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4330015070 con 0x555963c59720 2023-11-16T18:31:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.051+0000 7f433cff9700 1 -- 172.21.15.2:0/849129961 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43300053a0 con 0x555963c59720 2023-11-16T18:31:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.052+0000 7f43589e6b80 1 -- 172.21.15.2:0/849129961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 msgr2=0x555963c53560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.052+0000 7f43589e6b80 1 --2- 172.21.15.2:0/849129961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 0x555963c53560 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f4330010450 tx=0x7f4330010760 comp rx=0 tx=0).stop 2023-11-16T18:31:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.052+0000 7f43589e6b80 1 -- 172.21.15.2:0/849129961 shutdown_connections 2023-11-16T18:31:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.052+0000 7f43589e6b80 1 --2- 172.21.15.2:0/849129961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963aaf6b0 0x555963c591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.052+0000 7f43589e6b80 1 --2- 172.21.15.2:0/849129961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59b90 0x555963c53b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.052+0000 7f43589e6b80 1 --2- 172.21.15.2:0/849129961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 0x555963c53560 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.052+0000 7f43589e6b80 1 -- 172.21.15.2:0/849129961 >> 172.21.15.2:0/849129961 conn(0x555963c48f50 msgr2=0x555963c4e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.052+0000 7f43589e6b80 1 -- 172.21.15.2:0/849129961 shutdown_connections 2023-11-16T18:31:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.052+0000 7f43589e6b80 1 -- 172.21.15.2:0/849129961 wait complete. 2023-11-16T18:31:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f43589e6b80 1 -- start start 2023-11-16T18:31:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f43589e6b80 1 Processor -- start 2023-11-16T18:31:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f43589e6b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963aaf6b0 0x555963c5fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f43589e6b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59720 0x555963c5cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f43589e6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59b90 0x555963c5d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f43589e6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555963b46880 con 0x555963c59b90 2023-11-16T18:31:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f43589e6b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555963b46520 con 0x555963c59720 2023-11-16T18:31:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f43589e6b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555963a288c0 con 0x555963aaf6b0 2023-11-16T18:31:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f43589e6b80 10 client.0 _ll_register_callbacks cb 0x555963b64720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f4346d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59b90 0x555963c5d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f4346d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59b90 0x555963c5d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52876/0 (socket says 172.21.15.2:52876) 2023-11-16T18:31:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f4346d9d700 1 -- 172.21.15.2:0/2552040564 learned_addr learned my addr 172.21.15.2:0/2552040564 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f434659c700 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963aaf6b0 0x555963c5fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.053+0000 7f4345d9b700 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59720 0x555963c5cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f4346d9d700 1 -- 172.21.15.2:0/2552040564 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963aaf6b0 msgr2=0x555963c5fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f4346d9d700 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963aaf6b0 0x555963c5fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f4346d9d700 1 -- 172.21.15.2:0/2552040564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59720 msgr2=0x555963c5cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f4346d9d700 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59720 0x555963c5cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f4346d9d700 1 -- 172.21.15.2:0/2552040564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4330007c80 con 0x555963c59b90 2023-11-16T18:31:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f434659c700 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963aaf6b0 0x555963c5fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:31:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f4346d9d700 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59b90 0x555963c5d470 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f4340026000 tx=0x7f434000b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe8779c5e67d5c74 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f432dffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:31:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f432dffb700 1 -- 172.21.15.2:0/2552040564 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400a10f0 con 0x555963c59b90 2023-11-16T18:31:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f432dffb700 1 -- 172.21.15.2:0/2552040564 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f43400a35b0 con 0x555963c59b90 2023-11-16T18:31:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f43589e6b80 1 -- 172.21.15.2:0/2552040564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555963c46710 con 0x555963c59b90 2023-11-16T18:31:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f432dffb700 1 -- 172.21.15.2:0/2552040564 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4340099590 con 0x555963c59b90 2023-11-16T18:31:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f43589e6b80 10 client.16737 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f43589e6b80 1 -- 172.21.15.2:0/2552040564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x555963b6b130 con 0x555963c59b90 2023-11-16T18:31:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f432dffb700 1 -- 172.21.15.2:0/2552040564 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555963c46710 con 0x555963c59b90 2023-11-16T18:31:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f432dffb700 1 -- 172.21.15.2:0/2552040564 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f434009a6e0 con 0x555963c59b90 2023-11-16T18:31:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f43589e6b80 10 client.16737 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f43589e6b80 10 client.16737 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:31:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f43589e6b80 20 client.16737 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.054+0000 7f43589e6b80 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555963c6cd40 0x555963c6f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.055+0000 7f43589e6b80 4 client.16737 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:31:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.055+0000 7f43589e6b80 1 -- 172.21.15.2:0/2552040564 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555963b67040 con 0x555963c6cd40 2023-11-16T18:31:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.055+0000 7f434659c700 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555963c6cd40 0x555963c6f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.055+0000 7f434659c700 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555963c6cd40 0x555963c6f150 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f3e91a3ec9e5d4a server_cookie=d73b44d70b5f5869 in_seq=0 out_seq=0 2023-11-16T18:31:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.055+0000 7f432dffb700 10 client.16737 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f432dffb700 1 -- 172.21.15.2:0/2552040564 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555963b67040 con 0x555963c6cd40 2023-11-16T18:31:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f432dffb700 10 client.16737 handle_command_reply: tid=0 2023-11-16T18:31:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f43589e6b80 1 client.16737 shutdown 2023-11-16T18:31:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f43589e6b80 1 -- 172.21.15.2:0/2552040564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59b90 msgr2=0x555963c5d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f43589e6b80 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59b90 0x555963c5d470 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f4340026000 tx=0x7f434000b3e0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f43589e6b80 1 -- 172.21.15.2:0/2552040564 shutdown_connections 2023-11-16T18:31:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f43589e6b80 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555963c6cd40 0x555963c6f150 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f43589e6b80 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963aaf6b0 0x555963c5fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f43589e6b80 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59b90 0x555963c5d470 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f432dffb700 0 client.16737 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f43589e6b80 1 --2- 172.21.15.2:0/2552040564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59720 0x555963c5cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f43589e6b80 1 -- 172.21.15.2:0/2552040564 >> 172.21.15.2:0/2552040564 conn(0x555963c48f50 msgr2=0x555963c4d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f43589e6b80 1 -- 172.21.15.2:0/2552040564 shutdown_connections 2023-11-16T18:31:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f43589e6b80 1 -- 172.21.15.2:0/2552040564 wait complete. 2023-11-16T18:31:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.056+0000 7f43589e6b80 20 client.16737 trim_cache size 0 max 16384 2023-11-16T18:31:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.071+0000 7f43589e6b80 1 Processor -- start 2023-11-16T18:31:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.071+0000 7f43589e6b80 1 -- start start 2023-11-16T18:31:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.071+0000 7f43589e6b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963aaf6b0 0x555963c5cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.071+0000 7f43589e6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 0x555963c4df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.071+0000 7f43589e6b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963c59b90 0x555963c57e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.071+0000 7f43589e6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555963c53560 con 0x555963c59720 2023-11-16T18:31:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.071+0000 7f43589e6b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555963c6f7b0 con 0x555963aaf6b0 2023-11-16T18:31:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.071+0000 7f43589e6b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555963c57a10 con 0x555963c59b90 2023-11-16T18:31:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f4345d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 0x555963c4df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f4345d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 0x555963c4df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52882/0 (socket says 172.21.15.2:52882) 2023-11-16T18:31:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f4345d9b700 1 -- 172.21.15.2:0/605449699 learned_addr learned my addr 172.21.15.2:0/605449699 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f434659c700 1 --2- 172.21.15.2:0/605449699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963aaf6b0 0x555963c5cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f4346d9d700 1 --2- 172.21.15.2:0/605449699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963c59b90 0x555963c57e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f4345d9b700 1 -- 172.21.15.2:0/605449699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963c59b90 msgr2=0x555963c57e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f4345d9b700 1 --2- 172.21.15.2:0/605449699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963c59b90 0x555963c57e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f4345d9b700 1 -- 172.21.15.2:0/605449699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963aaf6b0 msgr2=0x555963c5cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f4345d9b700 1 --2- 172.21.15.2:0/605449699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963aaf6b0 0x555963c5cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f4345d9b700 1 -- 172.21.15.2:0/605449699 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555963c4a6f0 con 0x555963c59720 2023-11-16T18:31:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f4345d9b700 1 --2- 172.21.15.2:0/605449699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 0x555963c4df60 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4330007f70 tx=0x7f43300071e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2eed5f6d089b112 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f433d7fa700 1 -- 172.21.15.2:0/605449699 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330020040 con 0x555963c59720 2023-11-16T18:31:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f433d7fa700 1 -- 172.21.15.2:0/605449699 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4330005700 con 0x555963c59720 2023-11-16T18:31:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.072+0000 7f433d7fa700 1 -- 172.21.15.2:0/605449699 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433001a6e0 con 0x555963c59720 2023-11-16T18:31:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.073+0000 7f43589e6b80 1 -- 172.21.15.2:0/605449699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 msgr2=0x555963c4df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.073+0000 7f43589e6b80 1 --2- 172.21.15.2:0/605449699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 0x555963c4df60 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4330007f70 tx=0x7f43300071e0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.073+0000 7f43589e6b80 1 -- 172.21.15.2:0/605449699 shutdown_connections 2023-11-16T18:31:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.073+0000 7f43589e6b80 1 --2- 172.21.15.2:0/605449699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963c59b90 0x555963c57e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.073+0000 7f43589e6b80 1 --2- 172.21.15.2:0/605449699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963aaf6b0 0x555963c5cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.073+0000 7f43589e6b80 1 --2- 172.21.15.2:0/605449699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963c59720 0x555963c4df60 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.073+0000 7f43589e6b80 1 -- 172.21.15.2:0/605449699 >> 172.21.15.2:0/605449699 conn(0x555963c48f50 msgr2=0x555963c5c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.073+0000 7f43589e6b80 1 -- 172.21.15.2:0/605449699 shutdown_connections 2023-11-16T18:31:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.073+0000 7f43589e6b80 1 -- 172.21.15.2:0/605449699 wait complete. 2023-11-16T18:31:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.073+0000 7f43589e6b80 1 -- start start 2023-11-16T18:31:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.073+0000 7f43589e6b80 1 Processor -- start 2023-11-16T18:31:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f43589e6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963aaf6b0 0x555963c62050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f43589e6b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963c59720 0x555963c62590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f43589e6b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59b90 0x555963c60650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f43589e6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555963c60d20 con 0x555963aaf6b0 2023-11-16T18:31:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f43589e6b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555963c60e90 con 0x555963c59b90 2023-11-16T18:31:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f43589e6b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555963c61050 con 0x555963c59720 2023-11-16T18:31:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f43589e6b80 10 client.0 _ll_register_callbacks cb 0x7f434009b0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f434659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963aaf6b0 0x555963c62050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f434659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963aaf6b0 0x555963c62050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52886/0 (socket says 172.21.15.2:52886) 2023-11-16T18:31:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f434659c700 1 -- 172.21.15.2:0/183251208 learned_addr learned my addr 172.21.15.2:0/183251208 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f4345d9b700 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963c59720 0x555963c62590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f4346d9d700 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59b90 0x555963c60650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f434659c700 1 -- 172.21.15.2:0/183251208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963c59720 msgr2=0x555963c62590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f434659c700 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963c59720 0x555963c62590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f434659c700 1 -- 172.21.15.2:0/183251208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59b90 msgr2=0x555963c60650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f434659c700 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59b90 0x555963c60650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f434659c700 1 -- 172.21.15.2:0/183251208 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555963a265d0 con 0x555963aaf6b0 2023-11-16T18:31:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f4346d9d700 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59b90 0x555963c60650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:31:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f434659c700 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963aaf6b0 0x555963c62050 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x555963c6f9f0 tx=0x7f4338009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7484305552f378fa server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f432dffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:31:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f432dffb700 1 -- 172.21.15.2:0/183251208 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4338023050 con 0x555963aaf6b0 2023-11-16T18:31:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f432dffb700 1 -- 172.21.15.2:0/183251208 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f433802b070 con 0x555963aaf6b0 2023-11-16T18:31:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f43589e6b80 1 -- 172.21.15.2:0/183251208 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555963b73600 con 0x555963aaf6b0 2023-11-16T18:31:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.074+0000 7f432dffb700 1 -- 172.21.15.2:0/183251208 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4338025420 con 0x555963aaf6b0 2023-11-16T18:31:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.075+0000 7f43589e6b80 10 client.16743 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.075+0000 7f43589e6b80 1 -- 172.21.15.2:0/183251208 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5559639d2d80 con 0x555963aaf6b0 2023-11-16T18:31:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.075+0000 7f432dffb700 1 -- 172.21.15.2:0/183251208 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555963b73600 con 0x555963aaf6b0 2023-11-16T18:31:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.075+0000 7f432dffb700 1 -- 172.21.15.2:0/183251208 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f43380052a0 con 0x555963aaf6b0 2023-11-16T18:31:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.075+0000 7f43589e6b80 10 client.16743 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.075+0000 7f43589e6b80 10 client.16743 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:31:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.075+0000 7f43589e6b80 20 client.16743 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.075+0000 7f43589e6b80 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555963cc1d30 0x555963cc4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.075+0000 7f43589e6b80 4 client.16743 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:31:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.075+0000 7f43589e6b80 1 -- 172.21.15.2:0/183251208 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5559639f3f10 con 0x555963cc1d30 2023-11-16T18:31:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.075+0000 7f4345d9b700 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555963cc1d30 0x555963cc4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.076+0000 7f4345d9b700 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555963cc1d30 0x555963cc4140 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5661d48b083995d server_cookie=3b19518e0a0dfcf7 in_seq=0 out_seq=0 2023-11-16T18:31:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:08.076+0000 7f432dffb700 10 client.16743 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:08.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:08 smithi040 ceph-mon[100251]: pgmap v191: 129 pgs: 129 active+clean; 725 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 571 op/s 2023-11-16T18:31:08.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:08 smithi188 ceph-mon[100639]: pgmap v191: 129 pgs: 129 active+clean; 725 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 571 op/s 2023-11-16T18:31:08.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:08 smithi002 ceph-mon[98807]: pgmap v191: 129 pgs: 129 active+clean; 725 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 571 op/s 2023-11-16T18:31:09.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.145+0000 7f432dffb700 1 -- 172.21.15.2:0/183251208 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5559639f3f10 con 0x555963cc1d30 2023-11-16T18:31:09.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.145+0000 7f432dffb700 10 client.16743 handle_command_reply: tid=0 2023-11-16T18:31:09.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.146+0000 7f43589e6b80 1 client.16743 shutdown 2023-11-16T18:31:09.149 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.148+0000 7f43589e6b80 1 -- 172.21.15.2:0/183251208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963aaf6b0 msgr2=0x555963c62050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.148+0000 7f43589e6b80 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963aaf6b0 0x555963c62050 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x555963c6f9f0 tx=0x7f4338009260 comp rx=0 tx=0).stop 2023-11-16T18:31:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.149+0000 7f43589e6b80 1 -- 172.21.15.2:0/183251208 shutdown_connections 2023-11-16T18:31:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.149+0000 7f43589e6b80 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555963cc1d30 0x555963cc4140 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.149+0000 7f43589e6b80 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555963c59b90 0x555963c60650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.149+0000 7f43589e6b80 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555963aaf6b0 0x555963c62050 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.149+0000 7f43589e6b80 1 --2- 172.21.15.2:0/183251208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555963c59720 0x555963c62590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.149+0000 7f43589e6b80 1 -- 172.21.15.2:0/183251208 >> 172.21.15.2:0/183251208 conn(0x555963c48f50 msgr2=0x555963c586b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.149+0000 7f432dffb700 0 client.16743 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.149+0000 7f43589e6b80 1 -- 172.21.15.2:0/183251208 shutdown_connections 2023-11-16T18:31:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.149+0000 7f43589e6b80 1 -- 172.21.15.2:0/183251208 wait complete. 2023-11-16T18:31:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.150+0000 7f43589e6b80 20 client.16743 trim_cache size 0 max 16384 2023-11-16T18:31:09.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:31:09.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:31:09.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:31:09.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:31:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.152+0000 7f434c85a700 1 -- 172.21.15.2:0/2439253258 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f43340a12c0 msgr2=0x7f43340a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.152+0000 7f434c85a700 1 --2- 172.21.15.2:0/2439253258 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f43340a12c0 0x7f43340a3770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4338005950 tx=0x7f43380058e0 comp rx=0 tx=0).stop 2023-11-16T18:31:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.152+0000 7f434c85a700 1 -- 172.21.15.2:0/2439253258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4348104440 msgr2=0x7f434819a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.152+0000 7f434c85a700 1 --2- 172.21.15.2:0/2439253258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4348104440 0x7f434819a130 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f434000b770 tx=0x7f4340007870 comp rx=0 tx=0).stop 2023-11-16T18:31:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.153+0000 7f434c85a700 1 -- 172.21.15.2:0/2439253258 shutdown_connections 2023-11-16T18:31:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.153+0000 7f434c85a700 1 --2- 172.21.15.2:0/2439253258 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f43340a12c0 0x7f43340a3770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.153+0000 7f434c85a700 1 --2- 172.21.15.2:0/2439253258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43480ff400 0x7f43481950e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.153+0000 7f434c85a700 1 --2- 172.21.15.2:0/2439253258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4348104440 0x7f434819a130 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.153+0000 7f434c85a700 1 --2- 172.21.15.2:0/2439253258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43480ffda0 0x7f4348195620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.153+0000 7f434c85a700 1 -- 172.21.15.2:0/2439253258 >> 172.21.15.2:0/2439253258 conn(0x7f4348075fa0 msgr2=0x7f43480fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.153+0000 7f434c85a700 1 -- 172.21.15.2:0/2439253258 shutdown_connections 2023-11-16T18:31:09.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:09.154+0000 7f434c85a700 1 -- 172.21.15.2:0/2439253258 wait complete. 2023-11-16T18:31:10.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:10 smithi040 ceph-mon[100251]: pgmap v192: 129 pgs: 129 active+clean; 721 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.4 MiB/s wr, 582 op/s 2023-11-16T18:31:10.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:10 smithi188 ceph-mon[100639]: pgmap v192: 129 pgs: 129 active+clean; 721 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.4 MiB/s wr, 582 op/s 2023-11-16T18:31:10.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:10 smithi002 ceph-mon[98807]: pgmap v192: 129 pgs: 129 active+clean; 721 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.4 MiB/s wr, 582 op/s 2023-11-16T18:31:11.106 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T18:31:11.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.247+0000 7fa768571700 1 -- 172.21.15.2:0/3763110653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa760104140 msgr2=0x7fa760104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:11.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.247+0000 7fa768571700 1 --2- 172.21.15.2:0/3763110653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa760104140 0x7fa760104510 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fa7540099c0 tx=0x7fa754009cd0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.247+0000 7fa768571700 1 -- 172.21.15.2:0/3763110653 shutdown_connections 2023-11-16T18:31:11.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.247+0000 7fa768571700 1 --2- 172.21.15.2:0/3763110653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7601002a0 0x7fa76010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.247+0000 7fa768571700 1 --2- 172.21.15.2:0/3763110653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa760104140 0x7fa760104510 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.247+0000 7fa768571700 1 --2- 172.21.15.2:0/3763110653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7600ff8f0 0x7fa7600ffd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.247+0000 7fa768571700 1 -- 172.21.15.2:0/3763110653 >> 172.21.15.2:0/3763110653 conn(0x7fa760075880 msgr2=0x7fa760075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:11.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.247+0000 7fa768571700 1 -- 172.21.15.2:0/3763110653 shutdown_connections 2023-11-16T18:31:11.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.248+0000 7fa768571700 1 -- 172.21.15.2:0/3763110653 wait complete. 2023-11-16T18:31:11.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.248+0000 7fa768571700 1 Processor -- start 2023-11-16T18:31:11.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.248+0000 7fa768571700 1 -- start start 2023-11-16T18:31:11.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.248+0000 7fa768571700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7600ff8f0 0x7fa760197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.248+0000 7fa768571700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7601002a0 0x7fa7601976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.248+0000 7fa768571700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa760104140 0x7fa7601a21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.248+0000 7fa768571700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa760197fc0 con 0x7fa7600ff8f0 2023-11-16T18:31:11.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.248+0000 7fa768571700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7601a2730 con 0x7fa7601002a0 2023-11-16T18:31:11.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.248+0000 7fa768571700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7601a28a0 con 0x7fa760104140 2023-11-16T18:31:11.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.248+0000 7fa76630d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7600ff8f0 0x7fa760197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa76630d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7600ff8f0 0x7fa760197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52900/0 (socket says 172.21.15.2:52900) 2023-11-16T18:31:11.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa76630d700 1 -- 172.21.15.2:0/3945254675 learned_addr learned my addr 172.21.15.2:0/3945254675 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:11.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa765b0c700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7601002a0 0x7fa7601976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa766b0e700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa760104140 0x7fa7601a21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa76630d700 1 -- 172.21.15.2:0/3945254675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa760104140 msgr2=0x7fa7601a21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:11.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa76630d700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa760104140 0x7fa7601a21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa76630d700 1 -- 172.21.15.2:0/3945254675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7601002a0 msgr2=0x7fa7601976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:11.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa76630d700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7601002a0 0x7fa7601976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa76630d700 1 -- 172.21.15.2:0/3945254675 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7540095f0 con 0x7fa7600ff8f0 2023-11-16T18:31:11.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa76630d700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7600ff8f0 0x7fa760197170 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fa75400b580 tx=0x7fa75400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:11.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa7537fe700 1 -- 172.21.15.2:0/3945254675 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75401d070 con 0x7fa7600ff8f0 2023-11-16T18:31:11.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa768571700 1 -- 172.21.15.2:0/3945254675 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76010f1d0 con 0x7fa7600ff8f0 2023-11-16T18:31:11.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa7537fe700 1 -- 172.21.15.2:0/3945254675 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7540054f0 con 0x7fa7600ff8f0 2023-11-16T18:31:11.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa7537fe700 1 -- 172.21.15.2:0/3945254675 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754004b30 con 0x7fa7600ff8f0 2023-11-16T18:31:11.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.249+0000 7fa768571700 1 -- 172.21.15.2:0/3945254675 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76010f040 con 0x7fa7600ff8f0 2023-11-16T18:31:11.257 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.252+0000 7fa7537fe700 1 -- 172.21.15.2:0/3945254675 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fa75400fb80 con 0x7fa7600ff8f0 2023-11-16T18:31:11.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.252+0000 7fa7537fe700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa74c0a12c0 0x7fa74c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.252+0000 7fa7537fe700 1 -- 172.21.15.2:0/3945254675 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa7540cde40 con 0x7fa7600ff8f0 2023-11-16T18:31:11.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.252+0000 7fa765b0c700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa74c0a12c0 0x7fa74c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.253+0000 7fa765b0c700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa74c0a12c0 0x7fa74c0a3770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa7600ff090 tx=0x7fa75c009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:11.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.253+0000 7fa7746fdb80 1 Processor -- start 2023-11-16T18:31:11.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.253+0000 7fa7746fdb80 1 -- start start 2023-11-16T18:31:11.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.253+0000 7fa7746fdb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f4af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.253+0000 7fa7746fdb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4b470 0x55ea08f4f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.253+0000 7fa7746fdb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4f8b0 0x55ea08f4fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.253+0000 7fa7746fdb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea08e50060 con 0x55ea08f4ab60 2023-11-16T18:31:11.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.253+0000 7fa7746fdb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea08ea28a0 con 0x55ea08f4b470 2023-11-16T18:31:11.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.253+0000 7fa7746fdb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea08e9bd30 con 0x55ea08f4f8b0 2023-11-16T18:31:11.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa76630d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f4af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa76630d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f4af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52908/0 (socket says 172.21.15.2:52908) 2023-11-16T18:31:11.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa76630d700 1 -- 172.21.15.2:0/1198923438 learned_addr learned my addr 172.21.15.2:0/1198923438 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:11.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa765b0c700 1 --2- 172.21.15.2:0/1198923438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4b470 0x55ea08f4f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa766b0e700 1 --2- 172.21.15.2:0/1198923438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4f8b0 0x55ea08f4fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa76630d700 1 -- 172.21.15.2:0/1198923438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4f8b0 msgr2=0x55ea08f4fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:11.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa76630d700 1 --2- 172.21.15.2:0/1198923438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4f8b0 0x55ea08f4fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa76630d700 1 -- 172.21.15.2:0/1198923438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4b470 msgr2=0x55ea08f4f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:11.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa76630d700 1 --2- 172.21.15.2:0/1198923438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4b470 0x55ea08f4f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa76630d700 1 -- 172.21.15.2:0/1198923438 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea08d1d630 con 0x55ea08f4ab60 2023-11-16T18:31:11.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa76630d700 1 --2- 172.21.15.2:0/1198923438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f4af30 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa7540976a0 tx=0x7fa754009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc52c18348f508e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:11.259 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa750ff9700 1 -- 172.21.15.2:0/1198923438 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75409ac30 con 0x55ea08f4ab60 2023-11-16T18:31:11.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa750ff9700 1 -- 172.21.15.2:0/1198923438 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa75409ec60 con 0x55ea08f4ab60 2023-11-16T18:31:11.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.254+0000 7fa750ff9700 1 -- 172.21.15.2:0/1198923438 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75409a120 con 0x55ea08f4ab60 2023-11-16T18:31:11.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.255+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1198923438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 msgr2=0x55ea08f4af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:11.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.255+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1198923438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f4af30 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa7540976a0 tx=0x7fa754009310 comp rx=0 tx=0).stop 2023-11-16T18:31:11.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.255+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1198923438 shutdown_connections 2023-11-16T18:31:11.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.255+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1198923438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4f8b0 0x55ea08f4fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.255+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1198923438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f4af30 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.255+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1198923438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4b470 0x55ea08f4f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.255+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1198923438 >> 172.21.15.2:0/1198923438 conn(0x55ea08f40360 msgr2=0x55ea08f45b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:11.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.255+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1198923438 shutdown_connections 2023-11-16T18:31:11.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.255+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1198923438 wait complete. 2023-11-16T18:31:11.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.255+0000 7fa7746fdb80 1 -- start start 2023-11-16T18:31:11.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa7746fdb80 1 Processor -- start 2023-11-16T18:31:11.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa7746fdb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f56fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa7746fdb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4b470 0x55ea08f541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa7746fdb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4f8b0 0x55ea08f54790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa7746fdb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea08e3d8e0 con 0x55ea08f4ab60 2023-11-16T18:31:11.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa7746fdb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea08e3d580 con 0x55ea08f4b470 2023-11-16T18:31:11.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa7746fdb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea08d1f920 con 0x55ea08f4f8b0 2023-11-16T18:31:11.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa7746fdb80 10 client.0 _ll_register_callbacks cb 0x55ea08e5b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:11.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa76630d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f56fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa76630d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f56fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52914/0 (socket says 172.21.15.2:52914) 2023-11-16T18:31:11.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa76630d700 1 -- 172.21.15.2:0/2301072825 learned_addr learned my addr 172.21.15.2:0/2301072825 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:11.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa765b0c700 1 --2- 172.21.15.2:0/2301072825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4b470 0x55ea08f541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa766b0e700 1 --2- 172.21.15.2:0/2301072825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4f8b0 0x55ea08f54790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa76630d700 1 -- 172.21.15.2:0/2301072825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4f8b0 msgr2=0x55ea08f54790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:11.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa76630d700 1 --2- 172.21.15.2:0/2301072825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4f8b0 0x55ea08f54790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa76630d700 1 -- 172.21.15.2:0/2301072825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4b470 msgr2=0x55ea08f541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:11.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa76630d700 1 --2- 172.21.15.2:0/2301072825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4b470 0x55ea08f541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa76630d700 1 -- 172.21.15.2:0/2301072825 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76010f040 con 0x55ea08f4ab60 2023-11-16T18:31:11.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa76630d700 1 --2- 172.21.15.2:0/2301072825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f56fe0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa7540ce120 tx=0x7fa754005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f7dcb9cf99645d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:11.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.256+0000 7fa749ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:31:11.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa749ffb700 1 -- 172.21.15.2:0/2301072825 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75409a320 con 0x55ea08f4ab60 2023-11-16T18:31:11.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa749ffb700 1 -- 172.21.15.2:0/2301072825 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa7540a2b60 con 0x55ea08f4ab60 2023-11-16T18:31:11.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa7746fdb80 1 -- 172.21.15.2:0/2301072825 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea08f3db20 con 0x55ea08f4ab60 2023-11-16T18:31:11.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa749ffb700 1 -- 172.21.15.2:0/2301072825 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa75409acb0 con 0x55ea08f4ab60 2023-11-16T18:31:11.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa7746fdb80 10 client.16755 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:11.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa7746fdb80 1 -- 172.21.15.2:0/2301072825 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ea08e62190 con 0x55ea08f4ab60 2023-11-16T18:31:11.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa749ffb700 1 -- 172.21.15.2:0/2301072825 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ea08f3db20 con 0x55ea08f4ab60 2023-11-16T18:31:11.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa749ffb700 1 -- 172.21.15.2:0/2301072825 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa7540ba220 con 0x55ea08f4ab60 2023-11-16T18:31:11.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa7746fdb80 10 client.16755 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:11.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa7746fdb80 10 client.16755 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:31:11.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa7746fdb80 20 client.16755 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:11.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/2301072825 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ea08f64110 0x55ea08f664f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa7746fdb80 4 client.16755 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:31:11.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa7746fdb80 1 -- 172.21.15.2:0/2301072825 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea08e5e0a0 con 0x55ea08f64110 2023-11-16T18:31:11.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.257+0000 7fa765b0c700 1 --2- 172.21.15.2:0/2301072825 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ea08f64110 0x55ea08f664f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.258+0000 7fa749ffb700 10 client.16755 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:11.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.258+0000 7fa765b0c700 1 --2- 172.21.15.2:0/2301072825 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ea08f64110 0x55ea08f664f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42f27cab6fe4d5b2 server_cookie=35230161c5d44622 in_seq=0 out_seq=0 2023-11-16T18:31:11.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.259+0000 7fa749ffb700 1 -- 172.21.15.2:0/2301072825 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ea08e5e0a0 con 0x55ea08f64110 2023-11-16T18:31:11.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.259+0000 7fa749ffb700 10 client.16755 handle_command_reply: tid=0 2023-11-16T18:31:11.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.259+0000 7fa7746fdb80 1 client.16755 shutdown 2023-11-16T18:31:11.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.259+0000 7fa7746fdb80 1 -- 172.21.15.2:0/2301072825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 msgr2=0x55ea08f56fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:11.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.259+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/2301072825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f56fe0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa7540ce120 tx=0x7fa754005e80 comp rx=0 tx=0).stop 2023-11-16T18:31:11.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.259+0000 7fa7746fdb80 1 -- 172.21.15.2:0/2301072825 shutdown_connections 2023-11-16T18:31:11.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.259+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/2301072825 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ea08f64110 0x55ea08f664f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.259+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/2301072825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4f8b0 0x55ea08f54790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.259+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/2301072825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f56fe0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.259+0000 7fa749ffb700 0 client.16755 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:11.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.259+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/2301072825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4b470 0x55ea08f541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.259+0000 7fa7746fdb80 1 -- 172.21.15.2:0/2301072825 >> 172.21.15.2:0/2301072825 conn(0x55ea08f40360 msgr2=0x55ea08f50d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:11.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.259+0000 7fa7746fdb80 1 -- 172.21.15.2:0/2301072825 shutdown_connections 2023-11-16T18:31:11.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.260+0000 7fa7746fdb80 1 -- 172.21.15.2:0/2301072825 wait complete. 2023-11-16T18:31:11.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.260+0000 7fa7746fdb80 20 client.16755 trim_cache size 0 max 16384 2023-11-16T18:31:11.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa7746fdb80 1 Processor -- start 2023-11-16T18:31:11.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa7746fdb80 1 -- start start 2023-11-16T18:31:11.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa7746fdb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4ab60 0x55ea08f45be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa7746fdb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4b470 0x55ea08f5b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa7746fdb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4f8b0 0x55ea08f56b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa7746fdb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea08f60fc0 con 0x55ea08f4b470 2023-11-16T18:31:11.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa7746fdb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea08e5e0d0 con 0x55ea08f4f8b0 2023-11-16T18:31:11.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa7746fdb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea08f3ff60 con 0x55ea08f4ab60 2023-11-16T18:31:11.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa765b0c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4b470 0x55ea08f5b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa765b0c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4b470 0x55ea08f5b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52924/0 (socket says 172.21.15.2:52924) 2023-11-16T18:31:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa765b0c700 1 -- 172.21.15.2:0/1754815553 learned_addr learned my addr 172.21.15.2:0/1754815553 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa766b0e700 1 --2- 172.21.15.2:0/1754815553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4f8b0 0x55ea08f56b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa765b0c700 1 -- 172.21.15.2:0/1754815553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4ab60 msgr2=0x55ea08f45be0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T18:31:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa765b0c700 1 --2- 172.21.15.2:0/1754815553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4ab60 0x55ea08f45be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa765b0c700 1 -- 172.21.15.2:0/1754815553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4f8b0 msgr2=0x55ea08f56b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa765b0c700 1 --2- 172.21.15.2:0/1754815553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4f8b0 0x55ea08f56b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.277+0000 7fa765b0c700 1 -- 172.21.15.2:0/1754815553 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea08e61ee0 con 0x55ea08f4b470 2023-11-16T18:31:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.278+0000 7fa765b0c700 1 --2- 172.21.15.2:0/1754815553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4b470 0x55ea08f5b7b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x55ea08f558a0 tx=0x7fa75c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32535a383343731b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.278+0000 7fa7517fa700 1 -- 172.21.15.2:0/1754815553 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c022070 con 0x55ea08f4b470 2023-11-16T18:31:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.278+0000 7fa7517fa700 1 -- 172.21.15.2:0/1754815553 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa75c023040 con 0x55ea08f4b470 2023-11-16T18:31:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.278+0000 7fa7517fa700 1 -- 172.21.15.2:0/1754815553 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c003a80 con 0x55ea08f4b470 2023-11-16T18:31:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.278+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1754815553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4b470 msgr2=0x55ea08f5b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:11.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.278+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1754815553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4b470 0x55ea08f5b7b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x55ea08f558a0 tx=0x7fa75c014000 comp rx=0 tx=0).stop 2023-11-16T18:31:11.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.278+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1754815553 shutdown_connections 2023-11-16T18:31:11.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.278+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1754815553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4ab60 0x55ea08f45be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.278+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1754815553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4f8b0 0x55ea08f56b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.278+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1754815553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4b470 0x55ea08f5b7b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.278+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1754815553 >> 172.21.15.2:0/1754815553 conn(0x55ea08f40360 msgr2=0x55ea08f608b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:11.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.278+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1754815553 shutdown_connections 2023-11-16T18:31:11.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1754815553 wait complete. 2023-11-16T18:31:11.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa7746fdb80 1 -- start start 2023-11-16T18:31:11.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa7746fdb80 1 Processor -- start 2023-11-16T18:31:11.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa7746fdb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f5b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa7746fdb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4b470 0x55ea08f59c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa7746fdb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4f8b0 0x55ea08f5a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa7746fdb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea08f5a720 con 0x55ea08f4ab60 2023-11-16T18:31:11.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa7746fdb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea08f5a890 con 0x55ea08f4f8b0 2023-11-16T18:31:11.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa7746fdb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea08f5aa50 con 0x55ea08f4b470 2023-11-16T18:31:11.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa7746fdb80 10 client.0 _ll_register_callbacks cb 0x7fa75c0096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:11.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa76630d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f5b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa76630d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f5b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52928/0 (socket says 172.21.15.2:52928) 2023-11-16T18:31:11.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa76630d700 1 -- 172.21.15.2:0/1029242557 learned_addr learned my addr 172.21.15.2:0/1029242557 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:11.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa766b0e700 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4f8b0 0x55ea08f5a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.279+0000 7fa765b0c700 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4b470 0x55ea08f59c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa76630d700 1 -- 172.21.15.2:0/1029242557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4b470 msgr2=0x55ea08f59c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:11.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa76630d700 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4b470 0x55ea08f59c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa76630d700 1 -- 172.21.15.2:0/1029242557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4f8b0 msgr2=0x55ea08f5a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:11.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa76630d700 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4f8b0 0x55ea08f5a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa76630d700 1 -- 172.21.15.2:0/1029242557 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c017070 con 0x55ea08f4ab60 2023-11-16T18:31:11.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa765b0c700 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4b470 0x55ea08f59c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:31:11.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa76630d700 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f5b740 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fa75409a930 tx=0x7fa75400feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=564a837e12d4db88 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:11.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa749ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:31:11.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa749ffb700 1 -- 172.21.15.2:0/1029242557 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75409a590 con 0x55ea08f4ab60 2023-11-16T18:31:11.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa749ffb700 1 -- 172.21.15.2:0/1029242557 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa754099f20 con 0x55ea08f4ab60 2023-11-16T18:31:11.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1029242557 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa75409ec60 con 0x55ea08f4ab60 2023-11-16T18:31:11.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa749ffb700 1 -- 172.21.15.2:0/1029242557 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa75409ac30 con 0x55ea08f4ab60 2023-11-16T18:31:11.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa7746fdb80 10 client.16761 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:11.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1029242557 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ea08cb2c60 con 0x55ea08f4ab60 2023-11-16T18:31:11.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa749ffb700 1 -- 172.21.15.2:0/1029242557 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa75409ec60 con 0x55ea08f4ab60 2023-11-16T18:31:11.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa749ffb700 1 -- 172.21.15.2:0/1029242557 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa75409a220 con 0x55ea08f4ab60 2023-11-16T18:31:11.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa7746fdb80 10 client.16761 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:11.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa7746fdb80 10 client.16761 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:31:11.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa7746fdb80 20 client.16761 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:11.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ea08fb9000 0x55ea08fbb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa7746fdb80 4 client.16761 mds_command: new command op to 24473 tid=0[{"prefix": "scrub status"}] 2023-11-16T18:31:11.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.280+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1029242557 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ea08e6a660 con 0x55ea08fb9000 2023-11-16T18:31:11.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa765b0c700 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ea08fb9000 0x55ea08fbb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa749ffb700 10 client.16761 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:11.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa765b0c700 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ea08fb9000 0x55ea08fbb3e0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e0b811b3264d555 server_cookie=558631bfe440ffd3 in_seq=0 out_seq=0 2023-11-16T18:31:11.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa749ffb700 1 -- 172.21.15.2:0/1029242557 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ea08e6a660 con 0x55ea08fb9000 2023-11-16T18:31:11.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa749ffb700 10 client.16761 handle_command_reply: tid=0 2023-11-16T18:31:11.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa7746fdb80 1 client.16761 shutdown 2023-11-16T18:31:11.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1029242557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 msgr2=0x55ea08f5b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:11.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f5b740 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fa75409a930 tx=0x7fa75400feb0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1029242557 shutdown_connections 2023-11-16T18:31:11.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55ea08fb9000 0x55ea08fbb3e0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea08f4f8b0 0x55ea08f5a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa749ffb700 0 client.16761 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:11.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea08f4ab60 0x55ea08f5b740 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa7746fdb80 1 --2- 172.21.15.2:0/1029242557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ea08f4b470 0x55ea08f59c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1029242557 >> 172.21.15.2:0/1029242557 conn(0x55ea08f40360 msgr2=0x55ea08f4e790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:11.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.281+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1029242557 shutdown_connections 2023-11-16T18:31:11.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa7746fdb80 1 -- 172.21.15.2:0/1029242557 wait complete. 2023-11-16T18:31:11.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa7746fdb80 20 client.16761 trim_cache size 0 max 16384 2023-11-16T18:31:11.641 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:31:11.641 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "no active scrubs running", 2023-11-16T18:31:11.641 INFO:teuthology.orchestra.run.smithi002.stdout: "scrubs": {} 2023-11-16T18:31:11.641 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:31:11.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa768571700 1 -- 172.21.15.2:0/3945254675 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa74c0a12c0 msgr2=0x7fa74c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:11.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa768571700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa74c0a12c0 0x7fa74c0a3770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa7600ff090 tx=0x7fa75c009260 comp rx=0 tx=0).stop 2023-11-16T18:31:11.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa768571700 1 -- 172.21.15.2:0/3945254675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7600ff8f0 msgr2=0x7fa760197170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:11.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa768571700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7600ff8f0 0x7fa760197170 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fa75400b580 tx=0x7fa75400f690 comp rx=0 tx=0).stop 2023-11-16T18:31:11.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa768571700 1 -- 172.21.15.2:0/3945254675 shutdown_connections 2023-11-16T18:31:11.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa768571700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa74c0a12c0 0x7fa74c0a3770 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa768571700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa760104140 0x7fa7601a21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa768571700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7600ff8f0 0x7fa760197170 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa768571700 1 --2- 172.21.15.2:0/3945254675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7601002a0 0x7fa7601976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa768571700 1 -- 172.21.15.2:0/3945254675 >> 172.21.15.2:0/3945254675 conn(0x7fa760075880 msgr2=0x7fa7600fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:11.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa768571700 1 -- 172.21.15.2:0/3945254675 shutdown_connections 2023-11-16T18:31:11.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.282+0000 7fa768571700 1 -- 172.21.15.2:0/3945254675 wait complete. 2023-11-16T18:31:11.643 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T18:31:11.644 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:31:11.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.794+0000 7f9e0dd13700 1 -- 172.21.15.2:0/55189178 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e08105580 msgr2=0x7f9e0810d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:11.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.794+0000 7f9e0dd13700 1 --2- 172.21.15.2:0/55189178 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e08105580 0x7f9e0810d150 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f9e0000b0d0 tx=0x7f9e0000b3e0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.794+0000 7f9e0dd13700 1 -- 172.21.15.2:0/55189178 shutdown_connections 2023-11-16T18:31:11.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.794+0000 7f9e0dd13700 1 --2- 172.21.15.2:0/55189178 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e08105580 0x7f9e0810d150 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.794+0000 7f9e0dd13700 1 --2- 172.21.15.2:0/55189178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e08100540 0x7f9e08100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.794+0000 7f9e0dd13700 1 --2- 172.21.15.2:0/55189178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e08100ee0 0x7f9e08104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.794+0000 7f9e0dd13700 1 -- 172.21.15.2:0/55189178 >> 172.21.15.2:0/55189178 conn(0x7f9e08077590 msgr2=0x7f9e08077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:11.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.794+0000 7f9e0dd13700 1 -- 172.21.15.2:0/55189178 shutdown_connections 2023-11-16T18:31:11.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.794+0000 7f9e0dd13700 1 -- 172.21.15.2:0/55189178 wait complete. 2023-11-16T18:31:11.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.795+0000 7f9e0dd13700 1 Processor -- start 2023-11-16T18:31:11.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.795+0000 7f9e0dd13700 1 -- start start 2023-11-16T18:31:11.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.795+0000 7f9e0dd13700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e08100540 0x7f9e08196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.795+0000 7f9e0dd13700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e08100ee0 0x7f9e08196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.795+0000 7f9e0dd13700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e08105580 0x7f9e0819aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.795+0000 7f9e0dd13700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0819b4e0 con 0x7f9e08105580 2023-11-16T18:31:11.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.795+0000 7f9e0dd13700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0819b650 con 0x7f9e08100540 2023-11-16T18:31:11.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.795+0000 7f9e0dd13700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0819b810 con 0x7f9e08100ee0 2023-11-16T18:31:11.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.795+0000 7f9e07fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e08105580 0x7f9e0819aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.795+0000 7f9e077fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e08100540 0x7f9e08196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e07fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e08105580 0x7f9e0819aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52946/0 (socket says 172.21.15.2:52946) 2023-11-16T18:31:11.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e07fff700 1 -- 172.21.15.2:0/2390489462 learned_addr learned my addr 172.21.15.2:0/2390489462 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:11.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e06ffd700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e08100ee0 0x7f9e08196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e07fff700 1 -- 172.21.15.2:0/2390489462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e08100ee0 msgr2=0x7f9e08196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:11.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e07fff700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e08100ee0 0x7f9e08196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e07fff700 1 -- 172.21.15.2:0/2390489462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e08100540 msgr2=0x7f9e08196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:11.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e07fff700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e08100540 0x7f9e08196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:11.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e07fff700 1 -- 172.21.15.2:0/2390489462 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e00009d00 con 0x7f9e08105580 2023-11-16T18:31:11.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e06ffd700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e08100ee0 0x7f9e08196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:31:11.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e077fe700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e08100540 0x7f9e08196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e07fff700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e08105580 0x7f9e0819aea0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f9e00007870 tx=0x7f9e000078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e04ff9700 1 -- 172.21.15.2:0/2390489462 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0000e040 con 0x7f9e08105580 2023-11-16T18:31:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e04ff9700 1 -- 172.21.15.2:0/2390489462 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e00005070 con 0x7f9e08105580 2023-11-16T18:31:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e04ff9700 1 -- 172.21.15.2:0/2390489462 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0001b6a0 con 0x7f9e08105580 2023-11-16T18:31:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e0dd13700 1 -- 172.21.15.2:0/2390489462 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e08110350 con 0x7f9e08105580 2023-11-16T18:31:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.796+0000 7f9e0dd13700 1 -- 172.21.15.2:0/2390489462 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e081101c0 con 0x7f9e08105580 2023-11-16T18:31:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.797+0000 7f9e0dd13700 1 -- 172.21.15.2:0/2390489462 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e081104e0 con 0x7f9e08105580 2023-11-16T18:31:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.798+0000 7f9e04ff9700 1 -- 172.21.15.2:0/2390489462 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f9e000051e0 con 0x7f9e08105580 2023-11-16T18:31:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.799+0000 7f9e04ff9700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f9df00a1340 0x7f9df00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.799+0000 7f9e04ff9700 1 -- 172.21.15.2:0/2390489462 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f9e000ce570 con 0x7f9e08105580 2023-11-16T18:31:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.799+0000 7f9e077fe700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f9df00a1340 0x7f9df00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.799+0000 7f9e077fe700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f9df00a1340 0x7f9df00a37f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9df8011490 tx=0x7f9df8009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:11.806 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:11.802+0000 7f9e04ff9700 1 -- 172.21.15.2:0/2390489462 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9e000976a0 con 0x7f9e08105580 2023-11-16T18:31:12.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.050+0000 7f9e0dd13700 1 -- 172.21.15.2:0/2390489462 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e08067260 con 0x7f9e08105580 2023-11-16T18:31:12.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.051+0000 7f9e04ff9700 1 -- 172.21.15.2:0/2390489462 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+10783 (secure 0 0 0) 0x7f9e0009acc0 con 0x7f9e08105580 2023-11-16T18:31:12.054 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:31:12.054 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:28:27.507965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24439":{"gid":24439,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/264232322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":264232322},{"type":"v1","addr":"172.21.15.188:6837","nonce":264232322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":89,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:31:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.053+0000 7f9e0dd13700 1 -- 172.21.15.2:0/2390489462 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f9df00a1340 msgr2=0x7f9df00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.053+0000 7f9e0dd13700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f9df00a1340 0x7f9df00a37f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9df8011490 tx=0x7f9df8009260 comp rx=0 tx=0).stop 2023-11-16T18:31:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.053+0000 7f9e0dd13700 1 -- 172.21.15.2:0/2390489462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e08105580 msgr2=0x7f9e0819aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.053+0000 7f9e0dd13700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e08105580 0x7f9e0819aea0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f9e00007870 tx=0x7f9e000078a0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.053+0000 7f9e0dd13700 1 -- 172.21.15.2:0/2390489462 shutdown_connections 2023-11-16T18:31:12.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.053+0000 7f9e0dd13700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f9df00a1340 0x7f9df00a37f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.053+0000 7f9e0dd13700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e08105580 0x7f9e0819aea0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.053+0000 7f9e0dd13700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e08100540 0x7f9e08196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.053+0000 7f9e0dd13700 1 --2- 172.21.15.2:0/2390489462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e08100ee0 0x7f9e08196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.053+0000 7f9e0dd13700 1 -- 172.21.15.2:0/2390489462 >> 172.21.15.2:0/2390489462 conn(0x7f9e08077590 msgr2=0x7f9e080fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:12.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.053+0000 7f9e0dd13700 1 -- 172.21.15.2:0/2390489462 shutdown_connections 2023-11-16T18:31:12.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.053+0000 7f9e0dd13700 1 -- 172.21.15.2:0/2390489462 wait complete. 2023-11-16T18:31:12.059 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 29 2023-11-16T18:31:12.068 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-11-16T18:31:12.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.196+0000 7fbbff066700 1 -- 172.21.15.2:0/441588720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbf80ffa70 msgr2=0x7fbbf80ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:12.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.196+0000 7fbbff066700 1 --2- 172.21.15.2:0/441588720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbf80ffa70 0x7fbbf80ffee0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009990 tx=0x7fbbe4009ca0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.196+0000 7fbbff066700 1 -- 172.21.15.2:0/441588720 shutdown_connections 2023-11-16T18:31:12.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.196+0000 7fbbff066700 1 --2- 172.21.15.2:0/441588720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbf8105250 0x7fbbf8105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.196+0000 7fbbff066700 1 --2- 172.21.15.2:0/441588720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbf8100420 0x7fbbf810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.196+0000 7fbbff066700 1 --2- 172.21.15.2:0/441588720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbf80ffa70 0x7fbbf80ffee0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.196+0000 7fbbff066700 1 -- 172.21.15.2:0/441588720 >> 172.21.15.2:0/441588720 conn(0x7fbbf80763a0 msgr2=0x7fbbf80767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:12.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.196+0000 7fbbff066700 1 -- 172.21.15.2:0/441588720 shutdown_connections 2023-11-16T18:31:12.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.196+0000 7fbbff066700 1 -- 172.21.15.2:0/441588720 wait complete. 2023-11-16T18:31:12.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.197+0000 7fbbff066700 1 Processor -- start 2023-11-16T18:31:12.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.197+0000 7fbbff066700 1 -- start start 2023-11-16T18:31:12.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.197+0000 7fbbff066700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbf80ffa70 0x7fbbf8195360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.197+0000 7fbbff066700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbf8100420 0x7fbbf81958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.197+0000 7fbbff066700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbf8105250 0x7fbbf8199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.197+0000 7fbbff066700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf819a340 con 0x7fbbf8100420 2023-11-16T18:31:12.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.197+0000 7fbbff066700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf819a4b0 con 0x7fbbf80ffa70 2023-11-16T18:31:12.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.197+0000 7fbbff066700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf819a670 con 0x7fbbf8105250 2023-11-16T18:31:12.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.197+0000 7fbbf7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbf8100420 0x7fbbf81958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.197+0000 7fbbf7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbf8100420 0x7fbbf81958a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52958/0 (socket says 172.21.15.2:52958) 2023-11-16T18:31:12.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.197+0000 7fbbf7fff700 1 -- 172.21.15.2:0/2690462893 learned_addr learned my addr 172.21.15.2:0/2690462893 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:12.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.197+0000 7fbbfce02700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbf80ffa70 0x7fbbf8195360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.197+0000 7fbbfd603700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbf8105250 0x7fbbf8199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.198+0000 7fbbf7fff700 1 -- 172.21.15.2:0/2690462893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbf8105250 msgr2=0x7fbbf8199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:12.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.198+0000 7fbbf7fff700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbf8105250 0x7fbbf8199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.198+0000 7fbbf7fff700 1 -- 172.21.15.2:0/2690462893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbf80ffa70 msgr2=0x7fbbf8195360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:12.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.198+0000 7fbbf7fff700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbf80ffa70 0x7fbbf8195360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.198+0000 7fbbf7fff700 1 -- 172.21.15.2:0/2690462893 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe40095f0 con 0x7fbbf8100420 2023-11-16T18:31:12.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.198+0000 7fbbfd603700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbf8105250 0x7fbbf8199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:31:12.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.198+0000 7fbbfce02700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbf80ffa70 0x7fbbf8195360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:12.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.198+0000 7fbbf7fff700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbf8100420 0x7fbbf81958a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4005910 tx=0x7fbbe4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:12.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.198+0000 7fbbf5ffb700 1 -- 172.21.15.2:0/2690462893 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe401f070 con 0x7fbbf8100420 2023-11-16T18:31:12.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.198+0000 7fbbff066700 1 -- 172.21.15.2:0/2690462893 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf810f280 con 0x7fbbf8100420 2023-11-16T18:31:12.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.198+0000 7fbbf5ffb700 1 -- 172.21.15.2:0/2690462893 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbe400f460 con 0x7fbbf8100420 2023-11-16T18:31:12.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.198+0000 7fbbf5ffb700 1 -- 172.21.15.2:0/2690462893 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4025680 con 0x7fbbf8100420 2023-11-16T18:31:12.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.198+0000 7fbbff066700 1 -- 172.21.15.2:0/2690462893 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf810f0f0 con 0x7fbbf8100420 2023-11-16T18:31:12.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.201+0000 7fbbf5ffb700 1 -- 172.21.15.2:0/2690462893 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fbbe4023070 con 0x7fbbf8100420 2023-11-16T18:31:12.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.201+0000 7fbbf5ffb700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fbbe80a12c0 0x7fbbe80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.202+0000 7fbbf5ffb700 1 -- 172.21.15.2:0/2690462893 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fbbe40d0370 con 0x7fbbf8100420 2023-11-16T18:31:12.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.202+0000 7fbbfce02700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fbbe80a12c0 0x7fbbe80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.202+0000 7fbbfce02700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fbbe80a12c0 0x7fbbe80a3770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbbec009de0 tx=0x7fbbec009190 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:12.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.202+0000 7fbc0b1f2b80 1 Processor -- start 2023-11-16T18:31:12.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.202+0000 7fbc0b1f2b80 1 -- start start 2023-11-16T18:31:12.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbc0b1f2b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216e9b60 0x5654216e9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbc0b1f2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216ea470 0x5654216ee370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbc0b1f2b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216eed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbc0b1f2b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654215ef060 con 0x5654216ee8b0 2023-11-16T18:31:12.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbc0b1f2b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654216418a0 con 0x5654216e9b60 2023-11-16T18:31:12.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbc0b1f2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56542163ad30 con 0x5654216ea470 2023-11-16T18:31:12.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbfd603700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216eed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbfce02700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216e9b60 0x5654216e9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbfd603700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216eed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52968/0 (socket says 172.21.15.2:52968) 2023-11-16T18:31:12.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbfd603700 1 -- 172.21.15.2:0/2316664382 learned_addr learned my addr 172.21.15.2:0/2316664382 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:12.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbf7fff700 1 --2- 172.21.15.2:0/2316664382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216ea470 0x5654216ee370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbfd603700 1 -- 172.21.15.2:0/2316664382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216ea470 msgr2=0x5654216ee370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:12.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbfd603700 1 --2- 172.21.15.2:0/2316664382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216ea470 0x5654216ee370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbfd603700 1 -- 172.21.15.2:0/2316664382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216e9b60 msgr2=0x5654216e9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:12.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbfd603700 1 --2- 172.21.15.2:0/2316664382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216e9b60 0x5654216e9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbfd603700 1 -- 172.21.15.2:0/2316664382 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654214bc630 con 0x5654216ee8b0 2023-11-16T18:31:12.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbfce02700 1 --2- 172.21.15.2:0/2316664382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216e9b60 0x5654216e9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:12.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbf7fff700 1 --2- 172.21.15.2:0/2316664382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216ea470 0x5654216ee370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:31:12.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbfd603700 1 --2- 172.21.15.2:0/2316664382 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216eed00 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009440 tx=0x7fbbf0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683b3e91f2267f5d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:12.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbe2ffd700 1 -- 172.21.15.2:0/2316664382 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf000e070 con 0x5654216ee8b0 2023-11-16T18:31:12.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbe2ffd700 1 -- 172.21.15.2:0/2316664382 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbf000f040 con 0x5654216ee8b0 2023-11-16T18:31:12.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.203+0000 7fbbe2ffd700 1 -- 172.21.15.2:0/2316664382 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0015c30 con 0x5654216ee8b0 2023-11-16T18:31:12.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.204+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/2316664382 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 msgr2=0x5654216eed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:12.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.204+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/2316664382 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216eed00 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009440 tx=0x7fbbf0009920 comp rx=0 tx=0).stop 2023-11-16T18:31:12.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.204+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/2316664382 shutdown_connections 2023-11-16T18:31:12.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.204+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/2316664382 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216eed00 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.204+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/2316664382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216e9b60 0x5654216e9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.204+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/2316664382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216ea470 0x5654216ee370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.204+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/2316664382 >> 172.21.15.2:0/2316664382 conn(0x5654216df360 msgr2=0x5654216e4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:12.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.204+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/2316664382 shutdown_connections 2023-11-16T18:31:12.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.204+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/2316664382 wait complete. 2023-11-16T18:31:12.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.204+0000 7fbc0b1f2b80 1 -- start start 2023-11-16T18:31:12.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbc0b1f2b80 1 Processor -- start 2023-11-16T18:31:12.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbc0b1f2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216e9b60 0x5654216f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbc0b1f2b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 0x5654216f3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbc0b1f2b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216f3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbc0b1f2b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654215dc8e0 con 0x5654216ee8b0 2023-11-16T18:31:12.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbc0b1f2b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654215dc580 con 0x5654216ea470 2023-11-16T18:31:12.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbc0b1f2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654214be920 con 0x5654216e9b60 2023-11-16T18:31:12.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbc0b1f2b80 10 client.0 _ll_register_callbacks cb 0x5654215fa780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:12.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbbf7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 0x5654216f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbbfd603700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbbf7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 0x5654216f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43410/0 (socket says 172.21.15.2:43410) 2023-11-16T18:31:12.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbbfd603700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216f3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52974/0 (socket says 172.21.15.2:52974) 2023-11-16T18:31:12.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbbf7fff700 1 -- 172.21.15.2:0/478490100 learned_addr learned my addr 172.21.15.2:0/478490100 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:12.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbbfce02700 1 --2- 172.21.15.2:0/478490100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216e9b60 0x5654216f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbbf7fff700 1 -- 172.21.15.2:0/478490100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216e9b60 msgr2=0x5654216f5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:12.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbbf7fff700 1 --2- 172.21.15.2:0/478490100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216e9b60 0x5654216f5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbbf7fff700 1 -- 172.21.15.2:0/478490100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 msgr2=0x5654216f3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:12.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbbf7fff700 1 --2- 172.21.15.2:0/478490100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216f3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbbf7fff700 1 -- 172.21.15.2:0/478490100 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf00090f0 con 0x5654216ea470 2023-11-16T18:31:12.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.205+0000 7fbbfd603700 1 --2- 172.21.15.2:0/478490100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:12.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.206+0000 7fbbf7fff700 1 --2- 172.21.15.2:0/478490100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 0x5654216f3240 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4099b70 tx=0x7fbbe409d170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df4c01cff0e96ba9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:12.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.206+0000 7fbbd7fff700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:31:12.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.206+0000 7fbbd7fff700 1 -- 172.21.15.2:0/478490100 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe409d560 con 0x5654216ea470 2023-11-16T18:31:12.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.206+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/478490100 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5654216dcb20 con 0x5654216ea470 2023-11-16T18:31:12.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.206+0000 7fbbd7fff700 1 -- 172.21.15.2:0/478490100 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fbbe40cde40 con 0x5654216ea470 2023-11-16T18:31:12.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.206+0000 7fbbd7fff700 1 -- 172.21.15.2:0/478490100 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbe40d0150 con 0x5654216ea470 2023-11-16T18:31:12.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.206+0000 7fbc0b1f2b80 10 client.26294 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:12.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.206+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/478490100 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x565421601190 con 0x5654216ea470 2023-11-16T18:31:12.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.206+0000 7fbbd7fff700 1 -- 172.21.15.2:0/478490100 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbe40cb080 con 0x5654216ea470 2023-11-16T18:31:12.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.206+0000 7fbbd7fff700 1 -- 172.21.15.2:0/478490100 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fbbe40a5110 con 0x5654216ea470 2023-11-16T18:31:12.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.207+0000 7fbc0b1f2b80 10 client.26294 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:12.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.207+0000 7fbc0b1f2b80 10 client.26294 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2023-11-16T18:31:12.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.207+0000 7fbc0b1f2b80 20 client.26294 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:12.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.207+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/478490100 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x5654217034a0 0x5654217058b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.207+0000 7fbc0b1f2b80 4 client.26294 mds_command: new command op to 24446 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:31:12.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.207+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/478490100 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654215fd0a0 con 0x5654217034a0 2023-11-16T18:31:12.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.207+0000 7fbbfce02700 1 --2- 172.21.15.2:0/478490100 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x5654217034a0 0x5654217058b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.207+0000 7fbbfce02700 1 --2- 172.21.15.2:0/478490100 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x5654217034a0 0x5654217058b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac740a563ad7bcf1 server_cookie=62bf2a705dfb41e8 in_seq=0 out_seq=0 2023-11-16T18:31:12.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.207+0000 7fbbd7fff700 10 client.26294 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:31:12.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbbd7fff700 1 -- 172.21.15.2:0/478490100 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5654215fd0a0 con 0x5654217034a0 2023-11-16T18:31:12.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbbd7fff700 10 client.26294 handle_command_reply: tid=0 2023-11-16T18:31:12.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbc0b1f2b80 1 client.26294 shutdown 2023-11-16T18:31:12.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/478490100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 msgr2=0x5654216f3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:12.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/478490100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 0x5654216f3240 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4099b70 tx=0x7fbbe409d170 comp rx=0 tx=0).stop 2023-11-16T18:31:12.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/478490100 shutdown_connections 2023-11-16T18:31:12.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/478490100 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x5654217034a0 0x5654217058b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/478490100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216e9b60 0x5654216f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbbd7fff700 0 client.26294 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:31:12.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/478490100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/478490100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 0x5654216f3240 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/478490100 >> 172.21.15.2:0/478490100 conn(0x5654216df360 msgr2=0x5654216efd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:12.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/478490100 shutdown_connections 2023-11-16T18:31:12.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/478490100 wait complete. 2023-11-16T18:31:12.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.317+0000 7fbc0b1f2b80 20 client.26294 trim_cache size 0 max 16384 2023-11-16T18:31:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbc0b1f2b80 1 Processor -- start 2023-11-16T18:31:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbc0b1f2b80 1 -- start start 2023-11-16T18:31:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbc0b1f2b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216e9b60 0x5654216e4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbc0b1f2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216ea470 0x5654216fa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbc0b1f2b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216ff150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbc0b1f2b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654216f43b0 con 0x5654216ee8b0 2023-11-16T18:31:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbc0b1f2b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654215fd0a0 con 0x5654216e9b60 2023-11-16T18:31:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbc0b1f2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654216ed360 con 0x5654216ea470 2023-11-16T18:31:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbbfd603700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216ff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbbfd603700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216ff150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52982/0 (socket says 172.21.15.2:52982) 2023-11-16T18:31:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbbfd603700 1 -- 172.21.15.2:0/1907816772 learned_addr learned my addr 172.21.15.2:0/1907816772 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbbf7fff700 1 --2- 172.21.15.2:0/1907816772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216ea470 0x5654216fa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbbfce02700 1 --2- 172.21.15.2:0/1907816772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216e9b60 0x5654216e4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbbfd603700 1 -- 172.21.15.2:0/1907816772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216ea470 msgr2=0x5654216fa9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbbfd603700 1 --2- 172.21.15.2:0/1907816772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216ea470 0x5654216fa9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbbfd603700 1 -- 172.21.15.2:0/1907816772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216e9b60 msgr2=0x5654216e4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbbfd603700 1 --2- 172.21.15.2:0/1907816772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216e9b60 0x5654216e4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.332+0000 7fbbfd603700 1 -- 172.21.15.2:0/1907816772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654216e6240 con 0x5654216ee8b0 2023-11-16T18:31:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.333+0000 7fbbfd603700 1 --2- 172.21.15.2:0/1907816772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216ff150 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009410 tx=0x7fbbf0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c1f9592aadab14 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.333+0000 7fbbe37fe700 1 -- 172.21.15.2:0/1907816772 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf000e040 con 0x5654216ee8b0 2023-11-16T18:31:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.333+0000 7fbbe37fe700 1 -- 172.21.15.2:0/1907816772 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbf0003c00 con 0x5654216ee8b0 2023-11-16T18:31:12.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.333+0000 7fbbe37fe700 1 -- 172.21.15.2:0/1907816772 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00137b0 con 0x5654216ee8b0 2023-11-16T18:31:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.334+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/1907816772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 msgr2=0x5654216ff150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.334+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/1907816772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216ff150 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009410 tx=0x7fbbf0009b20 comp rx=0 tx=0).stop 2023-11-16T18:31:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.334+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/1907816772 shutdown_connections 2023-11-16T18:31:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.334+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/1907816772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654216ff150 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.334+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/1907816772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216e9b60 0x5654216e4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.334+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/1907816772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216ea470 0x5654216fa9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.334+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/1907816772 >> 172.21.15.2:0/1907816772 conn(0x5654216df360 msgr2=0x56542170a210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.334+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/1907816772 shutdown_connections 2023-11-16T18:31:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.334+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/1907816772 wait complete. 2023-11-16T18:31:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.334+0000 7fbc0b1f2b80 1 -- start start 2023-11-16T18:31:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.334+0000 7fbc0b1f2b80 1 Processor -- start 2023-11-16T18:31:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbc0b1f2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216e9b60 0x5654216f8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbc0b1f2b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 0x5654216f90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbc0b1f2b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654217503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbc0b1f2b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654217508e0 con 0x5654216ee8b0 2023-11-16T18:31:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbc0b1f2b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x565421750a50 con 0x5654216ea470 2023-11-16T18:31:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbc0b1f2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x565421750c10 con 0x5654216e9b60 2023-11-16T18:31:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbc0b1f2b80 10 client.0 _ll_register_callbacks cb 0x7fbbe409ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbbfd603700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654217503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbbf7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 0x5654216f90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbbfd603700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654217503a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52988/0 (socket says 172.21.15.2:52988) 2023-11-16T18:31:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbbf7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 0x5654216f90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43424/0 (socket says 172.21.15.2:43424) 2023-11-16T18:31:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbbfd603700 1 -- 172.21.15.2:0/3239995781 learned_addr learned my addr 172.21.15.2:0/3239995781 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbbfce02700 1 --2- 172.21.15.2:0/3239995781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216e9b60 0x5654216f8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbbf7fff700 1 -- 172.21.15.2:0/3239995781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216e9b60 msgr2=0x5654216f8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbbf7fff700 1 --2- 172.21.15.2:0/3239995781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216e9b60 0x5654216f8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbbf7fff700 1 -- 172.21.15.2:0/3239995781 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 msgr2=0x5654217503a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.335+0000 7fbbf7fff700 1 --2- 172.21.15.2:0/3239995781 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654217503a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.336+0000 7fbbf7fff700 1 -- 172.21.15.2:0/3239995781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654214bc630 con 0x5654216ea470 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.336+0000 7fbbf7fff700 1 --2- 172.21.15.2:0/3239995781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 0x5654216f90c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4099600 tx=0x7fbbe4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=818e3b913b9c86f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.336+0000 7fbbd7fff700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.336+0000 7fbbd7fff700 1 -- 172.21.15.2:0/3239995781 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe40bb330 con 0x5654216ea470 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.336+0000 7fbbd7fff700 1 -- 172.21.15.2:0/3239995781 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fbbe40cde40 con 0x5654216ea470 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.336+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/3239995781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbbf000f040 con 0x5654216ea470 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.337+0000 7fbbd7fff700 1 -- 172.21.15.2:0/3239995781 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbe4099440 con 0x5654216ea470 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.337+0000 7fbc0b1f2b80 10 client.26300 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.337+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/3239995781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x565421468de0 con 0x5654216ea470 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.337+0000 7fbbd7fff700 1 -- 172.21.15.2:0/3239995781 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbf000f040 con 0x5654216ea470 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.337+0000 7fbbd7fff700 1 -- 172.21.15.2:0/3239995781 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fbbe40c78a0 con 0x5654216ea470 2023-11-16T18:31:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.337+0000 7fbc0b1f2b80 10 client.26300 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:12.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.337+0000 7fbc0b1f2b80 10 client.26300 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2023-11-16T18:31:12.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.337+0000 7fbc0b1f2b80 20 client.26300 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:12.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.337+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/3239995781 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x565421757f00 0x56542175a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:12.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.337+0000 7fbc0b1f2b80 4 client.26300 mds_command: new command op to 24446 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:31:12.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.337+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/3239995781 --> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x565421609660 con 0x565421757f00 2023-11-16T18:31:12.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.337+0000 7fbbfce02700 1 --2- 172.21.15.2:0/3239995781 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x565421757f00 0x56542175a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:12.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.338+0000 7fbbd7fff700 10 client.26300 ms_handle_connect on v2:172.21.15.40:6836/248461642 2023-11-16T18:31:12.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.338+0000 7fbbfce02700 1 --2- 172.21.15.2:0/3239995781 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x565421757f00 0x56542175a2e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b4d8d6763c6f6873 server_cookie=af1d0f167d24730 in_seq=0 out_seq=0 2023-11-16T18:31:12.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:12 smithi002 ceph-mon[98807]: pgmap v193: 129 pgs: 129 active+clean; 721 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 196 op/s 2023-11-16T18:31:12.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:12 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2390489462' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:31:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:12 smithi040 ceph-mon[100251]: pgmap v193: 129 pgs: 129 active+clean; 721 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 196 op/s 2023-11-16T18:31:12.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:12 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2390489462' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:31:12.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:12 smithi188 ceph-mon[100639]: pgmap v193: 129 pgs: 129 active+clean; 721 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 196 op/s 2023-11-16T18:31:12.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:12 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2390489462' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:31:12.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.896+0000 7fbbd7fff700 1 -- 172.21.15.2:0/3239995781 <== mds.2 v2:172.21.15.40:6836/248461642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3321 (crc 0 0 0) 0x565421609660 con 0x565421757f00 2023-11-16T18:31:12.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.896+0000 7fbbd7fff700 10 client.26300 handle_command_reply: tid=0 2023-11-16T18:31:12.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.896+0000 7fbc0b1f2b80 1 client.26300 shutdown 2023-11-16T18:31:12.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.897+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/3239995781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 msgr2=0x5654216f90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:12.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.897+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/3239995781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 0x5654216f90c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4099600 tx=0x7fbbe4000bc0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.897+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/3239995781 shutdown_connections 2023-11-16T18:31:12.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.897+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/3239995781 >> [v2:172.21.15.40:6836/248461642,v1:172.21.15.40:6837/248461642] conn(0x565421757f00 0x56542175a2e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.897+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/3239995781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654216e9b60 0x5654216f8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.897+0000 7fbbd7fff700 0 client.26300 ms_handle_reset on v2:172.21.15.40:6836/248461642 2023-11-16T18:31:12.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.897+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/3239995781 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654216ee8b0 0x5654217503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.897+0000 7fbc0b1f2b80 1 --2- 172.21.15.2:0/3239995781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654216ea470 0x5654216f90c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.897+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/3239995781 >> 172.21.15.2:0/3239995781 conn(0x5654216df360 msgr2=0x5654216ec8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:12.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.897+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/3239995781 shutdown_connections 2023-11-16T18:31:12.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.897+0000 7fbc0b1f2b80 1 -- 172.21.15.2:0/3239995781 wait complete. 2023-11-16T18:31:12.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.897+0000 7fbc0b1f2b80 20 client.26300 trim_cache size 0 max 16384 2023-11-16T18:31:12.901 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2023-11-16T18:31:12.901 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.901 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.901 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 23268747, 2023-11-16T18:31:12.901 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642287, 2023-11-16T18:31:12.901 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9959" 2023-11-16T18:31:12.901 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:31:12.902 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.902 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.902 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 49177939, 2023-11-16T18:31:12.902 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642270, 2023-11-16T18:31:12.902 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9943" 2023-11-16T18:31:12.902 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:31:12.902 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.902 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.902 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 179423603, 2023-11-16T18:31:12.902 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642284, 2023-11-16T18:31:12.902 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9956" 2023-11-16T18:31:12.902 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:31:12.903 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.903 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.904 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 320634055, 2023-11-16T18:31:12.904 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642278, 2023-11-16T18:31:12.904 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9950" 2023-11-16T18:31:12.904 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:31:12.904 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.904 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.904 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 387783030, 2023-11-16T18:31:12.904 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642285, 2023-11-16T18:31:12.904 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9957" 2023-11-16T18:31:12.904 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:31:12.904 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.905 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.905 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 674883623, 2023-11-16T18:31:12.905 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642295, 2023-11-16T18:31:12.905 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9967" 2023-11-16T18:31:12.905 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:31:12.905 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.905 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.905 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 699703911, 2023-11-16T18:31:12.905 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642293, 2023-11-16T18:31:12.905 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9965" 2023-11-16T18:31:12.905 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:31:12.905 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.905 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.906 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 1384656058, 2023-11-16T18:31:12.906 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642274, 2023-11-16T18:31:12.906 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9946" 2023-11-16T18:31:12.906 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:31:12.906 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.906 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.906 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 1436072723, 2023-11-16T18:31:12.906 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642277, 2023-11-16T18:31:12.906 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9949" 2023-11-16T18:31:12.906 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:31:12.906 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.906 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.906 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 1562705912, 2023-11-16T18:31:12.907 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642281, 2023-11-16T18:31:12.907 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9953" 2023-11-16T18:31:12.907 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:31:12.907 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.907 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.907 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 2017776999, 2023-11-16T18:31:12.907 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642273, 2023-11-16T18:31:12.907 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9945" 2023-11-16T18:31:12.907 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:31:12.907 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.907 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.907 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 2424408919, 2023-11-16T18:31:12.907 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642279, 2023-11-16T18:31:12.908 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9951" 2023-11-16T18:31:12.908 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:31:12.908 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.908 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.908 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 3284816971, 2023-11-16T18:31:12.908 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642307, 2023-11-16T18:31:12.908 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9977" 2023-11-16T18:31:12.908 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:31:12.908 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:31:12.908 INFO:teuthology.orchestra.run.smithi002.stdout: "damage_type": "backtrace", 2023-11-16T18:31:12.908 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4058982802, 2023-11-16T18:31:12.908 INFO:teuthology.orchestra.run.smithi002.stdout: "ino": 1099511642306, 2023-11-16T18:31:12.920 INFO:teuthology.orchestra.run.smithi002.stdout: "path": "/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/file9398" 2023-11-16T18:31:12.920 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:31:12.920 INFO:teuthology.orchestra.run.smithi002.stdout:] 2023-11-16T18:31:12.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.898+0000 7fbbff066700 1 -- 172.21.15.2:0/2690462893 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fbbe80a12c0 msgr2=0x7fbbe80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:12.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.898+0000 7fbbff066700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fbbe80a12c0 0x7fbbe80a3770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbbec009de0 tx=0x7fbbec009190 comp rx=0 tx=0).stop 2023-11-16T18:31:12.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.898+0000 7fbbff066700 1 -- 172.21.15.2:0/2690462893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbf8100420 msgr2=0x7fbbf81958a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:12.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.898+0000 7fbbff066700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbf8100420 0x7fbbf81958a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4005910 tx=0x7fbbe4004730 comp rx=0 tx=0).stop 2023-11-16T18:31:12.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.898+0000 7fbbff066700 1 -- 172.21.15.2:0/2690462893 shutdown_connections 2023-11-16T18:31:12.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.898+0000 7fbbff066700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fbbe80a12c0 0x7fbbe80a3770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.898+0000 7fbbff066700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbf8105250 0x7fbbf8199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.898+0000 7fbbff066700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbf80ffa70 0x7fbbf8195360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:12.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.898+0000 7fbbff066700 1 --2- 172.21.15.2:0/2690462893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbf8100420 0x7fbbf81958a0 secure :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4005910 tx=0x7fbbe4004730 comp rx=0 tx=0).stop 2023-11-16T18:31:12.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.898+0000 7fbbff066700 1 -- 172.21.15.2:0/2690462893 >> 172.21.15.2:0/2690462893 conn(0x7fbbf80763a0 msgr2=0x7fbbf80fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:12.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.899+0000 7fbbff066700 1 -- 172.21.15.2:0/2690462893 shutdown_connections 2023-11-16T18:31:12.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:12.899+0000 7fbbff066700 1 -- 172.21.15.2:0/2690462893 wait complete. 2023-11-16T18:31:12.924 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-11-16T18:31:13.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.055+0000 7fa40e3ae700 1 -- 172.21.15.2:0/2360501755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4081009e0 msgr2=0x7fa408100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.055+0000 7fa40e3ae700 1 --2- 172.21.15.2:0/2360501755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4081009e0 0x7fa408100e50 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009990 tx=0x7fa3fc009ca0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.055+0000 7fa40e3ae700 1 -- 172.21.15.2:0/2360501755 shutdown_connections 2023-11-16T18:31:13.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.055+0000 7fa40e3ae700 1 --2- 172.21.15.2:0/2360501755 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4081064d0 0x7fa4081068a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.055+0000 7fa40e3ae700 1 --2- 172.21.15.2:0/2360501755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408101390 0x7fa40810d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.055+0000 7fa40e3ae700 1 --2- 172.21.15.2:0/2360501755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4081009e0 0x7fa408100e50 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.055+0000 7fa40e3ae700 1 -- 172.21.15.2:0/2360501755 >> 172.21.15.2:0/2360501755 conn(0x7fa408077530 msgr2=0x7fa408077930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:13.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.055+0000 7fa40e3ae700 1 -- 172.21.15.2:0/2360501755 shutdown_connections 2023-11-16T18:31:13.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.055+0000 7fa40e3ae700 1 -- 172.21.15.2:0/2360501755 wait complete. 2023-11-16T18:31:13.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.056+0000 7fa40e3ae700 1 Processor -- start 2023-11-16T18:31:13.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.056+0000 7fa40e3ae700 1 -- start start 2023-11-16T18:31:13.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.056+0000 7fa40e3ae700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4081009e0 0x7fa40819e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.056+0000 7fa40e3ae700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa408101390 0x7fa40819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.056+0000 7fa40e3ae700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4081064d0 0x7fa408198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.056+0000 7fa40e3ae700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa408198ca0 con 0x7fa4081064d0 2023-11-16T18:31:13.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.056+0000 7fa40e3ae700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa408198e10 con 0x7fa4081009e0 2023-11-16T18:31:13.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.056+0000 7fa40e3ae700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa408198fd0 con 0x7fa408101390 2023-11-16T18:31:13.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.056+0000 7fa40c94b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4081064d0 0x7fa408198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.056+0000 7fa407fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4081009e0 0x7fa40819e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.056+0000 7fa40c94b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4081064d0 0x7fa408198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53002/0 (socket says 172.21.15.2:53002) 2023-11-16T18:31:13.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.056+0000 7fa40c94b700 1 -- 172.21.15.2:0/3425417791 learned_addr learned my addr 172.21.15.2:0/3425417791 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:13.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa4077fe700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa408101390 0x7fa40819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa40c94b700 1 -- 172.21.15.2:0/3425417791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa408101390 msgr2=0x7fa40819eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa40c94b700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa408101390 0x7fa40819eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa40c94b700 1 -- 172.21.15.2:0/3425417791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4081009e0 msgr2=0x7fa40819e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa40c94b700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4081009e0 0x7fa40819e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa40c94b700 1 -- 172.21.15.2:0/3425417791 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc0095f0 con 0x7fa4081064d0 2023-11-16T18:31:13.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa407fff700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4081009e0 0x7fa40819e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:13.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa4077fe700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa408101390 0x7fa40819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:31:13.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa40c94b700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4081064d0 0x7fa408198630 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa40000ef30 tx=0x7fa40000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:13.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa4057fa700 1 -- 172.21.15.2:0/3425417791 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400018070 con 0x7fa4081064d0 2023-11-16T18:31:13.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa4057fa700 1 -- 172.21.15.2:0/3425417791 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa40000f040 con 0x7fa4081064d0 2023-11-16T18:31:13.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa40e3ae700 1 -- 172.21.15.2:0/3425417791 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4080749d0 con 0x7fa4081064d0 2023-11-16T18:31:13.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa4057fa700 1 -- 172.21.15.2:0/3425417791 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40000cdb0 con 0x7fa4081064d0 2023-11-16T18:31:13.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.057+0000 7fa40e3ae700 1 -- 172.21.15.2:0/3425417791 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa408074840 con 0x7fa4081064d0 2023-11-16T18:31:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.060+0000 7fa4057fa700 1 -- 172.21.15.2:0/3425417791 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fa400003a00 con 0x7fa4081064d0 2023-11-16T18:31:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.060+0000 7fa4057fa700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa3f00a1340 0x7fa3f00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.061+0000 7fa4057fa700 1 -- 172.21.15.2:0/3425417791 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa4000ccc40 con 0x7fa4081064d0 2023-11-16T18:31:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.061+0000 7fa407fff700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa3f00a1340 0x7fa3f00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.061+0000 7fa407fff700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa3f00a1340 0x7fa3f00a37f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009de0 tx=0x7fa3f8009190 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.061+0000 7fa41a53ab80 1 Processor -- start 2023-11-16T18:31:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.061+0000 7fa41a53ab80 1 -- start start 2023-11-16T18:31:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa41a53ab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9cef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa41a53ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 0x55e7fd9d3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa41a53ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa41a53ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7fd8d4060 con 0x55e7fd9d38b0 2023-11-16T18:31:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa41a53ab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7fd9268a0 con 0x55e7fd9ceb60 2023-11-16T18:31:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa41a53ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7fd91fd30 con 0x55e7fd9cf470 2023-11-16T18:31:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa40c94b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa407fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa40c94b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53010/0 (socket says 172.21.15.2:53010) 2023-11-16T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa40c94b700 1 -- 172.21.15.2:0/3891917157 learned_addr learned my addr 172.21.15.2:0/3891917157 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa4077fe700 1 --2- 172.21.15.2:0/3891917157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 0x55e7fd9d3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa40c94b700 1 -- 172.21.15.2:0/3891917157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 msgr2=0x55e7fd9d3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa40c94b700 1 --2- 172.21.15.2:0/3891917157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 0x55e7fd9d3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa40c94b700 1 -- 172.21.15.2:0/3891917157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 msgr2=0x55e7fd9cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa40c94b700 1 --2- 172.21.15.2:0/3891917157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9cef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa40c94b700 1 -- 172.21.15.2:0/3891917157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7fd7a1630 con 0x55e7fd9d38b0 2023-11-16T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa407fff700 1 --2- 172.21.15.2:0/3891917157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa40c94b700 1 --2- 172.21.15.2:0/3891917157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d3d00 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa4000090a0 tx=0x7fa400099570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c89aea114449c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa3ee7fc700 1 -- 172.21.15.2:0/3891917157 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400099a30 con 0x55e7fd9d38b0 2023-11-16T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa3ee7fc700 1 -- 172.21.15.2:0/3891917157 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa40009ae90 con 0x55e7fd9d38b0 2023-11-16T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.062+0000 7fa3ee7fc700 1 -- 172.21.15.2:0/3891917157 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400099e90 con 0x55e7fd9d38b0 2023-11-16T18:31:13.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.063+0000 7fa41a53ab80 1 -- 172.21.15.2:0/3891917157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 msgr2=0x55e7fd9d3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.063+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/3891917157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d3d00 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa4000090a0 tx=0x7fa400099570 comp rx=0 tx=0).stop 2023-11-16T18:31:13.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.063+0000 7fa41a53ab80 1 -- 172.21.15.2:0/3891917157 shutdown_connections 2023-11-16T18:31:13.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.063+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/3891917157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d3d00 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.063+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/3891917157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.063+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/3891917157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 0x55e7fd9d3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.063+0000 7fa41a53ab80 1 -- 172.21.15.2:0/3891917157 >> 172.21.15.2:0/3891917157 conn(0x55e7fd9c4360 msgr2=0x55e7fd9c9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:13.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.063+0000 7fa41a53ab80 1 -- 172.21.15.2:0/3891917157 shutdown_connections 2023-11-16T18:31:13.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.063+0000 7fa41a53ab80 1 -- 172.21.15.2:0/3891917157 wait complete. 2023-11-16T18:31:13.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.064+0000 7fa41a53ab80 1 -- start start 2023-11-16T18:31:13.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.064+0000 7fa41a53ab80 1 Processor -- start 2023-11-16T18:31:13.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.064+0000 7fa41a53ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9db0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.064+0000 7fa41a53ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9cf470 0x55e7fd9d8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.064+0000 7fa41a53ab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.064+0000 7fa41a53ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7fd8c18e0 con 0x55e7fd9cf470 2023-11-16T18:31:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.064+0000 7fa41a53ab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7fd8c1580 con 0x55e7fd9d38b0 2023-11-16T18:31:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.064+0000 7fa41a53ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7fd7a3920 con 0x55e7fd9ceb60 2023-11-16T18:31:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.064+0000 7fa41a53ab80 10 client.0 _ll_register_callbacks cb 0x55e7fd8df780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa4077fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9cf470 0x55e7fd9d8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa407fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9db0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa4077fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9cf470 0x55e7fd9d8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53014/0 (socket says 172.21.15.2:53014) 2023-11-16T18:31:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa4077fe700 1 -- 172.21.15.2:0/1799013170 learned_addr learned my addr 172.21.15.2:0/1799013170 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa40c94b700 1 --2- 172.21.15.2:0/1799013170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa407fff700 1 -- 172.21.15.2:0/1799013170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9d38b0 msgr2=0x55e7fd9d8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa407fff700 1 --2- 172.21.15.2:0/1799013170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa407fff700 1 -- 172.21.15.2:0/1799013170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9cf470 msgr2=0x55e7fd9d8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa407fff700 1 --2- 172.21.15.2:0/1799013170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9cf470 0x55e7fd9d8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa407fff700 1 -- 172.21.15.2:0/1799013170 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa408074840 con 0x55e7fd9ceb60 2023-11-16T18:31:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa407fff700 1 --2- 172.21.15.2:0/1799013170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9db0c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80098e0 tx=0x7fa3f8004860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dc6e98c8504a45 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa3e77fe700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:31:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa3e77fe700 1 -- 172.21.15.2:0/1799013170 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8013450 con 0x55e7fd9ceb60 2023-11-16T18:31:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.065+0000 7fa3e77fe700 1 -- 172.21.15.2:0/1799013170 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa3f80135b0 con 0x55e7fd9ceb60 2023-11-16T18:31:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.066+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1799013170 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e7fd9c1b20 con 0x55e7fd9ceb60 2023-11-16T18:31:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.066+0000 7fa3e77fe700 1 -- 172.21.15.2:0/1799013170 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3f8013770 con 0x55e7fd9ceb60 2023-11-16T18:31:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.066+0000 7fa41a53ab80 10 client.26176 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.066+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1799013170 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e7fd8e6190 con 0x55e7fd9ceb60 2023-11-16T18:31:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.066+0000 7fa3e77fe700 1 -- 172.21.15.2:0/1799013170 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e7fd9c1b20 con 0x55e7fd9ceb60 2023-11-16T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.066+0000 7fa3e77fe700 1 -- 172.21.15.2:0/1799013170 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa3f8021910 con 0x55e7fd9ceb60 2023-11-16T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.066+0000 7fa41a53ab80 10 client.26176 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.066+0000 7fa41a53ab80 10 client.26176 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-11-16T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.066+0000 7fa41a53ab80 20 client.26176 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.066+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1799013170 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55e7fd9e89f0 0x55e7fd9eae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.066+0000 7fa41a53ab80 4 client.26176 mds_command: new command op to 24460 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.066+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1799013170 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e7fd8e20a0 con 0x55e7fd9e89f0 2023-11-16T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.066+0000 7fa4077fe700 1 --2- 172.21.15.2:0/1799013170 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55e7fd9e89f0 0x55e7fd9eae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.067+0000 7fa3e77fe700 10 client.26176 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.067+0000 7fa4077fe700 1 --2- 172.21.15.2:0/1799013170 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55e7fd9e89f0 0x55e7fd9eae00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=56b175f25ea88f1f server_cookie=ab4de93b62e19a9b in_seq=0 out_seq=0 2023-11-16T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.067+0000 7fa3e77fe700 1 -- 172.21.15.2:0/1799013170 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e7fd8e20a0 con 0x55e7fd9e89f0 2023-11-16T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.067+0000 7fa3e77fe700 10 client.26176 handle_command_reply: tid=0 2023-11-16T18:31:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.067+0000 7fa41a53ab80 1 client.26176 shutdown 2023-11-16T18:31:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.068+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1799013170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9ceb60 msgr2=0x55e7fd9db0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.068+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1799013170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9db0c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80098e0 tx=0x7fa3f8004860 comp rx=0 tx=0).stop 2023-11-16T18:31:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.068+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1799013170 shutdown_connections 2023-11-16T18:31:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.068+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1799013170 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55e7fd9e89f0 0x55e7fd9eae00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.068+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1799013170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9db0c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.068+0000 7fa3e77fe700 0 client.26176 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:31:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.068+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1799013170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.068+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1799013170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9cf470 0x55e7fd9d8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.068+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1799013170 >> 172.21.15.2:0/1799013170 conn(0x55e7fd9c4360 msgr2=0x55e7fd9d4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.068+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1799013170 shutdown_connections 2023-11-16T18:31:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.068+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1799013170 wait complete. 2023-11-16T18:31:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.068+0000 7fa41a53ab80 20 client.26176 trim_cache size 0 max 16384 2023-11-16T18:31:13.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.082+0000 7fa41a53ab80 1 Processor -- start 2023-11-16T18:31:13.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.082+0000 7fa41a53ab80 1 -- start start 2023-11-16T18:31:13.085 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.082+0000 7fa41a53ab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9d6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.082+0000 7fa41a53ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 0x55e7fd9df9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.082+0000 7fa41a53ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9dac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.082+0000 7fa41a53ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7fd9d24b0 con 0x55e7fd9d38b0 2023-11-16T18:31:13.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.082+0000 7fa41a53ab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7fd9ca800 con 0x55e7fd9ceb60 2023-11-16T18:31:13.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.082+0000 7fa41a53ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7fd9cf900 con 0x55e7fd9cf470 2023-11-16T18:31:13.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.082+0000 7fa40c94b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9dac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa407fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9d6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa40c94b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9dac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53024/0 (socket says 172.21.15.2:53024) 2023-11-16T18:31:13.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa407fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9d6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43458/0 (socket says 172.21.15.2:43458) 2023-11-16T18:31:13.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa40c94b700 1 -- 172.21.15.2:0/1372885322 learned_addr learned my addr 172.21.15.2:0/1372885322 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:13.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa4077fe700 1 --2- 172.21.15.2:0/1372885322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 0x55e7fd9df9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa40c94b700 1 -- 172.21.15.2:0/1372885322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 msgr2=0x55e7fd9df9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa40c94b700 1 --2- 172.21.15.2:0/1372885322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 0x55e7fd9df9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa40c94b700 1 -- 172.21.15.2:0/1372885322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 msgr2=0x55e7fd9d6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa40c94b700 1 --2- 172.21.15.2:0/1372885322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9d6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa40c94b700 1 -- 172.21.15.2:0/1372885322 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7fd8e5ee0 con 0x55e7fd9d38b0 2023-11-16T18:31:13.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa407fff700 1 --2- 172.21.15.2:0/1372885322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9d6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:13.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa40c94b700 1 --2- 172.21.15.2:0/1372885322 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9dac60 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa400003f90 tx=0x7fa4000092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563d86306974fa94 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:13.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa3eeffd700 1 -- 172.21.15.2:0/1372885322 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400099a30 con 0x55e7fd9d38b0 2023-11-16T18:31:13.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa3eeffd700 1 -- 172.21.15.2:0/1372885322 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa40009da60 con 0x55e7fd9d38b0 2023-11-16T18:31:13.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.083+0000 7fa3eeffd700 1 -- 172.21.15.2:0/1372885322 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4000991a0 con 0x55e7fd9d38b0 2023-11-16T18:31:13.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.084+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1372885322 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 msgr2=0x55e7fd9dac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.084+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1372885322 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9dac60 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa400003f90 tx=0x7fa4000092e0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.084+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1372885322 shutdown_connections 2023-11-16T18:31:13.090 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.084+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1372885322 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9dac60 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.084+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1372885322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9d6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.084+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1372885322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 0x55e7fd9df9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.084+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1372885322 >> 172.21.15.2:0/1372885322 conn(0x55e7fd9c4360 msgr2=0x55e7fd9d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:13.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.084+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1372885322 shutdown_connections 2023-11-16T18:31:13.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.084+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1372885322 wait complete. 2023-11-16T18:31:13.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.084+0000 7fa41a53ab80 1 -- start start 2023-11-16T18:31:13.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa41a53ab80 1 Processor -- start 2023-11-16T18:31:13.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa41a53ab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9df9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa41a53ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 0x55e7fd9d6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa41a53ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa41a53ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7fd9d7720 con 0x55e7fd9d38b0 2023-11-16T18:31:13.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa41a53ab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7fd9d7890 con 0x55e7fd9ceb60 2023-11-16T18:31:13.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa41a53ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7fd9d7a50 con 0x55e7fd9cf470 2023-11-16T18:31:13.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa41a53ab80 10 client.0 _ll_register_callbacks cb 0x7fa3f8015550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:13.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa40c94b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa407fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9df9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa407fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9df9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43464/0 (socket says 172.21.15.2:43464) 2023-11-16T18:31:13.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa40c94b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53030/0 (socket says 172.21.15.2:53030) 2023-11-16T18:31:13.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa407fff700 1 -- 172.21.15.2:0/1048331570 learned_addr learned my addr 172.21.15.2:0/1048331570 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:13.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.085+0000 7fa4077fe700 1 --2- 172.21.15.2:0/1048331570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 0x55e7fd9d6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.086+0000 7fa407fff700 1 -- 172.21.15.2:0/1048331570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 msgr2=0x55e7fd9d6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.086+0000 7fa407fff700 1 --2- 172.21.15.2:0/1048331570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 0x55e7fd9d6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.086+0000 7fa407fff700 1 -- 172.21.15.2:0/1048331570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 msgr2=0x55e7fd9d7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.086+0000 7fa407fff700 1 --2- 172.21.15.2:0/1048331570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.086+0000 7fa407fff700 1 -- 172.21.15.2:0/1048331570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7fd8e6190 con 0x55e7fd9ceb60 2023-11-16T18:31:13.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.086+0000 7fa407fff700 1 --2- 172.21.15.2:0/1048331570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9df9b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8006bc0 tx=0x7fa3f8021650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9e32c54db5e119e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:13.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.086+0000 7fa3e77fe700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-11-16T18:31:13.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.086+0000 7fa3e77fe700 1 -- 172.21.15.2:0/1048331570 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8015b80 con 0x55e7fd9ceb60 2023-11-16T18:31:13.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.086+0000 7fa3e77fe700 1 -- 172.21.15.2:0/1048331570 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa3f80205d0 con 0x55e7fd9ceb60 2023-11-16T18:31:13.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.086+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1048331570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa40009ae90 con 0x55e7fd9ceb60 2023-11-16T18:31:13.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.086+0000 7fa3e77fe700 1 -- 172.21.15.2:0/1048331570 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3f80203e0 con 0x55e7fd9ceb60 2023-11-16T18:31:13.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.086+0000 7fa41a53ab80 10 client.26318 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:13.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.086+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1048331570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e7fd7a1630 con 0x55e7fd9ceb60 2023-11-16T18:31:13.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.087+0000 7fa3e77fe700 1 -- 172.21.15.2:0/1048331570 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa40009ae90 con 0x55e7fd9ceb60 2023-11-16T18:31:13.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.087+0000 7fa3e77fe700 1 -- 172.21.15.2:0/1048331570 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa3f802a5d0 con 0x55e7fd9ceb60 2023-11-16T18:31:13.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.087+0000 7fa41a53ab80 10 client.26318 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:13.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.087+0000 7fa41a53ab80 10 client.26318 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-11-16T18:31:13.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.087+0000 7fa41a53ab80 20 client.26318 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:13.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.087+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1048331570 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55e7fda3d1e0 0x55e7fda3f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.087+0000 7fa41a53ab80 4 client.26318 mds_command: new command op to 24460 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:31:13.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.087+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1048331570 --> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e7fd8ee660 con 0x55e7fda3d1e0 2023-11-16T18:31:13.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.087+0000 7fa4077fe700 1 --2- 172.21.15.2:0/1048331570 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55e7fda3d1e0 0x55e7fda3f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.089+0000 7fa4077fe700 1 --2- 172.21.15.2:0/1048331570 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55e7fda3d1e0 0x55e7fda3f5f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a2625e7846c96a71 server_cookie=8db8440fc219f7e4 in_seq=0 out_seq=0 2023-11-16T18:31:13.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.089+0000 7fa3e77fe700 10 client.26318 ms_handle_connect on v2:172.21.15.188:6838/1646912892 2023-11-16T18:31:13.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa3e77fe700 1 -- 172.21.15.2:0/1048331570 <== mds.1 v2:172.21.15.188:6838/1646912892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e7fd8ee660 con 0x55e7fda3d1e0 2023-11-16T18:31:13.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa3e77fe700 10 client.26318 handle_command_reply: tid=0 2023-11-16T18:31:13.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa41a53ab80 1 client.26318 shutdown 2023-11-16T18:31:13.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1048331570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 msgr2=0x55e7fd9df9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1048331570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9df9b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8006bc0 tx=0x7fa3f8021650 comp rx=0 tx=0).stop 2023-11-16T18:31:13.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1048331570 shutdown_connections 2023-11-16T18:31:13.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1048331570 >> [v2:172.21.15.188:6838/1646912892,v1:172.21.15.188:6839/1646912892] conn(0x55e7fda3d1e0 0x55e7fda3f5f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa3e77fe700 0 client.26318 ms_handle_reset on v2:172.21.15.188:6838/1646912892 2023-11-16T18:31:13.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1048331570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e7fd9d38b0 0x55e7fd9d7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1048331570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7fd9ceb60 0x55e7fd9df9b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa41a53ab80 1 --2- 172.21.15.2:0/1048331570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7fd9cf470 0x55e7fd9d6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1048331570 >> 172.21.15.2:0/1048331570 conn(0x55e7fd9c4360 msgr2=0x55e7fd9d28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:13.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1048331570 shutdown_connections 2023-11-16T18:31:13.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa41a53ab80 1 -- 172.21.15.2:0/1048331570 wait complete. 2023-11-16T18:31:13.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.090+0000 7fa41a53ab80 20 client.26318 trim_cache size 0 max 16384 2023-11-16T18:31:13.096 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:31:13.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.091+0000 7fa40e3ae700 1 -- 172.21.15.2:0/3425417791 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa3f00a1340 msgr2=0x7fa3f00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.091+0000 7fa40e3ae700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa3f00a1340 0x7fa3f00a37f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009de0 tx=0x7fa3f8009190 comp rx=0 tx=0).stop 2023-11-16T18:31:13.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.091+0000 7fa40e3ae700 1 -- 172.21.15.2:0/3425417791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4081064d0 msgr2=0x7fa408198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.091+0000 7fa40e3ae700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4081064d0 0x7fa408198630 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa40000ef30 tx=0x7fa40000c3a0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.091+0000 7fa40e3ae700 1 -- 172.21.15.2:0/3425417791 shutdown_connections 2023-11-16T18:31:13.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.091+0000 7fa40e3ae700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa3f00a1340 0x7fa3f00a37f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.091+0000 7fa40e3ae700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4081064d0 0x7fa408198630 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.091+0000 7fa40e3ae700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4081009e0 0x7fa40819e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.091+0000 7fa40e3ae700 1 --2- 172.21.15.2:0/3425417791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa408101390 0x7fa40819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.091+0000 7fa40e3ae700 1 -- 172.21.15.2:0/3425417791 >> 172.21.15.2:0/3425417791 conn(0x7fa408077530 msgr2=0x7fa4080fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:13.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.091+0000 7fa40e3ae700 1 -- 172.21.15.2:0/3425417791 shutdown_connections 2023-11-16T18:31:13.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.091+0000 7fa40e3ae700 1 -- 172.21.15.2:0/3425417791 wait complete. 2023-11-16T18:31:13.107 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T18:31:13.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.237+0000 7fde831ea700 1 -- 172.21.15.2:0/2156074258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde7c068770 msgr2=0x7fde7c10d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.237+0000 7fde831ea700 1 --2- 172.21.15.2:0/2156074258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde7c068770 0x7fde7c10d0e0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fde7400b0a0 tx=0x7fde7400b3b0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.237+0000 7fde831ea700 1 -- 172.21.15.2:0/2156074258 shutdown_connections 2023-11-16T18:31:13.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.237+0000 7fde831ea700 1 --2- 172.21.15.2:0/2156074258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde7c068770 0x7fde7c10d0e0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.237+0000 7fde831ea700 1 --2- 172.21.15.2:0/2156074258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde7c105250 0x7fde7c105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.237+0000 7fde831ea700 1 --2- 172.21.15.2:0/2156074258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde7c067dc0 0x7fde7c068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.237+0000 7fde831ea700 1 -- 172.21.15.2:0/2156074258 >> 172.21.15.2:0/2156074258 conn(0x7fde7c0763a0 msgr2=0x7fde7c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:13.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.237+0000 7fde831ea700 1 -- 172.21.15.2:0/2156074258 shutdown_connections 2023-11-16T18:31:13.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.238+0000 7fde831ea700 1 -- 172.21.15.2:0/2156074258 wait complete. 2023-11-16T18:31:13.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.238+0000 7fde831ea700 1 Processor -- start 2023-11-16T18:31:13.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.238+0000 7fde831ea700 1 -- start start 2023-11-16T18:31:13.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.238+0000 7fde831ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde7c067dc0 0x7fde7c0ffb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.238+0000 7fde831ea700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde7c068770 0x7fde7c100080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.238+0000 7fde831ea700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde7c105250 0x7fde7c1a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.238+0000 7fde831ea700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c100920 con 0x7fde7c067dc0 2023-11-16T18:31:13.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.238+0000 7fde831ea700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1a4ca0 con 0x7fde7c068770 2023-11-16T18:31:13.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.238+0000 7fde831ea700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1a4de0 con 0x7fde7c105250 2023-11-16T18:31:13.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.238+0000 7fde80f86700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde7c067dc0 0x7fde7c0ffb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde80f86700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde7c067dc0 0x7fde7c0ffb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53040/0 (socket says 172.21.15.2:53040) 2023-11-16T18:31:13.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde80f86700 1 -- 172.21.15.2:0/1844420598 learned_addr learned my addr 172.21.15.2:0/1844420598 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:13.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde7bfff700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde7c068770 0x7fde7c100080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde81787700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde7c105250 0x7fde7c1a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde80f86700 1 -- 172.21.15.2:0/1844420598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde7c105250 msgr2=0x7fde7c1a4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde80f86700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde7c105250 0x7fde7c1a4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde80f86700 1 -- 172.21.15.2:0/1844420598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde7c068770 msgr2=0x7fde7c100080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde80f86700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde7c068770 0x7fde7c100080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde80f86700 1 -- 172.21.15.2:0/1844420598 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde74009d00 con 0x7fde7c067dc0 2023-11-16T18:31:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde80f86700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde7c067dc0 0x7fde7c0ffb40 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00d9b0 tx=0x7fde6c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde79ffb700 1 -- 172.21.15.2:0/1844420598 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c004220 con 0x7fde7c067dc0 2023-11-16T18:31:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde79ffb700 1 -- 172.21.15.2:0/1844420598 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde6c007640 con 0x7fde7c067dc0 2023-11-16T18:31:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde831ea700 1 -- 172.21.15.2:0/1844420598 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7c110420 con 0x7fde7c067dc0 2023-11-16T18:31:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde79ffb700 1 -- 172.21.15.2:0/1844420598 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c005650 con 0x7fde7c067dc0 2023-11-16T18:31:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.239+0000 7fde831ea700 1 -- 172.21.15.2:0/1844420598 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c110290 con 0x7fde7c067dc0 2023-11-16T18:31:13.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.242+0000 7fde79ffb700 1 -- 172.21.15.2:0/1844420598 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fde6c004750 con 0x7fde7c067dc0 2023-11-16T18:31:13.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.243+0000 7fde79ffb700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fde600a12c0 0x7fde600a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.243+0000 7fde79ffb700 1 -- 172.21.15.2:0/1844420598 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fde6c0ccf10 con 0x7fde7c067dc0 2023-11-16T18:31:13.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.243+0000 7fde7bfff700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fde600a12c0 0x7fde600a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.243+0000 7fde7bfff700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fde600a12c0 0x7fde600a3770 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde7c101330 tx=0x7fde6800b410 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:13.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde8f376b80 1 Processor -- start 2023-11-16T18:31:13.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde8f376b80 1 -- start start 2023-11-16T18:31:13.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde8f376b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f1b60 0x55bddb2f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde8f376b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f2470 0x55bddb2f6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde8f376b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f68b0 0x55bddb2f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde8f376b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bddb1f7060 con 0x55bddb2f1b60 2023-11-16T18:31:13.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde8f376b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bddb2498a0 con 0x55bddb2f2470 2023-11-16T18:31:13.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde8f376b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bddb242d30 con 0x55bddb2f68b0 2023-11-16T18:31:13.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde80f86700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f1b60 0x55bddb2f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde80f86700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f1b60 0x55bddb2f1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53048/0 (socket says 172.21.15.2:53048) 2023-11-16T18:31:13.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde80f86700 1 -- 172.21.15.2:0/866746189 learned_addr learned my addr 172.21.15.2:0/866746189 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:13.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde7bfff700 1 --2- 172.21.15.2:0/866746189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f2470 0x55bddb2f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde81787700 1 --2- 172.21.15.2:0/866746189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f68b0 0x55bddb2f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde80f86700 1 -- 172.21.15.2:0/866746189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f68b0 msgr2=0x55bddb2f6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde80f86700 1 --2- 172.21.15.2:0/866746189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f68b0 0x55bddb2f6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde80f86700 1 -- 172.21.15.2:0/866746189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f2470 msgr2=0x55bddb2f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde80f86700 1 --2- 172.21.15.2:0/866746189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f2470 0x55bddb2f6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.244+0000 7fde80f86700 1 -- 172.21.15.2:0/866746189 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bddb0c4630 con 0x55bddb2f1b60 2023-11-16T18:31:13.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.245+0000 7fde80f86700 1 --2- 172.21.15.2:0/866746189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f1b60 0x55bddb2f1f30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fde6c096680 tx=0x7fde6c0995f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53fe86ef86cb3c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:13.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.245+0000 7fde72ffd700 1 -- 172.21.15.2:0/866746189 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c09dc40 con 0x55bddb2f1b60 2023-11-16T18:31:13.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.245+0000 7fde72ffd700 1 -- 172.21.15.2:0/866746189 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde6c0a2060 con 0x55bddb2f1b60 2023-11-16T18:31:13.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.245+0000 7fde72ffd700 1 -- 172.21.15.2:0/866746189 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c099100 con 0x55bddb2f1b60 2023-11-16T18:31:13.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.245+0000 7fde8f376b80 1 -- 172.21.15.2:0/866746189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f1b60 msgr2=0x55bddb2f1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.245+0000 7fde8f376b80 1 --2- 172.21.15.2:0/866746189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f1b60 0x55bddb2f1f30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fde6c096680 tx=0x7fde6c0995f0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.245+0000 7fde8f376b80 1 -- 172.21.15.2:0/866746189 shutdown_connections 2023-11-16T18:31:13.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.245+0000 7fde8f376b80 1 --2- 172.21.15.2:0/866746189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f68b0 0x55bddb2f6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.245+0000 7fde8f376b80 1 --2- 172.21.15.2:0/866746189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f1b60 0x55bddb2f1f30 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.245+0000 7fde8f376b80 1 --2- 172.21.15.2:0/866746189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f2470 0x55bddb2f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.246+0000 7fde8f376b80 1 -- 172.21.15.2:0/866746189 >> 172.21.15.2:0/866746189 conn(0x55bddb2e7360 msgr2=0x55bddb2ecb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:13.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.246+0000 7fde8f376b80 1 -- 172.21.15.2:0/866746189 shutdown_connections 2023-11-16T18:31:13.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.246+0000 7fde8f376b80 1 -- 172.21.15.2:0/866746189 wait complete. 2023-11-16T18:31:13.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.246+0000 7fde8f376b80 1 -- start start 2023-11-16T18:31:13.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.246+0000 7fde8f376b80 1 Processor -- start 2023-11-16T18:31:13.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.246+0000 7fde8f376b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f1b60 0x55bddb2fe0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.246+0000 7fde8f376b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb2fb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.246+0000 7fde8f376b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f68b0 0x55bddb2fba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.246+0000 7fde8f376b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bddb1e48e0 con 0x55bddb2f2470 2023-11-16T18:31:13.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.246+0000 7fde8f376b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bddb1e4580 con 0x55bddb2f68b0 2023-11-16T18:31:13.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.246+0000 7fde8f376b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bddb0c6920 con 0x55bddb2f1b60 2023-11-16T18:31:13.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.246+0000 7fde8f376b80 10 client.0 _ll_register_callbacks cb 0x55bddb202780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:13.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde7bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb2fb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde80f86700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f1b60 0x55bddb2fe0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde80f86700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f1b60 0x55bddb2fe0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.2:44584/0 (socket says 172.21.15.2:44584) 2023-11-16T18:31:13.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde7bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb2fb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53056/0 (socket says 172.21.15.2:53056) 2023-11-16T18:31:13.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde80f86700 1 -- 172.21.15.2:0/4009416897 learned_addr learned my addr 172.21.15.2:0/4009416897 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:13.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde81787700 1 --2- 172.21.15.2:0/4009416897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f68b0 0x55bddb2fba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde80f86700 1 -- 172.21.15.2:0/4009416897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f68b0 msgr2=0x55bddb2fba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde80f86700 1 --2- 172.21.15.2:0/4009416897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f68b0 0x55bddb2fba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde80f86700 1 -- 172.21.15.2:0/4009416897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 msgr2=0x55bddb2fb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde80f86700 1 --2- 172.21.15.2:0/4009416897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb2fb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde80f86700 1 -- 172.21.15.2:0/4009416897 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c110290 con 0x55bddb2f1b60 2023-11-16T18:31:13.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde7bfff700 1 --2- 172.21.15.2:0/4009416897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb2fb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:13.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde80f86700 1 --2- 172.21.15.2:0/4009416897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f1b60 0x55bddb2fe0a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde6c004440 tx=0x7fde6c0b2440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5a24e7f0f187536 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:13.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde5ffff700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-11-16T18:31:13.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde5ffff700 1 -- 172.21.15.2:0/4009416897 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c09dc40 con 0x55bddb2f1b60 2023-11-16T18:31:13.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde5ffff700 1 -- 172.21.15.2:0/4009416897 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fde6c099310 con 0x55bddb2f1b60 2023-11-16T18:31:13.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde8f376b80 1 -- 172.21.15.2:0/4009416897 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bddb2e4b20 con 0x55bddb2f1b60 2023-11-16T18:31:13.255 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.247+0000 7fde5ffff700 1 -- 172.21.15.2:0/4009416897 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde6c099e70 con 0x55bddb2f1b60 2023-11-16T18:31:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.248+0000 7fde8f376b80 10 client.26194 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.248+0000 7fde8f376b80 1 -- 172.21.15.2:0/4009416897 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55bddb209190 con 0x55bddb2f1b60 2023-11-16T18:31:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.248+0000 7fde5ffff700 1 -- 172.21.15.2:0/4009416897 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bddb2e4b20 con 0x55bddb2f1b60 2023-11-16T18:31:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.248+0000 7fde5ffff700 1 -- 172.21.15.2:0/4009416897 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fde6c095a90 con 0x55bddb2f1b60 2023-11-16T18:31:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.248+0000 7fde8f376b80 10 client.26194 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.248+0000 7fde8f376b80 10 client.26194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:31:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.248+0000 7fde8f376b80 20 client.26194 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.248+0000 7fde8f376b80 1 --2- 172.21.15.2:0/4009416897 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bddb30b290 0x55bddb30d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.248+0000 7fde8f376b80 4 client.26194 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:31:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.248+0000 7fde8f376b80 1 -- 172.21.15.2:0/4009416897 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bddb2050a0 con 0x55bddb30b290 2023-11-16T18:31:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.248+0000 7fde7bfff700 1 --2- 172.21.15.2:0/4009416897 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bddb30b290 0x55bddb30d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.249+0000 7fde5ffff700 10 client.26194 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:13.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.249+0000 7fde7bfff700 1 --2- 172.21.15.2:0/4009416897 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bddb30b290 0x55bddb30d670 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1713d124cfc0bd05 server_cookie=fa7be414cd8a8024 in_seq=0 out_seq=0 2023-11-16T18:31:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.257+0000 7fde5ffff700 1 -- 172.21.15.2:0/4009416897 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55bddb2050a0 con 0x55bddb30b290 2023-11-16T18:31:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.257+0000 7fde5ffff700 10 client.26194 handle_command_reply: tid=0 2023-11-16T18:31:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.257+0000 7fde8f376b80 1 client.26194 shutdown 2023-11-16T18:31:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.258+0000 7fde8f376b80 1 -- 172.21.15.2:0/4009416897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f1b60 msgr2=0x55bddb2fe0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.258+0000 7fde8f376b80 1 --2- 172.21.15.2:0/4009416897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f1b60 0x55bddb2fe0a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde6c004440 tx=0x7fde6c0b2440 comp rx=0 tx=0).stop 2023-11-16T18:31:13.260 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.258+0000 7fde8f376b80 1 -- 172.21.15.2:0/4009416897 shutdown_connections 2023-11-16T18:31:13.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.258+0000 7fde8f376b80 1 --2- 172.21.15.2:0/4009416897 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bddb30b290 0x55bddb30d670 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.258+0000 7fde8f376b80 1 --2- 172.21.15.2:0/4009416897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f1b60 0x55bddb2fe0a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.258+0000 7fde8f376b80 1 --2- 172.21.15.2:0/4009416897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f68b0 0x55bddb2fba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.258+0000 7fde8f376b80 1 --2- 172.21.15.2:0/4009416897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb2fb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.258+0000 7fde5ffff700 0 client.26194 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:13.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.258+0000 7fde8f376b80 1 -- 172.21.15.2:0/4009416897 >> 172.21.15.2:0/4009416897 conn(0x55bddb2e7360 msgr2=0x55bddb2f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:13.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.258+0000 7fde8f376b80 1 -- 172.21.15.2:0/4009416897 shutdown_connections 2023-11-16T18:31:13.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.258+0000 7fde8f376b80 1 -- 172.21.15.2:0/4009416897 wait complete. 2023-11-16T18:31:13.261 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.258+0000 7fde8f376b80 20 client.26194 trim_cache size 0 max 16384 2023-11-16T18:31:13.274 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.271+0000 7fde8f376b80 1 Processor -- start 2023-11-16T18:31:13.274 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.271+0000 7fde8f376b80 1 -- start start 2023-11-16T18:31:13.274 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.271+0000 7fde8f376b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f1b60 0x55bddb2ecbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.271+0000 7fde8f376b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb2fdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.271+0000 7fde8f376b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f68b0 0x55bddb2fabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.271+0000 7fde8f376b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bddb2e49a0 con 0x55bddb2f2470 2023-11-16T18:31:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.271+0000 7fde8f376b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bddb2f54b0 con 0x55bddb2f68b0 2023-11-16T18:31:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.271+0000 7fde8f376b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bddb2f5200 con 0x55bddb2f1b60 2023-11-16T18:31:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.271+0000 7fde7bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb2fdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.271+0000 7fde7bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb2fdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53064/0 (socket says 172.21.15.2:53064) 2023-11-16T18:31:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.271+0000 7fde81787700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f68b0 0x55bddb2fabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.271+0000 7fde7bfff700 1 -- 172.21.15.2:0/255531594 learned_addr learned my addr 172.21.15.2:0/255531594 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.271+0000 7fde80f86700 1 --2- 172.21.15.2:0/255531594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f1b60 0x55bddb2ecbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.272+0000 7fde7bfff700 1 -- 172.21.15.2:0/255531594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f1b60 msgr2=0x55bddb2ecbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.272+0000 7fde7bfff700 1 --2- 172.21.15.2:0/255531594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f1b60 0x55bddb2ecbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.272+0000 7fde7bfff700 1 -- 172.21.15.2:0/255531594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f68b0 msgr2=0x55bddb2fabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.272+0000 7fde7bfff700 1 --2- 172.21.15.2:0/255531594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f68b0 0x55bddb2fabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.272+0000 7fde7bfff700 1 -- 172.21.15.2:0/255531594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bddb2e8b00 con 0x55bddb2f2470 2023-11-16T18:31:13.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.272+0000 7fde7bfff700 1 --2- 172.21.15.2:0/255531594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb2fdbd0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fde68005d90 tx=0x7fde68013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed83e664a709cc51 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:13.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.272+0000 7fde737fe700 1 -- 172.21.15.2:0/255531594 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6802e070 con 0x55bddb2f2470 2023-11-16T18:31:13.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.272+0000 7fde737fe700 1 -- 172.21.15.2:0/255531594 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde68015070 con 0x55bddb2f2470 2023-11-16T18:31:13.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.272+0000 7fde737fe700 1 -- 172.21.15.2:0/255531594 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68024040 con 0x55bddb2f2470 2023-11-16T18:31:13.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.273+0000 7fde8f376b80 1 -- 172.21.15.2:0/255531594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 msgr2=0x55bddb2fdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.273+0000 7fde8f376b80 1 --2- 172.21.15.2:0/255531594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb2fdbd0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fde68005d90 tx=0x7fde68013000 comp rx=0 tx=0).stop 2023-11-16T18:31:13.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.273+0000 7fde8f376b80 1 -- 172.21.15.2:0/255531594 shutdown_connections 2023-11-16T18:31:13.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.273+0000 7fde8f376b80 1 --2- 172.21.15.2:0/255531594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f1b60 0x55bddb2ecbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.273+0000 7fde8f376b80 1 --2- 172.21.15.2:0/255531594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f68b0 0x55bddb2fabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.273+0000 7fde8f376b80 1 --2- 172.21.15.2:0/255531594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb2fdbd0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.273+0000 7fde8f376b80 1 -- 172.21.15.2:0/255531594 >> 172.21.15.2:0/255531594 conn(0x55bddb2e7360 msgr2=0x55bddb307b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:13.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.273+0000 7fde8f376b80 1 -- 172.21.15.2:0/255531594 shutdown_connections 2023-11-16T18:31:13.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.273+0000 7fde8f376b80 1 -- 172.21.15.2:0/255531594 wait complete. 2023-11-16T18:31:13.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.273+0000 7fde8f376b80 1 -- start start 2023-11-16T18:31:13.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.273+0000 7fde8f376b80 1 Processor -- start 2023-11-16T18:31:13.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde8f376b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f1b60 0x55bddb300140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde8f376b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb300680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde8f376b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f68b0 0x55bddb300be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde8f376b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bddb35e280 con 0x55bddb2f2470 2023-11-16T18:31:13.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde8f376b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bddb35e3f0 con 0x55bddb2f1b60 2023-11-16T18:31:13.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde8f376b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bddb35e5b0 con 0x55bddb2f68b0 2023-11-16T18:31:13.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde8f376b80 10 client.0 _ll_register_callbacks cb 0x7fde6800b990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:13.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde7bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb300680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde80f86700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f1b60 0x55bddb300140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde7bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb300680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53070/0 (socket says 172.21.15.2:53070) 2023-11-16T18:31:13.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde7bfff700 1 -- 172.21.15.2:0/4183656160 learned_addr learned my addr 172.21.15.2:0/4183656160 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:13.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde81787700 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f68b0 0x55bddb300be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde7bfff700 1 -- 172.21.15.2:0/4183656160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f68b0 msgr2=0x55bddb300be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde7bfff700 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f68b0 0x55bddb300be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.282 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde7bfff700 1 -- 172.21.15.2:0/4183656160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f1b60 msgr2=0x55bddb300140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:13.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde7bfff700 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f1b60 0x55bddb300140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde7bfff700 1 -- 172.21.15.2:0/4183656160 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde68010e30 con 0x55bddb2f2470 2023-11-16T18:31:13.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde81787700 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f68b0 0x55bddb300be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:31:13.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.274+0000 7fde80f86700 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f1b60 0x55bddb300140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:13.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde7bfff700 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb300680 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fde6800bb60 tx=0x7fde68029630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3bfd95761b0037c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:13.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde5ffff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:31:13.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde5ffff700 1 -- 172.21.15.2:0/4183656160 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68024070 con 0x55bddb2f2470 2023-11-16T18:31:13.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde5ffff700 1 -- 172.21.15.2:0/4183656160 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fde68017070 con 0x55bddb2f2470 2023-11-16T18:31:13.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde8f376b80 1 -- 172.21.15.2:0/4183656160 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde6c0a2060 con 0x55bddb2f2470 2023-11-16T18:31:13.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde5ffff700 1 -- 172.21.15.2:0/4183656160 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde6802e040 con 0x55bddb2f2470 2023-11-16T18:31:13.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde8f376b80 10 client.16821 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:13.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde8f376b80 1 -- 172.21.15.2:0/4183656160 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55bddb070de0 con 0x55bddb2f2470 2023-11-16T18:31:13.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde5ffff700 1 -- 172.21.15.2:0/4183656160 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde6c0a2060 con 0x55bddb2f2470 2023-11-16T18:31:13.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde5ffff700 1 -- 172.21.15.2:0/4183656160 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fde680117c0 con 0x55bddb2f2470 2023-11-16T18:31:13.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde8f376b80 10 client.16821 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:13.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde8f376b80 10 client.16821 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T18:31:13.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde8f376b80 20 client.16821 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:13.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde8f376b80 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bddb35fd50 0x55bddb362160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:13.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde8f376b80 4 client.16821 mds_command: new command op to 24473 tid=0[{"prefix": "damage ls"}] 2023-11-16T18:31:13.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde8f376b80 1 -- 172.21.15.2:0/4183656160 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bddb211660 con 0x55bddb35fd50 2023-11-16T18:31:13.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.275+0000 7fde80f86700 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bddb35fd50 0x55bddb362160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:13.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.276+0000 7fde5ffff700 10 client.16821 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:13.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.276+0000 7fde80f86700 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bddb35fd50 0x55bddb362160 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee69843ce932098 server_cookie=4c7c02e8915725c2 in_seq=0 out_seq=0 2023-11-16T18:31:13.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.289+0000 7fde5ffff700 1 -- 172.21.15.2:0/4183656160 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bddb211660 con 0x55bddb35fd50 2023-11-16T18:31:13.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde5ffff700 10 client.16821 handle_command_reply: tid=0 2023-11-16T18:31:13.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde8f376b80 1 client.16821 shutdown 2023-11-16T18:31:13.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde8f376b80 1 -- 172.21.15.2:0/4183656160 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 msgr2=0x55bddb300680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde8f376b80 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb300680 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fde6800bb60 tx=0x7fde68029630 comp rx=0 tx=0).stop 2023-11-16T18:31:13.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde8f376b80 1 -- 172.21.15.2:0/4183656160 shutdown_connections 2023-11-16T18:31:13.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde8f376b80 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x55bddb35fd50 0x55bddb362160 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde8f376b80 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bddb2f68b0 0x55bddb300be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde8f376b80 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bddb2f1b60 0x55bddb300140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde5ffff700 0 client.16821 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:13.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde8f376b80 1 --2- 172.21.15.2:0/4183656160 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55bddb2f2470 0x55bddb300680 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde8f376b80 1 -- 172.21.15.2:0/4183656160 >> 172.21.15.2:0/4183656160 conn(0x55bddb2e7360 msgr2=0x55bddb2faca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:13.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde8f376b80 1 -- 172.21.15.2:0/4183656160 shutdown_connections 2023-11-16T18:31:13.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde8f376b80 1 -- 172.21.15.2:0/4183656160 wait complete. 2023-11-16T18:31:13.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde8f376b80 20 client.16821 trim_cache size 0 max 16384 2023-11-16T18:31:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:[] 2023-11-16T18:31:13.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.290+0000 7fde831ea700 1 -- 172.21.15.2:0/1844420598 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fde600a12c0 msgr2=0x7fde600a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.291+0000 7fde831ea700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fde600a12c0 0x7fde600a3770 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde7c101330 tx=0x7fde6800b410 comp rx=0 tx=0).stop 2023-11-16T18:31:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.291+0000 7fde831ea700 1 -- 172.21.15.2:0/1844420598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde7c067dc0 msgr2=0x7fde7c0ffb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.291+0000 7fde831ea700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde7c067dc0 0x7fde7c0ffb40 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00d9b0 tx=0x7fde6c00dcc0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.291+0000 7fde831ea700 1 -- 172.21.15.2:0/1844420598 shutdown_connections 2023-11-16T18:31:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.291+0000 7fde831ea700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fde600a12c0 0x7fde600a3770 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.291+0000 7fde831ea700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde7c105250 0x7fde7c1a4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.291+0000 7fde831ea700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde7c067dc0 0x7fde7c0ffb40 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.291+0000 7fde831ea700 1 --2- 172.21.15.2:0/1844420598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde7c068770 0x7fde7c100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.291+0000 7fde831ea700 1 -- 172.21.15.2:0/1844420598 >> 172.21.15.2:0/1844420598 conn(0x7fde7c0763a0 msgr2=0x7fde7c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.291+0000 7fde831ea700 1 -- 172.21.15.2:0/1844420598 shutdown_connections 2023-11-16T18:31:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:31:13.291+0000 7fde831ea700 1 -- 172.21.15.2:0/1844420598 wait complete. 2023-11-16T18:31:13.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:13 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:31:13.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:13 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:31:13.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:13 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:31:13.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:13 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:31:13.305 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 23268747, 'ino': 1099511642287, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9959'}, {'damage_type': 'backtrace', 'id': 49177939, 'ino': 1099511642270, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9943'}, {'damage_type': 'backtrace', 'id': 179423603, 'ino': 1099511642284, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9956'}, {'damage_type': 'backtrace', 'id': 320634055, 'ino': 1099511642278, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9950'}, {'damage_type': 'backtrace', 'id': 387783030, 'ino': 1099511642285, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9957'}, {'damage_type': 'backtrace', 'id': 674883623, 'ino': 1099511642295, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9967'}, {'damage_type': 'backtrace', 'id': 699703911, 'ino': 1099511642293, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9965'}, {'damage_type': 'backtrace', 'id': 1384656058, 'ino': 1099511642274, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9946'}, {'damage_type': 'backtrace', 'id': 1436072723, 'ino': 1099511642277, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9949'}, {'damage_type': 'backtrace', 'id': 1562705912, 'ino': 1099511642281, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9953'}, {'damage_type': 'backtrace', 'id': 2017776999, 'ino': 1099511642273, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9945'}, {'damage_type': 'backtrace', 'id': 2424408919, 'ino': 1099511642279, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9951'}, {'damage_type': 'backtrace', 'id': 3284816971, 'ino': 1099511642307, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/dir5/file9977'}, {'damage_type': 'backtrace', 'id': 4058982802, 'ino': 1099511642306, 'path': '/volumes/_nogroup/sv_1/f8a49d1b-21bb-4a98-9f1b-adbf6a12333e/client.0/tmp/testdir/dir1/dir2/dir3/dir4/file9398'}] 2023-11-16T18:31:13.305 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2023-11-16T18:31:13.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:13 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:31:13.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:13 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:31:13.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:13 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:31:13.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:13 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:31:13.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:13 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:31:13.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:13 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:31:13.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:13 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:31:13.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:13 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:31:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:14 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:31:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:14 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:31:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:14 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:31:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:14 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:31:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:14 smithi040 ceph-mon[100251]: pgmap v194: 129 pgs: 129 active+clean; 711 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 330 op/s 2023-11-16T18:31:14.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:14 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:31:14.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:14 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:31:14.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:14 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:31:14.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:14 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:31:14.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:14 smithi188 ceph-mon[100639]: pgmap v194: 129 pgs: 129 active+clean; 711 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 330 op/s 2023-11-16T18:31:14.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:14 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:31:14.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:14 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:31:14.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:14 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:31:14.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:14 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:31:14.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:14 smithi002 ceph-mon[98807]: pgmap v194: 129 pgs: 129 active+clean; 711 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 330 op/s 2023-11-16T18:31:16.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:16 smithi040 ceph-mon[100251]: pgmap v195: 129 pgs: 129 active+clean; 711 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 981 KiB/s wr, 145 op/s 2023-11-16T18:31:16.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:16 smithi188 ceph-mon[100639]: pgmap v195: 129 pgs: 129 active+clean; 711 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 981 KiB/s wr, 145 op/s 2023-11-16T18:31:16.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:16 smithi002 ceph-mon[98807]: pgmap v195: 129 pgs: 129 active+clean; 711 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 981 KiB/s wr, 145 op/s 2023-11-16T18:31:18.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:18 smithi040 ceph-mon[100251]: pgmap v196: 129 pgs: 129 active+clean; 711 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 981 KiB/s wr, 145 op/s 2023-11-16T18:31:18.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:18 smithi040 ceph-mon[100251]: osdmap e92: 12 total, 12 up, 12 in 2023-11-16T18:31:18.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:18 smithi188 ceph-mon[100639]: pgmap v196: 129 pgs: 129 active+clean; 711 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 981 KiB/s wr, 145 op/s 2023-11-16T18:31:18.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:18 smithi188 ceph-mon[100639]: osdmap e92: 12 total, 12 up, 12 in 2023-11-16T18:31:18.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:18 smithi002 ceph-mon[98807]: pgmap v196: 129 pgs: 129 active+clean; 711 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 981 KiB/s wr, 145 op/s 2023-11-16T18:31:18.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:18 smithi002 ceph-mon[98807]: osdmap e92: 12 total, 12 up, 12 in 2023-11-16T18:31:20.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:20 smithi040 ceph-mon[100251]: pgmap v198: 129 pgs: 129 active+clean; 716 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 251 op/s 2023-11-16T18:31:20.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:20 smithi188 ceph-mon[100639]: pgmap v198: 129 pgs: 129 active+clean; 716 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 251 op/s 2023-11-16T18:31:20.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:20 smithi002 ceph-mon[98807]: pgmap v198: 129 pgs: 129 active+clean; 716 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 251 op/s 2023-11-16T18:31:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.268+0000 7fec3c763700 1 -- 172.21.15.2:0/3241124574 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec340ffda0 msgr2=0x7fec34103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.268+0000 7fec3c763700 1 --2- 172.21.15.2:0/3241124574 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec340ffda0 0x7fec34103d60 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fec240099c0 tx=0x7fec24009cd0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.269+0000 7fec3c763700 1 -- 172.21.15.2:0/3241124574 shutdown_connections 2023-11-16T18:31:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.269+0000 7fec3c763700 1 --2- 172.21.15.2:0/3241124574 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec34104440 0x7fec3410c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.269+0000 7fec3c763700 1 --2- 172.21.15.2:0/3241124574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec340ff400 0x7fec340ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.269+0000 7fec3c763700 1 --2- 172.21.15.2:0/3241124574 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec340ffda0 0x7fec34103d60 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.269+0000 7fec3c763700 1 -- 172.21.15.2:0/3241124574 >> 172.21.15.2:0/3241124574 conn(0x7fec34075fa0 msgr2=0x7fec340763a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.269+0000 7fec3c763700 1 -- 172.21.15.2:0/3241124574 shutdown_connections 2023-11-16T18:31:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.269+0000 7fec3c763700 1 -- 172.21.15.2:0/3241124574 wait complete. 2023-11-16T18:31:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.269+0000 7fec3c763700 1 Processor -- start 2023-11-16T18:31:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.269+0000 7fec3c763700 1 -- start start 2023-11-16T18:31:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3c763700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec340ff400 0x7fec34196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3c763700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec340ffda0 0x7fec34196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3c763700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec34104440 0x7fec3419adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3c763700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3419b3f0 con 0x7fec34104440 2023-11-16T18:31:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3c763700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3419b560 con 0x7fec340ff400 2023-11-16T18:31:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3c763700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3419b720 con 0x7fec340ffda0 2023-11-16T18:31:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3ad00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec34104440 0x7fec3419adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3a4ff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec340ff400 0x7fec34196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3ad00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec34104440 0x7fec3419adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53086/0 (socket says 172.21.15.2:53086) 2023-11-16T18:31:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3ad00700 1 -- 172.21.15.2:0/4175236379 learned_addr learned my addr 172.21.15.2:0/4175236379 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec39cfe700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec340ffda0 0x7fec34196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3ad00700 1 -- 172.21.15.2:0/4175236379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec340ffda0 msgr2=0x7fec34196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3ad00700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec340ffda0 0x7fec34196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3ad00700 1 -- 172.21.15.2:0/4175236379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec340ff400 msgr2=0x7fec34196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3ad00700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec340ff400 0x7fec34196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3ad00700 1 -- 172.21.15.2:0/4175236379 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec240095f0 con 0x7fec34104440 2023-11-16T18:31:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec39cfe700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec340ffda0 0x7fec34196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:31:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3ad00700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec34104440 0x7fec3419adb0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fec2c00efc0 tx=0x7fec2c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:22.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec2b7fe700 1 -- 172.21.15.2:0/4175236379 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c018070 con 0x7fec34104440 2023-11-16T18:31:22.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec2b7fe700 1 -- 172.21.15.2:0/4175236379 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec2c00f040 con 0x7fec34104440 2023-11-16T18:31:22.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3c763700 1 -- 172.21.15.2:0/4175236379 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3410f1c0 con 0x7fec34104440 2023-11-16T18:31:22.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec2b7fe700 1 -- 172.21.15.2:0/4175236379 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c00ccc0 con 0x7fec34104440 2023-11-16T18:31:22.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.270+0000 7fec3c763700 1 -- 172.21.15.2:0/4175236379 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3410f030 con 0x7fec34104440 2023-11-16T18:31:22.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.273+0000 7fec2b7fe700 1 -- 172.21.15.2:0/4175236379 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fec2c003a00 con 0x7fec34104440 2023-11-16T18:31:22.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.273+0000 7fec2b7fe700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fec200a12c0 0x7fec200a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.274+0000 7fec3a4ff700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fec200a12c0 0x7fec200a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.274+0000 7fec2b7fe700 1 -- 172.21.15.2:0/4175236379 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9570+0+0 (secure 0 0 0) 0x7fec2c0ccc10 con 0x7fec34104440 2023-11-16T18:31:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.274+0000 7fec3a4ff700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fec200a12c0 0x7fec200a3770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fec30009c60 tx=0x7fec30009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.274+0000 7fec488efb80 1 Processor -- start 2023-11-16T18:31:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec488efb80 1 -- start start 2023-11-16T18:31:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec488efb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb77a26b0 0x558fb794c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec488efb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 0x558fb7946560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec488efb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794cb90 0x558fb7946b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec488efb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fb784c000 con 0x558fb794c720 2023-11-16T18:31:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec488efb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fb789e840 con 0x558fb794cb90 2023-11-16T18:31:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec488efb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fb7897cd0 con 0x558fb77a26b0 2023-11-16T18:31:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec39cfe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 0x558fb7946560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec39cfe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 0x558fb7946560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53092/0 (socket says 172.21.15.2:53092) 2023-11-16T18:31:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec39cfe700 1 -- 172.21.15.2:0/1045677258 learned_addr learned my addr 172.21.15.2:0/1045677258 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec3ad00700 1 --2- 172.21.15.2:0/1045677258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794cb90 0x558fb7946b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec3a4ff700 1 --2- 172.21.15.2:0/1045677258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb77a26b0 0x558fb794c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec39cfe700 1 -- 172.21.15.2:0/1045677258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb77a26b0 msgr2=0x558fb794c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec39cfe700 1 --2- 172.21.15.2:0/1045677258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb77a26b0 0x558fb794c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec39cfe700 1 -- 172.21.15.2:0/1045677258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794cb90 msgr2=0x558fb7946b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec39cfe700 1 --2- 172.21.15.2:0/1045677258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794cb90 0x558fb7946b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec39cfe700 1 -- 172.21.15.2:0/1045677258 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fb77195d0 con 0x558fb794c720 2023-11-16T18:31:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec3ad00700 1 --2- 172.21.15.2:0/1045677258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794cb90 0x558fb7946b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec39cfe700 1 --2- 172.21.15.2:0/1045677258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 0x558fb7946560 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fec24000c00 tx=0x7fec24005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc6919483e666c0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.275+0000 7fec28ff9700 1 -- 172.21.15.2:0/1045677258 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24021040 con 0x558fb794c720 2023-11-16T18:31:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.276+0000 7fec28ff9700 1 -- 172.21.15.2:0/1045677258 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec24004800 con 0x558fb794c720 2023-11-16T18:31:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.276+0000 7fec28ff9700 1 -- 172.21.15.2:0/1045677258 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec240054e0 con 0x558fb794c720 2023-11-16T18:31:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.276+0000 7fec488efb80 1 -- 172.21.15.2:0/1045677258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 msgr2=0x558fb7946560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.276+0000 7fec488efb80 1 --2- 172.21.15.2:0/1045677258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 0x558fb7946560 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fec24000c00 tx=0x7fec24005f00 comp rx=0 tx=0).stop 2023-11-16T18:31:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.276+0000 7fec488efb80 1 -- 172.21.15.2:0/1045677258 shutdown_connections 2023-11-16T18:31:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.276+0000 7fec488efb80 1 --2- 172.21.15.2:0/1045677258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb77a26b0 0x558fb794c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.276+0000 7fec488efb80 1 --2- 172.21.15.2:0/1045677258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794cb90 0x558fb7946b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.276+0000 7fec488efb80 1 --2- 172.21.15.2:0/1045677258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 0x558fb7946560 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.276+0000 7fec488efb80 1 -- 172.21.15.2:0/1045677258 >> 172.21.15.2:0/1045677258 conn(0x558fb793bf50 msgr2=0x558fb7941270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.276+0000 7fec488efb80 1 -- 172.21.15.2:0/1045677258 shutdown_connections 2023-11-16T18:31:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.276+0000 7fec488efb80 1 -- 172.21.15.2:0/1045677258 wait complete. 2023-11-16T18:31:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec488efb80 1 -- start start 2023-11-16T18:31:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec488efb80 1 Processor -- start 2023-11-16T18:31:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec488efb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb77a26b0 0x558fb7955ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec488efb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 0x558fb7955ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec488efb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794cb90 0x558fb794fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec488efb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fb7839880 con 0x558fb794c720 2023-11-16T18:31:22.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec488efb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fb7839520 con 0x558fb77a26b0 2023-11-16T18:31:22.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec488efb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fb771b8c0 con 0x558fb794cb90 2023-11-16T18:31:22.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec488efb80 10 client.0 _ll_register_callbacks cb 0x558fb7857720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:22.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec39cfe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 0x558fb7955ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec39cfe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 0x558fb7955ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53098/0 (socket says 172.21.15.2:53098) 2023-11-16T18:31:22.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec39cfe700 1 -- 172.21.15.2:0/3252007796 learned_addr learned my addr 172.21.15.2:0/3252007796 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:22.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec3a4ff700 1 --2- 172.21.15.2:0/3252007796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb77a26b0 0x558fb7955ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec3ad00700 1 --2- 172.21.15.2:0/3252007796 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794cb90 0x558fb794fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec39cfe700 1 -- 172.21.15.2:0/3252007796 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794cb90 msgr2=0x558fb794fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:22.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec39cfe700 1 --2- 172.21.15.2:0/3252007796 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794cb90 0x558fb794fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec39cfe700 1 -- 172.21.15.2:0/3252007796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb77a26b0 msgr2=0x558fb7955ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:22.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec39cfe700 1 --2- 172.21.15.2:0/3252007796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb77a26b0 0x558fb7955ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.277+0000 7fec39cfe700 1 -- 172.21.15.2:0/3252007796 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3410c850 con 0x558fb794c720 2023-11-16T18:31:22.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec39cfe700 1 --2- 172.21.15.2:0/3252007796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 0x558fb7955ff0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fec240058e0 tx=0x7fec24005160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c4f8c1b83b249cb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:22.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec1dffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:31:22.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec1dffb700 1 -- 172.21.15.2:0/3252007796 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24021040 con 0x558fb794c720 2023-11-16T18:31:22.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec1dffb700 1 -- 172.21.15.2:0/3252007796 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9570+0+0 (secure 0 0 0) 0x7fec2401b880 con 0x558fb794c720 2023-11-16T18:31:22.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec488efb80 1 -- 172.21.15.2:0/3252007796 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558fb7939710 con 0x558fb794c720 2023-11-16T18:31:22.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec1dffb700 1 -- 172.21.15.2:0/3252007796 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec2401ba40 con 0x558fb794c720 2023-11-16T18:31:22.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec488efb80 10 client.16833 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:22.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec488efb80 1 -- 172.21.15.2:0/3252007796 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x558fb785e130 con 0x558fb794c720 2023-11-16T18:31:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec1dffb700 1 -- 172.21.15.2:0/3252007796 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558fb7939710 con 0x558fb794c720 2023-11-16T18:31:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec1dffb700 1 -- 172.21.15.2:0/3252007796 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fec2400f460 con 0x558fb794c720 2023-11-16T18:31:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec488efb80 10 client.16833 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec488efb80 10 client.16833 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:31:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec488efb80 20 client.16833 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec488efb80 1 --2- 172.21.15.2:0/3252007796 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x558fb795fa80 0x558fb7961e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec488efb80 4 client.16833 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:31:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.278+0000 7fec488efb80 1 -- 172.21.15.2:0/3252007796 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558fb785a040 con 0x558fb795fa80 2023-11-16T18:31:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.279+0000 7fec3a4ff700 1 --2- 172.21.15.2:0/3252007796 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x558fb795fa80 0x558fb7961e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.279+0000 7fec3a4ff700 1 --2- 172.21.15.2:0/3252007796 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x558fb795fa80 0x558fb7961e60 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2387c8bd40f96560 server_cookie=feb1333a4c7395cc in_seq=0 out_seq=0 2023-11-16T18:31:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.279+0000 7fec1dffb700 10 client.16833 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.333+0000 7fec1dffb700 1 -- 172.21.15.2:0/3252007796 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x558fb785a040 con 0x558fb795fa80 2023-11-16T18:31:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.333+0000 7fec1dffb700 10 client.16833 handle_command_reply: tid=0 2023-11-16T18:31:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.333+0000 7fec488efb80 1 client.16833 shutdown 2023-11-16T18:31:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.333+0000 7fec488efb80 1 -- 172.21.15.2:0/3252007796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 msgr2=0x558fb7955ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.333+0000 7fec488efb80 1 --2- 172.21.15.2:0/3252007796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 0x558fb7955ff0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fec240058e0 tx=0x7fec24005160 comp rx=0 tx=0).stop 2023-11-16T18:31:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.334+0000 7fec488efb80 1 -- 172.21.15.2:0/3252007796 shutdown_connections 2023-11-16T18:31:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.334+0000 7fec488efb80 1 --2- 172.21.15.2:0/3252007796 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x558fb795fa80 0x558fb7961e60 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.334+0000 7fec488efb80 1 --2- 172.21.15.2:0/3252007796 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794cb90 0x558fb794fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.334+0000 7fec1dffb700 0 client.16833 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.334+0000 7fec488efb80 1 --2- 172.21.15.2:0/3252007796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb77a26b0 0x558fb7955ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.334+0000 7fec488efb80 1 --2- 172.21.15.2:0/3252007796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb794c720 0x558fb7955ff0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.334+0000 7fec488efb80 1 -- 172.21.15.2:0/3252007796 >> 172.21.15.2:0/3252007796 conn(0x558fb793bf50 msgr2=0x558fb79407e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.334+0000 7fec488efb80 1 -- 172.21.15.2:0/3252007796 shutdown_connections 2023-11-16T18:31:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.334+0000 7fec488efb80 1 -- 172.21.15.2:0/3252007796 wait complete. 2023-11-16T18:31:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.334+0000 7fec488efb80 20 client.16833 trim_cache size 0 max 16384 2023-11-16T18:31:22.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.348+0000 7fec488efb80 1 Processor -- start 2023-11-16T18:31:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec488efb80 1 -- start start 2023-11-16T18:31:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec488efb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 0x558fb794b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec488efb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794c720 0x558fb7940f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec488efb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794cb90 0x558fb7946560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec488efb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fb79512e0 con 0x558fb77a26b0 2023-11-16T18:31:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec488efb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fb794e2c0 con 0x558fb794cb90 2023-11-16T18:31:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec488efb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fb794aa10 con 0x558fb794c720 2023-11-16T18:31:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec3a4ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 0x558fb794b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec3a4ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 0x558fb794b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53104/0 (socket says 172.21.15.2:53104) 2023-11-16T18:31:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec3a4ff700 1 -- 172.21.15.2:0/3076264550 learned_addr learned my addr 172.21.15.2:0/3076264550 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec39cfe700 1 --2- 172.21.15.2:0/3076264550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794c720 0x558fb7940f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec3ad00700 1 --2- 172.21.15.2:0/3076264550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794cb90 0x558fb7946560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec3a4ff700 1 -- 172.21.15.2:0/3076264550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794c720 msgr2=0x558fb7940f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec3a4ff700 1 --2- 172.21.15.2:0/3076264550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794c720 0x558fb7940f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec3a4ff700 1 -- 172.21.15.2:0/3076264550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794cb90 msgr2=0x558fb7946560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec3a4ff700 1 --2- 172.21.15.2:0/3076264550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794cb90 0x558fb7946560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec3a4ff700 1 -- 172.21.15.2:0/3076264550 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fb785de80 con 0x558fb77a26b0 2023-11-16T18:31:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.349+0000 7fec3a4ff700 1 --2- 172.21.15.2:0/3076264550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 0x558fb794b6b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fec30009f50 tx=0x7fec30014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67423b6ebd906b9a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.350+0000 7fec297fa700 1 -- 172.21.15.2:0/3076264550 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30022040 con 0x558fb77a26b0 2023-11-16T18:31:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.350+0000 7fec297fa700 1 -- 172.21.15.2:0/3076264550 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec30028e80 con 0x558fb77a26b0 2023-11-16T18:31:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.350+0000 7fec297fa700 1 -- 172.21.15.2:0/3076264550 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec300038e0 con 0x558fb77a26b0 2023-11-16T18:31:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.350+0000 7fec488efb80 1 -- 172.21.15.2:0/3076264550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 msgr2=0x558fb794b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.350+0000 7fec488efb80 1 --2- 172.21.15.2:0/3076264550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 0x558fb794b6b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fec30009f50 tx=0x7fec30014000 comp rx=0 tx=0).stop 2023-11-16T18:31:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.350+0000 7fec488efb80 1 -- 172.21.15.2:0/3076264550 shutdown_connections 2023-11-16T18:31:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.350+0000 7fec488efb80 1 --2- 172.21.15.2:0/3076264550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794cb90 0x558fb7946560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.350+0000 7fec488efb80 1 --2- 172.21.15.2:0/3076264550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 0x558fb794b6b0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.350+0000 7fec488efb80 1 --2- 172.21.15.2:0/3076264550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794c720 0x558fb7940f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.350+0000 7fec488efb80 1 -- 172.21.15.2:0/3076264550 >> 172.21.15.2:0/3076264550 conn(0x558fb793bf50 msgr2=0x558fb7950690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.350+0000 7fec488efb80 1 -- 172.21.15.2:0/3076264550 shutdown_connections 2023-11-16T18:31:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.350+0000 7fec488efb80 1 -- 172.21.15.2:0/3076264550 wait complete. 2023-11-16T18:31:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec488efb80 1 -- start start 2023-11-16T18:31:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec488efb80 1 Processor -- start 2023-11-16T18:31:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec488efb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 0x558fb7953c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec488efb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794c720 0x558fb79541a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec488efb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794cb90 0x558fb79b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec488efb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fb79548f0 con 0x558fb77a26b0 2023-11-16T18:31:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec488efb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fb79b2e20 con 0x558fb794c720 2023-11-16T18:31:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec488efb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fb79b2fe0 con 0x558fb794cb90 2023-11-16T18:31:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec488efb80 10 client.0 _ll_register_callbacks cb 0x7fec2c0093e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec3a4ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 0x558fb7953c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec3a4ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 0x558fb7953c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53110/0 (socket says 172.21.15.2:53110) 2023-11-16T18:31:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec3a4ff700 1 -- 172.21.15.2:0/414663933 learned_addr learned my addr 172.21.15.2:0/414663933 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec39cfe700 1 --2- 172.21.15.2:0/414663933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794c720 0x558fb79541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec3a4ff700 1 -- 172.21.15.2:0/414663933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794cb90 msgr2=0x558fb79b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec3a4ff700 1 --2- 172.21.15.2:0/414663933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794cb90 0x558fb79b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec3a4ff700 1 -- 172.21.15.2:0/414663933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794c720 msgr2=0x558fb79541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec3a4ff700 1 --2- 172.21.15.2:0/414663933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794c720 0x558fb79541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.351+0000 7fec3a4ff700 1 -- 172.21.15.2:0/414663933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec30017070 con 0x558fb77a26b0 2023-11-16T18:31:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec3a4ff700 1 --2- 172.21.15.2:0/414663933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 0x558fb7953c60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fec30028600 tx=0x7fec30028630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a99eaedcbb5c1c6a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec1dffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:31:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec1dffb700 1 -- 172.21.15.2:0/414663933 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30022070 con 0x558fb77a26b0 2023-11-16T18:31:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec1dffb700 1 -- 172.21.15.2:0/414663933 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9570+0+0 (secure 0 0 0) 0x7fec300289c0 con 0x558fb77a26b0 2023-11-16T18:31:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec488efb80 1 -- 172.21.15.2:0/414663933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558fb7866600 con 0x558fb77a26b0 2023-11-16T18:31:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec1dffb700 1 -- 172.21.15.2:0/414663933 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec3001ce90 con 0x558fb77a26b0 2023-11-16T18:31:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec488efb80 10 client.16839 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec488efb80 1 -- 172.21.15.2:0/414663933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x558fb76aec00 con 0x558fb77a26b0 2023-11-16T18:31:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec1dffb700 1 -- 172.21.15.2:0/414663933 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558fb7866600 con 0x558fb77a26b0 2023-11-16T18:31:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec1dffb700 1 -- 172.21.15.2:0/414663933 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fec30004930 con 0x558fb77a26b0 2023-11-16T18:31:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec488efb80 10 client.16839 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec488efb80 10 client.16839 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:31:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec488efb80 20 client.16839 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec488efb80 1 --2- 172.21.15.2:0/414663933 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x558fb79b4600 0x558fb79b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec488efb80 4 client.16839 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:31:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.352+0000 7fec488efb80 1 -- 172.21.15.2:0/414663933 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558fb76e6f10 con 0x558fb79b4600 2023-11-16T18:31:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.353+0000 7fec39cfe700 1 --2- 172.21.15.2:0/414663933 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x558fb79b4600 0x558fb79b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.353+0000 7fec1dffb700 10 client.16839 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:22.353+0000 7fec39cfe700 1 --2- 172.21.15.2:0/414663933 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x558fb79b4600 0x558fb79b6a10 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc8a943769eb13b9 server_cookie=95d5d0f1b9bca131 in_seq=0 out_seq=0 2023-11-16T18:31:22.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:22 smithi040 ceph-mon[100251]: pgmap v199: 129 pgs: 129 active+clean; 716 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 251 op/s 2023-11-16T18:31:22.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:22 smithi188 ceph-mon[100639]: pgmap v199: 129 pgs: 129 active+clean; 716 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 251 op/s 2023-11-16T18:31:22.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:22 smithi002 ceph-mon[98807]: pgmap v199: 129 pgs: 129 active+clean; 716 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 251 op/s 2023-11-16T18:31:23.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.317+0000 7fec1dffb700 1 -- 172.21.15.2:0/414663933 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558fb76e6f10 con 0x558fb79b4600 2023-11-16T18:31:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.317+0000 7fec1dffb700 10 client.16839 handle_command_reply: tid=0 2023-11-16T18:31:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.318+0000 7fec488efb80 1 client.16839 shutdown 2023-11-16T18:31:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.318+0000 7fec488efb80 1 -- 172.21.15.2:0/414663933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 msgr2=0x558fb7953c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.318+0000 7fec488efb80 1 --2- 172.21.15.2:0/414663933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 0x558fb7953c60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fec30028600 tx=0x7fec30028630 comp rx=0 tx=0).stop 2023-11-16T18:31:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.318+0000 7fec488efb80 1 -- 172.21.15.2:0/414663933 shutdown_connections 2023-11-16T18:31:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.318+0000 7fec488efb80 1 --2- 172.21.15.2:0/414663933 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x558fb79b4600 0x558fb79b6a10 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.318+0000 7fec488efb80 1 --2- 172.21.15.2:0/414663933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558fb794cb90 0x558fb79b28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.318+0000 7fec488efb80 1 --2- 172.21.15.2:0/414663933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558fb77a26b0 0x558fb7953c60 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.318+0000 7fec1dffb700 0 client.16839 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.318+0000 7fec488efb80 1 --2- 172.21.15.2:0/414663933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fb794c720 0x558fb79541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.318+0000 7fec488efb80 1 -- 172.21.15.2:0/414663933 >> 172.21.15.2:0/414663933 conn(0x558fb793bf50 msgr2=0x558fb794e510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.318+0000 7fec488efb80 1 -- 172.21.15.2:0/414663933 shutdown_connections 2023-11-16T18:31:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.318+0000 7fec488efb80 1 -- 172.21.15.2:0/414663933 wait complete. 2023-11-16T18:31:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.318+0000 7fec488efb80 20 client.16839 trim_cache size 0 max 16384 2023-11-16T18:31:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:31:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:31:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:31:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:31:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.319+0000 7fec3c763700 1 -- 172.21.15.2:0/4175236379 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fec200a12c0 msgr2=0x7fec200a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.319+0000 7fec3c763700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fec200a12c0 0x7fec200a3770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fec30009c60 tx=0x7fec30009260 comp rx=0 tx=0).stop 2023-11-16T18:31:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.319+0000 7fec3c763700 1 -- 172.21.15.2:0/4175236379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec34104440 msgr2=0x7fec3419adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.319+0000 7fec3c763700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec34104440 0x7fec3419adb0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fec2c00efc0 tx=0x7fec2c00c3a0 comp rx=0 tx=0).stop 2023-11-16T18:31:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.319+0000 7fec3c763700 1 -- 172.21.15.2:0/4175236379 shutdown_connections 2023-11-16T18:31:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.319+0000 7fec3c763700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fec200a12c0 0x7fec200a3770 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.319+0000 7fec3c763700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec34104440 0x7fec3419adb0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.319+0000 7fec3c763700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec340ff400 0x7fec34196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.319+0000 7fec3c763700 1 --2- 172.21.15.2:0/4175236379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec340ffda0 0x7fec34196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.319+0000 7fec3c763700 1 -- 172.21.15.2:0/4175236379 >> 172.21.15.2:0/4175236379 conn(0x7fec34075fa0 msgr2=0x7fec34068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.319+0000 7fec3c763700 1 -- 172.21.15.2:0/4175236379 shutdown_connections 2023-11-16T18:31:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:23.319+0000 7fec3c763700 1 -- 172.21.15.2:0/4175236379 wait complete. 2023-11-16T18:31:24.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:24 smithi040 ceph-mon[100251]: pgmap v200: 129 pgs: 129 active+clean; 723 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2023-11-16T18:31:24.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:24 smithi188 ceph-mon[100639]: pgmap v200: 129 pgs: 129 active+clean; 723 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2023-11-16T18:31:24.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:24 smithi002 ceph-mon[98807]: pgmap v200: 129 pgs: 129 active+clean; 723 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2023-11-16T18:31:26.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:26 smithi040 ceph-mon[100251]: pgmap v201: 129 pgs: 129 active+clean; 723 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2023-11-16T18:31:26.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:26 smithi188 ceph-mon[100639]: pgmap v201: 129 pgs: 129 active+clean; 723 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2023-11-16T18:31:26.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:26 smithi002 ceph-mon[98807]: pgmap v201: 129 pgs: 129 active+clean; 723 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2023-11-16T18:31:27.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:27 smithi002 ceph-mon[98807]: osdmap e93: 12 total, 12 up, 12 in 2023-11-16T18:31:27.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:27 smithi002 ceph-mon[98807]: pgmap v203: 129 pgs: 129 active+clean; 723 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2023-11-16T18:31:27.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:27 smithi040 ceph-mon[100251]: osdmap e93: 12 total, 12 up, 12 in 2023-11-16T18:31:27.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:27 smithi040 ceph-mon[100251]: pgmap v203: 129 pgs: 129 active+clean; 723 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2023-11-16T18:31:27.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:27 smithi188 ceph-mon[100639]: osdmap e93: 12 total, 12 up, 12 in 2023-11-16T18:31:27.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:27 smithi188 ceph-mon[100639]: pgmap v203: 129 pgs: 129 active+clean; 723 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2023-11-16T18:31:30.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:30 smithi040 ceph-mon[100251]: pgmap v204: 129 pgs: 129 active+clean; 708 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 320 op/s 2023-11-16T18:31:30.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:30 smithi188 ceph-mon[100639]: pgmap v204: 129 pgs: 129 active+clean; 708 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 320 op/s 2023-11-16T18:31:30.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:30 smithi002 ceph-mon[98807]: pgmap v204: 129 pgs: 129 active+clean; 708 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 320 op/s 2023-11-16T18:31:32.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:32 smithi040 ceph-mon[100251]: pgmap v205: 129 pgs: 129 active+clean; 708 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 320 op/s 2023-11-16T18:31:32.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:32 smithi188 ceph-mon[100639]: pgmap v205: 129 pgs: 129 active+clean; 708 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 320 op/s 2023-11-16T18:31:32.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:32 smithi002 ceph-mon[98807]: pgmap v205: 129 pgs: 129 active+clean; 708 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 320 op/s 2023-11-16T18:31:34.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:34 smithi040 ceph-mon[100251]: pgmap v206: 129 pgs: 129 active+clean; 718 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 404 op/s 2023-11-16T18:31:34.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:34 smithi188 ceph-mon[100639]: pgmap v206: 129 pgs: 129 active+clean; 718 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 404 op/s 2023-11-16T18:31:34.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:34 smithi002 ceph-mon[98807]: pgmap v206: 129 pgs: 129 active+clean; 718 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 404 op/s 2023-11-16T18:31:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.443+0000 7f1efb0c4700 1 -- 172.21.15.2:0/3514223843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef4105560 msgr2=0x7f1ef410d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.443+0000 7f1efb0c4700 1 --2- 172.21.15.2:0/3514223843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef4105560 0x7f1ef410d170 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1eec009f90 tx=0x7f1eec00b330 comp rx=0 tx=0).stop 2023-11-16T18:31:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.443+0000 7f1efb0c4700 1 -- 172.21.15.2:0/3514223843 shutdown_connections 2023-11-16T18:31:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.443+0000 7f1efb0c4700 1 --2- 172.21.15.2:0/3514223843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ef4100c50 0x7f1ef4101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.443+0000 7f1efb0c4700 1 --2- 172.21.15.2:0/3514223843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef4105560 0x7f1ef410d170 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.443+0000 7f1efb0c4700 1 --2- 172.21.15.2:0/3514223843 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ef41015f0 0x7f1ef4104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.443+0000 7f1efb0c4700 1 -- 172.21.15.2:0/3514223843 >> 172.21.15.2:0/3514223843 conn(0x7f1ef4077550 msgr2=0x7f1ef4077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.443+0000 7f1efb0c4700 1 -- 172.21.15.2:0/3514223843 shutdown_connections 2023-11-16T18:31:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.443+0000 7f1efb0c4700 1 -- 172.21.15.2:0/3514223843 wait complete. 2023-11-16T18:31:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1efb0c4700 1 Processor -- start 2023-11-16T18:31:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1efb0c4700 1 -- start start 2023-11-16T18:31:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1efb0c4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ef4100c50 0x7f1ef4072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1efb0c4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef41015f0 0x7f1ef4073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1efb0c4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ef4105560 0x7f1ef41a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1efb0c4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef4073b60 con 0x7f1ef4105560 2023-11-16T18:31:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1efb0c4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef41a4bc0 con 0x7f1ef41015f0 2023-11-16T18:31:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1efb0c4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef41a4d00 con 0x7f1ef4100c50 2023-11-16T18:31:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1ef9661700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ef4105560 0x7f1ef41a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1ef9661700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ef4105560 0x7f1ef41a4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53128/0 (socket says 172.21.15.2:53128) 2023-11-16T18:31:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1ef9661700 1 -- 172.21.15.2:0/3484608081 learned_addr learned my addr 172.21.15.2:0/3484608081 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1ef3fff700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef41015f0 0x7f1ef4073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1ef8e60700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ef4100c50 0x7f1ef4072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1ef9661700 1 -- 172.21.15.2:0/3484608081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ef4100c50 msgr2=0x7f1ef4072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1ef9661700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ef4100c50 0x7f1ef4072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1ef9661700 1 -- 172.21.15.2:0/3484608081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef41015f0 msgr2=0x7f1ef4073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1ef9661700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef41015f0 0x7f1ef4073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.444+0000 7f1ef9661700 1 -- 172.21.15.2:0/3484608081 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee00095f0 con 0x7f1ef4105560 2023-11-16T18:31:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.445+0000 7f1ef9661700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ef4105560 0x7f1ef41a4680 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f1eec00b6c0 tx=0x7f1eec009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.445+0000 7f1ef1ffb700 1 -- 172.21.15.2:0/3484608081 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec00e040 con 0x7f1ef4105560 2023-11-16T18:31:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.445+0000 7f1efb0c4700 1 -- 172.21.15.2:0/3484608081 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec009c40 con 0x7f1ef4105560 2023-11-16T18:31:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.445+0000 7f1efb0c4700 1 -- 172.21.15.2:0/3484608081 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef4110380 con 0x7f1ef4105560 2023-11-16T18:31:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.446+0000 7f1ef1ffb700 1 -- 172.21.15.2:0/3484608081 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1eec00bbf0 con 0x7f1ef4105560 2023-11-16T18:31:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.446+0000 7f1ef1ffb700 1 -- 172.21.15.2:0/3484608081 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec0052a0 con 0x7f1ef4105560 2023-11-16T18:31:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.448+0000 7f1ef1ffb700 1 -- 172.21.15.2:0/3484608081 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f1eec005400 con 0x7f1ef4105560 2023-11-16T18:31:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.448+0000 7f1ef1ffb700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f1ee40a1340 0x7f1ee40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.449+0000 7f1ef8e60700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f1ee40a1340 0x7f1ee40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.449+0000 7f1ef1ffb700 1 -- 172.21.15.2:0/3484608081 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f1eec0ce210 con 0x7f1ef4105560 2023-11-16T18:31:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.449+0000 7f1ef8e60700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f1ee40a1340 0x7f1ee40a37f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8005fd0 tx=0x7f1ee8005f60 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.449+0000 7f1f07250b80 1 Processor -- start 2023-11-16T18:31:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.449+0000 7f1f07250b80 1 -- start start 2023-11-16T18:31:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.449+0000 7f1f07250b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 0x5564abb5a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.449+0000 7f1f07250b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb54560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.449+0000 7f1f07250b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb54b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.449+0000 7f1f07250b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aba5a000 con 0x5564abb5a720 2023-11-16T18:31:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.449+0000 7f1f07250b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564abaac840 con 0x5564abb5ab90 2023-11-16T18:31:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.449+0000 7f1f07250b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564abaa5cd0 con 0x5564ab9b06b0 2023-11-16T18:31:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1ef3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb54560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1ef9661700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb54b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1ef9661700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb54b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43574/0 (socket says 172.21.15.2:43574) 2023-11-16T18:31:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1ef9661700 1 -- 172.21.15.2:0/261480374 learned_addr learned my addr 172.21.15.2:0/261480374 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1ef8e60700 1 --2- 172.21.15.2:0/261480374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 0x5564abb5a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1ef9661700 1 -- 172.21.15.2:0/261480374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 msgr2=0x5564abb5a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1ef9661700 1 --2- 172.21.15.2:0/261480374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 0x5564abb5a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1ef9661700 1 -- 172.21.15.2:0/261480374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 msgr2=0x5564abb54560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1ef9661700 1 --2- 172.21.15.2:0/261480374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb54560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1ef9661700 1 -- 172.21.15.2:0/261480374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564ab9275d0 con 0x5564abb5ab90 2023-11-16T18:31:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1ef9661700 1 --2- 172.21.15.2:0/261480374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb54b00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1eec097450 tx=0x7f1eec09adf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f05e508d3e7f3873 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1edeffd700 1 -- 172.21.15.2:0/261480374 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec01dea0 con 0x5564abb5ab90 2023-11-16T18:31:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1edeffd700 1 -- 172.21.15.2:0/261480374 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1eec09af50 con 0x5564abb5ab90 2023-11-16T18:31:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.450+0000 7f1edeffd700 1 -- 172.21.15.2:0/261480374 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec09b1b0 con 0x5564abb5ab90 2023-11-16T18:31:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.451+0000 7f1f07250b80 1 -- 172.21.15.2:0/261480374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 msgr2=0x5564abb54b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.451+0000 7f1f07250b80 1 --2- 172.21.15.2:0/261480374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb54b00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1eec097450 tx=0x7f1eec09adf0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.451+0000 7f1f07250b80 1 -- 172.21.15.2:0/261480374 shutdown_connections 2023-11-16T18:31:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.451+0000 7f1f07250b80 1 --2- 172.21.15.2:0/261480374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 0x5564abb5a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.451+0000 7f1f07250b80 1 --2- 172.21.15.2:0/261480374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb54b00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.451+0000 7f1f07250b80 1 --2- 172.21.15.2:0/261480374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb54560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.451+0000 7f1f07250b80 1 -- 172.21.15.2:0/261480374 >> 172.21.15.2:0/261480374 conn(0x5564abb49f50 msgr2=0x5564abb4f270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:36.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.451+0000 7f1f07250b80 1 -- 172.21.15.2:0/261480374 shutdown_connections 2023-11-16T18:31:36.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.451+0000 7f1f07250b80 1 -- 172.21.15.2:0/261480374 wait complete. 2023-11-16T18:31:36.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.451+0000 7f1f07250b80 1 -- start start 2023-11-16T18:31:36.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.451+0000 7f1f07250b80 1 Processor -- start 2023-11-16T18:31:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1f07250b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564ab9b06b0 0x5564abb63920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1f07250b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564abb5a720 0x5564abb63e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1f07250b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb5dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1f07250b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aba47880 con 0x5564ab9b06b0 2023-11-16T18:31:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1f07250b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aba47520 con 0x5564abb5ab90 2023-11-16T18:31:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1f07250b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ab9298c0 con 0x5564abb5a720 2023-11-16T18:31:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1f07250b80 10 client.0 _ll_register_callbacks cb 0x5564aba65720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1ef8e60700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564ab9b06b0 0x5564abb63920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1ef8e60700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564ab9b06b0 0x5564abb63920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53138/0 (socket says 172.21.15.2:53138) 2023-11-16T18:31:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1ef8e60700 1 -- 172.21.15.2:0/3148041766 learned_addr learned my addr 172.21.15.2:0/3148041766 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1ef3fff700 1 --2- 172.21.15.2:0/3148041766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564abb5a720 0x5564abb63e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1ef9661700 1 --2- 172.21.15.2:0/3148041766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb5dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1ef8e60700 1 -- 172.21.15.2:0/3148041766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564abb5a720 msgr2=0x5564abb63e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1ef8e60700 1 --2- 172.21.15.2:0/3148041766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564abb5a720 0x5564abb63e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1ef8e60700 1 -- 172.21.15.2:0/3148041766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 msgr2=0x5564abb5dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1ef8e60700 1 --2- 172.21.15.2:0/3148041766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb5dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.452+0000 7f1ef8e60700 1 -- 172.21.15.2:0/3148041766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef4110380 con 0x5564ab9b06b0 2023-11-16T18:31:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1ef8e60700 1 --2- 172.21.15.2:0/3148041766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564ab9b06b0 0x5564abb63920 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f1ee800ffe0 tx=0x7f1ee8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce43e0c75ba8743c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1ed3fff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:31:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1ed3fff700 1 -- 172.21.15.2:0/3148041766 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8028470 con 0x5564ab9b06b0 2023-11-16T18:31:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1ed3fff700 1 -- 172.21.15.2:0/3148041766 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f1ee80285d0 con 0x5564ab9b06b0 2023-11-16T18:31:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1f07250b80 1 -- 172.21.15.2:0/3148041766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5564abb47710 con 0x5564ab9b06b0 2023-11-16T18:31:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1ed3fff700 1 -- 172.21.15.2:0/3148041766 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ee8028790 con 0x5564ab9b06b0 2023-11-16T18:31:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1f07250b80 10 client.16851 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1f07250b80 1 -- 172.21.15.2:0/3148041766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5564aba6c130 con 0x5564ab9b06b0 2023-11-16T18:31:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1ed3fff700 1 -- 172.21.15.2:0/3148041766 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5564abb47710 con 0x5564ab9b06b0 2023-11-16T18:31:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1ed3fff700 1 -- 172.21.15.2:0/3148041766 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f1ee801e5f0 con 0x5564ab9b06b0 2023-11-16T18:31:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1f07250b80 10 client.16851 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1f07250b80 10 client.16851 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:31:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1f07250b80 20 client.16851 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1f07250b80 1 --2- 172.21.15.2:0/3148041766 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5564abb6df40 0x5564abb70350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1f07250b80 4 client.16851 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:31:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.453+0000 7f1f07250b80 1 -- 172.21.15.2:0/3148041766 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5564aba68040 con 0x5564abb6df40 2023-11-16T18:31:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.454+0000 7f1ef3fff700 1 --2- 172.21.15.2:0/3148041766 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5564abb6df40 0x5564abb70350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.454+0000 7f1ed3fff700 10 client.16851 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.454+0000 7f1ef3fff700 1 --2- 172.21.15.2:0/3148041766 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5564abb6df40 0x5564abb70350 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47af6bdffcea1ee1 server_cookie=4b58f0d64af473be in_seq=0 out_seq=0 2023-11-16T18:31:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1ed3fff700 1 -- 172.21.15.2:0/3148041766 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5564aba68040 con 0x5564abb6df40 2023-11-16T18:31:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1ed3fff700 10 client.16851 handle_command_reply: tid=0 2023-11-16T18:31:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1f07250b80 1 client.16851 shutdown 2023-11-16T18:31:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1f07250b80 1 -- 172.21.15.2:0/3148041766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564ab9b06b0 msgr2=0x5564abb63920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1f07250b80 1 --2- 172.21.15.2:0/3148041766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564ab9b06b0 0x5564abb63920 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f1ee800ffe0 tx=0x7f1ee8004750 comp rx=0 tx=0).stop 2023-11-16T18:31:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1f07250b80 1 -- 172.21.15.2:0/3148041766 shutdown_connections 2023-11-16T18:31:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1f07250b80 1 --2- 172.21.15.2:0/3148041766 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5564abb6df40 0x5564abb70350 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1f07250b80 1 --2- 172.21.15.2:0/3148041766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb5dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1ed3fff700 0 client.16851 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1f07250b80 1 --2- 172.21.15.2:0/3148041766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564ab9b06b0 0x5564abb63920 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1f07250b80 1 --2- 172.21.15.2:0/3148041766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564abb5a720 0x5564abb63e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1f07250b80 1 -- 172.21.15.2:0/3148041766 >> 172.21.15.2:0/3148041766 conn(0x5564abb49f50 msgr2=0x5564abb4e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1f07250b80 1 -- 172.21.15.2:0/3148041766 shutdown_connections 2023-11-16T18:31:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1f07250b80 1 -- 172.21.15.2:0/3148041766 wait complete. 2023-11-16T18:31:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.455+0000 7f1f07250b80 20 client.16851 trim_cache size 0 max 16384 2023-11-16T18:31:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.470+0000 7f1f07250b80 1 Processor -- start 2023-11-16T18:31:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.470+0000 7f1f07250b80 1 -- start start 2023-11-16T18:31:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.470+0000 7f1f07250b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 0x5564abb4ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.470+0000 7f1f07250b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb595b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.470+0000 7f1f07250b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb56c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.470+0000 7f1f07250b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564abb50bc0 con 0x5564abb5a720 2023-11-16T18:31:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.470+0000 7f1f07250b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564abb4bfd0 con 0x5564abb5ab90 2023-11-16T18:31:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.470+0000 7f1f07250b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564abb5a570 con 0x5564ab9b06b0 2023-11-16T18:31:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.470+0000 7f1ef3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.470+0000 7f1ef3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb595b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53148/0 (socket says 172.21.15.2:53148) 2023-11-16T18:31:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.470+0000 7f1ef3fff700 1 -- 172.21.15.2:0/1797462765 learned_addr learned my addr 172.21.15.2:0/1797462765 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.471+0000 7f1ef9661700 1 --2- 172.21.15.2:0/1797462765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb56c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.471+0000 7f1ef8e60700 1 --2- 172.21.15.2:0/1797462765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 0x5564abb4ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.471+0000 7f1ef3fff700 1 -- 172.21.15.2:0/1797462765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 msgr2=0x5564abb4ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.471+0000 7f1ef3fff700 1 --2- 172.21.15.2:0/1797462765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 0x5564abb4ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.471+0000 7f1ef3fff700 1 -- 172.21.15.2:0/1797462765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 msgr2=0x5564abb56c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.471+0000 7f1ef3fff700 1 --2- 172.21.15.2:0/1797462765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb56c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.471+0000 7f1ef3fff700 1 -- 172.21.15.2:0/1797462765 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564abb74f40 con 0x5564abb5a720 2023-11-16T18:31:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.471+0000 7f1ef3fff700 1 --2- 172.21.15.2:0/1797462765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb595b0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1ef41019c0 tx=0x7f1ee0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1e8f2e421a3a596 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.471+0000 7f1edf7fe700 1 -- 172.21.15.2:0/1797462765 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee0017040 con 0x5564abb5a720 2023-11-16T18:31:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.471+0000 7f1edf7fe700 1 -- 172.21.15.2:0/1797462765 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ee0018040 con 0x5564abb5a720 2023-11-16T18:31:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.471+0000 7f1edf7fe700 1 -- 172.21.15.2:0/1797462765 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee0004a90 con 0x5564abb5a720 2023-11-16T18:31:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.472+0000 7f1f07250b80 1 -- 172.21.15.2:0/1797462765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 msgr2=0x5564abb595b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.472+0000 7f1f07250b80 1 --2- 172.21.15.2:0/1797462765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb595b0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1ef41019c0 tx=0x7f1ee0009310 comp rx=0 tx=0).stop 2023-11-16T18:31:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.472+0000 7f1f07250b80 1 -- 172.21.15.2:0/1797462765 shutdown_connections 2023-11-16T18:31:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.472+0000 7f1f07250b80 1 --2- 172.21.15.2:0/1797462765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 0x5564abb4ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.472+0000 7f1f07250b80 1 --2- 172.21.15.2:0/1797462765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb56c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.472+0000 7f1f07250b80 1 --2- 172.21.15.2:0/1797462765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb595b0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.472+0000 7f1f07250b80 1 -- 172.21.15.2:0/1797462765 >> 172.21.15.2:0/1797462765 conn(0x5564abb49f50 msgr2=0x5564abb5b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.472+0000 7f1f07250b80 1 -- 172.21.15.2:0/1797462765 shutdown_connections 2023-11-16T18:31:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.472+0000 7f1f07250b80 1 -- 172.21.15.2:0/1797462765 wait complete. 2023-11-16T18:31:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.472+0000 7f1f07250b80 1 -- start start 2023-11-16T18:31:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.472+0000 7f1f07250b80 1 Processor -- start 2023-11-16T18:31:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.472+0000 7f1f07250b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 0x5564abb61410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.472+0000 7f1f07250b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb61950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1f07250b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb61e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1f07250b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564abbc0e80 con 0x5564abb5a720 2023-11-16T18:31:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1f07250b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564abbc0ff0 con 0x5564abb5ab90 2023-11-16T18:31:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1f07250b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564abbc11b0 con 0x5564ab9b06b0 2023-11-16T18:31:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1f07250b80 10 client.0 _ll_register_callbacks cb 0x7f1ee800fbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ef3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb61950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ef3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb61950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53154/0 (socket says 172.21.15.2:53154) 2023-11-16T18:31:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ef9661700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb61e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ef8e60700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 0x5564abb61410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ef3fff700 1 -- 172.21.15.2:0/1843681520 learned_addr learned my addr 172.21.15.2:0/1843681520 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ef3fff700 1 -- 172.21.15.2:0/1843681520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 msgr2=0x5564abb61410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ef3fff700 1 --2- 172.21.15.2:0/1843681520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 0x5564abb61410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ef3fff700 1 -- 172.21.15.2:0/1843681520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 msgr2=0x5564abb61e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ef3fff700 1 --2- 172.21.15.2:0/1843681520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb61e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ef3fff700 1 -- 172.21.15.2:0/1843681520 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee0021a50 con 0x5564abb5a720 2023-11-16T18:31:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ef9661700 1 --2- 172.21.15.2:0/1843681520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb61e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ef3fff700 1 --2- 172.21.15.2:0/1843681520 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb61950 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0009c30 tx=0x7f1ee0013de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953bd1603f714c04 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ed3fff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:31:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ed3fff700 1 -- 172.21.15.2:0/1843681520 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee0012b10 con 0x5564abb5a720 2023-11-16T18:31:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1ed3fff700 1 -- 172.21.15.2:0/1843681520 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f1ee00210f0 con 0x5564abb5a720 2023-11-16T18:31:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.473+0000 7f1f07250b80 1 -- 172.21.15.2:0/1843681520 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1eec09af50 con 0x5564abb5a720 2023-11-16T18:31:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1ed3fff700 1 -- 172.21.15.2:0/1843681520 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ee00212b0 con 0x5564abb5a720 2023-11-16T18:31:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1f07250b80 10 client.16857 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1f07250b80 1 -- 172.21.15.2:0/1843681520 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5564ab8d3d80 con 0x5564abb5a720 2023-11-16T18:31:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1ed3fff700 1 -- 172.21.15.2:0/1843681520 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1eec09af50 con 0x5564abb5a720 2023-11-16T18:31:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1ed3fff700 1 -- 172.21.15.2:0/1843681520 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f1ee0005170 con 0x5564abb5a720 2023-11-16T18:31:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1f07250b80 10 client.16857 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1f07250b80 10 client.16857 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:31:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1f07250b80 20 client.16857 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1f07250b80 1 --2- 172.21.15.2:0/1843681520 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5564abbc2870 0x5564abbc4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1f07250b80 4 client.16857 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:31:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1f07250b80 1 -- 172.21.15.2:0/1843681520 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5564aba74600 con 0x5564abbc2870 2023-11-16T18:31:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1ef8e60700 1 --2- 172.21.15.2:0/1843681520 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5564abbc2870 0x5564abbc4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1ed3fff700 10 client.16857 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:36.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.474+0000 7f1ef8e60700 1 --2- 172.21.15.2:0/1843681520 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5564abbc2870 0x5564abbc4c80 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93445d110d64fc84 server_cookie=b8d139af7d0245ff in_seq=0 out_seq=0 2023-11-16T18:31:36.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:36 smithi040 ceph-mon[100251]: pgmap v207: 129 pgs: 129 active+clean; 718 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 404 op/s 2023-11-16T18:31:36.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:36 smithi188 ceph-mon[100639]: pgmap v207: 129 pgs: 129 active+clean; 718 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 404 op/s 2023-11-16T18:31:36.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:36 smithi002 ceph-mon[98807]: pgmap v207: 129 pgs: 129 active+clean; 718 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 404 op/s 2023-11-16T18:31:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1ed3fff700 1 -- 172.21.15.2:0/1843681520 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5564aba74600 con 0x5564abbc2870 2023-11-16T18:31:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1ed3fff700 10 client.16857 handle_command_reply: tid=0 2023-11-16T18:31:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1f07250b80 1 client.16857 shutdown 2023-11-16T18:31:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1f07250b80 1 -- 172.21.15.2:0/1843681520 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 msgr2=0x5564abb61950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1f07250b80 1 --2- 172.21.15.2:0/1843681520 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb61950 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0009c30 tx=0x7f1ee0013de0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1f07250b80 1 -- 172.21.15.2:0/1843681520 shutdown_connections 2023-11-16T18:31:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1f07250b80 1 --2- 172.21.15.2:0/1843681520 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5564abbc2870 0x5564abbc4c80 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1f07250b80 1 --2- 172.21.15.2:0/1843681520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5564ab9b06b0 0x5564abb61410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1ed3fff700 0 client.16857 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1f07250b80 1 --2- 172.21.15.2:0/1843681520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564abb5ab90 0x5564abb61e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1f07250b80 1 --2- 172.21.15.2:0/1843681520 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564abb5a720 0x5564abb61950 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1f07250b80 1 -- 172.21.15.2:0/1843681520 >> 172.21.15.2:0/1843681520 conn(0x5564abb49f50 msgr2=0x5564abb595b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1f07250b80 1 -- 172.21.15.2:0/1843681520 shutdown_connections 2023-11-16T18:31:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.605+0000 7f1f07250b80 1 -- 172.21.15.2:0/1843681520 wait complete. 2023-11-16T18:31:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.606+0000 7f1f07250b80 20 client.16857 trim_cache size 0 max 16384 2023-11-16T18:31:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:31:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:31:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:31:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:31:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.606+0000 7f1efb0c4700 1 -- 172.21.15.2:0/3484608081 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f1ee40a1340 msgr2=0x7f1ee40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.606+0000 7f1efb0c4700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f1ee40a1340 0x7f1ee40a37f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8005fd0 tx=0x7f1ee8005f60 comp rx=0 tx=0).stop 2023-11-16T18:31:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.606+0000 7f1efb0c4700 1 -- 172.21.15.2:0/3484608081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ef4105560 msgr2=0x7f1ef41a4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.606+0000 7f1efb0c4700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ef4105560 0x7f1ef41a4680 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f1eec00b6c0 tx=0x7f1eec009510 comp rx=0 tx=0).stop 2023-11-16T18:31:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.606+0000 7f1efb0c4700 1 -- 172.21.15.2:0/3484608081 shutdown_connections 2023-11-16T18:31:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.606+0000 7f1efb0c4700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f1ee40a1340 0x7f1ee40a37f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.606+0000 7f1efb0c4700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ef4100c50 0x7f1ef4072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.606+0000 7f1efb0c4700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ef4105560 0x7f1ef41a4680 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.606+0000 7f1efb0c4700 1 --2- 172.21.15.2:0/3484608081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef41015f0 0x7f1ef4073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.606+0000 7f1efb0c4700 1 -- 172.21.15.2:0/3484608081 >> 172.21.15.2:0/3484608081 conn(0x7f1ef4077550 msgr2=0x7f1ef40fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.606+0000 7f1efb0c4700 1 -- 172.21.15.2:0/3484608081 shutdown_connections 2023-11-16T18:31:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:36.606+0000 7f1efb0c4700 1 -- 172.21.15.2:0/3484608081 wait complete. 2023-11-16T18:31:37.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:37 smithi002 ceph-mon[98807]: pgmap v208: 129 pgs: 129 active+clean; 718 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 375 op/s 2023-11-16T18:31:37.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:37 smithi040 ceph-mon[100251]: pgmap v208: 129 pgs: 129 active+clean; 718 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 375 op/s 2023-11-16T18:31:37.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:37 smithi188 ceph-mon[100639]: pgmap v208: 129 pgs: 129 active+clean; 718 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 375 op/s 2023-11-16T18:31:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:40 smithi040 ceph-mon[100251]: pgmap v209: 129 pgs: 129 active+clean; 715 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 544 op/s 2023-11-16T18:31:40.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:40 smithi188 ceph-mon[100639]: pgmap v209: 129 pgs: 129 active+clean; 715 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 544 op/s 2023-11-16T18:31:40.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:40 smithi002 ceph-mon[98807]: pgmap v209: 129 pgs: 129 active+clean; 715 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 544 op/s 2023-11-16T18:31:41.986 INFO:tasks.workunit.client.0.smithi002.stdout:cleaned up hierarchy 2023-11-16T18:31:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:42 smithi040 ceph-mon[100251]: pgmap v210: 129 pgs: 129 active+clean; 715 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 404 op/s 2023-11-16T18:31:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:42 smithi188 ceph-mon[100639]: pgmap v210: 129 pgs: 129 active+clean; 715 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 404 op/s 2023-11-16T18:31:42.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:42 smithi002 ceph-mon[98807]: pgmap v210: 129 pgs: 129 active+clean; 715 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 404 op/s 2023-11-16T18:31:44.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:44 smithi040 ceph-mon[100251]: pgmap v211: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 594 op/s 2023-11-16T18:31:44.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:44 smithi188 ceph-mon[100639]: pgmap v211: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 594 op/s 2023-11-16T18:31:44.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:44 smithi002 ceph-mon[98807]: pgmap v211: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 594 op/s 2023-11-16T18:31:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:46 smithi040 ceph-mon[100251]: pgmap v212: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 397 op/s 2023-11-16T18:31:46.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:46 smithi188 ceph-mon[100639]: pgmap v212: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 397 op/s 2023-11-16T18:31:46.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:46 smithi002 ceph-mon[98807]: pgmap v212: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 397 op/s 2023-11-16T18:31:47.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:47 smithi002 ceph-mon[98807]: pgmap v213: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 397 op/s 2023-11-16T18:31:47.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:47 smithi040 ceph-mon[100251]: pgmap v213: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 397 op/s 2023-11-16T18:31:47.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:47 smithi188 ceph-mon[100639]: pgmap v213: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 397 op/s 2023-11-16T18:31:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.729+0000 7f5ff9c0f700 1 -- 172.21.15.2:0/432502618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ff40ff3e0 msgr2=0x7f5ff40ff7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.729+0000 7f5ff9c0f700 1 --2- 172.21.15.2:0/432502618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ff40ff3e0 0x7f5ff40ff7b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40099c0 tx=0x7f5fe4009cd0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.730+0000 7f5ff9c0f700 1 -- 172.21.15.2:0/432502618 shutdown_connections 2023-11-16T18:31:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.730+0000 7f5ff9c0f700 1 --2- 172.21.15.2:0/432502618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ff4104420 0x7f5ff410bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.730+0000 7f5ff9c0f700 1 --2- 172.21.15.2:0/432502618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ff40ff3e0 0x7f5ff40ff7b0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.730+0000 7f5ff9c0f700 1 --2- 172.21.15.2:0/432502618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ff40ffd80 0x7f5ff4103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.730+0000 7f5ff9c0f700 1 -- 172.21.15.2:0/432502618 >> 172.21.15.2:0/432502618 conn(0x7f5ff40761d0 msgr2=0x7f5ff40765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.730+0000 7f5ff9c0f700 1 -- 172.21.15.2:0/432502618 shutdown_connections 2023-11-16T18:31:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.730+0000 7f5ff9c0f700 1 -- 172.21.15.2:0/432502618 wait complete. 2023-11-16T18:31:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.730+0000 7f5ff9c0f700 1 Processor -- start 2023-11-16T18:31:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.730+0000 7f5ff9c0f700 1 -- start start 2023-11-16T18:31:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff9c0f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ff40ff3e0 0x7f5ff4195320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff9c0f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ff40ffd80 0x7f5ff4195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff9c0f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ff4104420 0x7f5ff4199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff9c0f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff419a300 con 0x7f5ff4104420 2023-11-16T18:31:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff9c0f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff419a470 con 0x7f5ff40ffd80 2023-11-16T18:31:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff9c0f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff419a630 con 0x7f5ff40ff3e0 2023-11-16T18:31:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ff4104420 0x7f5ff4199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ff4104420 0x7f5ff4199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53170/0 (socket says 172.21.15.2:53170) 2023-11-16T18:31:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff3fff700 1 -- 172.21.15.2:0/3677408624 learned_addr learned my addr 172.21.15.2:0/3677408624 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff37fe700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ff40ff3e0 0x7f5ff4195320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff2ffd700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ff40ffd80 0x7f5ff4195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff3fff700 1 -- 172.21.15.2:0/3677408624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ff40ff3e0 msgr2=0x7f5ff4195320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff3fff700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ff40ff3e0 0x7f5ff4195320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff3fff700 1 -- 172.21.15.2:0/3677408624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ff40ffd80 msgr2=0x7f5ff4195860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff3fff700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ff40ffd80 0x7f5ff4195860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff3fff700 1 -- 172.21.15.2:0/3677408624 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe40095f0 con 0x7f5ff4104420 2023-11-16T18:31:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.731+0000 7f5ff3fff700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ff4104420 0x7f5ff4199cc0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5fec007ee0 tx=0x7f5fec00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.732+0000 7f5ff0ff9700 1 -- 172.21.15.2:0/3677408624 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec017070 con 0x7f5ff4104420 2023-11-16T18:31:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.732+0000 7f5ff0ff9700 1 -- 172.21.15.2:0/3677408624 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fec00f040 con 0x7f5ff4104420 2023-11-16T18:31:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.732+0000 7f5ff0ff9700 1 -- 172.21.15.2:0/3677408624 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec005580 con 0x7f5ff4104420 2023-11-16T18:31:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.732+0000 7f5ff9c0f700 1 -- 172.21.15.2:0/3677408624 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff410f290 con 0x7f5ff4104420 2023-11-16T18:31:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.732+0000 7f5ff9c0f700 1 -- 172.21.15.2:0/3677408624 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff410f100 con 0x7f5ff4104420 2023-11-16T18:31:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.734+0000 7f5ff0ff9700 1 -- 172.21.15.2:0/3677408624 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f5fec00db20 con 0x7f5ff4104420 2023-11-16T18:31:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.734+0000 7f5ff0ff9700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f5fdc0a12c0 0x7f5fdc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.734+0000 7f5ff0ff9700 1 -- 172.21.15.2:0/3677408624 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f5fec0cc7f0 con 0x7f5ff4104420 2023-11-16T18:31:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.734+0000 7f5ff37fe700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f5fdc0a12c0 0x7f5fdc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.735+0000 7f5ff37fe700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f5fdc0a12c0 0x7f5fdc0a3770 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4009990 tx=0x7f5fe4000bc0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.735+0000 7f6005d9bb80 1 Processor -- start 2023-11-16T18:31:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f6005d9bb80 1 -- start start 2023-11-16T18:31:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f6005d9bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb86276b0 0x555cb87cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f6005d9bb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87cc950 0x555cb87ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f6005d9bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87d9ac0 0x555cb87dbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f6005d9bb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb86d1000 con 0x555cb87cc950 2023-11-16T18:31:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f6005d9bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb8723840 con 0x555cb86276b0 2023-11-16T18:31:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f6005d9bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb871ccd0 con 0x555cb87d9ac0 2023-11-16T18:31:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f5ff2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87cc950 0x555cb87ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f5ff37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb86276b0 0x555cb87cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f5ff2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87cc950 0x555cb87ccda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53176/0 (socket says 172.21.15.2:53176) 2023-11-16T18:31:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f5ff2ffd700 1 -- 172.21.15.2:0/3977247143 learned_addr learned my addr 172.21.15.2:0/3977247143 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f5ff3fff700 1 --2- 172.21.15.2:0/3977247143 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87d9ac0 0x555cb87dbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f5ff2ffd700 1 -- 172.21.15.2:0/3977247143 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87d9ac0 msgr2=0x555cb87dbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f5ff2ffd700 1 --2- 172.21.15.2:0/3977247143 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87d9ac0 0x555cb87dbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f5ff2ffd700 1 -- 172.21.15.2:0/3977247143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb86276b0 msgr2=0x555cb87cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f5ff2ffd700 1 --2- 172.21.15.2:0/3977247143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb86276b0 0x555cb87cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f5ff2ffd700 1 -- 172.21.15.2:0/3977247143 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cb859e5d0 con 0x555cb87cc950 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f5ff3fff700 1 --2- 172.21.15.2:0/3977247143 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87d9ac0 0x555cb87dbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f5ff37fe700 1 --2- 172.21.15.2:0/3977247143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb86276b0 0x555cb87cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.736+0000 7f5ff2ffd700 1 --2- 172.21.15.2:0/3977247143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87cc950 0x555cb87ccda0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8009a60 tx=0x7f5fe8009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=329404055cd8b145 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.737+0000 7f5fd9ffb700 1 -- 172.21.15.2:0/3977247143 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe8009530 con 0x555cb87cc950 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.737+0000 7f5fd9ffb700 1 -- 172.21.15.2:0/3977247143 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fe800ed90 con 0x555cb87cc950 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.737+0000 7f5fd9ffb700 1 -- 172.21.15.2:0/3977247143 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe80123d0 con 0x555cb87cc950 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.737+0000 7f6005d9bb80 1 -- 172.21.15.2:0/3977247143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87cc950 msgr2=0x555cb87ccda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.737+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/3977247143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87cc950 0x555cb87ccda0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8009a60 tx=0x7f5fe8009d70 comp rx=0 tx=0).stop 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.737+0000 7f6005d9bb80 1 -- 172.21.15.2:0/3977247143 shutdown_connections 2023-11-16T18:31:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.737+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/3977247143 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87d9ac0 0x555cb87dbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.737+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/3977247143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb86276b0 0x555cb87cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.737+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/3977247143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87cc950 0x555cb87ccda0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.737+0000 7f6005d9bb80 1 -- 172.21.15.2:0/3977247143 >> 172.21.15.2:0/3977247143 conn(0x555cb87c0f50 msgr2=0x555cb87c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.737+0000 7f6005d9bb80 1 -- 172.21.15.2:0/3977247143 shutdown_connections 2023-11-16T18:31:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.737+0000 7f6005d9bb80 1 -- 172.21.15.2:0/3977247143 wait complete. 2023-11-16T18:31:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.738+0000 7f6005d9bb80 1 -- start start 2023-11-16T18:31:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.738+0000 7f6005d9bb80 1 Processor -- start 2023-11-16T18:31:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.738+0000 7f6005d9bb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 0x555cb87d9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.738+0000 7f6005d9bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 0x555cb87d6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.738+0000 7f6005d9bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb87d9ac0 0x555cb87d6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.738+0000 7f6005d9bb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb86be880 con 0x555cb86276b0 2023-11-16T18:31:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.738+0000 7f6005d9bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb86be520 con 0x555cb87d9ac0 2023-11-16T18:31:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.738+0000 7f6005d9bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb85a08c0 con 0x555cb87cc950 2023-11-16T18:31:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.738+0000 7f6005d9bb80 10 client.0 _ll_register_callbacks cb 0x555cb86dc720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.738+0000 7f5ff37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 0x555cb87d9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.738+0000 7f5ff37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 0x555cb87d9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53180/0 (socket says 172.21.15.2:53180) 2023-11-16T18:31:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5ff37fe700 1 -- 172.21.15.2:0/4012630169 learned_addr learned my addr 172.21.15.2:0/4012630169 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5ff3fff700 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb87d9ac0 0x555cb87d6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5ff2ffd700 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 0x555cb87d6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5ff37fe700 1 -- 172.21.15.2:0/4012630169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 msgr2=0x555cb87d6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5ff37fe700 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 0x555cb87d6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5ff37fe700 1 -- 172.21.15.2:0/4012630169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb87d9ac0 msgr2=0x555cb87d6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5ff37fe700 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb87d9ac0 0x555cb87d6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5ff37fe700 1 -- 172.21.15.2:0/4012630169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe8009690 con 0x555cb86276b0 2023-11-16T18:31:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5ff3fff700 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb87d9ac0 0x555cb87d6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5ff37fe700 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 0x555cb87d9480 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40099c0 tx=0x7f5fe4020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a913166d2400fd20 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5fd2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:31:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5fd2ffd700 1 -- 172.21.15.2:0/4012630169 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe402d040 con 0x555cb86276b0 2023-11-16T18:31:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5fd2ffd700 1 -- 172.21.15.2:0/4012630169 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f5fe40040c0 con 0x555cb86276b0 2023-11-16T18:31:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f6005d9bb80 1 -- 172.21.15.2:0/4012630169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555cb87be710 con 0x555cb86276b0 2023-11-16T18:31:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f5fd2ffd700 1 -- 172.21.15.2:0/4012630169 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fe4004500 con 0x555cb86276b0 2023-11-16T18:31:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f6005d9bb80 10 client.16869 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.739+0000 7f6005d9bb80 1 -- 172.21.15.2:0/4012630169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x555cb86e3130 con 0x555cb86276b0 2023-11-16T18:31:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.740+0000 7f5fd2ffd700 1 -- 172.21.15.2:0/4012630169 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555cb87be710 con 0x555cb86276b0 2023-11-16T18:31:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.740+0000 7f5fd2ffd700 1 -- 172.21.15.2:0/4012630169 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5fe400f4d0 con 0x555cb86276b0 2023-11-16T18:31:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.740+0000 7f6005d9bb80 10 client.16869 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.740+0000 7f6005d9bb80 10 client.16869 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:31:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.740+0000 7f6005d9bb80 20 client.16869 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.740+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555cb87e9450 0x555cb87eb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.740+0000 7f6005d9bb80 4 client.16869 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:31:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.740+0000 7f6005d9bb80 1 -- 172.21.15.2:0/4012630169 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555cb86df040 con 0x555cb87e9450 2023-11-16T18:31:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.740+0000 7f5ff2ffd700 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555cb87e9450 0x555cb87eb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.741+0000 7f5fd2ffd700 10 client.16869 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.741+0000 7f5ff2ffd700 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555cb87e9450 0x555cb87eb860 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=684ac45568163470 server_cookie=d715231b7d16a58f in_seq=0 out_seq=0 2023-11-16T18:31:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.741+0000 7f5fd2ffd700 1 -- 172.21.15.2:0/4012630169 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555cb86df040 con 0x555cb87e9450 2023-11-16T18:31:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.741+0000 7f5fd2ffd700 10 client.16869 handle_command_reply: tid=0 2023-11-16T18:31:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.741+0000 7f6005d9bb80 1 client.16869 shutdown 2023-11-16T18:31:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.742+0000 7f6005d9bb80 1 -- 172.21.15.2:0/4012630169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 msgr2=0x555cb87d9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.742+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 0x555cb87d9480 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40099c0 tx=0x7f5fe4020000 comp rx=0 tx=0).stop 2023-11-16T18:31:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.742+0000 7f6005d9bb80 1 -- 172.21.15.2:0/4012630169 shutdown_connections 2023-11-16T18:31:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.742+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555cb87e9450 0x555cb87eb860 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.742+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb87d9ac0 0x555cb87d6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.742+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 0x555cb87d9480 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.742+0000 7f5fd2ffd700 0 client.16869 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.742+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/4012630169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 0x555cb87d6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.742+0000 7f6005d9bb80 1 -- 172.21.15.2:0/4012630169 >> 172.21.15.2:0/4012630169 conn(0x555cb87c0f50 msgr2=0x555cb87cedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.742+0000 7f6005d9bb80 1 -- 172.21.15.2:0/4012630169 shutdown_connections 2023-11-16T18:31:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.742+0000 7f6005d9bb80 1 -- 172.21.15.2:0/4012630169 wait complete. 2023-11-16T18:31:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.742+0000 7f6005d9bb80 20 client.16869 trim_cache size 0 max 16384 2023-11-16T18:31:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f6005d9bb80 1 Processor -- start 2023-11-16T18:31:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f6005d9bb80 1 -- start start 2023-11-16T18:31:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f6005d9bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb86276b0 0x555cb87e04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f6005d9bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 0x555cb87d9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f6005d9bb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87d9ac0 0x555cb87c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f6005d9bb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb87c6a90 con 0x555cb87d9ac0 2023-11-16T18:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f6005d9bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb87e09e0 con 0x555cb86276b0 2023-11-16T18:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f6005d9bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb87cf3d0 con 0x555cb87cc950 2023-11-16T18:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f5ff3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87d9ac0 0x555cb87c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f5ff3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87d9ac0 0x555cb87c5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53192/0 (socket says 172.21.15.2:53192) 2023-11-16T18:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f5ff3fff700 1 -- 172.21.15.2:0/3499306925 learned_addr learned my addr 172.21.15.2:0/3499306925 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f5ff2ffd700 1 --2- 172.21.15.2:0/3499306925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 0x555cb87d9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f5ff37fe700 1 --2- 172.21.15.2:0/3499306925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb86276b0 0x555cb87e04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f5ff3fff700 1 -- 172.21.15.2:0/3499306925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 msgr2=0x555cb87d9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f5ff3fff700 1 --2- 172.21.15.2:0/3499306925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 0x555cb87d9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f5ff3fff700 1 -- 172.21.15.2:0/3499306925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb86276b0 msgr2=0x555cb87e04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f5ff3fff700 1 --2- 172.21.15.2:0/3499306925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb86276b0 0x555cb87e04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f5ff3fff700 1 -- 172.21.15.2:0/3499306925 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cb86e2e80 con 0x555cb87d9ac0 2023-11-16T18:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f5ff37fe700 1 --2- 172.21.15.2:0/3499306925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb86276b0 0x555cb87e04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.757+0000 7f5ff3fff700 1 --2- 172.21.15.2:0/3499306925 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87d9ac0 0x555cb87c5f50 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5fec0c9c30 tx=0x7f5fec084040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234a3b70afe06784 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.758+0000 7f5fda7fc700 1 -- 172.21.15.2:0/3499306925 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec0a1950 con 0x555cb87d9ac0 2023-11-16T18:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.758+0000 7f5fda7fc700 1 -- 172.21.15.2:0/3499306925 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fec0978a0 con 0x555cb87d9ac0 2023-11-16T18:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.758+0000 7f5fda7fc700 1 -- 172.21.15.2:0/3499306925 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec09d0b0 con 0x555cb87d9ac0 2023-11-16T18:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.758+0000 7f6005d9bb80 1 -- 172.21.15.2:0/3499306925 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87d9ac0 msgr2=0x555cb87c5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.758+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/3499306925 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87d9ac0 0x555cb87c5f50 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5fec0c9c30 tx=0x7f5fec084040 comp rx=0 tx=0).stop 2023-11-16T18:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.758+0000 7f6005d9bb80 1 -- 172.21.15.2:0/3499306925 shutdown_connections 2023-11-16T18:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.758+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/3499306925 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb87d9ac0 0x555cb87c5f50 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.758+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/3499306925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb86276b0 0x555cb87e04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.758+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/3499306925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 0x555cb87d9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.758+0000 7f6005d9bb80 1 -- 172.21.15.2:0/3499306925 >> 172.21.15.2:0/3499306925 conn(0x555cb87c0f50 msgr2=0x555cb87c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.758+0000 7f6005d9bb80 1 -- 172.21.15.2:0/3499306925 shutdown_connections 2023-11-16T18:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.758+0000 7f6005d9bb80 1 -- 172.21.15.2:0/3499306925 wait complete. 2023-11-16T18:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f6005d9bb80 1 -- start start 2023-11-16T18:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f6005d9bb80 1 Processor -- start 2023-11-16T18:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f6005d9bb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 0x555cb87dfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f6005d9bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 0x555cb87e0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f6005d9bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb87d9ac0 0x555cb87de0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f6005d9bb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb87de770 con 0x555cb86276b0 2023-11-16T18:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f6005d9bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb87de8e0 con 0x555cb87d9ac0 2023-11-16T18:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f6005d9bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb87deaa0 con 0x555cb87cc950 2023-11-16T18:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f6005d9bb80 10 client.0 _ll_register_callbacks cb 0x555cb86dc720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f5ff37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 0x555cb87dfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f5ff37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 0x555cb87dfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53194/0 (socket says 172.21.15.2:53194) 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f5ff2ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 0x555cb87e0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f5ff37fe700 1 -- 172.21.15.2:0/2603345959 learned_addr learned my addr 172.21.15.2:0/2603345959 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f5ff3fff700 1 --2- 172.21.15.2:0/2603345959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb87d9ac0 0x555cb87de0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f5ff37fe700 1 -- 172.21.15.2:0/2603345959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 msgr2=0x555cb87e0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f5ff37fe700 1 --2- 172.21.15.2:0/2603345959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 0x555cb87e0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f5ff37fe700 1 -- 172.21.15.2:0/2603345959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb87d9ac0 msgr2=0x555cb87de0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.759+0000 7f5ff37fe700 1 --2- 172.21.15.2:0/2603345959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb87d9ac0 0x555cb87de0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.760+0000 7f5ff37fe700 1 -- 172.21.15.2:0/2603345959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff410f100 con 0x555cb86276b0 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.760+0000 7f5ff3fff700 1 --2- 172.21.15.2:0/2603345959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb87d9ac0 0x555cb87de0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.760+0000 7f5ff37fe700 1 --2- 172.21.15.2:0/2603345959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 0x555cb87dfc40 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400f900 tx=0x7f5fe400f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfb0fd0bbaa9bf69 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.760+0000 7f5fd2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.760+0000 7f5fd2ffd700 1 -- 172.21.15.2:0/2603345959 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe402d040 con 0x555cb86276b0 2023-11-16T18:31:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.760+0000 7f5fd2ffd700 1 -- 172.21.15.2:0/2603345959 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f5fe4005350 con 0x555cb86276b0 2023-11-16T18:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.760+0000 7f6005d9bb80 1 -- 172.21.15.2:0/2603345959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5fe800ed90 con 0x555cb86276b0 2023-11-16T18:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.760+0000 7f5fd2ffd700 1 -- 172.21.15.2:0/2603345959 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fe400fa50 con 0x555cb86276b0 2023-11-16T18:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.760+0000 7f6005d9bb80 10 client.16875 fetch_fsmap learned FSMap version 29 2023-11-16T18:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.760+0000 7f6005d9bb80 1 -- 172.21.15.2:0/2603345959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x555cb8533c00 con 0x555cb86276b0 2023-11-16T18:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.760+0000 7f5fd2ffd700 1 -- 172.21.15.2:0/2603345959 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fe800ed90 con 0x555cb86276b0 2023-11-16T18:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.761+0000 7f5fd2ffd700 1 -- 172.21.15.2:0/2603345959 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5fe4047020 con 0x555cb86276b0 2023-11-16T18:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.761+0000 7f6005d9bb80 10 client.16875 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.761+0000 7f6005d9bb80 10 client.16875 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.761+0000 7f6005d9bb80 20 client.16875 populate_metadata read hostname 'smithi002' 2023-11-16T18:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.761+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/2603345959 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555cb883dff0 0x555cb8840400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.761+0000 7f6005d9bb80 4 client.16875 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.761+0000 7f6005d9bb80 1 -- 172.21.15.2:0/2603345959 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555cb86eb600 con 0x555cb883dff0 2023-11-16T18:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.761+0000 7f5ff2ffd700 1 --2- 172.21.15.2:0/2603345959 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555cb883dff0 0x555cb8840400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:31:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.761+0000 7f5ff2ffd700 1 --2- 172.21.15.2:0/2603345959 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555cb883dff0 0x555cb8840400 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bfdeb62c6db7b0c server_cookie=3344998dec68c4ef in_seq=0 out_seq=0 2023-11-16T18:31:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.761+0000 7f5fd2ffd700 10 client.16875 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f5fd2ffd700 1 -- 172.21.15.2:0/2603345959 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555cb86eb600 con 0x555cb883dff0 2023-11-16T18:31:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f5fd2ffd700 10 client.16875 handle_command_reply: tid=0 2023-11-16T18:31:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f6005d9bb80 1 client.16875 shutdown 2023-11-16T18:31:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f6005d9bb80 1 -- 172.21.15.2:0/2603345959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 msgr2=0x555cb87dfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/2603345959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 0x555cb87dfc40 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400f900 tx=0x7f5fe400f710 comp rx=0 tx=0).stop 2023-11-16T18:31:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f6005d9bb80 1 -- 172.21.15.2:0/2603345959 shutdown_connections 2023-11-16T18:31:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/2603345959 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x555cb883dff0 0x555cb8840400 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/2603345959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cb87d9ac0 0x555cb87de0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f5fd2ffd700 0 client.16875 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:31:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/2603345959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555cb86276b0 0x555cb87dfc40 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f6005d9bb80 1 --2- 172.21.15.2:0/2603345959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555cb87cc950 0x555cb87e0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f6005d9bb80 1 -- 172.21.15.2:0/2603345959 >> 172.21.15.2:0/2603345959 conn(0x555cb87c0f50 msgr2=0x555cb87c5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f6005d9bb80 1 -- 172.21.15.2:0/2603345959 shutdown_connections 2023-11-16T18:31:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f6005d9bb80 1 -- 172.21.15.2:0/2603345959 wait complete. 2023-11-16T18:31:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.921+0000 7f6005d9bb80 20 client.16875 trim_cache size 0 max 16384 2023-11-16T18:31:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:31:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:31:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:31:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:31:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.922+0000 7f5ff9c0f700 1 -- 172.21.15.2:0/3677408624 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f5fdc0a12c0 msgr2=0x7f5fdc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.922+0000 7f5ff9c0f700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f5fdc0a12c0 0x7f5fdc0a3770 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4009990 tx=0x7f5fe4000bc0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.922+0000 7f5ff9c0f700 1 -- 172.21.15.2:0/3677408624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ff4104420 msgr2=0x7f5ff4199cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:31:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.922+0000 7f5ff9c0f700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ff4104420 0x7f5ff4199cc0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5fec007ee0 tx=0x7f5fec00d3b0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.922+0000 7f5ff9c0f700 1 -- 172.21.15.2:0/3677408624 shutdown_connections 2023-11-16T18:31:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.922+0000 7f5ff9c0f700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f5fdc0a12c0 0x7f5fdc0a3770 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.922+0000 7f5ff9c0f700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ff40ff3e0 0x7f5ff4195320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.922+0000 7f5ff9c0f700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ff4104420 0x7f5ff4199cc0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.922+0000 7f5ff9c0f700 1 --2- 172.21.15.2:0/3677408624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ff40ffd80 0x7f5ff4195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:31:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.922+0000 7f5ff9c0f700 1 -- 172.21.15.2:0/3677408624 >> 172.21.15.2:0/3677408624 conn(0x7f5ff40761d0 msgr2=0x7f5ff40fc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:31:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.922+0000 7f5ff9c0f700 1 -- 172.21.15.2:0/3677408624 shutdown_connections 2023-11-16T18:31:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:31:49.922+0000 7f5ff9c0f700 1 -- 172.21.15.2:0/3677408624 wait complete. 2023-11-16T18:31:50.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:50 smithi040 ceph-mon[100251]: pgmap v214: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 486 op/s 2023-11-16T18:31:50.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:50 smithi188 ceph-mon[100639]: pgmap v214: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 486 op/s 2023-11-16T18:31:50.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:50 smithi002 ceph-mon[98807]: pgmap v214: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 486 op/s 2023-11-16T18:31:52.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:52 smithi040 ceph-mon[100251]: pgmap v215: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 278 op/s 2023-11-16T18:31:52.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:52 smithi188 ceph-mon[100639]: pgmap v215: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 278 op/s 2023-11-16T18:31:52.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:52 smithi002 ceph-mon[98807]: pgmap v215: 129 pgs: 129 active+clean; 724 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 278 op/s 2023-11-16T18:31:54.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:54 smithi040 ceph-mon[100251]: pgmap v216: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 362 op/s 2023-11-16T18:31:54.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:54 smithi188 ceph-mon[100639]: pgmap v216: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 362 op/s 2023-11-16T18:31:54.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:54 smithi002 ceph-mon[98807]: pgmap v216: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 362 op/s 2023-11-16T18:31:56.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:56 smithi040 ceph-mon[100251]: pgmap v217: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 188 KiB/s wr, 172 op/s 2023-11-16T18:31:56.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:56 smithi188 ceph-mon[100639]: pgmap v217: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 188 KiB/s wr, 172 op/s 2023-11-16T18:31:56.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:56 smithi002 ceph-mon[98807]: pgmap v217: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 188 KiB/s wr, 172 op/s 2023-11-16T18:31:57.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:31:57 smithi002 ceph-mon[98807]: pgmap v218: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 188 KiB/s wr, 172 op/s 2023-11-16T18:31:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:31:57 smithi040 ceph-mon[100251]: pgmap v218: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 188 KiB/s wr, 172 op/s 2023-11-16T18:31:57.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:31:57 smithi188 ceph-mon[100639]: pgmap v218: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 188 KiB/s wr, 172 op/s 2023-11-16T18:32:00.263 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T18:32:00.264 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T18:32:00.348 INFO:tasks.workunit:Running workunit fs/misc/filelock_deadlock.py... 2023-11-16T18:32:00.348 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/filelock_deadlock.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_deadlock.py 2023-11-16T18:32:00.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:00 smithi040 ceph-mon[100251]: pgmap v219: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 191 KiB/s wr, 180 op/s 2023-11-16T18:32:00.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:00 smithi188 ceph-mon[100639]: pgmap v219: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 191 KiB/s wr, 180 op/s 2023-11-16T18:32:00.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:00 smithi002 ceph-mon[98807]: pgmap v219: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 191 KiB/s wr, 180 op/s 2023-11-16T18:32:02.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:02 smithi040 ceph-mon[100251]: pgmap v220: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 59 KiB/s wr, 91 op/s 2023-11-16T18:32:02.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:02 smithi188 ceph-mon[100639]: pgmap v220: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 59 KiB/s wr, 91 op/s 2023-11-16T18:32:02.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:02 smithi002 ceph-mon[98807]: pgmap v220: 129 pgs: 129 active+clean; 721 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 59 KiB/s wr, 91 op/s 2023-11-16T18:32:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.043+0000 7fbbb32a5700 1 -- 172.21.15.2:0/322389680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac101500 msgr2=0x7fbbac1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.043+0000 7fbbb32a5700 1 --2- 172.21.15.2:0/322389680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac101500 0x7fbbac1018d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c009990 tx=0x7fbb9c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.043+0000 7fbbb32a5700 1 -- 172.21.15.2:0/322389680 shutdown_connections 2023-11-16T18:32:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.043+0000 7fbbb32a5700 1 --2- 172.21.15.2:0/322389680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbac10a840 0x7fbbac10cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.043+0000 7fbbb32a5700 1 --2- 172.21.15.2:0/322389680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac101500 0x7fbbac1018d0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.043+0000 7fbbb32a5700 1 --2- 172.21.15.2:0/322389680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbac101e10 0x7fbbac10a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.043+0000 7fbbb32a5700 1 -- 172.21.15.2:0/322389680 >> 172.21.15.2:0/322389680 conn(0x7fbbac075fd0 msgr2=0x7fbbac0763d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:32:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.043+0000 7fbbb32a5700 1 -- 172.21.15.2:0/322389680 shutdown_connections 2023-11-16T18:32:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.043+0000 7fbbb32a5700 1 -- 172.21.15.2:0/322389680 wait complete. 2023-11-16T18:32:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.044+0000 7fbbb32a5700 1 Processor -- start 2023-11-16T18:32:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.044+0000 7fbbb32a5700 1 -- start start 2023-11-16T18:32:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.044+0000 7fbbb32a5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac101500 0x7fbbac0ff150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.044+0000 7fbbb32a5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbac101e10 0x7fbbac0ff690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.044+0000 7fbbb32a5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbac10a840 0x7fbbac1a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.044+0000 7fbbb32a5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac0ffec0 con 0x7fbbac101500 2023-11-16T18:32:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.044+0000 7fbbb32a5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac1a4ac0 con 0x7fbbac10a840 2023-11-16T18:32:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.044+0000 7fbbb32a5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac1a4c00 con 0x7fbbac101e10 2023-11-16T18:32:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.044+0000 7fbbb1041700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac101500 0x7fbbac0ff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.044+0000 7fbbb1041700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac101500 0x7fbbac0ff150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53208/0 (socket says 172.21.15.2:53208) 2023-11-16T18:32:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.044+0000 7fbbb1041700 1 -- 172.21.15.2:0/1015116466 learned_addr learned my addr 172.21.15.2:0/1015116466 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:32:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbbb1842700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbac10a840 0x7fbbac1a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbbb0840700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbac101e10 0x7fbbac0ff690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbbb1041700 1 -- 172.21.15.2:0/1015116466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbac101e10 msgr2=0x7fbbac0ff690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbbb1041700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbac101e10 0x7fbbac0ff690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbbb1041700 1 -- 172.21.15.2:0/1015116466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbac10a840 msgr2=0x7fbbac1a4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbbb1041700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbac10a840 0x7fbbac1a4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbbb1041700 1 -- 172.21.15.2:0/1015116466 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9c0095f0 con 0x7fbbac101500 2023-11-16T18:32:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbbb1842700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbac10a840 0x7fbbac1a4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:32:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbbb0840700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbac101e10 0x7fbbac0ff690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:32:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbbb1041700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac101500 0x7fbbac0ff150 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c0098e0 tx=0x7fbb9c00f720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:32:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbba27fc700 1 -- 172.21.15.2:0/1015116466 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c01d070 con 0x7fbbac101500 2023-11-16T18:32:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbbb32a5700 1 -- 172.21.15.2:0/1015116466 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbac1102d0 con 0x7fbbac101500 2023-11-16T18:32:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbba27fc700 1 -- 172.21.15.2:0/1015116466 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb9c005510 con 0x7fbbac101500 2023-11-16T18:32:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbba27fc700 1 -- 172.21.15.2:0/1015116466 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c004c90 con 0x7fbbac101500 2023-11-16T18:32:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.045+0000 7fbbb32a5700 1 -- 172.21.15.2:0/1015116466 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbac110140 con 0x7fbbac101500 2023-11-16T18:32:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.047+0000 7fbba27fc700 1 -- 172.21.15.2:0/1015116466 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fbb9c00fc10 con 0x7fbbac101500 2023-11-16T18:32:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.048+0000 7fbba27fc700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fbb980a12c0 0x7fbb980a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.048+0000 7fbba27fc700 1 -- 172.21.15.2:0/1015116466 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9570+0+0 (secure 0 0 0) 0x7fbb9c0cdfb0 con 0x7fbbac101500 2023-11-16T18:32:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.048+0000 7fbbb0840700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fbb980a12c0 0x7fbb980a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.049+0000 7fbbb0840700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fbb980a12c0 0x7fbb980a3770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbba8005fd0 tx=0x7fbba8005ee0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:32:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.049+0000 7fbbbf431b80 1 Processor -- start 2023-11-16T18:32:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.049+0000 7fbbbf431b80 1 -- start start 2023-11-16T18:32:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbbf431b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a8386b0 0x557b0a9e21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbbf431b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a9e2720 0x557b0a9dc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbbf431b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2b90 0x557b0a9dcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbbf431b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0a8e2000 con 0x557b0a9e2b90 2023-11-16T18:32:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbbf431b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0a934840 con 0x557b0a8386b0 2023-11-16T18:32:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbbf431b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0a92dcd0 con 0x557b0a9e2720 2023-11-16T18:32:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbb1041700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a8386b0 0x557b0a9e21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbb1041700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a8386b0 0x557b0a9e21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.2:43654/0 (socket says 172.21.15.2:43654) 2023-11-16T18:32:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbb1041700 1 -- 172.21.15.2:0/2949155184 learned_addr learned my addr 172.21.15.2:0/2949155184 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:32:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbb1842700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2b90 0x557b0a9dcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbb0840700 1 --2- 172.21.15.2:0/2949155184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a9e2720 0x557b0a9dc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbb1041700 1 -- 172.21.15.2:0/2949155184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a9e2720 msgr2=0x557b0a9dc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbb1041700 1 --2- 172.21.15.2:0/2949155184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a9e2720 0x557b0a9dc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbb1041700 1 -- 172.21.15.2:0/2949155184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2b90 msgr2=0x557b0a9dcb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbb1041700 1 --2- 172.21.15.2:0/2949155184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2b90 0x557b0a9dcb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbb1041700 1 -- 172.21.15.2:0/2949155184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b0a7af5d0 con 0x557b0a8386b0 2023-11-16T18:32:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbb0840700 1 --2- 172.21.15.2:0/2949155184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a9e2720 0x557b0a9dc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:32:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.050+0000 7fbbb1842700 1 --2- 172.21.15.2:0/2949155184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2b90 0x557b0a9dcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:32:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.051+0000 7fbbb1041700 1 --2- 172.21.15.2:0/2949155184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a8386b0 0x557b0a9e21e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c097700 tx=0x7fbb9c09a900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=617983845b6e3c62 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:32:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.051+0000 7fbb977fe700 1 -- 172.21.15.2:0/2949155184 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c0a3050 con 0x557b0a8386b0 2023-11-16T18:32:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.051+0000 7fbb977fe700 1 -- 172.21.15.2:0/2949155184 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb9c099030 con 0x557b0a8386b0 2023-11-16T18:32:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.051+0000 7fbb977fe700 1 -- 172.21.15.2:0/2949155184 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c09a430 con 0x557b0a8386b0 2023-11-16T18:32:03.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.052+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2949155184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a8386b0 msgr2=0x557b0a9e21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:03.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.052+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2949155184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a8386b0 0x557b0a9e21e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c097700 tx=0x7fbb9c09a900 comp rx=0 tx=0).stop 2023-11-16T18:32:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.052+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2949155184 shutdown_connections 2023-11-16T18:32:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.052+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2949155184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2b90 0x557b0a9dcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.052+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2949155184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a8386b0 0x557b0a9e21e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.052+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2949155184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a9e2720 0x557b0a9dc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.052+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2949155184 >> 172.21.15.2:0/2949155184 conn(0x557b0a9d1f50 msgr2=0x557b0a9d7270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:32:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.052+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2949155184 shutdown_connections 2023-11-16T18:32:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.052+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2949155184 wait complete. 2023-11-16T18:32:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.052+0000 7fbbbf431b80 1 -- start start 2023-11-16T18:32:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.052+0000 7fbbbf431b80 1 Processor -- start 2023-11-16T18:32:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbbf431b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a8386b0 0x557b0a9eb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbbf431b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2720 0x557b0a9ebd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbbf431b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2b90 0x557b0a9e5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbbf431b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0a8cf880 con 0x557b0a9e2b90 2023-11-16T18:32:03.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbbf431b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0a8cf520 con 0x557b0a9e2720 2023-11-16T18:32:03.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbbf431b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0a7b18c0 con 0x557b0a8386b0 2023-11-16T18:32:03.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbbf431b80 10 client.0 _ll_register_callbacks cb 0x557b0a8ed720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:32:03.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbb1842700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2b90 0x557b0a9e5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbb1842700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2b90 0x557b0a9e5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53226/0 (socket says 172.21.15.2:53226) 2023-11-16T18:32:03.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbb1842700 1 -- 172.21.15.2:0/2177383341 learned_addr learned my addr 172.21.15.2:0/2177383341 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:32:03.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbb0840700 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2720 0x557b0a9ebd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbb1041700 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a8386b0 0x557b0a9eb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbb1842700 1 -- 172.21.15.2:0/2177383341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a8386b0 msgr2=0x557b0a9eb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbb1842700 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a8386b0 0x557b0a9eb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbb1842700 1 -- 172.21.15.2:0/2177383341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2720 msgr2=0x557b0a9ebd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbb1842700 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2720 0x557b0a9ebd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbb1842700 1 -- 172.21.15.2:0/2177383341 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbac110140 con 0x557b0a9e2b90 2023-11-16T18:32:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbb1041700 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a8386b0 0x557b0a9eb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:32:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbbb1842700 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2b90 0x557b0a9e5a50 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fbba4007b20 tx=0x7fbba4007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a85e0e00cc7fa62 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:32:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbb94ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:32:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.053+0000 7fbb94ff9700 1 -- 172.21.15.2:0/2177383341 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba4004db0 con 0x557b0a9e2b90 2023-11-16T18:32:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.054+0000 7fbb94ff9700 1 -- 172.21.15.2:0/2177383341 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9570+0+0 (secure 0 0 0) 0x7fbba4012980 con 0x557b0a9e2b90 2023-11-16T18:32:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.054+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2177383341 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557b0a9cf710 con 0x557b0a9e2b90 2023-11-16T18:32:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.054+0000 7fbb94ff9700 1 -- 172.21.15.2:0/2177383341 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbba4012b40 con 0x557b0a9e2b90 2023-11-16T18:32:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.054+0000 7fbbbf431b80 10 client.16887 fetch_fsmap learned FSMap version 29 2023-11-16T18:32:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.054+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2177383341 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557b0a8f4130 con 0x557b0a9e2b90 2023-11-16T18:32:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.054+0000 7fbb94ff9700 1 -- 172.21.15.2:0/2177383341 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557b0a9cf710 con 0x557b0a9e2b90 2023-11-16T18:32:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.054+0000 7fbb94ff9700 1 -- 172.21.15.2:0/2177383341 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fbba4014040 con 0x557b0a9e2b90 2023-11-16T18:32:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.054+0000 7fbbbf431b80 10 client.16887 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:32:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.054+0000 7fbbbf431b80 10 client.16887 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:32:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.054+0000 7fbbbf431b80 20 client.16887 populate_metadata read hostname 'smithi002' 2023-11-16T18:32:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.054+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557b0a9f5da0 0x557b0a9f81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.054+0000 7fbbbf431b80 4 client.16887 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:32:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.054+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2177383341 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b0a8f0040 con 0x557b0a9f5da0 2023-11-16T18:32:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.055+0000 7fbbb1041700 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557b0a9f5da0 0x557b0a9f81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.055+0000 7fbb94ff9700 10 client.16887 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:32:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.055+0000 7fbbb1041700 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557b0a9f5da0 0x557b0a9f81b0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c32b5cfc1b212d87 server_cookie=984842ac7f03f31c in_seq=0 out_seq=0 2023-11-16T18:32:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbb94ff9700 1 -- 172.21.15.2:0/2177383341 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557b0a8f0040 con 0x557b0a9f5da0 2023-11-16T18:32:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbb94ff9700 10 client.16887 handle_command_reply: tid=0 2023-11-16T18:32:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbbbf431b80 1 client.16887 shutdown 2023-11-16T18:32:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2177383341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2b90 msgr2=0x557b0a9e5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2b90 0x557b0a9e5a50 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fbba4007b20 tx=0x7fbba4007e30 comp rx=0 tx=0).stop 2023-11-16T18:32:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2177383341 shutdown_connections 2023-11-16T18:32:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557b0a9f5da0 0x557b0a9f81b0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a8386b0 0x557b0a9eb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2b90 0x557b0a9e5a50 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbb94ff9700 0 client.16887 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:32:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2177383341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2720 0x557b0a9ebd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2177383341 >> 172.21.15.2:0/2177383341 conn(0x557b0a9d1f50 msgr2=0x557b0a9d67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:32:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2177383341 shutdown_connections 2023-11-16T18:32:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2177383341 wait complete. 2023-11-16T18:32:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.056+0000 7fbbbf431b80 20 client.16887 trim_cache size 0 max 16384 2023-11-16T18:32:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.071+0000 7fbbbf431b80 1 Processor -- start 2023-11-16T18:32:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.071+0000 7fbbbf431b80 1 -- start start 2023-11-16T18:32:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbbf431b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a8386b0 0x557b0a9dfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbbf431b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2720 0x557b0a9d6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbbf431b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a9e2b90 0x557b0a9e2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbbf431b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0a9d8480 con 0x557b0a8386b0 2023-11-16T18:32:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbbf431b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0a9f3370 con 0x557b0a9e2720 2023-11-16T18:32:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbbf431b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0a9e3fd0 con 0x557b0a9e2b90 2023-11-16T18:32:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbb1041700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a8386b0 0x557b0a9dfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbb1041700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a8386b0 0x557b0a9dfe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53230/0 (socket says 172.21.15.2:53230) 2023-11-16T18:32:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbb1041700 1 -- 172.21.15.2:0/4111872152 learned_addr learned my addr 172.21.15.2:0/4111872152 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:32:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbb1041700 1 -- 172.21.15.2:0/4111872152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a9e2b90 msgr2=0x557b0a9e2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbb1041700 1 --2- 172.21.15.2:0/4111872152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a9e2b90 0x557b0a9e2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbb0840700 1 --2- 172.21.15.2:0/4111872152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2720 0x557b0a9d6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbb1041700 1 -- 172.21.15.2:0/4111872152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2720 msgr2=0x557b0a9d6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbb1041700 1 --2- 172.21.15.2:0/4111872152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2720 0x557b0a9d6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbb1041700 1 -- 172.21.15.2:0/4111872152 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b0a9d36f0 con 0x557b0a8386b0 2023-11-16T18:32:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbbb1041700 1 --2- 172.21.15.2:0/4111872152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a8386b0 0x557b0a9dfe60 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c09a990 tx=0x7fbb9c003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7aead46fb2c3520 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:32:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbb97fff700 1 -- 172.21.15.2:0/4111872152 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c0b3280 con 0x557b0a8386b0 2023-11-16T18:32:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbb97fff700 1 -- 172.21.15.2:0/4111872152 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb9c0b33e0 con 0x557b0a8386b0 2023-11-16T18:32:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.072+0000 7fbb97fff700 1 -- 172.21.15.2:0/4111872152 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c0a2e20 con 0x557b0a8386b0 2023-11-16T18:32:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.073+0000 7fbbbf431b80 1 -- 172.21.15.2:0/4111872152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a8386b0 msgr2=0x557b0a9dfe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.073+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/4111872152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a8386b0 0x557b0a9dfe60 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c09a990 tx=0x7fbb9c003680 comp rx=0 tx=0).stop 2023-11-16T18:32:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.073+0000 7fbbbf431b80 1 -- 172.21.15.2:0/4111872152 shutdown_connections 2023-11-16T18:32:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.073+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/4111872152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a9e2b90 0x557b0a9e2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.073+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/4111872152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a8386b0 0x557b0a9dfe60 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.073+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/4111872152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2720 0x557b0a9d6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.073+0000 7fbbbf431b80 1 -- 172.21.15.2:0/4111872152 >> 172.21.15.2:0/4111872152 conn(0x557b0a9d1f50 msgr2=0x557b0a9e6380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.073+0000 7fbbbf431b80 1 -- 172.21.15.2:0/4111872152 shutdown_connections 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.073+0000 7fbbbf431b80 1 -- 172.21.15.2:0/4111872152 wait complete. 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.073+0000 7fbbbf431b80 1 -- start start 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.073+0000 7fbbbf431b80 1 Processor -- start 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbbf431b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a8386b0 0x557b0a9e9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbbf431b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2720 0x557b0a9e98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbbf431b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2b90 0x557b0a9e9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbbf431b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0aa48c20 con 0x557b0a9e2720 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbbf431b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0aa48d60 con 0x557b0a9e2b90 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbbf431b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0aa48ed0 con 0x557b0a8386b0 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbbf431b80 10 client.0 _ll_register_callbacks cb 0x557b0a8ed720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:32:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbb0840700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2720 0x557b0a9e98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbb0840700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2720 0x557b0a9e98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53238/0 (socket says 172.21.15.2:53238) 2023-11-16T18:32:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbb0840700 1 -- 172.21.15.2:0/2034756615 learned_addr learned my addr 172.21.15.2:0/2034756615 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:32:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbb1842700 1 --2- 172.21.15.2:0/2034756615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2b90 0x557b0a9e9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbb0840700 1 -- 172.21.15.2:0/2034756615 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a8386b0 msgr2=0x557b0a9e9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbb0840700 1 --2- 172.21.15.2:0/2034756615 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a8386b0 0x557b0a9e9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbb0840700 1 -- 172.21.15.2:0/2034756615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2b90 msgr2=0x557b0a9e9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbb0840700 1 --2- 172.21.15.2:0/2034756615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2b90 0x557b0a9e9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbb0840700 1 -- 172.21.15.2:0/2034756615 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b0a7af5d0 con 0x557b0a9e2720 2023-11-16T18:32:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbb1842700 1 --2- 172.21.15.2:0/2034756615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2b90 0x557b0a9e9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:32:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbbb0840700 1 --2- 172.21.15.2:0/2034756615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2720 0x557b0a9e98b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fbba8009410 tx=0x7fbba80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367ca36907a6b28a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:32:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbb94ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:32:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.074+0000 7fbb94ff9700 1 -- 172.21.15.2:0/2034756615 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba8004220 con 0x557b0a9e2720 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbb94ff9700 1 -- 172.21.15.2:0/2034756615 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9570+0+0 (secure 0 0 0) 0x7fbba8004380 con 0x557b0a9e2720 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2034756615 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb9c099030 con 0x557b0a9e2720 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbb94ff9700 1 -- 172.21.15.2:0/2034756615 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbba8004540 con 0x557b0a9e2720 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbbbf431b80 10 client.16893 fetch_fsmap learned FSMap version 29 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2034756615 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557b0a75bd80 con 0x557b0a9e2720 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbb94ff9700 1 -- 172.21.15.2:0/2034756615 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb9c099030 con 0x557b0a9e2720 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbb94ff9700 1 -- 172.21.15.2:0/2034756615 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fbba8003a50 con 0x557b0a9e2720 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbbbf431b80 10 client.16893 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbbbf431b80 10 client.16893 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbbbf431b80 20 client.16893 populate_metadata read hostname 'smithi002' 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2034756615 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557b0aa4a6b0 0x557b0aa4cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbbbf431b80 4 client.16893 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2034756615 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557b0a8fc600 con 0x557b0aa4a6b0 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.075+0000 7fbbb1041700 1 --2- 172.21.15.2:0/2034756615 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557b0aa4a6b0 0x557b0aa4cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.076+0000 7fbb94ff9700 10 client.16893 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:32:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.076+0000 7fbbb1041700 1 --2- 172.21.15.2:0/2034756615 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557b0aa4a6b0 0x557b0aa4cac0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ae6de4d6a805556 server_cookie=40ef7e2552c8aee6 in_seq=0 out_seq=0 2023-11-16T18:32:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbb94ff9700 1 -- 172.21.15.2:0/2034756615 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557b0a8fc600 con 0x557b0aa4a6b0 2023-11-16T18:32:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbb94ff9700 10 client.16893 handle_command_reply: tid=0 2023-11-16T18:32:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbbbf431b80 1 client.16893 shutdown 2023-11-16T18:32:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2034756615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2720 msgr2=0x557b0a9e98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2034756615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2720 0x557b0a9e98b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fbba8009410 tx=0x7fbba80047c0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2034756615 shutdown_connections 2023-11-16T18:32:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2034756615 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x557b0aa4a6b0 0x557b0aa4cac0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2034756615 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b0a8386b0 0x557b0a9e9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbb94ff9700 0 client.16893 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:32:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2034756615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b0a9e2b90 0x557b0a9e9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbbbf431b80 1 --2- 172.21.15.2:0/2034756615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b0a9e2720 0x557b0a9e98b0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2034756615 >> 172.21.15.2:0/2034756615 conn(0x557b0a9d1f50 msgr2=0x557b0a9dfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:32:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2034756615 shutdown_connections 2023-11-16T18:32:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbbbf431b80 1 -- 172.21.15.2:0/2034756615 wait complete. 2023-11-16T18:32:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.271+0000 7fbbbf431b80 20 client.16893 trim_cache size 0 max 16384 2023-11-16T18:32:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:32:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:32:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:32:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:32:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.272+0000 7fbbb32a5700 1 -- 172.21.15.2:0/1015116466 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fbb980a12c0 msgr2=0x7fbb980a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.272+0000 7fbbb32a5700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fbb980a12c0 0x7fbb980a3770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbba8005fd0 tx=0x7fbba8005ee0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.272+0000 7fbbb32a5700 1 -- 172.21.15.2:0/1015116466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac101500 msgr2=0x7fbbac0ff150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.272+0000 7fbbb32a5700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac101500 0x7fbbac0ff150 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c0098e0 tx=0x7fbb9c00f720 comp rx=0 tx=0).stop 2023-11-16T18:32:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.272+0000 7fbbb32a5700 1 -- 172.21.15.2:0/1015116466 shutdown_connections 2023-11-16T18:32:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.272+0000 7fbbb32a5700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fbb980a12c0 0x7fbb980a3770 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.272+0000 7fbbb32a5700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbac10a840 0x7fbbac1a4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.272+0000 7fbbb32a5700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac101500 0x7fbbac0ff150 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.272+0000 7fbbb32a5700 1 --2- 172.21.15.2:0/1015116466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbac101e10 0x7fbbac0ff690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.272+0000 7fbbb32a5700 1 -- 172.21.15.2:0/1015116466 >> 172.21.15.2:0/1015116466 conn(0x7fbbac075fd0 msgr2=0x7fbbac1022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:32:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.272+0000 7fbbb32a5700 1 -- 172.21.15.2:0/1015116466 shutdown_connections 2023-11-16T18:32:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:03.272+0000 7fbbb32a5700 1 -- 172.21.15.2:0/1015116466 wait complete. 2023-11-16T18:32:03.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:03 smithi040 ceph-mon[100251]: osdmap e94: 12 total, 12 up, 12 in 2023-11-16T18:32:03.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:03 smithi188 ceph-mon[100639]: osdmap e94: 12 total, 12 up, 12 in 2023-11-16T18:32:03.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:03 smithi002 ceph-mon[98807]: osdmap e94: 12 total, 12 up, 12 in 2023-11-16T18:32:04.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:04 smithi040 ceph-mon[100251]: pgmap v222: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 142 KiB/s wr, 38 op/s 2023-11-16T18:32:04.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:04 smithi188 ceph-mon[100639]: pgmap v222: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 142 KiB/s wr, 38 op/s 2023-11-16T18:32:04.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:04 smithi002 ceph-mon[98807]: pgmap v222: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 142 KiB/s wr, 38 op/s 2023-11-16T18:32:06.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:06 smithi040 ceph-mon[100251]: pgmap v223: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 142 KiB/s wr, 38 op/s 2023-11-16T18:32:06.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:06 smithi188 ceph-mon[100639]: pgmap v223: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 142 KiB/s wr, 38 op/s 2023-11-16T18:32:06.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:06 smithi002 ceph-mon[98807]: pgmap v223: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 142 KiB/s wr, 38 op/s 2023-11-16T18:32:07.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:07 smithi002 ceph-mon[98807]: pgmap v224: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 142 KiB/s wr, 38 op/s 2023-11-16T18:32:07.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:07 smithi040 ceph-mon[100251]: pgmap v224: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 142 KiB/s wr, 38 op/s 2023-11-16T18:32:07.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:07 smithi188 ceph-mon[100639]: pgmap v224: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 142 KiB/s wr, 38 op/s 2023-11-16T18:32:10.485 INFO:tasks.workunit.client.0.smithi002.stdout:ok 2023-11-16T18:32:10.486 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T18:32:10.486 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T18:32:10.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:10 smithi040 ceph-mon[100251]: pgmap v225: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 183 KiB/s wr, 138 op/s 2023-11-16T18:32:10.515 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:10 smithi002 ceph-mon[98807]: pgmap v225: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 183 KiB/s wr, 138 op/s 2023-11-16T18:32:10.545 INFO:tasks.workunit:Running workunit fs/misc/filelock_interrupt.py... 2023-11-16T18:32:10.612 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/filelock_interrupt.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_interrupt.py 2023-11-16T18:32:10.613 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:10 smithi188 ceph-mon[100639]: pgmap v225: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 183 KiB/s wr, 138 op/s 2023-11-16T18:32:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:12 smithi040 ceph-mon[100251]: pgmap v226: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 183 KiB/s wr, 138 op/s 2023-11-16T18:32:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:12 smithi040 ceph-mon[100251]: osdmap e95: 12 total, 12 up, 12 in 2023-11-16T18:32:12.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:12 smithi188 ceph-mon[100639]: pgmap v226: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 183 KiB/s wr, 138 op/s 2023-11-16T18:32:12.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:12 smithi188 ceph-mon[100639]: osdmap e95: 12 total, 12 up, 12 in 2023-11-16T18:32:12.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:12 smithi002 ceph-mon[98807]: pgmap v226: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 183 KiB/s wr, 138 op/s 2023-11-16T18:32:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:12 smithi002 ceph-mon[98807]: osdmap e95: 12 total, 12 up, 12 in 2023-11-16T18:32:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:14 smithi040 ceph-mon[100251]: pgmap v228: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 70 KiB/s wr, 114 op/s 2023-11-16T18:32:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:14 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:32:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:14 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:32:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:14 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:32:14.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:14 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:32:14.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:14 smithi188 ceph-mon[100639]: pgmap v228: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 70 KiB/s wr, 114 op/s 2023-11-16T18:32:14.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:14 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:32:14.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:14 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:32:14.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:14 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:32:14.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:14 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:32:14.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:14 smithi002 ceph-mon[98807]: pgmap v228: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 70 KiB/s wr, 114 op/s 2023-11-16T18:32:14.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:14 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:32:14.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:14 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:32:14.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:14 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:32:14.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:14 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:32:15.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:15 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:32:15.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:15 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:32:15.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:15 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:32:15.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:15 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:32:15.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:15 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:32:15.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:15 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:32:15.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:15 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:32:15.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:15 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:32:15.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:15 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:32:15.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:15 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:32:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:15 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:32:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:15 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:32:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.399+0000 7fa586655700 1 -- 172.21.15.2:0/625845718 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5801015f0 msgr2=0x7fa580104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.399+0000 7fa586655700 1 --2- 172.21.15.2:0/625845718 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5801015f0 0x7fa580104e80 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa5700099c0 tx=0x7fa570009cd0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.399+0000 7fa586655700 1 -- 172.21.15.2:0/625845718 shutdown_connections 2023-11-16T18:32:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.399+0000 7fa586655700 1 --2- 172.21.15.2:0/625845718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa580100c50 0x7fa580101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.399+0000 7fa586655700 1 --2- 172.21.15.2:0/625845718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa580105560 0x7fa58010d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.399+0000 7fa586655700 1 --2- 172.21.15.2:0/625845718 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5801015f0 0x7fa580104e80 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.399+0000 7fa586655700 1 -- 172.21.15.2:0/625845718 >> 172.21.15.2:0/625845718 conn(0x7fa5800774f0 msgr2=0x7fa5800778f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:32:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.399+0000 7fa586655700 1 -- 172.21.15.2:0/625845718 shutdown_connections 2023-11-16T18:32:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.399+0000 7fa586655700 1 -- 172.21.15.2:0/625845718 wait complete. 2023-11-16T18:32:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.400+0000 7fa586655700 1 Processor -- start 2023-11-16T18:32:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.400+0000 7fa586655700 1 -- start start 2023-11-16T18:32:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa586655700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa580100c50 0x7fa5801964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa586655700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5801015f0 0x7fa5801969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa586655700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa580105560 0x7fa58019ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa586655700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58019b480 con 0x7fa580105560 2023-11-16T18:32:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa586655700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58019b5f0 con 0x7fa5801015f0 2023-11-16T18:32:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa586655700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58019b7b0 con 0x7fa580100c50 2023-11-16T18:32:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa584bf2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa580105560 0x7fa58019ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa584bf2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa580105560 0x7fa58019ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53252/0 (socket says 172.21.15.2:53252) 2023-11-16T18:32:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa584bf2700 1 -- 172.21.15.2:0/1750837837 learned_addr learned my addr 172.21.15.2:0/1750837837 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:32:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa57ffff700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa580100c50 0x7fa5801964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa57f7fe700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5801015f0 0x7fa5801969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa584bf2700 1 -- 172.21.15.2:0/1750837837 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa580100c50 msgr2=0x7fa5801964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa584bf2700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa580100c50 0x7fa5801964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa584bf2700 1 -- 172.21.15.2:0/1750837837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5801015f0 msgr2=0x7fa5801969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa584bf2700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5801015f0 0x7fa5801969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa584bf2700 1 -- 172.21.15.2:0/1750837837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5700095f0 con 0x7fa580105560 2023-11-16T18:32:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa57f7fe700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5801015f0 0x7fa5801969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:32:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa584bf2700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa580105560 0x7fa58019ae40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa57800ada0 tx=0x7fa57800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:32:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa57d7fa700 1 -- 172.21.15.2:0/1750837837 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57800f070 con 0x7fa580105560 2023-11-16T18:32:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa57d7fa700 1 -- 172.21.15.2:0/1750837837 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa578007b30 con 0x7fa580105560 2023-11-16T18:32:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa586655700 1 -- 172.21.15.2:0/1750837837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa580110320 con 0x7fa580105560 2023-11-16T18:32:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa57d7fa700 1 -- 172.21.15.2:0/1750837837 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5780053e0 con 0x7fa580105560 2023-11-16T18:32:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.401+0000 7fa586655700 1 -- 172.21.15.2:0/1750837837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa580110190 con 0x7fa580105560 2023-11-16T18:32:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.404+0000 7fa57d7fa700 1 -- 172.21.15.2:0/1750837837 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fa57800c980 con 0x7fa580105560 2023-11-16T18:32:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.404+0000 7fa57d7fa700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa56c0a1340 0x7fa56c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.404+0000 7fa57d7fa700 1 -- 172.21.15.2:0/1750837837 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9482+0+0 (secure 0 0 0) 0x7fa5780cc610 con 0x7fa580105560 2023-11-16T18:32:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.404+0000 7fa57ffff700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa56c0a1340 0x7fa56c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.405+0000 7fa57ffff700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa56c0a1340 0x7fa56c0a37f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa568005950 tx=0x7fa56800b410 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:32:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.405+0000 7fa5927e1b80 1 Processor -- start 2023-11-16T18:32:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.405+0000 7fa5927e1b80 1 -- start start 2023-11-16T18:32:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.405+0000 7fa5927e1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 0x5610cbf1a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.405+0000 7fa5927e1b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf14560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.405+0000 7fa5927e1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1ab90 0x5610cbf14b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.405+0000 7fa5927e1b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610cbe1a000 con 0x5610cbf1a720 2023-11-16T18:32:16.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.405+0000 7fa5927e1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610cbe6c840 con 0x5610cbf1ab90 2023-11-16T18:32:16.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.405+0000 7fa5927e1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610cbe65cd0 con 0x5610cbd706b0 2023-11-16T18:32:16.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa57f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf14560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa57f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf14560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53258/0 (socket says 172.21.15.2:53258) 2023-11-16T18:32:16.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa57f7fe700 1 -- 172.21.15.2:0/3307625739 learned_addr learned my addr 172.21.15.2:0/3307625739 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:32:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa57ffff700 1 --2- 172.21.15.2:0/3307625739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 0x5610cbf1a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa584bf2700 1 --2- 172.21.15.2:0/3307625739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1ab90 0x5610cbf14b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa57f7fe700 1 -- 172.21.15.2:0/3307625739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 msgr2=0x5610cbf1a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa57f7fe700 1 --2- 172.21.15.2:0/3307625739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 0x5610cbf1a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa57f7fe700 1 -- 172.21.15.2:0/3307625739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1ab90 msgr2=0x5610cbf14b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa57f7fe700 1 --2- 172.21.15.2:0/3307625739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1ab90 0x5610cbf14b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa57f7fe700 1 -- 172.21.15.2:0/3307625739 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610cbce75d0 con 0x5610cbf1a720 2023-11-16T18:32:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa584bf2700 1 --2- 172.21.15.2:0/3307625739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1ab90 0x5610cbf14b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:32:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa57f7fe700 1 --2- 172.21.15.2:0/3307625739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf14560 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa570000c00 tx=0x7fa570005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db4c27e55c716a46 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:32:16.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa5767fc700 1 -- 172.21.15.2:0/3307625739 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570005230 con 0x5610cbf1a720 2023-11-16T18:32:16.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa5767fc700 1 -- 172.21.15.2:0/3307625739 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa57001e040 con 0x5610cbf1a720 2023-11-16T18:32:16.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.406+0000 7fa5767fc700 1 -- 172.21.15.2:0/3307625739 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5700047e0 con 0x5610cbf1a720 2023-11-16T18:32:16.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.407+0000 7fa5927e1b80 1 -- 172.21.15.2:0/3307625739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 msgr2=0x5610cbf14560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:16.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.407+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/3307625739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf14560 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa570000c00 tx=0x7fa570005c90 comp rx=0 tx=0).stop 2023-11-16T18:32:16.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.407+0000 7fa5927e1b80 1 -- 172.21.15.2:0/3307625739 shutdown_connections 2023-11-16T18:32:16.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.407+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/3307625739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 0x5610cbf1a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.407+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/3307625739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1ab90 0x5610cbf14b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.407+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/3307625739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf14560 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.407+0000 7fa5927e1b80 1 -- 172.21.15.2:0/3307625739 >> 172.21.15.2:0/3307625739 conn(0x5610cbf09f50 msgr2=0x5610cbf0f270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:32:16.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.407+0000 7fa5927e1b80 1 -- 172.21.15.2:0/3307625739 shutdown_connections 2023-11-16T18:32:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.407+0000 7fa5927e1b80 1 -- 172.21.15.2:0/3307625739 wait complete. 2023-11-16T18:32:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.408+0000 7fa5927e1b80 1 -- start start 2023-11-16T18:32:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.408+0000 7fa5927e1b80 1 Processor -- start 2023-11-16T18:32:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.408+0000 7fa5927e1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 0x5610cbf23a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.408+0000 7fa5927e1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1a720 0x5610cbf23f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.408+0000 7fa5927e1b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1ab90 0x5610cbf1dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.408+0000 7fa5927e1b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610cbe07880 con 0x5610cbf1ab90 2023-11-16T18:32:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.408+0000 7fa5927e1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610cbe07520 con 0x5610cbf1a720 2023-11-16T18:32:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.408+0000 7fa5927e1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610cbce98c0 con 0x5610cbd706b0 2023-11-16T18:32:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.408+0000 7fa5927e1b80 10 client.0 _ll_register_callbacks cb 0x5610cbe25720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:32:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.408+0000 7fa584bf2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1ab90 0x5610cbf1dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.408+0000 7fa584bf2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1ab90 0x5610cbf1dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53266/0 (socket says 172.21.15.2:53266) 2023-11-16T18:32:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.408+0000 7fa584bf2700 1 -- 172.21.15.2:0/419314638 learned_addr learned my addr 172.21.15.2:0/419314638 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:32:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa57f7fe700 1 --2- 172.21.15.2:0/419314638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1a720 0x5610cbf23f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa584bf2700 1 -- 172.21.15.2:0/419314638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 msgr2=0x5610cbf23a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa584bf2700 1 --2- 172.21.15.2:0/419314638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 0x5610cbf23a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa584bf2700 1 -- 172.21.15.2:0/419314638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1a720 msgr2=0x5610cbf23f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa584bf2700 1 --2- 172.21.15.2:0/419314638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1a720 0x5610cbf23f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa584bf2700 1 -- 172.21.15.2:0/419314638 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58010d9b0 con 0x5610cbf1ab90 2023-11-16T18:32:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa584bf2700 1 --2- 172.21.15.2:0/419314638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1ab90 0x5610cbf1dd10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fa578095e20 tx=0x7fa578004bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18285f87a48e8fbf server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:32:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa55f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:32:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa55f7fe700 1 -- 172.21.15.2:0/419314638 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5780993b0 con 0x5610cbf1ab90 2023-11-16T18:32:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa55f7fe700 1 -- 172.21.15.2:0/419314638 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9482+0+0 (secure 0 0 0) 0x7fa5780cc030 con 0x5610cbf1ab90 2023-11-16T18:32:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa5927e1b80 1 -- 172.21.15.2:0/419314638 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5610cbf07710 con 0x5610cbf1ab90 2023-11-16T18:32:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa55f7fe700 1 -- 172.21.15.2:0/419314638 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa57809d3e0 con 0x5610cbf1ab90 2023-11-16T18:32:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa5927e1b80 10 client.16905 fetch_fsmap learned FSMap version 29 2023-11-16T18:32:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa5927e1b80 1 -- 172.21.15.2:0/419314638 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5610cbe2c130 con 0x5610cbf1ab90 2023-11-16T18:32:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.409+0000 7fa55f7fe700 1 -- 172.21.15.2:0/419314638 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5610cbf07710 con 0x5610cbf1ab90 2023-11-16T18:32:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.410+0000 7fa55f7fe700 1 -- 172.21.15.2:0/419314638 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa5780cc1f0 con 0x5610cbf1ab90 2023-11-16T18:32:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.410+0000 7fa5927e1b80 10 client.16905 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:32:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.410+0000 7fa5927e1b80 10 client.16905 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:32:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.410+0000 7fa5927e1b80 20 client.16905 populate_metadata read hostname 'smithi002' 2023-11-16T18:32:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.410+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/419314638 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5610cbf2deb0 0x5610cbf302c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.410+0000 7fa5927e1b80 4 client.16905 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T18:32:16.526 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.410+0000 7fa5927e1b80 1 -- 172.21.15.2:0/419314638 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5610cbe28040 con 0x5610cbf2deb0 2023-11-16T18:32:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.410+0000 7fa57ffff700 1 --2- 172.21.15.2:0/419314638 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5610cbf2deb0 0x5610cbf302c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.410+0000 7fa55f7fe700 10 client.16905 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:32:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.410+0000 7fa57ffff700 1 --2- 172.21.15.2:0/419314638 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5610cbf2deb0 0x5610cbf302c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1b65f2cb16f126f server_cookie=72b5efbf424809c4 in_seq=0 out_seq=0 2023-11-16T18:32:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.411+0000 7fa55f7fe700 1 -- 172.21.15.2:0/419314638 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5610cbe28040 con 0x5610cbf2deb0 2023-11-16T18:32:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.411+0000 7fa55f7fe700 10 client.16905 handle_command_reply: tid=0 2023-11-16T18:32:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.411+0000 7fa5927e1b80 1 client.16905 shutdown 2023-11-16T18:32:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.411+0000 7fa5927e1b80 1 -- 172.21.15.2:0/419314638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1ab90 msgr2=0x5610cbf1dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.411+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/419314638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1ab90 0x5610cbf1dd10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fa578095e20 tx=0x7fa578004bc0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.411+0000 7fa5927e1b80 1 -- 172.21.15.2:0/419314638 shutdown_connections 2023-11-16T18:32:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.411+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/419314638 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5610cbf2deb0 0x5610cbf302c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.411+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/419314638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 0x5610cbf23a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.411+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/419314638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1ab90 0x5610cbf1dd10 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.411+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/419314638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1a720 0x5610cbf23f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.411+0000 7fa5927e1b80 1 -- 172.21.15.2:0/419314638 >> 172.21.15.2:0/419314638 conn(0x5610cbf09f50 msgr2=0x5610cbf0e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:32:16.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.412+0000 7fa55f7fe700 0 client.16905 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:32:16.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.412+0000 7fa5927e1b80 1 -- 172.21.15.2:0/419314638 shutdown_connections 2023-11-16T18:32:16.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.412+0000 7fa5927e1b80 1 -- 172.21.15.2:0/419314638 wait complete. 2023-11-16T18:32:16.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.412+0000 7fa5927e1b80 20 client.16905 trim_cache size 0 max 16384 2023-11-16T18:32:16.529 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:16 smithi040 ceph-mon[100251]: pgmap v229: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 70 KiB/s wr, 114 op/s 2023-11-16T18:32:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.426+0000 7fa5927e1b80 1 Processor -- start 2023-11-16T18:32:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.426+0000 7fa5927e1b80 1 -- start start 2023-11-16T18:32:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa5927e1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbd706b0 0x5610cbf0ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa5927e1b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf1d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa5927e1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbf1ab90 0x5610cbf1daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa5927e1b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610cbf09b70 con 0x5610cbf1a720 2023-11-16T18:32:16.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa5927e1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610cbf1c290 con 0x5610cbd706b0 2023-11-16T18:32:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa5927e1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610cbf189c0 con 0x5610cbf1ab90 2023-11-16T18:32:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa57f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf1d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa57f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf1d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53272/0 (socket says 172.21.15.2:53272) 2023-11-16T18:32:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa57f7fe700 1 -- 172.21.15.2:0/3464861337 learned_addr learned my addr 172.21.15.2:0/3464861337 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:32:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa57ffff700 1 --2- 172.21.15.2:0/3464861337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbd706b0 0x5610cbf0ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa584bf2700 1 --2- 172.21.15.2:0/3464861337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbf1ab90 0x5610cbf1daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa57f7fe700 1 -- 172.21.15.2:0/3464861337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbf1ab90 msgr2=0x5610cbf1daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa57f7fe700 1 --2- 172.21.15.2:0/3464861337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbf1ab90 0x5610cbf1daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa57f7fe700 1 -- 172.21.15.2:0/3464861337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbd706b0 msgr2=0x5610cbf0ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa57f7fe700 1 --2- 172.21.15.2:0/3464861337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbd706b0 0x5610cbf0ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa57f7fe700 1 -- 172.21.15.2:0/3464861337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610cbe2be80 con 0x5610cbf1a720 2023-11-16T18:32:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa584bf2700 1 --2- 172.21.15.2:0/3464861337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbf1ab90 0x5610cbf1daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:32:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa57f7fe700 1 --2- 172.21.15.2:0/3464861337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf1d5b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fa570009990 tx=0x7fa570005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd1ce3b8f2fba3b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:32:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.427+0000 7fa576ffd700 1 -- 172.21.15.2:0/3464861337 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57000b990 con 0x5610cbf1a720 2023-11-16T18:32:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.428+0000 7fa576ffd700 1 -- 172.21.15.2:0/3464861337 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa570005500 con 0x5610cbf1a720 2023-11-16T18:32:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.428+0000 7fa576ffd700 1 -- 172.21.15.2:0/3464861337 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570004a40 con 0x5610cbf1a720 2023-11-16T18:32:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.428+0000 7fa5927e1b80 1 -- 172.21.15.2:0/3464861337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 msgr2=0x5610cbf1d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.428+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/3464861337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf1d5b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fa570009990 tx=0x7fa570005fd0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.428+0000 7fa5927e1b80 1 -- 172.21.15.2:0/3464861337 shutdown_connections 2023-11-16T18:32:16.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.428+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/3464861337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbf1ab90 0x5610cbf1daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.428+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/3464861337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbd706b0 0x5610cbf0ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.428+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/3464861337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf1d5b0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.428+0000 7fa5927e1b80 1 -- 172.21.15.2:0/3464861337 >> 172.21.15.2:0/3464861337 conn(0x5610cbf09f50 msgr2=0x5610cbf17e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:32:16.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.428+0000 7fa5927e1b80 1 -- 172.21.15.2:0/3464861337 shutdown_connections 2023-11-16T18:32:16.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.428+0000 7fa5927e1b80 1 -- 172.21.15.2:0/3464861337 wait complete. 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.428+0000 7fa5927e1b80 1 -- start start 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa5927e1b80 1 Processor -- start 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa5927e1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 0x5610cbf23990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa5927e1b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf23ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa5927e1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1ab90 0x5610cbf217c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa5927e1b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610cbf21ed0 con 0x5610cbf1a720 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa5927e1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610cbf22040 con 0x5610cbf1ab90 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa5927e1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610cbf22200 con 0x5610cbd706b0 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa5927e1b80 10 client.0 _ll_register_callbacks cb 0x7fa578098fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa57f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf23ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa57f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf23ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53278/0 (socket says 172.21.15.2:53278) 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa57f7fe700 1 -- 172.21.15.2:0/1370486783 learned_addr learned my addr 172.21.15.2:0/1370486783 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa584bf2700 1 --2- 172.21.15.2:0/1370486783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1ab90 0x5610cbf217c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa57ffff700 1 --2- 172.21.15.2:0/1370486783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 0x5610cbf23990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa57f7fe700 1 -- 172.21.15.2:0/1370486783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 msgr2=0x5610cbf23990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa57f7fe700 1 --2- 172.21.15.2:0/1370486783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 0x5610cbf23990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa57f7fe700 1 -- 172.21.15.2:0/1370486783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1ab90 msgr2=0x5610cbf217c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T18:32:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.429+0000 7fa57f7fe700 1 --2- 172.21.15.2:0/1370486783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1ab90 0x5610cbf217c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa57f7fe700 1 -- 172.21.15.2:0/1370486783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5610cbce75d0 con 0x5610cbf1a720 2023-11-16T18:32:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa57f7fe700 1 --2- 172.21.15.2:0/1370486783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf23ed0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fa57000b600 tx=0x7fa57000bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f8f32912650190 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:32:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa55f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T18:32:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa55f7fe700 1 -- 172.21.15.2:0/1370486783 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570003d90 con 0x5610cbf1a720 2023-11-16T18:32:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa55f7fe700 1 -- 172.21.15.2:0/1370486783 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9482+0+0 (secure 0 0 0) 0x7fa570003ef0 con 0x5610cbf1a720 2023-11-16T18:32:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa5927e1b80 1 -- 172.21.15.2:0/1370486783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa57001e040 con 0x5610cbf1a720 2023-11-16T18:32:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa55f7fe700 1 -- 172.21.15.2:0/1370486783 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5700040b0 con 0x5610cbf1a720 2023-11-16T18:32:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa5927e1b80 10 client.16911 fetch_fsmap learned FSMap version 29 2023-11-16T18:32:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa5927e1b80 1 -- 172.21.15.2:0/1370486783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5610cbc7cc00 con 0x5610cbf1a720 2023-11-16T18:32:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa55f7fe700 1 -- 172.21.15.2:0/1370486783 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa57001e040 con 0x5610cbf1a720 2023-11-16T18:32:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa55f7fe700 1 -- 172.21.15.2:0/1370486783 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa57003d030 con 0x5610cbf1a720 2023-11-16T18:32:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa5927e1b80 10 client.16911 fetch_fsmap finished waiting for FSMap version 29 2023-11-16T18:32:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa5927e1b80 10 client.16911 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T18:32:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa5927e1b80 20 client.16911 populate_metadata read hostname 'smithi002' 2023-11-16T18:32:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/1370486783 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5610cbf81ac0 0x5610cbf83ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:32:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa5927e1b80 4 client.16911 mds_command: new command op to 24473 tid=0[{"prefix": "flush journal"}] 2023-11-16T18:32:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.430+0000 7fa5927e1b80 1 -- 172.21.15.2:0/1370486783 --> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5610cbe34600 con 0x5610cbf81ac0 2023-11-16T18:32:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.431+0000 7fa57ffff700 1 --2- 172.21.15.2:0/1370486783 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5610cbf81ac0 0x5610cbf83ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:32:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.431+0000 7fa55f7fe700 10 client.16911 ms_handle_connect on v2:172.21.15.40:6838/3162585010 2023-11-16T18:32:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.431+0000 7fa57ffff700 1 --2- 172.21.15.2:0/1370486783 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5610cbf81ac0 0x5610cbf83ed0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ba3c98fad7f0cc2 server_cookie=73a9a2443d13b430 in_seq=0 out_seq=0 2023-11-16T18:32:16.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:16 smithi188 ceph-mon[100639]: pgmap v229: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 70 KiB/s wr, 114 op/s 2023-11-16T18:32:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.570+0000 7fa55f7fe700 1 -- 172.21.15.2:0/1370486783 <== mds.0 v2:172.21.15.40:6838/3162585010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5610cbe34600 con 0x5610cbf81ac0 2023-11-16T18:32:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.570+0000 7fa55f7fe700 10 client.16911 handle_command_reply: tid=0 2023-11-16T18:32:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.570+0000 7fa5927e1b80 1 client.16911 shutdown 2023-11-16T18:32:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.571+0000 7fa5927e1b80 1 -- 172.21.15.2:0/1370486783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 msgr2=0x5610cbf23ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.571+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/1370486783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf23ed0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fa57000b600 tx=0x7fa57000bcd0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.571+0000 7fa5927e1b80 1 -- 172.21.15.2:0/1370486783 shutdown_connections 2023-11-16T18:32:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.571+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/1370486783 >> [v2:172.21.15.40:6838/3162585010,v1:172.21.15.40:6839/3162585010] conn(0x5610cbf81ac0 0x5610cbf83ed0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.571+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/1370486783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610cbd706b0 0x5610cbf23990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.571+0000 7fa55f7fe700 0 client.16911 ms_handle_reset on v2:172.21.15.40:6838/3162585010 2023-11-16T18:32:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.571+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/1370486783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610cbf1ab90 0x5610cbf217c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.571+0000 7fa5927e1b80 1 --2- 172.21.15.2:0/1370486783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5610cbf1a720 0x5610cbf23ed0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.571+0000 7fa5927e1b80 1 -- 172.21.15.2:0/1370486783 >> 172.21.15.2:0/1370486783 conn(0x5610cbf09f50 msgr2=0x5610cbf18dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:32:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.571+0000 7fa5927e1b80 1 -- 172.21.15.2:0/1370486783 shutdown_connections 2023-11-16T18:32:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.571+0000 7fa5927e1b80 1 -- 172.21.15.2:0/1370486783 wait complete. 2023-11-16T18:32:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.571+0000 7fa5927e1b80 20 client.16911 trim_cache size 0 max 16384 2023-11-16T18:32:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T18:32:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T18:32:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T18:32:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T18:32:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.572+0000 7fa586655700 1 -- 172.21.15.2:0/1750837837 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa56c0a1340 msgr2=0x7fa56c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.572+0000 7fa586655700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa56c0a1340 0x7fa56c0a37f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa568005950 tx=0x7fa56800b410 comp rx=0 tx=0).stop 2023-11-16T18:32:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.572+0000 7fa586655700 1 -- 172.21.15.2:0/1750837837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa580105560 msgr2=0x7fa58019ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:32:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.572+0000 7fa586655700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa580105560 0x7fa58019ae40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa57800ada0 tx=0x7fa57800c330 comp rx=0 tx=0).stop 2023-11-16T18:32:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.572+0000 7fa586655700 1 -- 172.21.15.2:0/1750837837 shutdown_connections 2023-11-16T18:32:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.572+0000 7fa586655700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa56c0a1340 0x7fa56c0a37f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.572+0000 7fa586655700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa580100c50 0x7fa5801964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.572+0000 7fa586655700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa580105560 0x7fa58019ae40 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.572+0000 7fa586655700 1 --2- 172.21.15.2:0/1750837837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5801015f0 0x7fa5801969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:32:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.572+0000 7fa586655700 1 -- 172.21.15.2:0/1750837837 >> 172.21.15.2:0/1750837837 conn(0x7fa5800774f0 msgr2=0x7fa5800feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:32:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.572+0000 7fa586655700 1 -- 172.21.15.2:0/1750837837 shutdown_connections 2023-11-16T18:32:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T18:32:16.572+0000 7fa586655700 1 -- 172.21.15.2:0/1750837837 wait complete. 2023-11-16T18:32:16.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:16 smithi002 ceph-mon[98807]: pgmap v229: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 70 KiB/s wr, 114 op/s 2023-11-16T18:32:17.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:17 smithi188 ceph-mon[100639]: pgmap v230: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 70 KiB/s wr, 114 op/s 2023-11-16T18:32:17.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:17 smithi188 ceph-mon[100639]: daemon mds.c restarted 2023-11-16T18:32:17.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:17 smithi188 ceph-mon[100639]: osdmap e96: 12 total, 12 up, 12 in 2023-11-16T18:32:17.550 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:32:17 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-c[126467]: ignoring --setuser ceph since I am not root 2023-11-16T18:32:17.550 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:32:17 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-c[126467]: ignoring --setgroup ceph since I am not root 2023-11-16T18:32:17.550 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:32:17 smithi188 ceph-mds[126491]: -- 172.21.15.188:0/982634770 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5580175c3520 con 0x558017599000 2023-11-16T18:32:17.550 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:32:17 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-c[126467]: starting mds.c at 2023-11-16T18:32:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:17 smithi002 ceph-mon[98807]: pgmap v230: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 70 KiB/s wr, 114 op/s 2023-11-16T18:32:17.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:17 smithi002 ceph-mon[98807]: daemon mds.c restarted 2023-11-16T18:32:17.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:17 smithi002 ceph-mon[98807]: osdmap e96: 12 total, 12 up, 12 in 2023-11-16T18:32:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:17 smithi040 ceph-mon[100251]: pgmap v230: 129 pgs: 129 active+clean; 722 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 70 KiB/s wr, 114 op/s 2023-11-16T18:32:17.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:17 smithi040 ceph-mon[100251]: daemon mds.c restarted 2023-11-16T18:32:17.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:17 smithi040 ceph-mon[100251]: osdmap e96: 12 total, 12 up, 12 in 2023-11-16T18:32:18.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:18 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.188:6836/3118705556,v1:172.21.15.188:6837/3118705556] up:boot 2023-11-16T18:32:18.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:18 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2023-11-16T18:32:18.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:18 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T18:32:18.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:18 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:32:18.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:18 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.188:6836/3118705556,v1:172.21.15.188:6837/3118705556] up:boot 2023-11-16T18:32:18.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:18 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2023-11-16T18:32:18.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:18 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T18:32:18.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:18 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:32:18.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:18 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.188:6836/3118705556,v1:172.21.15.188:6837/3118705556] up:boot 2023-11-16T18:32:18.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:18 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2023-11-16T18:32:18.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:18 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T18:32:18.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:18 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:32:19.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:19 smithi002 ceph-mon[98807]: pgmap v232: 129 pgs: 129 active+clean; 718 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 47 KiB/s wr, 141 op/s 2023-11-16T18:32:19.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:19 smithi040 ceph-mon[100251]: pgmap v232: 129 pgs: 129 active+clean; 718 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 47 KiB/s wr, 141 op/s 2023-11-16T18:32:19.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:19 smithi188 ceph-mon[100639]: pgmap v232: 129 pgs: 129 active+clean; 718 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 47 KiB/s wr, 141 op/s 2023-11-16T18:32:20.672 INFO:tasks.workunit.client.0.smithi002.stdout:ok 2023-11-16T18:32:20.692 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T18:32:20.692 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T18:32:20.789 INFO:tasks.workunit:Running workunit fs/misc/general_vxattrs.sh... 2023-11-16T18:32:20.789 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/general_vxattrs.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh 2023-11-16T18:32:20.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir -p dir 2023-11-16T18:32:20.823 INFO:tasks.workunit.client.0.smithi002.stderr:+ getfattr -n ceph.dir.pin dir 2023-11-16T18:32:20.823 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'ceph.dir.pin="-1"' 2023-11-16T18:32:20.824 INFO:tasks.workunit.client.0.smithi002.stdout:ceph.dir.pin="-1" 2023-11-16T18:32:20.824 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.pin dir 2023-11-16T18:32:20.825 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-16T18:32:20.825 INFO:tasks.workunit.client.0.smithi002.stdout:setfattr: dir: Invalid argument 2023-11-16T18:32:20.826 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.pin -v 1 dir 2023-11-16T18:32:20.828 INFO:tasks.workunit.client.0.smithi002.stderr:+ getfattr -n ceph.dir.pin dir 2023-11-16T18:32:20.828 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'ceph.dir.pin="1"' 2023-11-16T18:32:20.829 INFO:tasks.workunit.client.0.smithi002.stdout:ceph.dir.pin="1" 2023-11-16T18:32:20.829 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -x ceph.dir.pin dir 2023-11-16T18:32:20.831 INFO:tasks.workunit.client.0.smithi002.stderr:+ getfattr -n ceph.dir.pin dir 2023-11-16T18:32:20.831 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'ceph.dir.pin="-1"' 2023-11-16T18:32:20.832 INFO:tasks.workunit.client.0.smithi002.stdout:ceph.dir.pin="-1" 2023-11-16T18:32:20.833 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.pin.distributed dir 2023-11-16T18:32:20.833 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-16T18:32:20.834 INFO:tasks.workunit.client.0.smithi002.stdout:setfattr: dir: Invalid argument 2023-11-16T18:32:20.834 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.pin.distributed -v 1 dir 2023-11-16T18:32:20.835 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -x ceph.dir.pin.distributed dir 2023-11-16T18:32:20.837 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.pin.random dir 2023-11-16T18:32:20.838 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-16T18:32:20.841 INFO:tasks.workunit.client.0.smithi002.stdout:setfattr: dir: Invalid argument 2023-11-16T18:32:20.841 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.pin.random -v 0.01 dir 2023-11-16T18:32:20.841 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -x ceph.dir.pin.random dir 2023-11-16T18:32:20.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.subvolume dir 2023-11-16T18:32:20.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-16T18:32:20.844 INFO:tasks.workunit.client.0.smithi002.stdout:setfattr: dir: Invalid argument 2023-11-16T18:32:20.845 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.subvolume -v 1 dir 2023-11-16T18:32:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:32:20.855 INFO:tasks.workunit.client.0.smithi002.stderr:setfattr: dir: Invalid argument 2023-11-16T18:32:20.855 INFO:tasks.workunit:Stopping ['fs/misc'] on client.0... 2023-11-16T18:32:20.855 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-11-16T18:32:21.873 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2023-11-16T18:32:21.999 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=29017a6ed58943658eccf9f240370877 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2023-11-16T18:32:22.001 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2023-11-16T18:32:22.011 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T18:32:22.023 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2023-11-16T18:32:22.034 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2023-11-16T18:32:22.034 INFO:teuthology.task.exec:Executing custom commands... 2023-11-16T18:32:22.034 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T18:32:22.034 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2023-11-16T18:32:22.099 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2023-11-16T18:32:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:22 smithi040 ceph-mon[100251]: pgmap v233: 129 pgs: 129 active+clean; 718 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 38 KiB/s wr, 115 op/s 2023-11-16T18:32:22.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:22 smithi188 ceph-mon[100639]: pgmap v233: 129 pgs: 129 active+clean; 718 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 38 KiB/s wr, 115 op/s 2023-11-16T18:32:22.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:22 smithi002 ceph-mon[98807]: pgmap v233: 129 pgs: 129 active+clean; 718 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 38 KiB/s wr, 115 op/s 2023-11-16T18:32:24.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:24 smithi040 ceph-mon[100251]: pgmap v234: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 114 op/s 2023-11-16T18:32:24.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:24 smithi188 ceph-mon[100639]: pgmap v234: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 114 op/s 2023-11-16T18:32:24.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:24 smithi002 ceph-mon[98807]: pgmap v234: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 114 op/s 2023-11-16T18:32:26.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:26 smithi040 ceph-mon[100251]: pgmap v235: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 114 op/s 2023-11-16T18:32:26.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:26 smithi188 ceph-mon[100639]: pgmap v235: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 114 op/s 2023-11-16T18:32:26.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:26 smithi002 ceph-mon[98807]: pgmap v235: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 114 op/s 2023-11-16T18:32:27.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:27 smithi002 ceph-mon[98807]: pgmap v236: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 114 op/s 2023-11-16T18:32:27.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:27 smithi040 ceph-mon[100251]: pgmap v236: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 114 op/s 2023-11-16T18:32:27.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:27 smithi188 ceph-mon[100639]: pgmap v236: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 114 op/s 2023-11-16T18:32:30.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:30 smithi040 ceph-mon[100251]: pgmap v237: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 34 op/s 2023-11-16T18:32:30.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:30 smithi188 ceph-mon[100639]: pgmap v237: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 34 op/s 2023-11-16T18:32:30.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:30 smithi002 ceph-mon[98807]: pgmap v237: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 34 op/s 2023-11-16T18:32:32.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:32 smithi040 ceph-mon[100251]: pgmap v238: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2023-11-16T18:32:32.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:32 smithi188 ceph-mon[100639]: pgmap v238: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2023-11-16T18:32:32.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:32 smithi002 ceph-mon[98807]: pgmap v238: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2023-11-16T18:32:34.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:34 smithi040 ceph-mon[100251]: pgmap v239: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.9 KiB/s wr, 12 op/s 2023-11-16T18:32:34.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:34 smithi188 ceph-mon[100639]: pgmap v239: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.9 KiB/s wr, 12 op/s 2023-11-16T18:32:34.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:34 smithi002 ceph-mon[98807]: pgmap v239: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.9 KiB/s wr, 12 op/s 2023-11-16T18:32:36.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:36 smithi040 ceph-mon[100251]: pgmap v240: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2023-11-16T18:32:36.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:36 smithi188 ceph-mon[100639]: pgmap v240: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2023-11-16T18:32:36.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:36 smithi002 ceph-mon[98807]: pgmap v240: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2023-11-16T18:32:37.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:37 smithi002 ceph-mon[98807]: pgmap v241: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2023-11-16T18:32:37.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:37 smithi040 ceph-mon[100251]: pgmap v241: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2023-11-16T18:32:37.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:37 smithi188 ceph-mon[100639]: pgmap v241: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2023-11-16T18:32:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:40 smithi040 ceph-mon[100251]: pgmap v242: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.5 KiB/s wr, 10 op/s 2023-11-16T18:32:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:40 smithi188 ceph-mon[100639]: pgmap v242: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.5 KiB/s wr, 10 op/s 2023-11-16T18:32:40.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:40 smithi002 ceph-mon[98807]: pgmap v242: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.5 KiB/s wr, 10 op/s 2023-11-16T18:32:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:42 smithi040 ceph-mon[100251]: pgmap v243: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2023-11-16T18:32:42.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:42 smithi188 ceph-mon[100639]: pgmap v243: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2023-11-16T18:32:42.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:42 smithi002 ceph-mon[98807]: pgmap v243: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2023-11-16T18:32:44.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:44 smithi040 ceph-mon[100251]: pgmap v244: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2023-11-16T18:32:44.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:44 smithi188 ceph-mon[100639]: pgmap v244: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2023-11-16T18:32:44.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:44 smithi002 ceph-mon[98807]: pgmap v244: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2023-11-16T18:32:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:46 smithi040 ceph-mon[100251]: pgmap v245: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2023-11-16T18:32:46.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:46 smithi188 ceph-mon[100639]: pgmap v245: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2023-11-16T18:32:46.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:46 smithi002 ceph-mon[98807]: pgmap v245: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2023-11-16T18:32:47.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:47 smithi002 ceph-mon[98807]: pgmap v246: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2023-11-16T18:32:47.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:47 smithi040 ceph-mon[100251]: pgmap v246: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2023-11-16T18:32:47.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:47 smithi188 ceph-mon[100639]: pgmap v246: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2023-11-16T18:32:50.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:50 smithi040 ceph-mon[100251]: pgmap v247: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.4 KiB/s wr, 9 op/s 2023-11-16T18:32:50.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:50 smithi188 ceph-mon[100639]: pgmap v247: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.4 KiB/s wr, 9 op/s 2023-11-16T18:32:50.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:50 smithi002 ceph-mon[98807]: pgmap v247: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.4 KiB/s wr, 9 op/s 2023-11-16T18:32:52.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:52 smithi040 ceph-mon[100251]: pgmap v248: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T18:32:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:52 smithi188 ceph-mon[100639]: pgmap v248: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T18:32:52.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:52 smithi002 ceph-mon[98807]: pgmap v248: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T18:32:54.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:54 smithi040 ceph-mon[100251]: pgmap v249: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 8 op/s 2023-11-16T18:32:54.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:54 smithi188 ceph-mon[100639]: pgmap v249: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 8 op/s 2023-11-16T18:32:54.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:54 smithi002 ceph-mon[98807]: pgmap v249: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 8 op/s 2023-11-16T18:32:56.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:56 smithi040 ceph-mon[100251]: pgmap v250: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T18:33:08.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:32:57 smithi040 ceph-mon[100251]: pgmap v251: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T18:33:08.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:00 smithi040 ceph-mon[100251]: pgmap v252: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 8 op/s 2023-11-16T18:33:08.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:02 smithi040 ceph-mon[100251]: pgmap v253: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:08.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:04 smithi040 ceph-mon[100251]: pgmap v254: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-11-16T18:33:08.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:06 smithi040 ceph-mon[100251]: pgmap v255: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-11-16T18:33:08.237 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:56 smithi002 ceph-mon[98807]: pgmap v250: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T18:33:08.237 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:32:57 smithi002 ceph-mon[98807]: pgmap v251: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T18:33:08.279 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:00 smithi002 ceph-mon[98807]: pgmap v252: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 8 op/s 2023-11-16T18:33:08.310 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:02 smithi002 ceph-mon[98807]: pgmap v253: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:08.310 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:04 smithi002 ceph-mon[98807]: pgmap v254: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-11-16T18:33:08.310 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:06 smithi002 ceph-mon[98807]: pgmap v255: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-11-16T18:33:08.310 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:56 smithi188 ceph-mon[100639]: pgmap v250: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T18:33:08.310 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:32:57 smithi188 ceph-mon[100639]: pgmap v251: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T18:33:08.311 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:00 smithi188 ceph-mon[100639]: pgmap v252: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 8 op/s 2023-11-16T18:33:08.311 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:02 smithi188 ceph-mon[100639]: pgmap v253: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:08.311 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:04 smithi188 ceph-mon[100639]: pgmap v254: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-11-16T18:33:08.311 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:06 smithi188 ceph-mon[100639]: pgmap v255: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-11-16T18:33:08.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:08 smithi040 ceph-mon[100251]: pgmap v256: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-11-16T18:33:08.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:08 smithi040 ceph-mon[100251]: osdmap e97: 12 total, 12 up, 12 in 2023-11-16T18:33:08.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:08 smithi188 ceph-mon[100639]: pgmap v256: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-11-16T18:33:08.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:08 smithi188 ceph-mon[100639]: osdmap e97: 12 total, 12 up, 12 in 2023-11-16T18:33:08.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:08 smithi002 ceph-mon[98807]: pgmap v256: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-11-16T18:33:08.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:08 smithi002 ceph-mon[98807]: osdmap e97: 12 total, 12 up, 12 in 2023-11-16T18:33:10.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:10 smithi040 ceph-mon[100251]: pgmap v258: 129 pgs: 37 active+clean+snaptrim_wait, 80 active+clean, 12 active+clean+snaptrim; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:33:10.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:10 smithi188 ceph-mon[100639]: pgmap v258: 129 pgs: 37 active+clean+snaptrim_wait, 80 active+clean, 12 active+clean+snaptrim; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:33:10.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:10 smithi002 ceph-mon[98807]: pgmap v258: 129 pgs: 37 active+clean+snaptrim_wait, 80 active+clean, 12 active+clean+snaptrim; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:33:12.489 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:12 smithi040 ceph-mon[100251]: pgmap v259: 129 pgs: 37 active+clean+snaptrim_wait, 80 active+clean, 12 active+clean+snaptrim; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:33:12.551 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:12 smithi188 ceph-mon[100639]: pgmap v259: 129 pgs: 37 active+clean+snaptrim_wait, 80 active+clean, 12 active+clean+snaptrim; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:33:12.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:12 smithi002 ceph-mon[98807]: pgmap v259: 129 pgs: 37 active+clean+snaptrim_wait, 80 active+clean, 12 active+clean+snaptrim; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:33:14.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:14 smithi040 ceph-mon[100251]: pgmap v260: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:33:14.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:14 smithi188 ceph-mon[100639]: pgmap v260: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:33:14.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:14 smithi002 ceph-mon[98807]: pgmap v260: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:33:15.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:15 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:33:15.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:15 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:33:15.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:15 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:33:15.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:15 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:33:15.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:15 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:33:15.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:15 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:33:15.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:15 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:33:15.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:15 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:33:15.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:15 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:33:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:15 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:33:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:15 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:33:15.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:15 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:33:16.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:16 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:33:16.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:16 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:33:16.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:16 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:33:16.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:16 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:33:16.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:16 smithi040 ceph-mon[100251]: pgmap v261: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:33:16.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:16 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:33:16.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:16 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:33:16.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:16 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:33:16.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:16 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:33:16.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:16 smithi188 ceph-mon[100639]: pgmap v261: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:33:16.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:16 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:33:16.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:16 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:33:16.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:16 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:33:16.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:16 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:33:16.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:16 smithi002 ceph-mon[98807]: pgmap v261: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:33:17.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:17 smithi002 ceph-mon[98807]: osdmap e98: 12 total, 12 up, 12 in 2023-11-16T18:33:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:17 smithi002 ceph-mon[98807]: pgmap v263: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 206 B/s wr, 6 op/s 2023-11-16T18:33:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:17 smithi040 ceph-mon[100251]: osdmap e98: 12 total, 12 up, 12 in 2023-11-16T18:33:17.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:17 smithi040 ceph-mon[100251]: pgmap v263: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 206 B/s wr, 6 op/s 2023-11-16T18:33:17.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:17 smithi188 ceph-mon[100639]: osdmap e98: 12 total, 12 up, 12 in 2023-11-16T18:33:17.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:17 smithi188 ceph-mon[100639]: pgmap v263: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 206 B/s wr, 6 op/s 2023-11-16T18:33:20.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:20 smithi040 ceph-mon[100251]: pgmap v264: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-11-16T18:33:20.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:20 smithi188 ceph-mon[100639]: pgmap v264: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-11-16T18:33:20.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:20 smithi002 ceph-mon[98807]: pgmap v264: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-11-16T18:33:22.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:22 smithi040 ceph-mon[100251]: pgmap v265: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-11-16T18:33:22.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:22 smithi188 ceph-mon[100639]: pgmap v265: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-11-16T18:33:22.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:22 smithi002 ceph-mon[98807]: pgmap v265: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-11-16T18:33:24.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:24 smithi040 ceph-mon[100251]: pgmap v266: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:33:24.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:24 smithi188 ceph-mon[100639]: pgmap v266: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:33:24.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:24 smithi002 ceph-mon[98807]: pgmap v266: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:33:26.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:26 smithi040 ceph-mon[100251]: pgmap v267: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:33:26.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:26 smithi188 ceph-mon[100639]: pgmap v267: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:33:26.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:26 smithi002 ceph-mon[98807]: pgmap v267: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:33:27.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:27 smithi040 ceph-mon[100251]: pgmap v268: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-11-16T18:33:27.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:27 smithi188 ceph-mon[100639]: pgmap v268: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-11-16T18:33:27.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:27 smithi002 ceph-mon[98807]: pgmap v268: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-11-16T18:33:30.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:30 smithi040 ceph-mon[100251]: pgmap v269: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:33:30.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:30 smithi188 ceph-mon[100639]: pgmap v269: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:33:30.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:30 smithi002 ceph-mon[98807]: pgmap v269: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:33:32.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:32 smithi040 ceph-mon[100251]: pgmap v270: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:33:32.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:32 smithi188 ceph-mon[100639]: pgmap v270: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:33:32.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:32 smithi002 ceph-mon[98807]: pgmap v270: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:33:34.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:34 smithi040 ceph-mon[100251]: pgmap v271: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:33:34.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:34 smithi188 ceph-mon[100639]: pgmap v271: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:33:34.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:34 smithi002 ceph-mon[98807]: pgmap v271: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:33:36.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:36 smithi040 ceph-mon[100251]: pgmap v272: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:33:36.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:36 smithi188 ceph-mon[100639]: pgmap v272: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:33:36.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:36 smithi002 ceph-mon[98807]: pgmap v272: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:33:37.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:37 smithi040 ceph-mon[100251]: pgmap v273: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:33:37.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:37 smithi188 ceph-mon[100639]: pgmap v273: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:33:37.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:37 smithi002 ceph-mon[98807]: pgmap v273: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:33:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:40 smithi040 ceph-mon[100251]: pgmap v274: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:33:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:40 smithi188 ceph-mon[100639]: pgmap v274: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:33:40.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:40 smithi002 ceph-mon[98807]: pgmap v274: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:33:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:42 smithi040 ceph-mon[100251]: pgmap v275: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:42.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:42 smithi188 ceph-mon[100639]: pgmap v275: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:42.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:42 smithi002 ceph-mon[98807]: pgmap v275: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:44.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:44 smithi040 ceph-mon[100251]: pgmap v276: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:33:44.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:44 smithi188 ceph-mon[100639]: pgmap v276: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:33:44.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:44 smithi002 ceph-mon[98807]: pgmap v276: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:33:46.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:46 smithi040 ceph-mon[100251]: pgmap v277: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:46.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:46 smithi188 ceph-mon[100639]: pgmap v277: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:46.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:46 smithi002 ceph-mon[98807]: pgmap v277: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:47.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:47 smithi040 ceph-mon[100251]: pgmap v278: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:47.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:47 smithi188 ceph-mon[100639]: pgmap v278: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:47.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:47 smithi002 ceph-mon[98807]: pgmap v278: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:50.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:50 smithi040 ceph-mon[100251]: pgmap v279: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:33:50.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:50 smithi188 ceph-mon[100639]: pgmap v279: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:33:50.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:50 smithi002 ceph-mon[98807]: pgmap v279: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:33:52.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:52 smithi040 ceph-mon[100251]: pgmap v280: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:52 smithi188 ceph-mon[100639]: pgmap v280: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:52.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:52 smithi002 ceph-mon[98807]: pgmap v280: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:54.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:54 smithi040 ceph-mon[100251]: pgmap v281: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:33:54.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:54 smithi188 ceph-mon[100639]: pgmap v281: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:33:54.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:54 smithi002 ceph-mon[98807]: pgmap v281: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:33:56.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:56 smithi040 ceph-mon[100251]: pgmap v282: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:56.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:56 smithi188 ceph-mon[100639]: pgmap v282: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:56.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:56 smithi002 ceph-mon[98807]: pgmap v282: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:57.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:33:57 smithi040 ceph-mon[100251]: pgmap v283: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:57.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:33:57 smithi188 ceph-mon[100639]: pgmap v283: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:33:57.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:33:57 smithi002 ceph-mon[98807]: pgmap v283: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:00.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:00 smithi040 ceph-mon[100251]: pgmap v284: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:00.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:00 smithi188 ceph-mon[100639]: pgmap v284: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:00.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:00 smithi002 ceph-mon[98807]: pgmap v284: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:02.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:02 smithi040 ceph-mon[100251]: pgmap v285: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:02.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:02 smithi188 ceph-mon[100639]: pgmap v285: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:02.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:02 smithi002 ceph-mon[98807]: pgmap v285: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:04.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:04 smithi040 ceph-mon[100251]: pgmap v286: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-11-16T18:34:04.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:04 smithi188 ceph-mon[100639]: pgmap v286: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-11-16T18:34:04.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:04 smithi002 ceph-mon[98807]: pgmap v286: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-11-16T18:34:06.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:06 smithi040 ceph-mon[100251]: pgmap v287: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-11-16T18:34:06.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:06 smithi188 ceph-mon[100639]: pgmap v287: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-11-16T18:34:06.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:06 smithi002 ceph-mon[98807]: pgmap v287: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-11-16T18:34:08.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:08 smithi040 ceph-mon[100251]: pgmap v288: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-11-16T18:34:08.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:08 smithi040 ceph-mon[100251]: osdmap e99: 12 total, 12 up, 12 in 2023-11-16T18:34:08.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:08 smithi188 ceph-mon[100639]: pgmap v288: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-11-16T18:34:08.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:08 smithi188 ceph-mon[100639]: osdmap e99: 12 total, 12 up, 12 in 2023-11-16T18:34:08.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:08 smithi002 ceph-mon[98807]: pgmap v288: 129 pgs: 129 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-11-16T18:34:08.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:08 smithi002 ceph-mon[98807]: osdmap e99: 12 total, 12 up, 12 in 2023-11-16T18:34:10.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:10 smithi040 ceph-mon[100251]: pgmap v290: 129 pgs: 10 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 85 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:34:10.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:10 smithi188 ceph-mon[100639]: pgmap v290: 129 pgs: 10 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 85 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:34:10.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:10 smithi002 ceph-mon[98807]: pgmap v290: 129 pgs: 10 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 85 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:34:12.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:12 smithi040 ceph-mon[100251]: pgmap v291: 129 pgs: 10 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 85 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:34:12.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:12 smithi188 ceph-mon[100639]: pgmap v291: 129 pgs: 10 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 85 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:34:12.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:12 smithi002 ceph-mon[98807]: pgmap v291: 129 pgs: 10 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 85 active+clean; 719 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:34:14.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:14 smithi188 ceph-mon[100639]: pgmap v292: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 719 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:34:14.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:14 smithi002 ceph-mon[98807]: pgmap v292: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 719 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:34:14.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:14 smithi040 ceph-mon[100251]: pgmap v292: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 719 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:34:15.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:15 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:34:15.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:15 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:34:15.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:15 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:34:16.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:16 smithi188 ceph-mon[100639]: pgmap v293: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 719 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:34:16.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:16 smithi002 ceph-mon[98807]: pgmap v293: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 719 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:34:16.737 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:16 smithi040 ceph-mon[100251]: pgmap v293: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 719 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:34:17.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:17 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:34:17.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:17 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:34:17.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:17 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:34:17.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:17 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:34:17.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:17 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:34:17.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:17 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:34:17.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:17 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:34:17.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:17 smithi188 ceph-mon[100639]: pgmap v294: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 719 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:34:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:17 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:34:17.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:17 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:34:17.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:17 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:34:17.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:17 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:34:17.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:17 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:34:17.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:17 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:34:17.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:17 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:34:17.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:17 smithi002 ceph-mon[98807]: pgmap v294: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 719 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:34:17.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:17 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:34:17.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:17 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:34:17.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:17 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:34:17.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:17 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:34:17.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:17 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:34:17.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:17 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:34:17.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:17 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:34:17.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:17 smithi040 ceph-mon[100251]: pgmap v294: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 719 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:34:20.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:20 smithi040 ceph-mon[100251]: pgmap v295: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 719 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 171 B/s wr, 8 op/s 2023-11-16T18:34:20.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:20 smithi188 ceph-mon[100639]: pgmap v295: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 719 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 171 B/s wr, 8 op/s 2023-11-16T18:34:20.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:20 smithi002 ceph-mon[98807]: pgmap v295: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 719 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 171 B/s wr, 8 op/s 2023-11-16T18:34:22.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:22 smithi040 ceph-mon[100251]: pgmap v296: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 719 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:34:22.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:22 smithi188 ceph-mon[100639]: pgmap v296: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 719 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:34:22.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:22 smithi002 ceph-mon[98807]: pgmap v296: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 719 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:34:24.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:24 smithi040 ceph-mon[100251]: pgmap v297: 129 pgs: 21 active+clean+snaptrim, 66 active+clean+snaptrim_wait, 42 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:34:24.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:24 smithi188 ceph-mon[100639]: pgmap v297: 129 pgs: 21 active+clean+snaptrim, 66 active+clean+snaptrim_wait, 42 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:34:24.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:24 smithi002 ceph-mon[98807]: pgmap v297: 129 pgs: 21 active+clean+snaptrim, 66 active+clean+snaptrim_wait, 42 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:34:26.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:26 smithi188 ceph-mon[100639]: pgmap v298: 129 pgs: 21 active+clean+snaptrim, 66 active+clean+snaptrim_wait, 42 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:26.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:26 smithi002 ceph-mon[98807]: pgmap v298: 129 pgs: 21 active+clean+snaptrim, 66 active+clean+snaptrim_wait, 42 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:26.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:26 smithi040 ceph-mon[100251]: pgmap v298: 129 pgs: 21 active+clean+snaptrim, 66 active+clean+snaptrim_wait, 42 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:27.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:27 smithi040 ceph-mon[100251]: pgmap v299: 129 pgs: 19 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 49 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:27.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:27 smithi188 ceph-mon[100639]: pgmap v299: 129 pgs: 19 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 49 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:27.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:27 smithi002 ceph-mon[98807]: pgmap v299: 129 pgs: 19 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 49 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:30.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:30 smithi040 ceph-mon[100251]: pgmap v300: 129 pgs: 13 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 74 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:30.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:30 smithi188 ceph-mon[100639]: pgmap v300: 129 pgs: 13 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 74 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:30.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:30 smithi002 ceph-mon[98807]: pgmap v300: 129 pgs: 13 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 74 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:31.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:31 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:31.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:31 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:31.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:31 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:32.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:32 smithi040 ceph-mon[100251]: pgmap v301: 129 pgs: 13 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 74 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:32.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:32 smithi188 ceph-mon[100639]: pgmap v301: 129 pgs: 13 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 74 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:32.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:32 smithi002 ceph-mon[98807]: pgmap v301: 129 pgs: 13 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 74 active+clean; 719 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:34.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:34 smithi040 ceph-mon[100251]: pgmap v302: 129 pgs: 129 active+clean; 719 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:34.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:34 smithi188 ceph-mon[100639]: pgmap v302: 129 pgs: 129 active+clean; 719 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:34.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:34 smithi002 ceph-mon[98807]: pgmap v302: 129 pgs: 129 active+clean; 719 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:36.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:36 smithi188 ceph-mon[100639]: pgmap v303: 129 pgs: 129 active+clean; 719 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:36.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:36 smithi002 ceph-mon[98807]: pgmap v303: 129 pgs: 129 active+clean; 719 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:36.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:36 smithi040 ceph-mon[100251]: pgmap v303: 129 pgs: 129 active+clean; 719 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:37.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:37 smithi040 ceph-mon[100251]: osdmap e100: 12 total, 12 up, 12 in 2023-11-16T18:34:37.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:37 smithi040 ceph-mon[100251]: pgmap v305: 129 pgs: 129 active+clean; 719 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:37.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:37 smithi188 ceph-mon[100639]: osdmap e100: 12 total, 12 up, 12 in 2023-11-16T18:34:37.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:37 smithi188 ceph-mon[100639]: pgmap v305: 129 pgs: 129 active+clean; 719 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:37.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:37 smithi002 ceph-mon[98807]: osdmap e100: 12 total, 12 up, 12 in 2023-11-16T18:34:37.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:37 smithi002 ceph-mon[98807]: pgmap v305: 129 pgs: 129 active+clean; 719 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:40 smithi040 ceph-mon[100251]: pgmap v306: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:40 smithi188 ceph-mon[100639]: pgmap v306: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:40.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:40 smithi002 ceph-mon[98807]: pgmap v306: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:41.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:41 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:41.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:41 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:41.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:41 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:42 smithi040 ceph-mon[100251]: pgmap v307: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:42.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:42 smithi188 ceph-mon[100639]: pgmap v307: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:42 smithi002 ceph-mon[98807]: pgmap v307: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:43.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:43 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:43.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:43 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:43.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:43 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:44.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:44 smithi188 ceph-mon[100639]: pgmap v308: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:44.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:44 smithi002 ceph-mon[98807]: pgmap v308: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:44.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:44 smithi040 ceph-mon[100251]: pgmap v308: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:46.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:46 smithi188 ceph-mon[100639]: pgmap v309: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:46.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:46 smithi002 ceph-mon[98807]: pgmap v309: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:46.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:46 smithi040 ceph-mon[100251]: pgmap v309: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:34:48.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:48 smithi188 ceph-mon[100639]: pgmap v310: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-11-16T18:34:48.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:48 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:48 smithi002 ceph-mon[98807]: pgmap v310: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-11-16T18:34:48.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:48 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:48.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:48 smithi040 ceph-mon[100251]: pgmap v310: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-11-16T18:34:48.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:48 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:50.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:50 smithi188 ceph-mon[100639]: pgmap v311: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:50.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:50 smithi002 ceph-mon[98807]: pgmap v311: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:50.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:50 smithi040 ceph-mon[100251]: pgmap v311: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:52 smithi188 ceph-mon[100639]: pgmap v312: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:52.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:52 smithi002 ceph-mon[98807]: pgmap v312: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:52.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:52 smithi040 ceph-mon[100251]: pgmap v312: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:34:53.179 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-11-16T18:34:53.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.318+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1136277685 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa00ffdd0 msgr2=0x7fcaa0103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:53.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.318+0000 7fcaa51d9700 1 --2- 172.21.15.2:0/1136277685 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa00ffdd0 0x7fcaa0103d90 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fca940099c0 tx=0x7fca94009cd0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.318+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1136277685 shutdown_connections 2023-11-16T18:34:53.322 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.318+0000 7fcaa51d9700 1 --2- 172.21.15.2:0/1136277685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcaa0104470 0x7fcaa010c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.322 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.318+0000 7fcaa51d9700 1 --2- 172.21.15.2:0/1136277685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa00ff430 0x7fcaa00ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.322 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.318+0000 7fcaa51d9700 1 --2- 172.21.15.2:0/1136277685 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa00ffdd0 0x7fcaa0103d90 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.322 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.318+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1136277685 >> 172.21.15.2:0/1136277685 conn(0x7fcaa0076370 msgr2=0x7fcaa0076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:53.322 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.318+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1136277685 shutdown_connections 2023-11-16T18:34:53.322 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.318+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1136277685 wait complete. 2023-11-16T18:34:53.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.319+0000 7fcaa51d9700 1 Processor -- start 2023-11-16T18:34:53.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.319+0000 7fcaa51d9700 1 -- start start 2023-11-16T18:34:53.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fcaa51d9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcaa00ff430 0x7fcaa0197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:53.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fcaa51d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa00ffdd0 0x7fcaa0197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:53.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fcaa51d9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa0104470 0x7fcaa01a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:53.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fcaa51d9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa0198090 con 0x7fcaa00ffdd0 2023-11-16T18:34:53.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fcaa51d9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa01a2800 con 0x7fcaa0104470 2023-11-16T18:34:53.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fcaa51d9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa01a2970 con 0x7fcaa00ff430 2023-11-16T18:34:53.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fca9e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa00ffdd0 0x7fcaa0197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:53.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fca9e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa00ffdd0 0x7fcaa0197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53292/0 (socket says 172.21.15.2:53292) 2023-11-16T18:34:53.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fca9e59c700 1 -- 172.21.15.2:0/1597340306 learned_addr learned my addr 172.21.15.2:0/1597340306 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:34:53.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fca9ed9d700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcaa00ff430 0x7fcaa0197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:53.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fca9f59e700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa0104470 0x7fcaa01a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:53.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fca9e59c700 1 -- 172.21.15.2:0/1597340306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcaa00ff430 msgr2=0x7fcaa0197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:53.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fca9e59c700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcaa00ff430 0x7fcaa0197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fca9e59c700 1 -- 172.21.15.2:0/1597340306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa0104470 msgr2=0x7fcaa01a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:53.326 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fca9e59c700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa0104470 0x7fcaa01a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.327 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fca9e59c700 1 -- 172.21.15.2:0/1597340306 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca940095f0 con 0x7fcaa00ffdd0 2023-11-16T18:34:53.327 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fca9ed9d700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcaa00ff430 0x7fcaa0197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:34:53.327 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fca9e59c700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa00ffdd0 0x7fcaa0197780 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fca94009990 tx=0x7fca94005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:53.327 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.320+0000 7fca87fff700 1 -- 172.21.15.2:0/1597340306 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9401d070 con 0x7fcaa00ffdd0 2023-11-16T18:34:53.327 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.321+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1597340306 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa010f290 con 0x7fcaa00ffdd0 2023-11-16T18:34:53.327 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.321+0000 7fca87fff700 1 -- 172.21.15.2:0/1597340306 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca94004090 con 0x7fcaa00ffdd0 2023-11-16T18:34:53.327 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.321+0000 7fca87fff700 1 -- 172.21.15.2:0/1597340306 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca94022d00 con 0x7fcaa00ffdd0 2023-11-16T18:34:53.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.321+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1597340306 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa010f100 con 0x7fcaa00ffdd0 2023-11-16T18:34:53.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.321+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1597340306 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa010f420 con 0x7fcaa00ffdd0 2023-11-16T18:34:53.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.323+0000 7fca87fff700 1 -- 172.21.15.2:0/1597340306 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fca94022470 con 0x7fcaa00ffdd0 2023-11-16T18:34:53.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.323+0000 7fca87fff700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fca880a12c0 0x7fca880a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:53.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.323+0000 7fca9ed9d700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fca880a12c0 0x7fca880a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:53.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.323+0000 7fca87fff700 1 -- 172.21.15.2:0/1597340306 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9482+0+0 (secure 0 0 0) 0x7fca940ce630 con 0x7fcaa00ffdd0 2023-11-16T18:34:53.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.324+0000 7fca9ed9d700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fca880a12c0 0x7fca880a3770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fca90005950 tx=0x7fca900058e0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:53.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.327+0000 7fca87fff700 1 -- 172.21.15.2:0/1597340306 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fca94097710 con 0x7fcaa00ffdd0 2023-11-16T18:34:53.555 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.553+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1597340306 --> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7fcaa00fd140 con 0x7fca880a12c0 2023-11-16T18:34:53.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.564+0000 7fca87fff700 1 -- 172.21.15.2:0/1597340306 <== mgr.15594 v2:172.21.15.2:6800/2202170290 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+335 (secure 0 0 0) 0x7fcaa00fd140 con 0x7fca880a12c0 2023-11-16T18:34:53.566 INFO:teuthology.orchestra.run.smithi002.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-11-16T00:00:00", "created": "2023-11-16T18:24:51", "first": "2023-11-16T18:25:00", "last": "2023-11-16T18:34:00", "last_pruned": "2023-11-16T18:34:00", "created_count": 10, "pruned_count": 4, "active": true} 2023-11-16T18:34:53.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.566+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1597340306 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fca880a12c0 msgr2=0x7fca880a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.566+0000 7fcaa51d9700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fca880a12c0 0x7fca880a3770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fca90005950 tx=0x7fca900058e0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.566+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1597340306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa00ffdd0 msgr2=0x7fcaa0197780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.566+0000 7fcaa51d9700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa00ffdd0 0x7fcaa0197780 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fca94009990 tx=0x7fca94005e80 comp rx=0 tx=0).stop 2023-11-16T18:34:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.566+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1597340306 shutdown_connections 2023-11-16T18:34:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.566+0000 7fcaa51d9700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fca880a12c0 0x7fca880a3770 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.566+0000 7fcaa51d9700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcaa00ff430 0x7fcaa0197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.566+0000 7fcaa51d9700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa0104470 0x7fcaa01a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.566+0000 7fcaa51d9700 1 --2- 172.21.15.2:0/1597340306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa00ffdd0 0x7fcaa0197780 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.566+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1597340306 >> 172.21.15.2:0/1597340306 conn(0x7fcaa0076370 msgr2=0x7fcaa00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.566+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1597340306 shutdown_connections 2023-11-16T18:34:53.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.566+0000 7fcaa51d9700 1 -- 172.21.15.2:0/1597340306 wait complete. 2023-11-16T18:34:53.581 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-11-16T18:34:53.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.728+0000 7eff0b082700 1 -- 172.21.15.2:0/1015661863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff04104460 msgr2=0x7eff0410c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:53.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.728+0000 7eff0b082700 1 --2- 172.21.15.2:0/1015661863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff04104460 0x7eff0410c030 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7efefc00b0d0 tx=0x7efefc00b3e0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.728+0000 7eff0b082700 1 -- 172.21.15.2:0/1015661863 shutdown_connections 2023-11-16T18:34:53.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.728+0000 7eff0b082700 1 --2- 172.21.15.2:0/1015661863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff040ff420 0x7eff040ff7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.728+0000 7eff0b082700 1 --2- 172.21.15.2:0/1015661863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff04104460 0x7eff0410c030 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.728+0000 7eff0b082700 1 --2- 172.21.15.2:0/1015661863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff040ffdc0 0x7eff04103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.728+0000 7eff0b082700 1 -- 172.21.15.2:0/1015661863 >> 172.21.15.2:0/1015661863 conn(0x7eff04075fd0 msgr2=0x7eff040763d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:53.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.728+0000 7eff0b082700 1 -- 172.21.15.2:0/1015661863 shutdown_connections 2023-11-16T18:34:53.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.728+0000 7eff0b082700 1 -- 172.21.15.2:0/1015661863 wait complete. 2023-11-16T18:34:53.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.728+0000 7eff0b082700 1 Processor -- start 2023-11-16T18:34:53.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.728+0000 7eff0b082700 1 -- start start 2023-11-16T18:34:53.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff0b082700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff040ff420 0x7eff04197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:53.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff0b082700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff040ffdc0 0x7eff041978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:53.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff0b082700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff04104460 0x7eff041a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:53.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff0b082700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff041a2950 con 0x7eff040ff420 2023-11-16T18:34:53.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff0b082700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff041a2a90 con 0x7eff04104460 2023-11-16T18:34:53.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff0b082700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff041a2bd0 con 0x7eff040ffdc0 2023-11-16T18:34:53.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff08e1e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff040ff420 0x7eff04197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:53.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff08e1e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff040ff420 0x7eff04197390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53304/0 (socket says 172.21.15.2:53304) 2023-11-16T18:34:53.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff08e1e700 1 -- 172.21.15.2:0/4126742359 learned_addr learned my addr 172.21.15.2:0/4126742359 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:34:53.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff0961f700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff04104460 0x7eff041a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:53.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff08e1e700 1 -- 172.21.15.2:0/4126742359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff040ffdc0 msgr2=0x7eff041978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:53.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff08e1e700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff040ffdc0 0x7eff041978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff08e1e700 1 -- 172.21.15.2:0/4126742359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff04104460 msgr2=0x7eff041a2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:53.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff08e1e700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff04104460 0x7eff041a2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:53.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff08e1e700 1 -- 172.21.15.2:0/4126742359 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc009d00 con 0x7eff040ff420 2023-11-16T18:34:53.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff0961f700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff04104460 0x7eff041a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:34:53.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff08e1e700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff040ff420 0x7eff04197390 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7efef800b530 tx=0x7efef800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:53.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff01ffb700 1 -- 172.21.15.2:0/4126742359 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef8011830 con 0x7eff040ff420 2023-11-16T18:34:53.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff01ffb700 1 -- 172.21.15.2:0/4126742359 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efef8011e50 con 0x7eff040ff420 2023-11-16T18:34:53.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff0b082700 1 -- 172.21.15.2:0/4126742359 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0410f2a0 con 0x7eff040ff420 2023-11-16T18:34:53.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.729+0000 7eff01ffb700 1 -- 172.21.15.2:0/4126742359 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef800f2e0 con 0x7eff040ff420 2023-11-16T18:34:53.736 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.730+0000 7eff0b082700 1 -- 172.21.15.2:0/4126742359 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0410f110 con 0x7eff040ff420 2023-11-16T18:34:53.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.731+0000 7eff0b082700 1 -- 172.21.15.2:0/4126742359 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0410f430 con 0x7eff040ff420 2023-11-16T18:34:53.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.732+0000 7eff01ffb700 1 -- 172.21.15.2:0/4126742359 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7efef8010580 con 0x7eff040ff420 2023-11-16T18:34:53.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.732+0000 7eff01ffb700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7efef40a1340 0x7efef40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:53.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.732+0000 7eff03fff700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7efef40a1340 0x7efef40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:53.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.732+0000 7eff01ffb700 1 -- 172.21.15.2:0/4126742359 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9482+0+0 (secure 0 0 0) 0x7efef80cca70 con 0x7eff040ff420 2023-11-16T18:34:53.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.733+0000 7eff03fff700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7efef40a1340 0x7efef40a37f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efef0005950 tx=0x7efef00058e0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:53.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.737+0000 7eff01ffb700 1 -- 172.21.15.2:0/4126742359 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efef8095bd0 con 0x7eff040ff420 2023-11-16T18:34:53.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.963+0000 7eff0b082700 1 -- 172.21.15.2:0/4126742359 --> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7eff040fd3c0 con 0x7efef40a1340 2023-11-16T18:34:53.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.973+0000 7eff01ffb700 1 -- 172.21.15.2:0/4126742359 <== mgr.15594 v2:172.21.15.2:6800/2202170290 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7eff040fd3c0 con 0x7efef40a1340 2023-11-16T18:34:53.976 INFO:teuthology.orchestra.run.smithi002.stdout:/ 1M 6M3h 2023-11-16T18:34:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.975+0000 7eff0b082700 1 -- 172.21.15.2:0/4126742359 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7efef40a1340 msgr2=0x7efef40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.975+0000 7eff0b082700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7efef40a1340 0x7efef40a37f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efef0005950 tx=0x7efef00058e0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.975+0000 7eff0b082700 1 -- 172.21.15.2:0/4126742359 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff040ff420 msgr2=0x7eff04197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.975+0000 7eff0b082700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff040ff420 0x7eff04197390 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7efef800b530 tx=0x7efef800b8f0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.976+0000 7eff0b082700 1 -- 172.21.15.2:0/4126742359 shutdown_connections 2023-11-16T18:34:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.976+0000 7eff0b082700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7efef40a1340 0x7efef40a37f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.976+0000 7eff0b082700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff04104460 0x7eff041a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.976+0000 7eff0b082700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff040ff420 0x7eff04197390 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.976+0000 7eff0b082700 1 --2- 172.21.15.2:0/4126742359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff040ffdc0 0x7eff041978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.976+0000 7eff0b082700 1 -- 172.21.15.2:0/4126742359 >> 172.21.15.2:0/4126742359 conn(0x7eff04075fd0 msgr2=0x7eff040fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:54.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.976+0000 7eff0b082700 1 -- 172.21.15.2:0/4126742359 shutdown_connections 2023-11-16T18:34:54.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:53.976+0000 7eff0b082700 1 -- 172.21.15.2:0/4126742359 wait complete. 2023-11-16T18:34:54.049 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2023-11-16T18:34:54.060 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2023-11-16T18:34:54.061 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2023-11-16T18:34:54.065 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2023-11-16T18:34:54.106 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:34:54.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.233+0000 7fa798439700 1 -- 172.21.15.2:0/3493346025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7900ff320 msgr2=0x7fa79010b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.233+0000 7fa798439700 1 --2- 172.21.15.2:0/3493346025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7900ff320 0x7fa79010b7b0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa78800b0a0 tx=0x7fa78800b3b0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.233+0000 7fa798439700 1 -- 172.21.15.2:0/3493346025 shutdown_connections 2023-11-16T18:34:54.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.233+0000 7fa798439700 1 --2- 172.21.15.2:0/3493346025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7900ff320 0x7fa79010b7b0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.233+0000 7fa798439700 1 --2- 172.21.15.2:0/3493346025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa790104980 0x7fa790104d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.233+0000 7fa798439700 1 --2- 172.21.15.2:0/3493346025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7900fe970 0x7fa7900fede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.233+0000 7fa798439700 1 -- 172.21.15.2:0/3493346025 >> 172.21.15.2:0/3493346025 conn(0x7fa790075820 msgr2=0x7fa790075c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:54.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.233+0000 7fa798439700 1 -- 172.21.15.2:0/3493346025 shutdown_connections 2023-11-16T18:34:54.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.233+0000 7fa798439700 1 -- 172.21.15.2:0/3493346025 wait complete. 2023-11-16T18:34:54.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa798439700 1 Processor -- start 2023-11-16T18:34:54.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa798439700 1 -- start start 2023-11-16T18:34:54.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa798439700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7900fe970 0x7fa79010d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:54.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa798439700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7900ff320 0x7fa79010d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:54.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa798439700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa790104980 0x7fa7901a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:54.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa798439700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79010dd80 con 0x7fa790104980 2023-11-16T18:34:54.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa798439700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901a4be0 con 0x7fa7900ff320 2023-11-16T18:34:54.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa798439700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901a4d20 con 0x7fa7900fe970 2023-11-16T18:34:54.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa7969d6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa790104980 0x7fa7901a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:54.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa7969d6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa790104980 0x7fa7901a46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53322/0 (socket says 172.21.15.2:53322) 2023-11-16T18:34:54.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa7969d6700 1 -- 172.21.15.2:0/3680832117 learned_addr learned my addr 172.21.15.2:0/3680832117 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:34:54.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa7961d5700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7900fe970 0x7fa79010d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:54.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa7969d6700 1 -- 172.21.15.2:0/3680832117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7900fe970 msgr2=0x7fa79010d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa7969d6700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7900fe970 0x7fa79010d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa7959d4700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7900ff320 0x7fa79010d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:54.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa7969d6700 1 -- 172.21.15.2:0/3680832117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7900ff320 msgr2=0x7fa79010d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa7969d6700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7900ff320 0x7fa79010d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa7969d6700 1 -- 172.21.15.2:0/3680832117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa788009d00 con 0x7fa790104980 2023-11-16T18:34:54.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa7961d5700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7900fe970 0x7fa79010d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:34:54.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.234+0000 7fa7969d6700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa790104980 0x7fa7901a46a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fa78800b070 tx=0x7fa788004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:54.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.235+0000 7fa7877fe700 1 -- 172.21.15.2:0/3680832117 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78800e040 con 0x7fa790104980 2023-11-16T18:34:54.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.235+0000 7fa7877fe700 1 -- 172.21.15.2:0/3680832117 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa78801db40 con 0x7fa790104980 2023-11-16T18:34:54.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.235+0000 7fa798439700 1 -- 172.21.15.2:0/3680832117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa790110390 con 0x7fa790104980 2023-11-16T18:34:54.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.235+0000 7fa7877fe700 1 -- 172.21.15.2:0/3680832117 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78801bcc0 con 0x7fa790104980 2023-11-16T18:34:54.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.235+0000 7fa798439700 1 -- 172.21.15.2:0/3680832117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa790110200 con 0x7fa790104980 2023-11-16T18:34:54.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.236+0000 7fa798439700 1 -- 172.21.15.2:0/3680832117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa790110520 con 0x7fa790104980 2023-11-16T18:34:54.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.237+0000 7fa7877fe700 1 -- 172.21.15.2:0/3680832117 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fa78801dcb0 con 0x7fa790104980 2023-11-16T18:34:54.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.237+0000 7fa7877fe700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa77c0a12c0 0x7fa77c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:54.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.237+0000 7fa7877fe700 1 -- 172.21.15.2:0/3680832117 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9482+0+0 (secure 0 0 0) 0x7fa7880cf2a0 con 0x7fa790104980 2023-11-16T18:34:54.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.237+0000 7fa7961d5700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa77c0a12c0 0x7fa77c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:54.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.238+0000 7fa7961d5700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa77c0a12c0 0x7fa77c0a3770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa780005fd0 tx=0x7fa780005dc0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:54.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.241+0000 7fa7877fe700 1 -- 172.21.15.2:0/3680832117 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa788098380 con 0x7fa790104980 2023-11-16T18:34:54.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.442+0000 7fa798439700 1 -- 172.21.15.2:0/3680832117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa790067f40 con 0x7fa790104980 2023-11-16T18:34:54.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.444+0000 7fa7877fe700 1 -- 172.21.15.2:0/3680832117 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+25124 (secure 0 0 0) 0x7fa78809b9a0 con 0x7fa790104980 2023-11-16T18:34:54.446 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:34:54.447 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":100,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:34:36.382836+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":99,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":99,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6800/3369431713":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/1472878779":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/1347603606":"2023-11-17T18:24:50.949710+0000","172.21.15.2:6801/3369431713":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/3335793429":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/3821858662":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:34:54.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.446+0000 7fa798439700 1 -- 172.21.15.2:0/3680832117 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa77c0a12c0 msgr2=0x7fa77c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.446+0000 7fa798439700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa77c0a12c0 0x7fa77c0a3770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa780005fd0 tx=0x7fa780005dc0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.446+0000 7fa798439700 1 -- 172.21.15.2:0/3680832117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa790104980 msgr2=0x7fa7901a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.446+0000 7fa798439700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa790104980 0x7fa7901a46a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fa78800b070 tx=0x7fa788004b00 comp rx=0 tx=0).stop 2023-11-16T18:34:54.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.446+0000 7fa798439700 1 -- 172.21.15.2:0/3680832117 shutdown_connections 2023-11-16T18:34:54.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.446+0000 7fa798439700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fa77c0a12c0 0x7fa77c0a3770 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.446+0000 7fa798439700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7900fe970 0x7fa79010d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.446+0000 7fa798439700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa790104980 0x7fa7901a46a0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.446+0000 7fa798439700 1 --2- 172.21.15.2:0/3680832117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7900ff320 0x7fa79010d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.446+0000 7fa798439700 1 -- 172.21.15.2:0/3680832117 >> 172.21.15.2:0/3680832117 conn(0x7fa790075820 msgr2=0x7fa7901057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:54.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.446+0000 7fa798439700 1 -- 172.21.15.2:0/3680832117 shutdown_connections 2023-11-16T18:34:54.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.446+0000 7fa798439700 1 -- 172.21.15.2:0/3680832117 wait complete. 2023-11-16T18:34:54.461 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T18:22:44.902889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T18:22:46.187275+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 99, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T18:22:49.539148+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 99, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T18:34:54.462 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T18:34:54.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:54 smithi188 ceph-mon[100639]: pgmap v313: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:54.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:54 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:54.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.588+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/1443208077 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f540ff3f0 msgr2=0x7f3f54104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.588+0000 7f3f5b0c0700 1 --2- 172.21.15.2:0/1443208077 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f540ff3f0 0x7f3f54104ec0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f3f400099c0 tx=0x7f3f40009cd0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.588+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/1443208077 shutdown_connections 2023-11-16T18:34:54.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.588+0000 7f3f5b0c0700 1 --2- 172.21.15.2:0/1443208077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f540fea50 0x7f3f540fee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.588+0000 7f3f5b0c0700 1 --2- 172.21.15.2:0/1443208077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f541055f0 0x7f3f5410d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.588+0000 7f3f5b0c0700 1 --2- 172.21.15.2:0/1443208077 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f540ff3f0 0x7f3f54104ec0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.588+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/1443208077 >> 172.21.15.2:0/1443208077 conn(0x7f3f54075880 msgr2=0x7f3f54075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:54.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.588+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/1443208077 shutdown_connections 2023-11-16T18:34:54.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.588+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/1443208077 wait complete. 2023-11-16T18:34:54.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f5b0c0700 1 Processor -- start 2023-11-16T18:34:54.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f5b0c0700 1 -- start start 2023-11-16T18:34:54.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f5b0c0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f540fea50 0x7f3f54196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:54.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f5b0c0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f540ff3f0 0x7f3f54196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:54.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f5b0c0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f541055f0 0x7f3f5419acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:54.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f5b0c0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5419b2f0 con 0x7f3f540fea50 2023-11-16T18:34:54.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f5b0c0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5419b460 con 0x7f3f541055f0 2023-11-16T18:34:54.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f5b0c0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5419b620 con 0x7f3f540ff3f0 2023-11-16T18:34:54.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f58e5c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f540fea50 0x7f3f54196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:54.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f58e5c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f540fea50 0x7f3f54196310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53332/0 (socket says 172.21.15.2:53332) 2023-11-16T18:34:54.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f58e5c700 1 -- 172.21.15.2:0/4008378783 learned_addr learned my addr 172.21.15.2:0/4008378783 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:34:54.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f5965d700 1 --2- 172.21.15.2:0/4008378783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f541055f0 0x7f3f5419acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:54.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f58e5c700 1 -- 172.21.15.2:0/4008378783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f540ff3f0 msgr2=0x7f3f54196850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f58e5c700 1 --2- 172.21.15.2:0/4008378783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f540ff3f0 0x7f3f54196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f58e5c700 1 -- 172.21.15.2:0/4008378783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f541055f0 msgr2=0x7f3f5419acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f58e5c700 1 --2- 172.21.15.2:0/4008378783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f541055f0 0x7f3f5419acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f58e5c700 1 -- 172.21.15.2:0/4008378783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f400095f0 con 0x7f3f540fea50 2023-11-16T18:34:54.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f58e5c700 1 --2- 172.21.15.2:0/4008378783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f540fea50 0x7f3f54196310 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800b530 tx=0x7f3f4800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:54.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.589+0000 7f3f51ffb700 1 -- 172.21.15.2:0/4008378783 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48011830 con 0x7f3f540fea50 2023-11-16T18:34:54.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.590+0000 7f3f51ffb700 1 -- 172.21.15.2:0/4008378783 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f48011e50 con 0x7f3f540fea50 2023-11-16T18:34:54.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.590+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/4008378783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f54110330 con 0x7f3f540fea50 2023-11-16T18:34:54.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.590+0000 7f3f51ffb700 1 -- 172.21.15.2:0/4008378783 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4800f2e0 con 0x7f3f540fea50 2023-11-16T18:34:54.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.590+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/4008378783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f541101a0 con 0x7f3f540fea50 2023-11-16T18:34:54.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.591+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/4008378783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f541104c0 con 0x7f3f540fea50 2023-11-16T18:34:54.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.592+0000 7f3f51ffb700 1 -- 172.21.15.2:0/4008378783 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f3f48010580 con 0x7f3f540fea50 2023-11-16T18:34:54.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.593+0000 7f3f51ffb700 1 --2- 172.21.15.2:0/4008378783 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f3f440a1340 0x7f3f440a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:54.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.593+0000 7f3f53fff700 1 --2- 172.21.15.2:0/4008378783 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f3f440a1340 0x7f3f440a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:54.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.593+0000 7f3f51ffb700 1 -- 172.21.15.2:0/4008378783 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f3f480ccb30 con 0x7f3f540fea50 2023-11-16T18:34:54.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.593+0000 7f3f53fff700 1 --2- 172.21.15.2:0/4008378783 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f3f440a1340 0x7f3f440a37f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3f40009990 tx=0x7f3f40000bc0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:54.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.597+0000 7f3f51ffb700 1 -- 172.21.15.2:0/4008378783 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3f48095c30 con 0x7f3f540fea50 2023-11-16T18:34:54.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:54 smithi002 ceph-mon[98807]: pgmap v313: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:54.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:54 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:54.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:54 smithi040 ceph-mon[100251]: pgmap v313: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:34:54.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:54 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:34:54.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.794+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/4008378783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f541a33e0 con 0x7f3f540fea50 2023-11-16T18:34:54.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.794+0000 7f3f51ffb700 1 -- 172.21.15.2:0/4008378783 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f48099250 con 0x7f3f540fea50 2023-11-16T18:34:54.797 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:34:54.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.796+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/4008378783 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f3f440a1340 msgr2=0x7f3f440a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.797+0000 7f3f5b0c0700 1 --2- 172.21.15.2:0/4008378783 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f3f440a1340 0x7f3f440a37f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3f40009990 tx=0x7f3f40000bc0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.797+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/4008378783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f540fea50 msgr2=0x7f3f54196310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.797+0000 7f3f5b0c0700 1 --2- 172.21.15.2:0/4008378783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f540fea50 0x7f3f54196310 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800b530 tx=0x7f3f4800b8f0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.797+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/4008378783 shutdown_connections 2023-11-16T18:34:54.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.797+0000 7f3f5b0c0700 1 --2- 172.21.15.2:0/4008378783 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f3f440a1340 0x7f3f440a37f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.797+0000 7f3f5b0c0700 1 --2- 172.21.15.2:0/4008378783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f541055f0 0x7f3f5419acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.797+0000 7f3f5b0c0700 1 --2- 172.21.15.2:0/4008378783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f540fea50 0x7f3f54196310 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.797+0000 7f3f5b0c0700 1 --2- 172.21.15.2:0/4008378783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f540ff3f0 0x7f3f54196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.797+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/4008378783 >> 172.21.15.2:0/4008378783 conn(0x7f3f54075880 msgr2=0x7f3f540fc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:54.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.797+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/4008378783 shutdown_connections 2023-11-16T18:34:54.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.797+0000 7f3f5b0c0700 1 -- 172.21.15.2:0/4008378783 wait complete. 2023-11-16T18:34:54.811 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T18:34:54.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.940+0000 7f5fc7304700 1 -- 172.21.15.2:0/3516587646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fc00ff8f0 msgr2=0x7f5fc00ffd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.940+0000 7f5fc7304700 1 --2- 172.21.15.2:0/3516587646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fc00ff8f0 0x7f5fc00ffd60 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00099c0 tx=0x7f5fb0009cd0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.940+0000 7f5fc7304700 1 -- 172.21.15.2:0/3516587646 shutdown_connections 2023-11-16T18:34:54.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.940+0000 7f5fc7304700 1 --2- 172.21.15.2:0/3516587646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fc01002a0 0x7f5fc010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.940+0000 7f5fc7304700 1 --2- 172.21.15.2:0/3516587646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc0104140 0x7f5fc0104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.940+0000 7f5fc7304700 1 --2- 172.21.15.2:0/3516587646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fc00ff8f0 0x7f5fc00ffd60 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.940+0000 7f5fc7304700 1 -- 172.21.15.2:0/3516587646 >> 172.21.15.2:0/3516587646 conn(0x7f5fc0075880 msgr2=0x7f5fc0075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:54.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.940+0000 7f5fc7304700 1 -- 172.21.15.2:0/3516587646 shutdown_connections 2023-11-16T18:34:54.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.940+0000 7f5fc7304700 1 -- 172.21.15.2:0/3516587646 wait complete. 2023-11-16T18:34:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc7304700 1 Processor -- start 2023-11-16T18:34:54.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc7304700 1 -- start start 2023-11-16T18:34:54.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc7304700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc00ff8f0 0x7f5fc0195240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:54.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc7304700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fc01002a0 0x7f5fc0195780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:54.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc7304700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fc0104140 0x7f5fc0199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:54.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc7304700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc019a220 con 0x7f5fc0104140 2023-11-16T18:34:54.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc7304700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc019a390 con 0x7f5fc00ff8f0 2023-11-16T18:34:54.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc7304700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc019a550 con 0x7f5fc01002a0 2023-11-16T18:34:54.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc58a1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fc0104140 0x7f5fc0199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:54.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc58a1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fc0104140 0x7f5fc0199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53350/0 (socket says 172.21.15.2:53350) 2023-11-16T18:34:54.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc58a1700 1 -- 172.21.15.2:0/613481822 learned_addr learned my addr 172.21.15.2:0/613481822 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:34:54.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc50a0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc00ff8f0 0x7f5fc0195240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:54.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc489f700 1 --2- 172.21.15.2:0/613481822 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fc01002a0 0x7f5fc0195780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:54.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc58a1700 1 -- 172.21.15.2:0/613481822 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fc01002a0 msgr2=0x7f5fc0195780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc58a1700 1 --2- 172.21.15.2:0/613481822 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fc01002a0 0x7f5fc0195780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.941+0000 7f5fc58a1700 1 -- 172.21.15.2:0/613481822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc00ff8f0 msgr2=0x7f5fc0195240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:54.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.942+0000 7f5fc58a1700 1 --2- 172.21.15.2:0/613481822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc00ff8f0 0x7f5fc0195240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:54.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.942+0000 7f5fc58a1700 1 -- 172.21.15.2:0/613481822 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb00095f0 con 0x7f5fc0104140 2023-11-16T18:34:54.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.942+0000 7f5fc58a1700 1 --2- 172.21.15.2:0/613481822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fc0104140 0x7f5fc0199be0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800b9e0 tx=0x7f5fb800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:54.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.942+0000 7f5fb67fc700 1 -- 172.21.15.2:0/613481822 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb800d8e0 con 0x7f5fc0104140 2023-11-16T18:34:54.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.942+0000 7f5fb67fc700 1 -- 172.21.15.2:0/613481822 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fb8009880 con 0x7f5fc0104140 2023-11-16T18:34:54.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.942+0000 7f5fb67fc700 1 -- 172.21.15.2:0/613481822 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb80143d0 con 0x7f5fc0104140 2023-11-16T18:34:54.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.942+0000 7f5fc7304700 1 -- 172.21.15.2:0/613481822 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc010f270 con 0x7f5fc0104140 2023-11-16T18:34:54.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.942+0000 7f5fc7304700 1 -- 172.21.15.2:0/613481822 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc010f0e0 con 0x7f5fc0104140 2023-11-16T18:34:54.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.943+0000 7f5fc7304700 1 -- 172.21.15.2:0/613481822 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc010f400 con 0x7f5fc0104140 2023-11-16T18:34:54.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.945+0000 7f5fb67fc700 1 -- 172.21.15.2:0/613481822 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f5fb800da40 con 0x7f5fc0104140 2023-11-16T18:34:54.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.945+0000 7f5fb67fc700 1 --2- 172.21.15.2:0/613481822 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f5fac0a12c0 0x7f5fac0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:54.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.945+0000 7f5fc50a0700 1 --2- 172.21.15.2:0/613481822 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f5fac0a12c0 0x7f5fac0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:54.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.945+0000 7f5fb67fc700 1 -- 172.21.15.2:0/613481822 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f5fb8011070 con 0x7f5fc0104140 2023-11-16T18:34:54.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.946+0000 7f5fc50a0700 1 --2- 172.21.15.2:0/613481822 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f5fac0a12c0 0x7f5fac0a3770 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc005950 tx=0x7f5fbc00b410 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:54.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:54.949+0000 7f5fb67fc700 1 -- 172.21.15.2:0/613481822 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5fb804e730 con 0x7f5fc0104140 2023-11-16T18:34:55.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.153+0000 7f5fc7304700 1 -- 172.21.15.2:0/613481822 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5fc01a22b0 con 0x7f5fc0104140 2023-11-16T18:34:55.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.153+0000 7f5fb67fc700 1 -- 172.21.15.2:0/613481822 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5fb8095c40 con 0x7f5fc0104140 2023-11-16T18:34:55.156 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:34:55.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.155+0000 7f5fc7304700 1 -- 172.21.15.2:0/613481822 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f5fac0a12c0 msgr2=0x7f5fac0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:55.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.155+0000 7f5fc7304700 1 --2- 172.21.15.2:0/613481822 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f5fac0a12c0 0x7f5fac0a3770 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc005950 tx=0x7f5fbc00b410 comp rx=0 tx=0).stop 2023-11-16T18:34:55.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.155+0000 7f5fc7304700 1 -- 172.21.15.2:0/613481822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fc0104140 msgr2=0x7f5fc0199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:55.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.155+0000 7f5fc7304700 1 --2- 172.21.15.2:0/613481822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fc0104140 0x7f5fc0199be0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800b9e0 tx=0x7f5fb800bcf0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.155+0000 7f5fc7304700 1 -- 172.21.15.2:0/613481822 shutdown_connections 2023-11-16T18:34:55.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.155+0000 7f5fc7304700 1 --2- 172.21.15.2:0/613481822 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f5fac0a12c0 0x7f5fac0a3770 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.155+0000 7f5fc7304700 1 --2- 172.21.15.2:0/613481822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fc0104140 0x7f5fc0199be0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.155+0000 7f5fc7304700 1 --2- 172.21.15.2:0/613481822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc00ff8f0 0x7f5fc0195240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.155+0000 7f5fc7304700 1 --2- 172.21.15.2:0/613481822 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fc01002a0 0x7f5fc0195780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.155+0000 7f5fc7304700 1 -- 172.21.15.2:0/613481822 >> 172.21.15.2:0/613481822 conn(0x7f5fc0075880 msgr2=0x7f5fc00fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:55.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.155+0000 7f5fc7304700 1 -- 172.21.15.2:0/613481822 shutdown_connections 2023-11-16T18:34:55.159 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.155+0000 7f5fc7304700 1 -- 172.21.15.2:0/613481822 wait complete. 2023-11-16T18:34:55.170 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T18:34:55.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.298+0000 7eff9a52a700 1 -- 172.21.15.2:0/1617312889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff94069030 msgr2=0x7eff940694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:55.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.298+0000 7eff9a52a700 1 --2- 172.21.15.2:0/1617312889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff94069030 0x7eff940694a0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7eff7c0099c0 tx=0x7eff7c009cd0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.298+0000 7eff9a52a700 1 -- 172.21.15.2:0/1617312889 shutdown_connections 2023-11-16T18:34:55.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.298+0000 7eff9a52a700 1 --2- 172.21.15.2:0/1617312889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff940699e0 0x7eff9410d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.298+0000 7eff9a52a700 1 --2- 172.21.15.2:0/1617312889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff94105250 0x7eff94105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.298+0000 7eff9a52a700 1 --2- 172.21.15.2:0/1617312889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff94069030 0x7eff940694a0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.298+0000 7eff9a52a700 1 -- 172.21.15.2:0/1617312889 >> 172.21.15.2:0/1617312889 conn(0x7eff94076370 msgr2=0x7eff94076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:55.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.298+0000 7eff9a52a700 1 -- 172.21.15.2:0/1617312889 shutdown_connections 2023-11-16T18:34:55.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.298+0000 7eff9a52a700 1 -- 172.21.15.2:0/1617312889 wait complete. 2023-11-16T18:34:55.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.298+0000 7eff9a52a700 1 Processor -- start 2023-11-16T18:34:55.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.298+0000 7eff9a52a700 1 -- start start 2023-11-16T18:34:55.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff9a52a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff94069030 0x7eff94196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:55.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff9a52a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff940699e0 0x7eff94196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:55.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff9a52a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff94105250 0x7eff9419acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:55.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff9a52a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9419b2f0 con 0x7eff940699e0 2023-11-16T18:34:55.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff9a52a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9419b460 con 0x7eff94069030 2023-11-16T18:34:55.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff9a52a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9419b620 con 0x7eff94105250 2023-11-16T18:34:55.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff937fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff940699e0 0x7eff94196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:55.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff937fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff940699e0 0x7eff94196850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53362/0 (socket says 172.21.15.2:53362) 2023-11-16T18:34:55.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff937fe700 1 -- 172.21.15.2:0/1799784171 learned_addr learned my addr 172.21.15.2:0/1799784171 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:34:55.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff93fff700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff94069030 0x7eff94196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:55.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff98ac7700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff94105250 0x7eff9419acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:55.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff937fe700 1 -- 172.21.15.2:0/1799784171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff94105250 msgr2=0x7eff9419acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:55.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff937fe700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff94105250 0x7eff9419acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff937fe700 1 -- 172.21.15.2:0/1799784171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff94069030 msgr2=0x7eff94196310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:55.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff937fe700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff94069030 0x7eff94196310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff937fe700 1 -- 172.21.15.2:0/1799784171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c0095f0 con 0x7eff940699e0 2023-11-16T18:34:55.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.299+0000 7eff98ac7700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff94105250 0x7eff9419acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:34:55.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.300+0000 7eff937fe700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff940699e0 0x7eff94196850 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7eff7c009990 tx=0x7eff7c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:55.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.300+0000 7eff917fa700 1 -- 172.21.15.2:0/1799784171 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c01d070 con 0x7eff940699e0 2023-11-16T18:34:55.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.300+0000 7eff917fa700 1 -- 172.21.15.2:0/1799784171 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff7c003f90 con 0x7eff940699e0 2023-11-16T18:34:55.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.300+0000 7eff9a52a700 1 -- 172.21.15.2:0/1799784171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff94110380 con 0x7eff940699e0 2023-11-16T18:34:55.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.300+0000 7eff917fa700 1 -- 172.21.15.2:0/1799784171 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c022ce0 con 0x7eff940699e0 2023-11-16T18:34:55.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.300+0000 7eff9a52a700 1 -- 172.21.15.2:0/1799784171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff941101f0 con 0x7eff940699e0 2023-11-16T18:34:55.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.301+0000 7eff9a52a700 1 -- 172.21.15.2:0/1799784171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff94110510 con 0x7eff940699e0 2023-11-16T18:34:55.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.302+0000 7eff917fa700 1 -- 172.21.15.2:0/1799784171 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7eff7c022470 con 0x7eff940699e0 2023-11-16T18:34:55.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.302+0000 7eff917fa700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7eff800a1340 0x7eff800a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:55.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.303+0000 7eff917fa700 1 -- 172.21.15.2:0/1799784171 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9482+0+0 (secure 0 0 0) 0x7eff7c0ce690 con 0x7eff940699e0 2023-11-16T18:34:55.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.303+0000 7eff93fff700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7eff800a1340 0x7eff800a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:55.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.303+0000 7eff93fff700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7eff800a1340 0x7eff800a37f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7eff84005950 tx=0x7eff8400b410 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:55.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.306+0000 7eff917fa700 1 -- 172.21.15.2:0/1799784171 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7eff7c097770 con 0x7eff940699e0 2023-11-16T18:34:55.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.503+0000 7eff9a52a700 1 -- 172.21.15.2:0/1799784171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7eff941a3380 con 0x7eff940699e0 2023-11-16T18:34:55.507 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.504+0000 7eff917fa700 1 -- 172.21.15.2:0/1799784171 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7eff7c09ad90 con 0x7eff940699e0 2023-11-16T18:34:55.507 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T18:34:55.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.506+0000 7eff9a52a700 1 -- 172.21.15.2:0/1799784171 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7eff800a1340 msgr2=0x7eff800a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:55.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.506+0000 7eff9a52a700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7eff800a1340 0x7eff800a37f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7eff84005950 tx=0x7eff8400b410 comp rx=0 tx=0).stop 2023-11-16T18:34:55.508 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.506+0000 7eff9a52a700 1 -- 172.21.15.2:0/1799784171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff940699e0 msgr2=0x7eff94196850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:55.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.506+0000 7eff9a52a700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff940699e0 0x7eff94196850 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7eff7c009990 tx=0x7eff7c009580 comp rx=0 tx=0).stop 2023-11-16T18:34:55.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.506+0000 7eff9a52a700 1 -- 172.21.15.2:0/1799784171 shutdown_connections 2023-11-16T18:34:55.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.506+0000 7eff9a52a700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7eff800a1340 0x7eff800a37f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.506+0000 7eff9a52a700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff94105250 0x7eff9419acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.506+0000 7eff9a52a700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff94069030 0x7eff94196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.506+0000 7eff9a52a700 1 --2- 172.21.15.2:0/1799784171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff940699e0 0x7eff94196850 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.509 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.506+0000 7eff9a52a700 1 -- 172.21.15.2:0/1799784171 >> 172.21.15.2:0/1799784171 conn(0x7eff94076370 msgr2=0x7eff940fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:55.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.506+0000 7eff9a52a700 1 -- 172.21.15.2:0/1799784171 shutdown_connections 2023-11-16T18:34:55.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.506+0000 7eff9a52a700 1 -- 172.21.15.2:0/1799784171 wait complete. 2023-11-16T18:34:55.519 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:55 smithi002 ceph-mon[98807]: from='client.16920 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:34:55.519 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:55 smithi002 ceph-mon[98807]: from='client.16926 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:34:55.519 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:55 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3680832117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:34:55.520 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:55 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/4008378783' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:34:55.520 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:55 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/613481822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:34:55.521 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T18:34:55.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:55 smithi188 ceph-mon[100639]: from='client.16920 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:34:55.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:55 smithi188 ceph-mon[100639]: from='client.16926 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:34:55.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:55 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3680832117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:34:55.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:55 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/4008378783' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:34:55.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:55 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/613481822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:34:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.647+0000 7fc25e7e7700 1 -- 172.21.15.2:0/229083810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258105250 msgr2=0x7fc258105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.647+0000 7fc25e7e7700 1 --2- 172.21.15.2:0/229083810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258105250 0x7fc258105620 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fc240009990 tx=0x7fc240009ca0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.647+0000 7fc25e7e7700 1 -- 172.21.15.2:0/229083810 shutdown_connections 2023-11-16T18:34:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.647+0000 7fc25e7e7700 1 --2- 172.21.15.2:0/229083810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258100420 0x7fc25810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.647+0000 7fc25e7e7700 1 --2- 172.21.15.2:0/229083810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258105250 0x7fc258105620 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.647+0000 7fc25e7e7700 1 --2- 172.21.15.2:0/229083810 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2580ffa70 0x7fc2580ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.647+0000 7fc25e7e7700 1 -- 172.21.15.2:0/229083810 >> 172.21.15.2:0/229083810 conn(0x7fc2580763a0 msgr2=0x7fc2580767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.647+0000 7fc25e7e7700 1 -- 172.21.15.2:0/229083810 shutdown_connections 2023-11-16T18:34:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.647+0000 7fc25e7e7700 1 -- 172.21.15.2:0/229083810 wait complete. 2023-11-16T18:34:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc25e7e7700 1 Processor -- start 2023-11-16T18:34:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc25e7e7700 1 -- start start 2023-11-16T18:34:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc25e7e7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2580ffa70 0x7fc258197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc25e7e7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258100420 0x7fc258197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc25e7e7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc258105250 0x7fc2581a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc25e7e7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc258197fa0 con 0x7fc258100420 2023-11-16T18:34:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc25e7e7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2581a2710 con 0x7fc2580ffa70 2023-11-16T18:34:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc25e7e7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2581a2880 con 0x7fc258105250 2023-11-16T18:34:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc2577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258100420 0x7fc258197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc2577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258100420 0x7fc258197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53376/0 (socket says 172.21.15.2:53376) 2023-11-16T18:34:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc2577fe700 1 -- 172.21.15.2:0/1092344516 learned_addr learned my addr 172.21.15.2:0/1092344516 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:34:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc257fff700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2580ffa70 0x7fc258197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc25cd84700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc258105250 0x7fc2581a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc2577fe700 1 -- 172.21.15.2:0/1092344516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc258105250 msgr2=0x7fc2581a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc2577fe700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc258105250 0x7fc2581a21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc2577fe700 1 -- 172.21.15.2:0/1092344516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2580ffa70 msgr2=0x7fc258197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc2577fe700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2580ffa70 0x7fc258197150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc2577fe700 1 -- 172.21.15.2:0/1092344516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2400095f0 con 0x7fc258100420 2023-11-16T18:34:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc25cd84700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc258105250 0x7fc2581a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:34:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.648+0000 7fc257fff700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2580ffa70 0x7fc258197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:34:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.649+0000 7fc2577fe700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258100420 0x7fc258197690 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fc24800d950 tx=0x7fc24800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.649+0000 7fc2557fa700 1 -- 172.21.15.2:0/1092344516 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc248009b40 con 0x7fc258100420 2023-11-16T18:34:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.649+0000 7fc2557fa700 1 -- 172.21.15.2:0/1092344516 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc248009ca0 con 0x7fc258100420 2023-11-16T18:34:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.649+0000 7fc25e7e7700 1 -- 172.21.15.2:0/1092344516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc25810f1b0 con 0x7fc258100420 2023-11-16T18:34:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.649+0000 7fc2557fa700 1 -- 172.21.15.2:0/1092344516 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24800f620 con 0x7fc258100420 2023-11-16T18:34:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.649+0000 7fc25e7e7700 1 -- 172.21.15.2:0/1092344516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25810f020 con 0x7fc258100420 2023-11-16T18:34:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.650+0000 7fc25e7e7700 1 -- 172.21.15.2:0/1092344516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25810f340 con 0x7fc258100420 2023-11-16T18:34:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.651+0000 7fc2557fa700 1 -- 172.21.15.2:0/1092344516 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fc248010460 con 0x7fc258100420 2023-11-16T18:34:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.651+0000 7fc2557fa700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc2440a1340 0x7fc2440a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.652+0000 7fc257fff700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc2440a1340 0x7fc2440a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.652+0000 7fc2557fa700 1 -- 172.21.15.2:0/1092344516 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9482+0+0 (secure 0 0 0) 0x7fc2480cd440 con 0x7fc258100420 2023-11-16T18:34:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.652+0000 7fc257fff700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc2440a1340 0x7fc2440a37f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc24000b5c0 tx=0x7fc240005fb0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:55.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.656+0000 7fc2557fa700 1 -- 172.21.15.2:0/1092344516 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc248096430 con 0x7fc258100420 2023-11-16T18:34:55.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:55 smithi040 ceph-mon[100251]: from='client.16920 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:34:55.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:55 smithi040 ceph-mon[100251]: from='client.16926 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T18:34:55.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:55 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3680832117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:34:55.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:55 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/4008378783' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:34:55.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:55 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/613481822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:34:55.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.855+0000 7fc25e7e7700 1 -- 172.21.15.2:0/1092344516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc2581a33e0 con 0x7fc258100420 2023-11-16T18:34:55.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.856+0000 7fc2557fa700 1 -- 172.21.15.2:0/1092344516 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v100) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc248099a50 con 0x7fc258100420 2023-11-16T18:34:55.859 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:34:55.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.858+0000 7fc25e7e7700 1 -- 172.21.15.2:0/1092344516 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc2440a1340 msgr2=0x7fc2440a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:55.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.858+0000 7fc25e7e7700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc2440a1340 0x7fc2440a37f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc24000b5c0 tx=0x7fc240005fb0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.858+0000 7fc25e7e7700 1 -- 172.21.15.2:0/1092344516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258100420 msgr2=0x7fc258197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:55.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.858+0000 7fc25e7e7700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258100420 0x7fc258197690 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fc24800d950 tx=0x7fc24800dd10 comp rx=0 tx=0).stop 2023-11-16T18:34:55.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.858+0000 7fc25e7e7700 1 -- 172.21.15.2:0/1092344516 shutdown_connections 2023-11-16T18:34:55.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.858+0000 7fc25e7e7700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc2440a1340 0x7fc2440a37f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.858+0000 7fc25e7e7700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc258105250 0x7fc2581a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.858+0000 7fc25e7e7700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2580ffa70 0x7fc258197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.858+0000 7fc25e7e7700 1 --2- 172.21.15.2:0/1092344516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258100420 0x7fc258197690 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:55.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.858+0000 7fc25e7e7700 1 -- 172.21.15.2:0/1092344516 >> 172.21.15.2:0/1092344516 conn(0x7fc2580763a0 msgr2=0x7fc2580fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:55.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.858+0000 7fc25e7e7700 1 -- 172.21.15.2:0/1092344516 shutdown_connections 2023-11-16T18:34:55.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.858+0000 7fc25e7e7700 1 -- 172.21.15.2:0/1092344516 wait complete. 2023-11-16T18:34:55.873 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2023-11-16T18:34:56.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.997+0000 7fdafe463700 1 -- 172.21.15.2:0/4251852033 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaf810cb90 msgr2=0x7fdaf810ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:56.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.997+0000 7fdafe463700 1 --2- 172.21.15.2:0/4251852033 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaf810cb90 0x7fdaf810ef70 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000b0a0 tx=0x7fdaf000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:34:56.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.997+0000 7fdafe463700 1 -- 172.21.15.2:0/4251852033 shutdown_connections 2023-11-16T18:34:56.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.997+0000 7fdafe463700 1 --2- 172.21.15.2:0/4251852033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaf8072e10 0x7fdaf80731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:56.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.997+0000 7fdafe463700 1 --2- 172.21.15.2:0/4251852033 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaf810cb90 0x7fdaf810ef70 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:56.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.997+0000 7fdafe463700 1 --2- 172.21.15.2:0/4251852033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf8073720 0x7fdaf810c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:56.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.997+0000 7fdafe463700 1 -- 172.21.15.2:0/4251852033 >> 172.21.15.2:0/4251852033 conn(0x7fdaf8077590 msgr2=0x7fdaf8077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:34:56.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.997+0000 7fdafe463700 1 -- 172.21.15.2:0/4251852033 shutdown_connections 2023-11-16T18:34:56.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.997+0000 7fdafe463700 1 -- 172.21.15.2:0/4251852033 wait complete. 2023-11-16T18:34:56.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdafe463700 1 Processor -- start 2023-11-16T18:34:56.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdafe463700 1 -- start start 2023-11-16T18:34:56.003 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdafe463700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaf8072e10 0x7fdaf8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:56.003 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdafe463700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaf8073720 0x7fdaf8198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:56.003 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdafe463700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf810cb90 0x7fdaf819bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:56.003 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdafe463700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf819c5f0 con 0x7fdaf8072e10 2023-11-16T18:34:56.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdafe463700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf819c760 con 0x7fdaf810cb90 2023-11-16T18:34:56.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdafe463700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf819c920 con 0x7fdaf8073720 2023-11-16T18:34:56.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdaf7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaf8072e10 0x7fdaf8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:56.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdaf7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaf8072e10 0x7fdaf8198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53388/0 (socket says 172.21.15.2:53388) 2023-11-16T18:34:56.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdaf7fff700 1 -- 172.21.15.2:0/2787757545 learned_addr learned my addr 172.21.15.2:0/2787757545 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:34:56.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdafca00700 1 --2- 172.21.15.2:0/2787757545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf810cb90 0x7fdaf819bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:56.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdaf7fff700 1 -- 172.21.15.2:0/2787757545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaf8073720 msgr2=0x7fdaf8198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:56.004 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdaf7fff700 1 --2- 172.21.15.2:0/2787757545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaf8073720 0x7fdaf8198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:56.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdaf7fff700 1 -- 172.21.15.2:0/2787757545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf810cb90 msgr2=0x7fdaf819bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:34:56.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdaf7fff700 1 --2- 172.21.15.2:0/2787757545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf810cb90 0x7fdaf819bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:34:56.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.998+0000 7fdaf7fff700 1 -- 172.21.15.2:0/2787757545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf0009d00 con 0x7fdaf8072e10 2023-11-16T18:34:56.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.999+0000 7fdaf7fff700 1 --2- 172.21.15.2:0/2787757545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaf8072e10 0x7fdaf8198700 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fdae800b550 tx=0x7fdae800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:56.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.999+0000 7fdaf57fa700 1 -- 172.21.15.2:0/2787757545 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae8011830 con 0x7fdaf8072e10 2023-11-16T18:34:56.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.999+0000 7fdafe463700 1 -- 172.21.15.2:0/2787757545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf81125d0 con 0x7fdaf8072e10 2023-11-16T18:34:56.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.999+0000 7fdaf57fa700 1 -- 172.21.15.2:0/2787757545 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdae8011e50 con 0x7fdaf8072e10 2023-11-16T18:34:56.005 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.999+0000 7fdaf57fa700 1 -- 172.21.15.2:0/2787757545 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae800f2e0 con 0x7fdaf8072e10 2023-11-16T18:34:56.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:55.999+0000 7fdafe463700 1 -- 172.21.15.2:0/2787757545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf8112440 con 0x7fdaf8072e10 2023-11-16T18:34:56.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.000+0000 7fdafe463700 1 -- 172.21.15.2:0/2787757545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf8112760 con 0x7fdaf8072e10 2023-11-16T18:34:56.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.001+0000 7fdaf57fa700 1 -- 172.21.15.2:0/2787757545 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fdae8010580 con 0x7fdaf8072e10 2023-11-16T18:34:56.006 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.002+0000 7fdaf57fa700 1 --2- 172.21.15.2:0/2787757545 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdae40a12c0 0x7fdae40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:34:56.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.002+0000 7fdaf57fa700 1 -- 172.21.15.2:0/2787757545 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9482+0+0 (secure 0 0 0) 0x7fdae80ccb40 con 0x7fdaf8072e10 2023-11-16T18:34:56.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.002+0000 7fdaf77fe700 1 --2- 172.21.15.2:0/2787757545 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdae40a12c0 0x7fdae40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:34:56.007 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.002+0000 7fdaf77fe700 1 --2- 172.21.15.2:0/2787757545 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdae40a12c0 0x7fdae40a3770 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdae0005950 tx=0x7fdae00058e0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:34:56.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.006+0000 7fdaf57fa700 1 -- 172.21.15.2:0/2787757545 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdae8095c40 con 0x7fdaf8072e10 2023-11-16T18:34:56.264 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.261+0000 7fdafe463700 1 -- 172.21.15.2:0/2787757545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7fdaf8067260 con 0x7fdaf8072e10 2023-11-16T18:34:56.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.270+0000 7fdaf57fa700 1 -- 172.21.15.2:0/2787757545 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) v1 ==== 81+0+733766 (secure 0 0 0) 0x7fdae8099260 con 0x7fdaf8072e10 2023-11-16T18:34:56.284 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:34:56.284 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:34:56.285 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 25, 2023-11-16T18:34:56.285 INFO:teuthology.orchestra.run.smithi002.stdout: "active_gid": 15594, 2023-11-16T18:34:56.285 INFO:teuthology.orchestra.run.smithi002.stdout: "active_name": "x", 2023-11-16T18:34:56.285 INFO:teuthology.orchestra.run.smithi002.stdout: "active_addrs": { 2023-11-16T18:34:56.285 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T18:34:56.285 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:56.285 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T18:34:56.286 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6800", 2023-11-16T18:34:56.286 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2202170290 2023-11-16T18:34:56.286 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.286 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:56.286 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-16T18:34:56.286 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6801", 2023-11-16T18:34:56.286 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2202170290 2023-11-16T18:34:56.287 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:56.287 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T18:34:56.287 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.287 INFO:teuthology.orchestra.run.smithi002.stdout: "active_addr": "172.21.15.2:6801/2202170290", 2023-11-16T18:34:56.287 INFO:teuthology.orchestra.run.smithi002.stdout: "active_change": "2023-11-16T18:24:50.950116+0000", 2023-11-16T18:34:56.287 INFO:teuthology.orchestra.run.smithi002.stdout: "active_mgr_features": 4540138322906710015, 2023-11-16T18:34:56.287 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2023-11-16T18:34:56.287 INFO:teuthology.orchestra.run.smithi002.stdout: "standbys": [ 2023-11-16T18:34:56.288 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:56.288 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 25156, 2023-11-16T18:34:56.288 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "z", 2023-11-16T18:34:56.288 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_features": 4540138322906710015, 2023-11-16T18:34:56.288 INFO:teuthology.orchestra.run.smithi002.stdout: "available_modules": [ 2023-11-16T18:34:56.288 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:56.288 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alerts", 2023-11-16T18:34:56.288 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:56.288 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:56.289 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:56.289 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:34:56.289 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:34:56.289 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:56.289 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.289 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.289 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:34:56.289 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.290 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.290 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.290 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How frequently to reexamine health status", 2023-11-16T18:34:56.290 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.290 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.290 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.290 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.290 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:56.291 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:56.291 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.291 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.291 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.291 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:56.291 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.291 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.291 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:56.292 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:56.292 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:56.292 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:56.292 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:56.292 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:56.292 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:56.292 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:56.292 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:56.293 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.293 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.293 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.293 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.293 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:56.293 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:56.293 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:56.293 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.293 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.294 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:56.294 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.294 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.294 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.294 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:56.294 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.294 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.294 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.295 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.295 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:56.295 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:56.295 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.295 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.295 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.295 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:56.295 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.296 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.296 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:56.296 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:56.296 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:56.296 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:56.296 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:56.296 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:56.296 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:56.297 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:56.297 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:56.297 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.297 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.297 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.297 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.297 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:56.298 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:56.298 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:56.298 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.298 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.298 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:56.298 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.298 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.298 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:56.299 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.299 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.299 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.299 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.299 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_destination": { 2023-11-16T18:34:56.299 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", 2023-11-16T18:34:56.299 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.299 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.300 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.300 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:56.300 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.300 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.300 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.300 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email address to send alerts to", 2023-11-16T18:34:56.300 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.300 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.301 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.301 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.301 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_from_name": { 2023-11-16T18:34:56.301 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_from_name", 2023-11-16T18:34:56.301 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.301 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.301 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.301 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "Ceph", 2023-11-16T18:34:56.302 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.302 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.302 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.302 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email From: name", 2023-11-16T18:34:56.302 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.302 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.302 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.302 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.303 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_host": { 2023-11-16T18:34:56.303 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_host", 2023-11-16T18:34:56.303 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.303 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.303 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.303 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:56.303 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.303 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.304 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.304 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP server", 2023-11-16T18:34:56.304 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.304 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.304 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.304 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.304 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_password": { 2023-11-16T18:34:56.304 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_password", 2023-11-16T18:34:56.305 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.305 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.305 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.305 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:56.305 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.305 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.305 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.305 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Password to authenticate with", 2023-11-16T18:34:56.305 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.306 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.306 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.306 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.306 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_port": { 2023-11-16T18:34:56.306 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_port", 2023-11-16T18:34:56.306 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:56.306 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.307 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.307 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", 2023-11-16T18:34:56.307 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.307 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.307 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.307 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP port", 2023-11-16T18:34:56.307 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.307 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.308 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.308 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.308 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_sender": { 2023-11-16T18:34:56.308 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_sender", 2023-11-16T18:34:56.308 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.308 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.308 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.309 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:56.309 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.309 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.309 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.309 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP envelope sender", 2023-11-16T18:34:56.309 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.309 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.309 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.310 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.310 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_ssl": { 2023-11-16T18:34:56.310 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_ssl", 2023-11-16T18:34:56.310 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:56.310 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.310 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.310 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:56.311 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.311 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.311 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.311 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use SSL to connect to SMTP server", 2023-11-16T18:34:56.311 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.311 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.311 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.311 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.311 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_user": { 2023-11-16T18:34:56.312 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_user", 2023-11-16T18:34:56.312 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.312 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.312 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.312 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:56.312 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.312 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.312 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.313 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "User to authenticate as", 2023-11-16T18:34:56.313 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:56.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:56.313 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.313 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:56.314 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "balancer", 2023-11-16T18:34:56.314 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:56.314 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:56.314 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:56.314 INFO:teuthology.orchestra.run.smithi002.stdout: "active": { 2023-11-16T18:34:56.314 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "active", 2023-11-16T18:34:56.314 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:56.315 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.315 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.315 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:56.315 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.315 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.315 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.315 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically balance PGs across cluster", 2023-11-16T18:34:56.315 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.315 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.316 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.316 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.316 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_time": { 2023-11-16T18:34:56.316 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_time", 2023-11-16T18:34:56.316 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.316 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.316 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.316 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0000", 2023-11-16T18:34:56.317 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.317 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.317 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.317 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "beginning time of day to automatically balance", 2023-11-16T18:34:56.317 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T18:34:56.317 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.317 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.317 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.318 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_weekday": { 2023-11-16T18:34:56.318 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_weekday", 2023-11-16T18:34:56.318 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:56.318 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.318 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.318 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:34:56.318 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2023-11-16T18:34:56.318 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2023-11-16T18:34:56.319 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.319 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-11-16T18:34:56.319 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T18:34:56.319 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.319 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.319 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.319 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_max_iterations": { 2023-11-16T18:34:56.319 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", 2023-11-16T18:34:56.320 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:56.320 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.320 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.320 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "25", 2023-11-16T18:34:56.320 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T18:34:56.320 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "250", 2023-11-16T18:34:56.320 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.320 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-11-16T18:34:56.320 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.321 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.321 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.321 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.321 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_metrics": { 2023-11-16T18:34:56.321 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_metrics", 2023-11-16T18:34:56.321 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.321 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.321 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.322 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "pgs,objects,bytes", 2023-11-16T18:34:56.322 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.322 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.322 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.322 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-11-16T18:34:56.322 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-11-16T18:34:56.322 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.322 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.323 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.323 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_step": { 2023-11-16T18:34:56.323 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_step", 2023-11-16T18:34:56.323 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:56.323 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.323 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.323 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.5", 2023-11-16T18:34:56.323 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0.001", 2023-11-16T18:34:56.324 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "0.999", 2023-11-16T18:34:56.324 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.324 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "aggressiveness of optimization", 2023-11-16T18:34:56.324 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-11-16T18:34:56.324 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.324 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.324 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.324 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { 2023-11-16T18:34:56.325 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", 2023-11-16T18:34:56.325 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.325 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.325 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.325 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2359", 2023-11-16T18:34:56.325 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.325 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.325 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.326 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ending time of day to automatically balance", 2023-11-16T18:34:56.326 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T18:34:56.326 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.326 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.326 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.326 INFO:teuthology.orchestra.run.smithi002.stdout: "end_weekday": { 2023-11-16T18:34:56.326 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_weekday", 2023-11-16T18:34:56.326 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:56.327 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.327 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.327 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:34:56.327 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2023-11-16T18:34:56.327 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2023-11-16T18:34:56.327 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.327 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-11-16T18:34:56.327 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T18:34:56.328 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.328 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.328 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.328 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:56.328 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:56.328 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.328 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.328 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.328 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:56.329 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.329 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.329 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:56.329 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:56.329 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:56.329 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:56.329 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:56.329 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:56.330 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:56.330 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:56.330 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:56.330 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.330 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.330 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.330 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.331 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:56.331 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:56.331 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:56.331 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.331 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.331 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:56.331 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.332 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.332 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.332 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:56.332 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.332 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.335 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.367 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.405 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:56.405 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:56.405 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.405 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.405 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.406 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:56.406 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.693 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.693 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:56.693 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:56.693 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:56.693 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:56.693 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:56.693 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:56.694 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:56.694 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:56.694 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:56.694 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.694 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.694 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.694 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.694 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:56.695 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:56.695 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:56.695 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.695 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.695 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:56.695 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.695 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.695 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.696 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:56.696 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.696 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.696 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.696 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.696 INFO:teuthology.orchestra.run.smithi002.stdout: "min_score": { 2023-11-16T18:34:56.696 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_score", 2023-11-16T18:34:56.696 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:56.696 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.697 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.697 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:34:56.697 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.697 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.697 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.697 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-11-16T18:34:56.697 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.697 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.698 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.768 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.768 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2023-11-16T18:34:56.768 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2023-11-16T18:34:56.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.768 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.768 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.768 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "upmap", 2023-11-16T18:34:56.768 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.769 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.769 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:56.769 INFO:teuthology.orchestra.run.smithi002.stdout: "crush-compat", 2023-11-16T18:34:56.769 INFO:teuthology.orchestra.run.smithi002.stdout: "none", 2023-11-16T18:34:56.769 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap" 2023-11-16T18:34:56.769 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:56.769 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Balancer mode", 2023-11-16T18:34:56.769 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.769 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.770 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.770 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.770 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_ids": { 2023-11-16T18:34:56.770 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_ids", 2023-11-16T18:34:56.770 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.771 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.771 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.771 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:56.771 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.772 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.772 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.786 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-11-16T18:34:56.786 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.787 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.787 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.787 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.787 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:34:56.788 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:34:56.788 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:56.788 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.788 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.789 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:34:56.789 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.789 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.789 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.790 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-11-16T18:34:56.790 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.790 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.790 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.791 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_deviation": { 2023-11-16T18:34:56.791 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_deviation", 2023-11-16T18:34:56.791 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:56.791 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.815 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.815 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T18:34:56.816 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T18:34:56.816 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.816 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.817 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "deviation below which no optimization is attempted", 2023-11-16T18:34:56.817 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-11-16T18:34:56.817 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.817 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.817 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.818 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_optimizations": { 2023-11-16T18:34:56.818 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_optimizations", 2023-11-16T18:34:56.818 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:56.819 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.819 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.819 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:56.819 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.837 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.837 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.837 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-11-16T18:34:56.838 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.838 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.838 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.838 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:56.839 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:56.839 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.839 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:56.840 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephadm", 2023-11-16T18:34:56.840 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:56.840 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:56.840 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:56.841 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_down_multiplier": { 2023-11-16T18:34:56.841 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_down_multiplier", 2023-11-16T18:34:56.841 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:56.841 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.842 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.842 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2023-11-16T18:34:56.842 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.842 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.843 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.843 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-11-16T18:34:56.844 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.844 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.844 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.844 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.844 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_refresh_rate": { 2023-11-16T18:34:56.844 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_refresh_rate", 2023-11-16T18:34:56.844 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:56.844 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.845 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.845 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "20", 2023-11-16T18:34:56.845 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.845 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.845 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.845 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-11-16T18:34:56.845 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.846 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.846 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.846 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.846 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_starting_port": { 2023-11-16T18:34:56.846 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_starting_port", 2023-11-16T18:34:56.846 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:56.846 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.847 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.847 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4721", 2023-11-16T18:34:56.847 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.847 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.847 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-11-16T18:34:56.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.849 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.849 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.850 INFO:teuthology.orchestra.run.smithi002.stdout: "alertmanager_web_password": { 2023-11-16T18:34:56.850 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alertmanager_web_password", 2023-11-16T18:34:56.850 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.850 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.850 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.850 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:56.850 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.850 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.851 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.851 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Alertmanager web password", 2023-11-16T18:34:56.851 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.851 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.851 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.851 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.851 INFO:teuthology.orchestra.run.smithi002.stdout: "alertmanager_web_user": { 2023-11-16T18:34:56.852 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alertmanager_web_user", 2023-11-16T18:34:56.852 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.852 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.852 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.852 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:56.852 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.852 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.886 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.906 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Alertmanager web user", 2023-11-16T18:34:56.924 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.926 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.931 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.931 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.931 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_ptrace": { 2023-11-16T18:34:56.931 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_ptrace", 2023-11-16T18:34:56.931 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:56.931 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.932 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.932 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:56.932 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.933 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.933 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.933 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-11-16T18:34:56.933 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-11-16T18:34:56.933 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.934 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.934 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.934 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_interval": { 2023-11-16T18:34:56.934 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_interval", 2023-11-16T18:34:56.934 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:56.934 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.934 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.934 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:34:56.935 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.935 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.935 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.935 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to autotune daemon memory", 2023-11-16T18:34:56.935 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.935 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.935 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.935 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.936 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { 2023-11-16T18:34:56.936 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-11-16T18:34:56.936 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:56.936 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.936 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.936 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.7", 2023-11-16T18:34:56.936 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.936 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.937 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.937 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-11-16T18:34:56.937 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.937 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.937 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.937 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.937 INFO:teuthology.orchestra.run.smithi002.stdout: "cgroups_split": { 2023-11-16T18:34:56.938 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cgroups_split", 2023-11-16T18:34:56.938 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:56.938 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.938 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.938 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:56.938 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.938 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.939 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.939 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-11-16T18:34:56.939 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.939 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.939 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.939 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.939 INFO:teuthology.orchestra.run.smithi002.stdout: "config_checks_enabled": { 2023-11-16T18:34:56.939 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_checks_enabled", 2023-11-16T18:34:56.940 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:56.958 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.958 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.958 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:56.958 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.959 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.959 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.959 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-11-16T18:34:56.959 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.959 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.959 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.960 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.960 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-11-16T18:34:56.960 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_dashboard", 2023-11-16T18:34:56.960 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:56.960 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.960 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.960 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:56.960 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.961 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.961 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.961 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-11-16T18:34:56.961 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.961 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.961 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.961 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.962 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_alertmanager": { 2023-11-16T18:34:56.962 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_alertmanager", 2023-11-16T18:34:56.962 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.962 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.962 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.962 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-11-16T18:34:56.962 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.962 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.963 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.963 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T18:34:56.963 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.963 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.963 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.963 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.963 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_base": { 2023-11-16T18:34:56.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_base", 2023-11-16T18:34:56.964 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.964 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.964 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:56.964 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph", 2023-11-16T18:34:56.964 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.964 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.964 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.965 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Container image name, without the tag", 2023-11-16T18:34:56.965 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.966 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.966 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.966 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.966 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_elasticsearch": { 2023-11-16T18:34:56.966 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_elasticsearch", 2023-11-16T18:34:56.966 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.966 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.967 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.967 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-11-16T18:34:56.967 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.967 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.967 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.967 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "elasticsearch container image", 2023-11-16T18:34:56.967 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.968 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.968 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.969 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.969 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_grafana": { 2023-11-16T18:34:56.969 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_grafana", 2023-11-16T18:34:56.969 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.969 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.969 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.969 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-11-16T18:34:56.970 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.970 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.970 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.970 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T18:34:56.970 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.970 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.970 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.971 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.971 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_haproxy": { 2023-11-16T18:34:56.971 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_haproxy", 2023-11-16T18:34:56.971 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.971 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.971 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.972 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-11-16T18:34:56.972 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.972 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.972 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.972 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "HAproxy container image", 2023-11-16T18:34:56.972 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.972 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.972 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.973 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.973 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_agent": { 2023-11-16T18:34:56.973 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_agent", 2023-11-16T18:34:56.973 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.973 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.973 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.973 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-11-16T18:34:56.974 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.974 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.974 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.974 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger agent container image", 2023-11-16T18:34:56.974 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.974 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.974 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.975 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.975 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_collector": { 2023-11-16T18:34:56.975 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", 2023-11-16T18:34:56.975 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.975 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.975 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.975 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-11-16T18:34:56.976 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.976 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.976 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.976 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger collector container image", 2023-11-16T18:34:56.976 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.976 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.976 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.977 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.977 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_query": { 2023-11-16T18:34:56.977 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_query", 2023-11-16T18:34:56.977 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.977 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.977 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.977 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-11-16T18:34:56.977 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.978 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.978 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.978 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger query container image", 2023-11-16T18:34:56.978 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.978 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.978 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.978 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.978 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_keepalived": { 2023-11-16T18:34:56.979 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_keepalived", 2023-11-16T18:34:56.979 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.979 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.979 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.979 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-11-16T18:34:56.979 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.979 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.980 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.980 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Keepalived container image", 2023-11-16T18:34:56.980 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.980 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.980 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.980 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.980 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_loki": { 2023-11-16T18:34:56.981 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_loki", 2023-11-16T18:34:56.981 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.981 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.981 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.981 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-11-16T18:34:56.981 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.981 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.981 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.982 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Loki container image", 2023-11-16T18:34:56.982 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.982 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.982 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.982 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.982 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_node_exporter": { 2023-11-16T18:34:56.982 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_node_exporter", 2023-11-16T18:34:56.982 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.983 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.983 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.983 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-11-16T18:34:56.983 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.983 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.983 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.983 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T18:34:56.983 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.984 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.984 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.984 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.984 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_prometheus": { 2023-11-16T18:34:56.984 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_prometheus", 2023-11-16T18:34:56.984 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.984 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.984 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.985 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-11-16T18:34:56.985 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.985 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.985 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.985 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T18:34:56.985 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.985 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.985 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.986 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.986 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_promtail": { 2023-11-16T18:34:56.986 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_promtail", 2023-11-16T18:34:56.986 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.986 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.986 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.986 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-11-16T18:34:56.986 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.987 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.987 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.987 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Promtail container image", 2023-11-16T18:34:56.988 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.988 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.988 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.988 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.988 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_snmp_gateway": { 2023-11-16T18:34:56.988 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_snmp_gateway", 2023-11-16T18:34:56.988 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.989 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.989 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.989 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-11-16T18:34:56.989 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.989 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.989 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.989 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SNMP Gateway container image", 2023-11-16T18:34:56.989 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.990 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.990 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.990 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.990 INFO:teuthology.orchestra.run.smithi002.stdout: "container_init": { 2023-11-16T18:34:56.990 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_init", 2023-11-16T18:34:56.990 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:56.990 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.991 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.991 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:56.991 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.991 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.991 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.991 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Run podman/docker with `--init`", 2023-11-16T18:34:56.991 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.991 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.992 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.992 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.992 INFO:teuthology.orchestra.run.smithi002.stdout: "daemon_cache_timeout": { 2023-11-16T18:34:56.992 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "daemon_cache_timeout", 2023-11-16T18:34:56.992 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:56.992 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.992 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.993 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:34:56.993 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.993 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.993 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.993 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-11-16T18:34:56.993 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.993 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.993 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.994 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.994 INFO:teuthology.orchestra.run.smithi002.stdout: "default_cephadm_command_timeout": { 2023-11-16T18:34:56.994 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_cephadm_command_timeout", 2023-11-16T18:34:56.994 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:56.994 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.995 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "900", 2023-11-16T18:34:56.995 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.995 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.995 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.995 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-11-16T18:34:56.995 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.995 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.995 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.996 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.996 INFO:teuthology.orchestra.run.smithi002.stdout: "default_registry": { 2023-11-16T18:34:56.996 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_registry", 2023-11-16T18:34:56.996 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:56.996 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.996 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.996 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io", 2023-11-16T18:34:56.996 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.997 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.997 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.997 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-11-16T18:34:56.997 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.997 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.997 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.997 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.998 INFO:teuthology.orchestra.run.smithi002.stdout: "device_cache_timeout": { 2023-11-16T18:34:56.998 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_cache_timeout", 2023-11-16T18:34:56.998 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:56.998 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:56.998 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:56.998 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1800", 2023-11-16T18:34:56.998 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:56.998 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:56.999 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:56.999 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache device inventory", 2023-11-16T18:34:56.999 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:56.999 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:56.999 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:56.999 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:56.999 INFO:teuthology.orchestra.run.smithi002.stdout: "device_enhanced_scan": { 2023-11-16T18:34:56.999 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_enhanced_scan", 2023-11-16T18:34:57.000 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.000 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.000 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.000 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.000 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.000 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.000 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.001 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-11-16T18:34:57.001 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.001 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.001 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.001 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.001 INFO:teuthology.orchestra.run.smithi002.stdout: "facts_cache_timeout": { 2023-11-16T18:34:57.001 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "facts_cache_timeout", 2023-11-16T18:34:57.002 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:57.002 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.002 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.002 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:34:57.002 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.002 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.003 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.003 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache host facts data", 2023-11-16T18:34:57.003 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.003 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.003 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.003 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.003 INFO:teuthology.orchestra.run.smithi002.stdout: "host_check_interval": { 2023-11-16T18:34:57.003 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "host_check_interval", 2023-11-16T18:34:57.004 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:57.004 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.004 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.004 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:34:57.004 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.004 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.004 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.005 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to perform a host check", 2023-11-16T18:34:57.005 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.005 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.005 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.005 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.005 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:57.005 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:57.005 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.006 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.006 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.006 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.006 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.006 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.006 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.006 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.007 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.007 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.007 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.007 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:57.007 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:57.007 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.007 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.007 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.008 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.008 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.008 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.008 INFO:teuthology.orchestra.run.smithi002.stdout: "log_refresh_metadata": { 2023-11-16T18:34:57.008 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_refresh_metadata", 2023-11-16T18:34:57.008 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.008 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.008 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.009 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.009 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.009 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.009 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.009 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-11-16T18:34:57.009 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.009 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.010 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.010 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.010 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:57.010 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:57.010 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.010 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.010 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.010 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.011 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.011 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.011 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.011 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-11-16T18:34:57.011 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.011 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.011 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.011 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.012 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:57.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:57.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.012 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.012 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.012 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:57.012 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.012 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.013 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.013 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.013 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.013 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.013 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.013 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:57.013 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:57.013 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.014 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.014 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.014 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.014 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.014 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.014 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:57.014 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:57.015 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.015 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.015 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.015 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.015 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.015 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.015 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.016 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.016 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.016 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.016 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.016 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.016 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf": { 2023-11-16T18:34:57.016 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-11-16T18:34:57.016 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.017 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.017 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.017 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.017 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.017 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.017 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.018 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-11-16T18:34:57.018 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.018 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.018 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.018 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.018 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-11-16T18:34:57.018 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-11-16T18:34:57.019 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.019 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.019 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.019 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "*", 2023-11-16T18:34:57.019 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.019 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.019 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.020 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-11-16T18:34:57.020 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.020 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.020 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.020 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.020 INFO:teuthology.orchestra.run.smithi002.stdout: "max_count_per_host": { 2023-11-16T18:34:57.020 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_count_per_host", 2023-11-16T18:34:57.021 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.021 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.021 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.021 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:57.021 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.021 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.021 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.021 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of daemons per service per host", 2023-11-16T18:34:57.022 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.022 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.022 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.022 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.022 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { 2023-11-16T18:34:57.022 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", 2023-11-16T18:34:57.022 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.023 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.023 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.023 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:57.023 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.023 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.023 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.023 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-11-16T18:34:57.024 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.024 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.024 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.024 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.024 INFO:teuthology.orchestra.run.smithi002.stdout: "migration_current": { 2023-11-16T18:34:57.024 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "migration_current", 2023-11-16T18:34:57.024 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.024 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.025 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.025 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.025 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.025 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.025 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.025 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "internal - do not modify", 2023-11-16T18:34:57.025 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.026 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.031 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.031 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.032 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2023-11-16T18:34:57.032 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2023-11-16T18:34:57.033 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.033 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.033 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "root", 2023-11-16T18:34:57.033 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.033 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.034 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm-package", 2023-11-16T18:34:57.034 INFO:teuthology.orchestra.run.smithi002.stdout: "root" 2023-11-16T18:34:57.034 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.034 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "mode for remote execution of cephadm", 2023-11-16T18:34:57.034 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.034 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.034 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.035 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.035 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_alerts_path": { 2023-11-16T18:34:57.035 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_alerts_path", 2023-11-16T18:34:57.035 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.035 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.035 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.035 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-11-16T18:34:57.035 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.036 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.036 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.036 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-11-16T18:34:57.036 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.036 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.036 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.036 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.037 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_web_password": { 2023-11-16T18:34:57.037 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_web_password", 2023-11-16T18:34:57.037 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.037 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.037 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.037 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:57.037 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.058 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.058 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.058 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus web password", 2023-11-16T18:34:57.058 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.059 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.059 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.059 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.059 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_web_user": { 2023-11-16T18:34:57.059 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_web_user", 2023-11-16T18:34:57.059 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.059 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.059 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.060 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:57.060 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.060 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.060 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.060 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus web user", 2023-11-16T18:34:57.060 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.060 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.060 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.061 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.061 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_insecure": { 2023-11-16T18:34:57.061 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_insecure", 2023-11-16T18:34:57.061 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.061 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.061 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.061 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.061 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.062 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.062 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.062 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-11-16T18:34:57.062 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.062 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.062 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.062 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.062 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_password": { 2023-11-16T18:34:57.063 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_password", 2023-11-16T18:34:57.063 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.063 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.063 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.063 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.063 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.063 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.063 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.064 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-11-16T18:34:57.064 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.064 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.064 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.064 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.064 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_url": { 2023-11-16T18:34:57.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_url", 2023-11-16T18:34:57.064 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.065 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.065 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.065 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.065 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.065 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.065 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.065 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-11-16T18:34:57.065 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.066 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.066 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.066 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.066 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { 2023-11-16T18:34:57.066 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", 2023-11-16T18:34:57.066 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.066 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.067 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.067 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.067 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.067 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.067 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.067 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-11-16T18:34:57.067 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.068 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.068 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.068 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.068 INFO:teuthology.orchestra.run.smithi002.stdout: "secure_monitoring_stack": { 2023-11-16T18:34:57.068 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "secure_monitoring_stack", 2023-11-16T18:34:57.068 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.068 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.068 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.069 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.069 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.069 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.069 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.069 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-11-16T18:34:57.069 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.069 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.069 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.069 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.070 INFO:teuthology.orchestra.run.smithi002.stdout: "service_discovery_port": { 2023-11-16T18:34:57.070 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "service_discovery_port", 2023-11-16T18:34:57.070 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.070 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.070 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.070 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8765", 2023-11-16T18:34:57.070 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.070 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.071 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.071 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "cephadm service discovery port", 2023-11-16T18:34:57.071 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.071 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.071 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.071 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.071 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { 2023-11-16T18:34:57.071 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-11-16T18:34:57.072 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.072 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.072 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.072 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.072 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.072 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.072 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.072 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-11-16T18:34:57.073 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.073 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.073 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.073 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.073 INFO:teuthology.orchestra.run.smithi002.stdout: "use_agent": { 2023-11-16T18:34:57.073 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_agent", 2023-11-16T18:34:57.073 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.073 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.073 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.074 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.074 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.074 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.074 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.074 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-11-16T18:34:57.074 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.074 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.074 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.075 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.075 INFO:teuthology.orchestra.run.smithi002.stdout: "use_repo_digest": { 2023-11-16T18:34:57.075 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_repo_digest", 2023-11-16T18:34:57.075 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.075 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.075 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.075 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.075 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.076 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.076 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.076 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-11-16T18:34:57.076 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.076 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.076 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.076 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_failed_host_check": { 2023-11-16T18:34:57.077 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_failed_host_check", 2023-11-16T18:34:57.077 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.077 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.077 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.077 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.077 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.077 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.077 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.078 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if the host check fails", 2023-11-16T18:34:57.078 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.078 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.078 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.078 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.078 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_daemons": { 2023-11-16T18:34:57.078 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_daemons", 2023-11-16T18:34:57.078 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.079 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.079 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.079 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.079 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.079 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.079 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.079 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-11-16T18:34:57.079 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.079 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.080 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.080 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_hosts": { 2023-11-16T18:34:57.080 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_hosts", 2023-11-16T18:34:57.080 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.080 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.080 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.080 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.081 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.081 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.081 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.081 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-11-16T18:34:57.081 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.081 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.081 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.081 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:57.082 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:57.082 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.082 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:57.082 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crash", 2023-11-16T18:34:57.082 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:57.088 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:57.088 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:57.089 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:57.089 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:57.136 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.136 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.136 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.136 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.136 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.136 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.136 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.137 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.137 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.137 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.137 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.137 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:57.137 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:57.137 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.137 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.137 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.138 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.138 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.138 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.138 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:57.138 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:57.138 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.138 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.138 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.139 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.139 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.139 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.153 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.153 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.153 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.153 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.153 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.153 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.153 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:57.153 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:57.154 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.154 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.154 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.154 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:57.154 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.154 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.154 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.154 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.155 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.155 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.155 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.155 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:57.155 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:57.155 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.155 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.155 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.155 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.156 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.156 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.156 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:57.156 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:57.156 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.156 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.156 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.157 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.157 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.157 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.157 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.157 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.157 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.157 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.157 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.157 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.158 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { 2023-11-16T18:34:57.158 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-11-16T18:34:57.197 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:57.197 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.197 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.197 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "31536000", 2023-11-16T18:34:57.197 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.197 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.197 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.198 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain crashes before pruning them", 2023-11-16T18:34:57.198 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.198 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.198 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.198 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.198 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_recent_interval": { 2023-11-16T18:34:57.198 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_recent_interval", 2023-11-16T18:34:57.198 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:57.199 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.199 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.199 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1209600", 2023-11-16T18:34:57.199 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.199 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.199 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.199 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "time interval in which to warn about recent crashes", 2023-11-16T18:34:57.199 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.200 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.200 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.200 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:57.200 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:57.200 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.200 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:57.200 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dashboard", 2023-11-16T18:34:57.200 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:57.200 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:57.201 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:57.201 INFO:teuthology.orchestra.run.smithi002.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-11-16T18:34:57.201 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-11-16T18:34:57.201 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.201 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.201 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.201 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:57.202 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.202 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.202 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.202 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.202 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.202 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.202 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.202 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.203 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_HOST": { 2023-11-16T18:34:57.203 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_HOST", 2023-11-16T18:34:57.203 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.203 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.203 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.203 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.203 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.203 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.204 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.204 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.204 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.204 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.204 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.204 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.204 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-11-16T18:34:57.204 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-11-16T18:34:57.205 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.205 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.205 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.205 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.205 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.205 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.205 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.205 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.206 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.206 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.206 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.206 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.206 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_ENABLED": { 2023-11-16T18:34:57.206 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_ENABLED", 2023-11-16T18:34:57.206 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.206 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.207 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.207 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.207 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.207 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.207 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.207 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.207 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.207 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.208 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.208 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-11-16T18:34:57.208 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-11-16T18:34:57.208 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.208 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.208 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.209 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.209 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.209 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.209 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.209 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.209 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.209 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.209 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.210 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.210 INFO:teuthology.orchestra.run.smithi002.stdout: "ENABLE_BROWSABLE_API": { 2023-11-16T18:34:57.210 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ENABLE_BROWSABLE_API", 2023-11-16T18:34:57.210 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.210 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.210 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.210 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.210 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.211 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.211 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.211 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.211 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.211 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.211 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.211 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.211 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-11-16T18:34:57.211 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-11-16T18:34:57.212 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.212 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.212 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.212 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.212 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.212 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.212 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.213 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.213 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.213 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.213 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.213 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.213 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-11-16T18:34:57.213 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-11-16T18:34:57.213 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.214 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.214 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.214 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.214 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.214 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.214 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.214 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.214 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.215 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.215 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.215 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.215 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-11-16T18:34:57.215 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-11-16T18:34:57.215 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.215 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.215 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.215 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.216 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.216 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.216 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.216 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.216 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.216 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.216 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.217 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.217 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-11-16T18:34:57.217 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-11-16T18:34:57.217 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.217 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.217 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.217 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.217 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.218 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.218 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.218 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.218 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.218 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.218 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.218 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.219 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_NFS": { 2023-11-16T18:34:57.219 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-11-16T18:34:57.219 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.219 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.219 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.219 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.219 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.219 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.220 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.220 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.220 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.220 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.220 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.220 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.220 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RBD": { 2023-11-16T18:34:57.226 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-11-16T18:34:57.227 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.227 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.227 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.227 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.227 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.227 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.252 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.319 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.320 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.320 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.320 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.320 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.320 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-11-16T18:34:57.320 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-11-16T18:34:57.320 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.321 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.321 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.321 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.321 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.321 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.321 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.321 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.321 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.321 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.322 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.322 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.322 INFO:teuthology.orchestra.run.smithi002.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-11-16T18:34:57.322 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-11-16T18:34:57.322 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.322 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.322 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.322 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.323 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.323 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.323 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.323 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.323 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.323 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.323 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.324 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.392 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_PASSWORD": { 2023-11-16T18:34:57.392 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_PASSWORD", 2023-11-16T18:34:57.392 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.393 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.393 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.393 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:57.393 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.393 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.393 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.393 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.393 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.394 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.394 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.394 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.394 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-11-16T18:34:57.394 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-11-16T18:34:57.394 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.394 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.395 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.395 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.395 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.395 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.395 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.395 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.395 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.395 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.395 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.396 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.396 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_URL": { 2023-11-16T18:34:57.396 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_URL", 2023-11-16T18:34:57.396 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.396 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.396 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.396 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.396 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.397 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.397 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.397 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.397 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.397 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.397 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.397 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.397 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_USERNAME": { 2023-11-16T18:34:57.398 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_USERNAME", 2023-11-16T18:34:57.398 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.398 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.398 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.398 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:57.398 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.398 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.398 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.399 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.399 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.399 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.399 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.399 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.399 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-11-16T18:34:57.399 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-11-16T18:34:57.399 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.400 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.400 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.400 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.400 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.400 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.400 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.400 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.400 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.401 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.401 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.401 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.401 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-11-16T18:34:57.401 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-11-16T18:34:57.401 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.401 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.401 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.401 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.402 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.402 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.402 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.402 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.402 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.402 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.402 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.402 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.403 INFO:teuthology.orchestra.run.smithi002.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-11-16T18:34:57.403 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-11-16T18:34:57.403 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.403 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.403 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.403 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.403 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.403 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.404 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.404 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.404 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.404 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.404 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.404 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.404 INFO:teuthology.orchestra.run.smithi002.stdout: "ISSUE_TRACKER_API_KEY": { 2023-11-16T18:34:57.404 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-11-16T18:34:57.405 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.405 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.405 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.405 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.405 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.405 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.405 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.405 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.405 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.406 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.406 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.406 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.406 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { 2023-11-16T18:34:57.406 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", 2023-11-16T18:34:57.406 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.406 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.406 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.407 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.407 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.407 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.407 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.407 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.407 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.408 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.408 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.408 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.408 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-11-16T18:34:57.408 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-11-16T18:34:57.409 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.409 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.409 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.409 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.409 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.409 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.409 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.409 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.410 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.410 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.410 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.410 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.410 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-11-16T18:34:57.410 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-11-16T18:34:57.410 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.410 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.411 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.411 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.411 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.411 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.411 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.412 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.412 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.412 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.412 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.412 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.412 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-11-16T18:34:57.412 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-11-16T18:34:57.413 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.413 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.413 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.413 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.413 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.413 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.413 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.413 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.414 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.414 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.414 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.414 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.414 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-11-16T18:34:57.414 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-11-16T18:34:57.414 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.414 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.415 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.415 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.415 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.415 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.415 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.415 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.415 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.415 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.416 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.416 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.416 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-11-16T18:34:57.416 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-11-16T18:34:57.416 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.416 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.416 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.416 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.417 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.417 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.417 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.417 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.417 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.417 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.417 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-11-16T18:34:57.418 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-11-16T18:34:57.418 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.418 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.418 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.418 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.418 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.418 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.418 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.419 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.419 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.419 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.419 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.419 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.419 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-11-16T18:34:57.419 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-11-16T18:34:57.420 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.420 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.420 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.420 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.420 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.420 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.420 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.420 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.421 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.421 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.421 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.421 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.421 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-11-16T18:34:57.421 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-11-16T18:34:57.421 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.421 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.422 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.422 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.422 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.422 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.422 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.422 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.422 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.422 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.423 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.423 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.423 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_ENABLED": { 2023-11-16T18:34:57.423 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_ENABLED", 2023-11-16T18:34:57.423 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.423 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.423 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.423 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.424 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.424 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.424 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.424 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.424 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.424 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.424 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.424 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.424 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-11-16T18:34:57.425 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-11-16T18:34:57.425 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.425 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.425 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.425 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-11-16T18:34:57.425 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.425 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.425 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.426 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.426 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.426 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.426 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.426 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.426 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-11-16T18:34:57.426 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-11-16T18:34:57.426 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.427 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.427 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.427 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:57.427 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.427 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.427 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.427 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.427 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.428 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.428 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.428 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.428 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-11-16T18:34:57.428 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-11-16T18:34:57.428 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.428 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.428 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.428 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8", 2023-11-16T18:34:57.429 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.429 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.429 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.429 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.429 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.429 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.429 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.429 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.430 INFO:teuthology.orchestra.run.smithi002.stdout: "REST_REQUESTS_TIMEOUT": { 2023-11-16T18:34:57.430 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-11-16T18:34:57.430 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.430 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.430 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.430 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "45", 2023-11-16T18:34:57.430 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.430 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.431 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.431 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.431 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.431 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.431 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.431 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ACCESS_KEY": { 2023-11-16T18:34:57.432 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ACCESS_KEY", 2023-11-16T18:34:57.432 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.432 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.432 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.432 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.432 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.432 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.432 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.432 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.433 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.433 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.433 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.433 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.433 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-11-16T18:34:57.433 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-11-16T18:34:57.433 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.433 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.434 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.434 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:57.434 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.434 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.434 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.434 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.434 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.434 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.435 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.435 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.435 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SECRET_KEY": { 2023-11-16T18:34:57.435 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SECRET_KEY", 2023-11-16T18:34:57.435 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.435 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.435 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.435 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.436 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.436 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.436 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.436 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.436 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.436 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.436 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.436 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.436 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SSL_VERIFY": { 2023-11-16T18:34:57.437 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SSL_VERIFY", 2023-11-16T18:34:57.437 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.437 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.437 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.437 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.437 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.437 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.438 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.438 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.438 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.438 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.438 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.438 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-11-16T18:34:57.438 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-11-16T18:34:57.438 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.439 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.439 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.439 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:34:57.439 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.439 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.439 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.439 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.439 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.440 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.440 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.440 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.440 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-11-16T18:34:57.440 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-11-16T18:34:57.440 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.440 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.440 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.441 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:57.441 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.441 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.441 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.441 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.441 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.441 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.441 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.441 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.442 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-11-16T18:34:57.442 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-11-16T18:34:57.442 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.442 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.442 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.442 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T18:34:57.442 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.443 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.443 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.443 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.443 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.443 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.443 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.443 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.443 INFO:teuthology.orchestra.run.smithi002.stdout: "cross_origin_url": { 2023-11-16T18:34:57.443 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cross_origin_url", 2023-11-16T18:34:57.444 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.444 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.444 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.444 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.444 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.444 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.444 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.444 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.445 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.445 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.445 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.445 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.445 INFO:teuthology.orchestra.run.smithi002.stdout: "crt_file": { 2023-11-16T18:34:57.445 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crt_file", 2023-11-16T18:34:57.445 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.445 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.446 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.446 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.446 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.446 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.446 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.446 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.446 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.446 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.447 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.447 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.447 INFO:teuthology.orchestra.run.smithi002.stdout: "debug": { 2023-11-16T18:34:57.447 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "debug", 2023-11-16T18:34:57.447 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.447 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.447 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.447 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.448 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.448 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.448 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.448 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable/disable debug options", 2023-11-16T18:34:57.448 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.448 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.448 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.448 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.449 INFO:teuthology.orchestra.run.smithi002.stdout: "jwt_token_ttl": { 2023-11-16T18:34:57.449 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "jwt_token_ttl", 2023-11-16T18:34:57.449 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.449 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.449 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.449 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "28800", 2023-11-16T18:34:57.449 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.449 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.449 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.450 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.450 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.450 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.450 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.450 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.450 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2023-11-16T18:34:57.450 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2023-11-16T18:34:57.450 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.451 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.451 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.451 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.451 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.451 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.451 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.451 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.457 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.457 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.457 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.457 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:57.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:57.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.457 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.458 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.458 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.458 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.458 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.458 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.458 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.458 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.458 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.461 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:57.461 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:57.462 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.462 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.462 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.462 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.462 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.463 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.463 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:57.463 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:57.463 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.463 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.463 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.463 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.463 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.464 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.464 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.464 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.464 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.464 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.464 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.464 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.464 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:57.464 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:57.465 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.465 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.465 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.465 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:57.465 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.465 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.465 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.466 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.466 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.466 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.466 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.466 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:57.466 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:57.466 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.467 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.467 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.467 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.467 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.467 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.467 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:57.467 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:57.467 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.467 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.468 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.468 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.468 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.468 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.468 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.468 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.468 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.468 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.469 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.469 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.469 INFO:teuthology.orchestra.run.smithi002.stdout: "motd": { 2023-11-16T18:34:57.469 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "motd", 2023-11-16T18:34:57.469 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.469 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.469 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.469 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.470 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.470 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.470 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.470 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "The message of the day", 2023-11-16T18:34:57.470 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.470 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.470 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.471 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.471 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect_resolve_ip_addr": { 2023-11-16T18:34:57.471 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "redirect_resolve_ip_addr", 2023-11-16T18:34:57.471 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.471 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.471 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.471 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.471 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.472 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.472 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.472 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.472 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.472 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.472 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.472 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.472 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T18:34:57.473 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T18:34:57.478 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.505 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.506 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.506 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-11-16T18:34:57.506 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.506 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.506 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.506 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.506 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.506 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.507 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.507 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.507 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T18:34:57.507 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T18:34:57.507 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.507 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.507 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.508 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8080", 2023-11-16T18:34:57.508 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.508 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.508 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.508 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.508 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.508 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.508 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.509 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.509 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2023-11-16T18:34:57.509 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2023-11-16T18:34:57.509 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.509 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.509 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.509 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.509 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.510 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.510 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.510 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.510 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.510 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.510 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.510 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.510 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl_server_port": { 2023-11-16T18:34:57.510 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", 2023-11-16T18:34:57.511 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.511 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.511 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.511 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8443", 2023-11-16T18:34:57.511 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.511 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.511 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.511 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.512 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.512 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.512 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.512 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.512 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2023-11-16T18:34:57.512 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2023-11-16T18:34:57.512 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.512 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.513 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.513 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "redirect", 2023-11-16T18:34:57.513 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.513 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.513 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.513 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.513 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect" 2023-11-16T18:34:57.513 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.513 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.514 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.514 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.514 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.514 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.514 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2023-11-16T18:34:57.514 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2023-11-16T18:34:57.514 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.514 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.515 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.515 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2023-11-16T18:34:57.515 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-11-16T18:34:57.515 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2023-11-16T18:34:57.515 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.515 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.515 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.515 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.516 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.516 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.516 INFO:teuthology.orchestra.run.smithi002.stdout: "url_prefix": { 2023-11-16T18:34:57.516 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url_prefix", 2023-11-16T18:34:57.516 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.516 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.516 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.516 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.517 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.517 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.517 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.517 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.517 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.517 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.517 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.517 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:57.517 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:57.518 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.518 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:57.518 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", 2023-11-16T18:34:57.518 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:57.518 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:57.518 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:57.518 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_monitoring": { 2023-11-16T18:34:57.518 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_monitoring", 2023-11-16T18:34:57.519 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.519 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.519 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.519 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.519 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.519 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.520 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "monitor device health metrics", 2023-11-16T18:34:57.520 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.520 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.520 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.520 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.520 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:57.520 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:57.520 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.521 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.521 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.521 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.521 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.521 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.521 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.521 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.521 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.521 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.522 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.522 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:57.522 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:57.522 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.522 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.522 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.522 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.523 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.523 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.523 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:57.523 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:57.523 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.523 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.523 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.523 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.523 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.524 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.524 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.524 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.524 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.524 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.524 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.524 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.524 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:57.525 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:57.525 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.525 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.525 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.525 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:57.525 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.525 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.525 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.526 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.526 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.526 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.526 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.526 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:57.526 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:57.526 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.526 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.527 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.527 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.527 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.527 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.527 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:57.527 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:57.527 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.527 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.527 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.528 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.528 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.528 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.528 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.528 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.528 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.528 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.528 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.529 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.529 INFO:teuthology.orchestra.run.smithi002.stdout: "mark_out_threshold": { 2023-11-16T18:34:57.529 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mark_out_threshold", 2023-11-16T18:34:57.529 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:57.529 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.529 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.529 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2419200", 2023-11-16T18:34:57.529 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.530 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.530 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.530 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-11-16T18:34:57.530 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.530 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.530 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.530 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.530 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-11-16T18:34:57.531 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_name", 2023-11-16T18:34:57.531 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.531 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.531 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.531 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "device_health_metrics", 2023-11-16T18:34:57.531 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.531 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.531 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.531 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name of pool in which to store device health metrics", 2023-11-16T18:34:57.532 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.532 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.532 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.532 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.532 INFO:teuthology.orchestra.run.smithi002.stdout: "retention_period": { 2023-11-16T18:34:57.532 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retention_period", 2023-11-16T18:34:57.532 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:57.532 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.533 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.533 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15552000", 2023-11-16T18:34:57.533 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.533 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.533 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.533 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain device health metrics", 2023-11-16T18:34:57.533 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.533 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.533 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.534 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.534 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_frequency": { 2023-11-16T18:34:57.534 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_frequency", 2023-11-16T18:34:57.534 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:57.534 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.534 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.534 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2023-11-16T18:34:57.534 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.535 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.535 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.535 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to scrape device health metrics", 2023-11-16T18:34:57.535 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.535 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.535 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.535 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.535 INFO:teuthology.orchestra.run.smithi002.stdout: "self_heal": { 2023-11-16T18:34:57.536 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "self_heal", 2023-11-16T18:34:57.536 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.536 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.536 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.536 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:57.536 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.536 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.536 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-11-16T18:34:57.537 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.537 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.537 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.537 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.537 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:34:57.537 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:34:57.537 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:57.537 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.538 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.538 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:34:57.538 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.538 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.538 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.538 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and check device health", 2023-11-16T18:34:57.538 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.538 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.538 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.539 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.539 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_threshold": { 2023-11-16T18:34:57.539 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_threshold", 2023-11-16T18:34:57.539 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:57.539 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.539 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.539 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7257600", 2023-11-16T18:34:57.539 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.540 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.540 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.540 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-11-16T18:34:57.540 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.540 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.540 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.540 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:57.540 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:57.540 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.541 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:57.541 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "diskprediction_local", 2023-11-16T18:34:57.541 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:57.541 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:57.546 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:57.546 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:57.547 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:57.644 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.644 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.644 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.644 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.644 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.645 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.645 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.645 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.645 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.645 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.645 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.645 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:57.645 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:57.646 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.646 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.646 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.646 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.646 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.646 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.646 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:57.646 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:57.647 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.647 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.647 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.647 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.651 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.651 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.651 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.652 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.652 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.653 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.653 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.654 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.654 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:57.655 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:57.655 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.657 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.657 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:57.658 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.659 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.659 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.660 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.660 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.661 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.662 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.662 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:57.666 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:57.668 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.668 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.669 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.670 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.670 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.670 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.670 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:57.671 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:57.671 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.671 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.671 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.671 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.671 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.671 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.671 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.672 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.672 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.672 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.672 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.672 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.672 INFO:teuthology.orchestra.run.smithi002.stdout: "predict_interval": { 2023-11-16T18:34:57.672 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predict_interval", 2023-11-16T18:34:57.672 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.673 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.673 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.673 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2023-11-16T18:34:57.673 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.673 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.673 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.673 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.674 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.674 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.674 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.674 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.674 INFO:teuthology.orchestra.run.smithi002.stdout: "predictor_model": { 2023-11-16T18:34:57.674 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predictor_model", 2023-11-16T18:34:57.674 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.674 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.675 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.675 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "prophetstor", 2023-11-16T18:34:57.675 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.675 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.675 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.675 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.675 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.675 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.676 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.676 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.676 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:34:57.676 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:34:57.676 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.676 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.676 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.677 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:34:57.677 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.677 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.677 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.677 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.677 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.677 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.677 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.678 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:57.678 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:57.678 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.678 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:57.678 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "influx", 2023-11-16T18:34:57.678 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": false, 2023-11-16T18:34:57.678 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "influxdb python module not found", 2023-11-16T18:34:57.678 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:57.681 INFO:teuthology.orchestra.run.smithi002.stdout: "batch_size": { 2023-11-16T18:34:57.681 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "batch_size", 2023-11-16T18:34:57.681 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.681 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.682 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.682 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5000", 2023-11-16T18:34:57.682 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.683 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.683 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.683 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-11-16T18:34:57.683 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.684 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "database": { 2023-11-16T18:34:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "database", 2023-11-16T18:34:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "ceph", 2023-11-16T18:34:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-11-16T18:34:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": { 2023-11-16T18:34:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "hostname", 2023-11-16T18:34:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.712 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.712 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.712 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server hostname", 2023-11-16T18:34:57.712 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.712 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.712 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.712 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.712 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:34:57.713 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:34:57.733 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:57.733 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.734 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.734 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "30", 2023-11-16T18:34:57.734 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "5", 2023-11-16T18:34:57.734 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.734 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.734 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-11-16T18:34:57.735 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.735 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.735 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.735 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.735 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:57.735 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:57.735 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.735 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.736 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.736 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.736 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.736 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.736 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.736 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.737 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.737 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.737 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.737 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:57.737 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:57.737 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.737 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.737 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.738 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.738 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.738 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.837 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:57.837 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:57.837 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.837 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.837 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.837 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.838 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.838 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.838 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.838 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.838 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.838 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.838 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.838 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.839 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:57.839 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:57.839 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.839 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.839 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.839 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:57.839 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.839 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.840 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.840 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.840 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.840 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.840 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.840 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:57.840 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:57.840 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.841 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.841 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.841 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.841 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.841 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.841 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:57.841 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:57.841 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.841 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.842 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.842 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.842 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.842 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.842 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.842 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.842 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.842 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.843 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.843 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.843 INFO:teuthology.orchestra.run.smithi002.stdout: "password": { 2023-11-16T18:34:57.843 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "password", 2023-11-16T18:34:57.843 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.843 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.843 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.843 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.844 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.844 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.844 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.844 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "password of InfluxDB server user", 2023-11-16T18:34:57.844 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.844 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.844 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.844 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.845 INFO:teuthology.orchestra.run.smithi002.stdout: "port": { 2023-11-16T18:34:57.845 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "port", 2023-11-16T18:34:57.845 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.845 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.845 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.845 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8086", 2023-11-16T18:34:57.845 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.845 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.846 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.846 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server port", 2023-11-16T18:34:57.846 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.846 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.846 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.846 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.846 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2023-11-16T18:34:57.846 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2023-11-16T18:34:57.847 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.847 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.847 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.847 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "false", 2023-11-16T18:34:57.847 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.847 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.847 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T18:34:57.848 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.848 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.848 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.848 INFO:teuthology.orchestra.run.smithi002.stdout: "threads": { 2023-11-16T18:34:57.848 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threads", 2023-11-16T18:34:57.848 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:57.848 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.848 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.849 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T18:34:57.849 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T18:34:57.849 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "32", 2023-11-16T18:34:57.849 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.849 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-11-16T18:34:57.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.849 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.850 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.925 INFO:teuthology.orchestra.run.smithi002.stdout: "username": { 2023-11-16T18:34:57.926 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "username", 2023-11-16T18:34:57.926 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.926 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.926 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.927 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.927 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.927 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.928 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.928 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "username of InfluxDB server user", 2023-11-16T18:34:57.928 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.928 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.929 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.929 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.929 INFO:teuthology.orchestra.run.smithi002.stdout: "verify_ssl": { 2023-11-16T18:34:57.929 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "verify_ssl", 2023-11-16T18:34:57.930 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.930 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.930 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:57.930 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "true", 2023-11-16T18:34:57.931 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.931 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.931 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.931 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T18:34:57.932 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.932 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.932 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.932 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:57.933 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:57.933 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.933 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:57.933 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "insights", 2023-11-16T18:34:57.934 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:57.934 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:57.934 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:57.934 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:57.935 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:57.935 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.935 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.935 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.936 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:57.936 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.936 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.936 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.937 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.937 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.937 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.938 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:57.939 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:57.939 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:57.939 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:57.946 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.946 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.947 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.947 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.947 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.947 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:57.948 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:57.948 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:57.948 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.948 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.949 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:57.949 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.949 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.949 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:57.950 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:57.950 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:57.950 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:57.950 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:57.951 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:57.951 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:57.951 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:57.951 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:57.952 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:57.952 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:57.988 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:57.989 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:57.989 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:57.989 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:57.989 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:57.989 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:57.989 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:57.989 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.017 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.017 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.018 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.018 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.018 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.020 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.021 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.048 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.048 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.048 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.049 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.049 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.049 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.049 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.049 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.049 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.049 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.050 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.050 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.050 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.050 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.050 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.050 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.050 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.050 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.051 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "iostat", 2023-11-16T18:34:58.051 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.051 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.051 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.051 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.051 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.051 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.051 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.052 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.052 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.052 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.052 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.052 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.052 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.052 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.052 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.053 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.053 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.053 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.053 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.053 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.053 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.053 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.054 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.054 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.054 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.054 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.054 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.054 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.054 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.054 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.055 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.055 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.055 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.055 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.055 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.055 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.055 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.055 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.056 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.056 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.056 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.056 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.056 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.056 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.056 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.056 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.057 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.057 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.057 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.057 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.057 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.057 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.057 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.057 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.058 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.058 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.058 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.058 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.058 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.058 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.058 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.058 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.059 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.059 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.059 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.059 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.059 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.059 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.059 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.059 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.060 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.060 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.060 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.060 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.060 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.060 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "k8sevents", 2023-11-16T18:34:58.060 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.060 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.061 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.061 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_event_retention_days": { 2023-11-16T18:34:58.061 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ceph_event_retention_days", 2023-11-16T18:34:58.061 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.061 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.061 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.061 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7", 2023-11-16T18:34:58.061 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.062 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.062 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.062 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Days to hold ceph event information within local cache", 2023-11-16T18:34:58.062 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.062 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.062 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.062 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.062 INFO:teuthology.orchestra.run.smithi002.stdout: "config_check_secs": { 2023-11-16T18:34:58.063 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_check_secs", 2023-11-16T18:34:58.063 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.063 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.063 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.063 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:58.063 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "10", 2023-11-16T18:34:58.063 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.063 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.064 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-11-16T18:34:58.064 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.064 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.064 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.064 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.064 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.064 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.065 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.065 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.065 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.065 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.065 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.065 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.065 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.065 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.066 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.066 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.066 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.066 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.066 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.066 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.066 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.066 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.067 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.067 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.067 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.067 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.067 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.067 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.067 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.067 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.068 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.068 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.068 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.068 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.068 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.068 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.068 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.068 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.069 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.069 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.069 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.069 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.069 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.069 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.069 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.069 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.070 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.070 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.070 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.070 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.070 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.070 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.070 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.070 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.071 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.073 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.074 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.074 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.074 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.074 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.074 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.074 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.074 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.075 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.075 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.075 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.075 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.075 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.075 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.075 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.075 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.076 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.076 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.076 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.076 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.076 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "localpool", 2023-11-16T18:34:58.076 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.076 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.077 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.077 INFO:teuthology.orchestra.run.smithi002.stdout: "failure_domain": { 2023-11-16T18:34:58.077 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "failure_domain", 2023-11-16T18:34:58.077 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.077 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.077 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.077 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "host", 2023-11-16T18:34:58.077 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.078 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.078 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.078 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "failure domain for any created local pool", 2023-11-16T18:34:58.078 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-11-16T18:34:58.078 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.078 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.078 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.078 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.079 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.079 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.079 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.079 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.079 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.079 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.079 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.079 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.080 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.080 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.080 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.080 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.080 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.080 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.080 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.080 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.081 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.081 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.081 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.081 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.081 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.081 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.081 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.081 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.081 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.082 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.082 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.082 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.082 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.082 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.082 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.082 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.083 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.083 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.083 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.083 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.083 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.083 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.083 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.084 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.084 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.084 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.084 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.084 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.084 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.084 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.084 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.084 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.085 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.085 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.085 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.085 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.085 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.085 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.085 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.085 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.086 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.086 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.086 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.086 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.086 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.086 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.086 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.086 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.087 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.087 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.087 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.087 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.087 INFO:teuthology.orchestra.run.smithi002.stdout: "min_size": { 2023-11-16T18:34:58.087 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_size", 2023-11-16T18:34:58.087 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.087 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.088 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.088 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.088 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.088 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.088 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.088 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default min_size for any created local pool", 2023-11-16T18:34:58.088 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-11-16T18:34:58.088 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.089 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.089 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.089 INFO:teuthology.orchestra.run.smithi002.stdout: "num_rep": { 2023-11-16T18:34:58.089 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "num_rep", 2023-11-16T18:34:58.089 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.089 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.089 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.089 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3", 2023-11-16T18:34:58.090 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.090 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.090 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.090 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default replica count for any created local pool", 2023-11-16T18:34:58.090 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.090 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.090 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.090 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.091 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num": { 2023-11-16T18:34:58.091 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_num", 2023-11-16T18:34:58.091 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.091 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.091 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.091 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "128", 2023-11-16T18:34:58.091 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.091 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.092 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.092 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default pg_num for any created local pool", 2023-11-16T18:34:58.092 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.092 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.092 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.092 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.092 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-11-16T18:34:58.092 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prefix", 2023-11-16T18:34:58.093 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.093 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.093 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.093 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.093 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.093 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.093 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.093 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name prefix for any created local pool", 2023-11-16T18:34:58.093 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.094 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.094 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.094 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.094 INFO:teuthology.orchestra.run.smithi002.stdout: "subtree": { 2023-11-16T18:34:58.094 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subtree", 2023-11-16T18:34:58.094 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.094 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.094 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.095 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "rack", 2023-11-16T18:34:58.095 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.095 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.095 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.095 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "CRUSH level for which to create a local pool", 2023-11-16T18:34:58.095 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-11-16T18:34:58.095 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.095 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.096 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.096 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.096 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.096 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.096 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mds_autoscaler", 2023-11-16T18:34:58.096 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.096 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.097 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.097 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.097 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.097 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.097 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.097 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.097 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.097 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.098 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.098 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.098 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.098 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.098 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.098 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.098 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.098 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.099 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.099 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.099 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.099 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.099 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.099 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.120 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.120 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.120 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.120 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.120 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.120 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.121 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.121 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.121 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.121 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.121 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.121 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.121 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.121 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.122 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.122 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.122 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.122 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.122 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.122 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.122 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.123 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.123 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.123 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.123 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.123 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.123 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.123 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.123 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.124 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.124 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.124 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.124 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.124 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.124 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.124 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.124 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.125 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.125 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.125 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.125 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.125 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.125 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.144 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.146 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.146 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.146 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.147 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.147 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.147 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.147 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.147 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirroring", 2023-11-16T18:34:58.147 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.147 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.147 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.148 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.148 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.148 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.166 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.166 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.166 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.166 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.166 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.167 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.167 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.167 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.167 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.167 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.167 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.167 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.168 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.168 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.168 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.169 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.170 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.170 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.170 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.171 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.171 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.172 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.172 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.173 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.173 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.173 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.174 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.174 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.174 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.174 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.174 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.174 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.174 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.175 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.175 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.175 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.175 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.175 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.176 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.176 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.176 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.176 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.176 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.176 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.176 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.177 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.177 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.177 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.177 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.177 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.177 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.177 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.177 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.178 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.178 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.179 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.179 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.179 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.179 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.179 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.179 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.179 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.179 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.180 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.180 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.180 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.180 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.180 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.180 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "nfs", 2023-11-16T18:34:58.180 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.180 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.180 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.181 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.181 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.181 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.181 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.181 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.181 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.181 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.181 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.182 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.182 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.182 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.182 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.182 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.182 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.182 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.182 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.183 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.183 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.183 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.183 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.183 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.183 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.183 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.183 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.184 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.184 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.184 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.184 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.184 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.184 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.184 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.184 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.186 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.186 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.186 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.186 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.186 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.186 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.186 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.187 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.187 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.187 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.187 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.187 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.187 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.187 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.187 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.188 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.188 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.188 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.188 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.188 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.188 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.188 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.188 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.188 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.189 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.189 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.189 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.189 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.189 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.189 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.189 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.189 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.190 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.190 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2023-11-16T18:34:58.190 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.190 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.190 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.190 INFO:teuthology.orchestra.run.smithi002.stdout: "fail_fs": { 2023-11-16T18:34:58.190 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "fail_fs", 2023-11-16T18:34:58.190 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.191 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.191 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.191 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.191 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.191 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.191 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.191 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-11-16T18:34:58.191 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.192 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.192 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.192 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.192 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.202 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.202 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.202 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.202 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.202 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.203 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.203 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.203 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.203 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.203 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.203 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.203 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.203 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.204 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.204 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.204 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.204 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.204 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.204 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.204 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.204 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.204 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.205 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.205 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.205 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.205 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.205 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.205 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.205 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.205 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.206 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.206 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.206 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.206 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.206 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.206 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.206 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.206 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.207 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.207 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.207 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.207 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.207 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.207 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.207 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.207 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.208 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.208 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.208 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.208 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.208 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.208 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.208 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.208 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.209 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.209 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.209 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.209 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.209 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.209 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.209 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.209 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.210 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.210 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.210 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.223 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.223 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.224 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.224 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.224 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator": { 2023-11-16T18:34:58.224 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2023-11-16T18:34:58.224 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.224 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.224 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.224 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.224 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.225 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.225 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.225 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm", 2023-11-16T18:34:58.225 INFO:teuthology.orchestra.run.smithi002.stdout: "rook", 2023-11-16T18:34:58.225 INFO:teuthology.orchestra.run.smithi002.stdout: "test_orchestrator" 2023-11-16T18:34:58.225 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.225 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Orchestrator backend", 2023-11-16T18:34:58.225 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_perf_query", 2023-11-16T18:34:58.226 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.227 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.228 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.229 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.229 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.229 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.229 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.229 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.229 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.229 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.229 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.230 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.230 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.230 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.230 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.230 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.230 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.230 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.230 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.231 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.231 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.231 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.231 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.231 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.231 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.231 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.231 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.232 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.232 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.232 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.232 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.232 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.232 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.232 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.232 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.233 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.233 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.233 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.233 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.233 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.233 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.233 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.234 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.234 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.234 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.234 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.234 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.234 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.234 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.234 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.235 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.235 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.235 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.235 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.235 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.235 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.235 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.235 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.236 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.236 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.236 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.236 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_support", 2023-11-16T18:34:58.236 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.236 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.236 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.236 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.236 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.237 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.237 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.237 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.237 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.237 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.237 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.237 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.237 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.238 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.238 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.238 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.238 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.238 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.238 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.238 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.238 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.239 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.239 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.239 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.239 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.239 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.239 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.239 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.239 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.240 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.240 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.240 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.240 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.240 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.240 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.240 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.240 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.241 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.241 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.241 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.241 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.241 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.241 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.241 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.241 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.241 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.242 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.242 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.242 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.242 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.242 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.242 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.242 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.242 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.243 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.243 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.243 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.243 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.243 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.243 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.243 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.243 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.244 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.244 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.244 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.244 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.244 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.244 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.244 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.244 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.244 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.245 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.245 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.245 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.245 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.245 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.245 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_autoscaler", 2023-11-16T18:34:58.245 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.245 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.246 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.246 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.246 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.246 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.246 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.246 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.246 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.246 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.247 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.247 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.247 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.247 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.247 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.247 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.247 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.247 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.248 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.248 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.248 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.248 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.248 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.248 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.248 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.249 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.249 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.249 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.249 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.249 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.249 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.249 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.249 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.249 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.250 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.250 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.250 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.269 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.269 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.269 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.269 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.269 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.269 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.269 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.270 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.270 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.270 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.270 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.270 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.270 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.270 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.270 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.270 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.271 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.271 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.271 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.271 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.271 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.271 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.271 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.271 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.272 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.272 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.272 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.272 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.272 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.272 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.272 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.272 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.273 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.273 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.273 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.273 INFO:teuthology.orchestra.run.smithi002.stdout: "noautoscale": { 2023-11-16T18:34:58.273 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "noautoscale", 2023-11-16T18:34:58.273 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.273 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.273 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.273 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.274 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.274 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.274 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.274 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "global autoscale flag", 2023-11-16T18:34:58.274 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-11-16T18:34:58.274 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.274 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.274 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.275 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:34:58.275 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:34:58.275 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.275 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.275 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:34:58.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.277 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.277 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.278 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.278 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.278 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.278 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.278 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.278 INFO:teuthology.orchestra.run.smithi002.stdout: "threshold": { 2023-11-16T18:34:58.278 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threshold", 2023-11-16T18:34:58.279 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:58.279 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.279 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.279 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2023-11-16T18:34:58.279 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1.0", 2023-11-16T18:34:58.279 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.279 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.279 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "scaling threshold", 2023-11-16T18:34:58.279 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-11-16T18:34:58.280 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.280 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.280 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.280 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.280 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.280 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "progress", 2023-11-16T18:34:58.280 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.281 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.281 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.281 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_pg_recovery_event": { 2023-11-16T18:34:58.281 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_pg_recovery_event", 2023-11-16T18:34:58.281 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.281 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.281 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.281 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.282 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.282 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.282 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.282 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow the module to show pg recovery progress", 2023-11-16T18:34:58.282 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.282 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.282 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.282 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.283 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2023-11-16T18:34:58.283 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2023-11-16T18:34:58.283 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.283 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.283 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.283 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.283 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.283 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.284 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.284 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.284 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.284 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.284 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.284 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.284 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.284 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.284 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.285 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.285 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.285 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.285 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.285 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.285 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.285 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.285 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.286 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.286 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.286 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.286 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.286 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.286 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.286 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.286 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.287 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.287 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.287 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.287 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.287 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.287 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.287 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.287 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.287 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.288 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.288 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.288 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.288 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.288 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.288 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.288 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.288 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.289 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.289 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.289 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.289 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.289 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.289 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.289 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.289 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.290 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.290 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.290 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.290 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.290 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.290 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.290 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.290 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.291 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.291 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.291 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.291 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.291 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.291 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.291 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.291 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.292 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.292 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.292 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.292 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.292 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.292 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.292 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.293 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.293 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.293 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.293 INFO:teuthology.orchestra.run.smithi002.stdout: "max_completed_events": { 2023-11-16T18:34:58.293 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_completed_events", 2023-11-16T18:34:58.293 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.293 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.293 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "50", 2023-11-16T18:34:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "number of past completed events to remember", 2023-11-16T18:34:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:34:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:34:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T18:34:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.296 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.296 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.296 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long the module is going to sleep", 2023-11-16T18:34:58.296 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.296 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.296 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.296 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.296 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.297 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.297 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.297 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus", 2023-11-16T18:34:58.297 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.297 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.297 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.297 INFO:teuthology.orchestra.run.smithi002.stdout: "cache": { 2023-11-16T18:34:58.297 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", 2023-11-16T18:34:58.297 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.298 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.298 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.298 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.298 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.298 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.298 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.299 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.299 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.299 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.299 INFO:teuthology.orchestra.run.smithi002.stdout: "exclude_perf_counters": { 2023-11-16T18:34:58.299 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "exclude_perf_counters", 2023-11-16T18:34:58.299 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.299 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.300 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.300 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.300 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.300 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.300 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.300 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-11-16T18:34:58.300 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-11-16T18:34:58.301 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.301 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.301 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.301 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.301 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.301 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.301 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.301 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.301 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.302 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.302 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.302 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.302 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.302 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.302 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.302 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.302 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.303 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.303 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.303 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.303 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.303 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.303 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.303 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.303 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.304 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.304 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.304 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.304 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.304 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.304 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.304 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.304 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.305 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.305 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.305 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.305 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.305 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.305 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.305 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.305 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.306 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.306 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.306 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.306 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.306 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.306 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.306 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.306 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.307 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.307 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.307 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.307 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.307 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.307 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.307 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.307 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.308 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.308 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.308 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.308 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.308 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.308 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.308 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.308 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.309 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.309 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.309 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.309 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.309 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.309 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.309 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.309 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.310 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools": { 2023-11-16T18:34:58.310 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools", 2023-11-16T18:34:58.310 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.310 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.310 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.310 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.310 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.310 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.311 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.311 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.311 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.311 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.311 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.311 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.311 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools_refresh_interval": { 2023-11-16T18:34:58.311 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-11-16T18:34:58.312 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.312 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.312 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.312 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300", 2023-11-16T18:34:58.312 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.312 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.312 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.312 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.312 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.313 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.313 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_interval": { 2023-11-16T18:34:58.313 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_interval", 2023-11-16T18:34:58.313 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:58.313 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.313 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.314 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15.0", 2023-11-16T18:34:58.314 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.314 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.314 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.314 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.314 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.314 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.315 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.315 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.315 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T18:34:58.315 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T18:34:58.315 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.315 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.315 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.315 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-11-16T18:34:58.316 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.316 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.316 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.316 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-11-16T18:34:58.316 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.316 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.316 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.316 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.317 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T18:34:58.317 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T18:34:58.317 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.317 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.317 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.317 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "9283", 2023-11-16T18:34:58.317 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.317 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.318 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.318 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-11-16T18:34:58.318 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.318 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.318 INFO:teuthology.orchestra.run.smithi002.stdout: "stale_cache_strategy": { 2023-11-16T18:34:58.318 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stale_cache_strategy", 2023-11-16T18:34:58.319 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.319 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.319 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.319 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "log", 2023-11-16T18:34:58.319 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.319 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.319 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.319 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.320 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.320 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.320 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.320 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.320 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2023-11-16T18:34:58.320 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2023-11-16T18:34:58.320 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.320 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.321 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.321 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "default", 2023-11-16T18:34:58.321 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.321 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.321 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.321 INFO:teuthology.orchestra.run.smithi002.stdout: "default", 2023-11-16T18:34:58.321 INFO:teuthology.orchestra.run.smithi002.stdout: "error" 2023-11-16T18:34:58.321 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.322 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.322 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.322 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.322 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.322 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.323 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2023-11-16T18:34:58.323 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2023-11-16T18:34:58.323 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.323 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.324 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.324 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2023-11-16T18:34:58.324 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-11-16T18:34:58.324 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2023-11-16T18:34:58.324 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.325 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.325 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.325 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.325 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.326 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.326 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.326 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.326 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.326 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_support", 2023-11-16T18:34:58.327 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.327 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.327 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.327 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.327 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.327 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.328 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.328 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.328 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.328 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.329 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.329 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.329 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.329 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.329 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.330 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.330 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.330 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.330 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.331 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.331 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.331 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.331 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.332 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.332 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.332 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.332 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.332 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.333 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.333 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.333 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.333 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.333 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.334 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.334 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.334 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.334 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.334 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.334 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.334 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.335 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.335 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.335 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.335 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.335 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.335 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.335 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.335 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.335 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.336 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.336 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.336 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.336 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.336 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.336 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.336 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.336 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.337 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.337 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.337 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.337 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.337 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.337 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.337 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.338 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.338 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.338 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.338 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.338 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.338 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.338 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.338 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.338 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_snap_create": { 2023-11-16T18:34:58.339 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_snap_create", 2023-11-16T18:34:58.339 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.339 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.339 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.339 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:58.339 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.339 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.339 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.340 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.340 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.340 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.340 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.340 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.340 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { 2023-11-16T18:34:58.340 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", 2023-11-16T18:34:58.340 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.341 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.341 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.341 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.341 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.341 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.341 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.341 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.341 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.342 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.342 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.342 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.342 INFO:teuthology.orchestra.run.smithi002.stdout: "trash_purge_schedule": { 2023-11-16T18:34:58.342 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "trash_purge_schedule", 2023-11-16T18:34:58.342 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.342 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.342 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.342 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.343 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.343 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.343 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.343 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.343 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.343 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.343 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.343 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.344 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.344 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.344 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.344 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "restful", 2023-11-16T18:34:58.344 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.344 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.344 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.344 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_auth": { 2023-11-16T18:34:58.345 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_auth", 2023-11-16T18:34:58.345 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.345 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.345 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.345 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.345 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.345 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.345 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.346 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.346 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.346 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.346 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.346 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.346 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2023-11-16T18:34:58.346 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2023-11-16T18:34:58.346 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.346 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.347 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.347 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.347 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.347 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.347 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.348 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.348 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.348 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.348 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.348 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.348 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.349 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.349 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.349 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.349 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.349 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.350 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.350 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.350 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.350 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.350 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.350 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.350 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.350 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.350 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.351 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.351 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.351 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.351 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.351 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.351 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.351 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.351 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.352 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.352 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.352 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.352 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.352 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.352 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.352 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.352 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.353 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.353 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.353 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.353 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.353 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.353 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.353 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.353 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.354 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.354 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.354 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.354 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.354 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.354 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.355 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.355 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.355 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.355 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.355 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.355 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.355 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.355 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.356 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.356 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.356 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.356 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.356 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.356 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.356 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.356 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.357 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.357 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.357 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.357 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.357 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.357 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.357 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.358 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T18:34:58.358 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T18:34:58.358 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.358 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.358 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.358 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.358 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.358 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.359 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.359 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.359 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.359 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.359 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.359 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.359 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T18:34:58.359 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T18:34:58.360 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.360 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.360 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.360 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.360 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.360 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.360 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.360 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.360 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.361 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.361 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.361 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.361 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.361 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.361 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.361 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rgw", 2023-11-16T18:34:58.361 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.361 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.362 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.362 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.362 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.362 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.362 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.362 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.362 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.362 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.363 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.363 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.363 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.363 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.363 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.363 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.363 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.363 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.364 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.364 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.364 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.364 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.364 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.364 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.364 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.364 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.364 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.365 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.365 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.365 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.365 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.365 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.365 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.365 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.365 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.366 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.366 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.366 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.366 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.366 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.366 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.366 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.366 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.367 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.367 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.367 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.367 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.367 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.367 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.367 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.367 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.367 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.368 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.368 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.368 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.368 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.368 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.368 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.368 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.368 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.369 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.369 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.369 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.369 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.369 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.369 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.369 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.369 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.369 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.370 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.370 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.370 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.370 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.370 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.370 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.370 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.370 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rook", 2023-11-16T18:34:58.371 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.371 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.371 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.371 INFO:teuthology.orchestra.run.smithi002.stdout: "drive_group_interval": { 2023-11-16T18:34:58.371 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "drive_group_interval", 2023-11-16T18:34:58.371 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:58.371 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.371 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.372 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300.0", 2023-11-16T18:34:58.372 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.372 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.372 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.372 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-11-16T18:34:58.372 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.372 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.372 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.372 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.373 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.373 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.373 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.373 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.373 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.373 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.373 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.373 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.374 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.374 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.374 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.374 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.374 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.374 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.374 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.374 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.375 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.375 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.375 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.375 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.375 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.375 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.375 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.375 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.376 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.376 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.376 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.376 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.376 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.376 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.376 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.376 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.377 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.377 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.377 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.377 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.377 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.377 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.377 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.377 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.378 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.378 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.378 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.378 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.378 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.378 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.378 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.378 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.379 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.379 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.379 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.379 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.379 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.379 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.379 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.379 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.380 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.380 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.380 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.380 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.380 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.380 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.380 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.380 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.381 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.381 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.381 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.381 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.381 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.381 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.381 INFO:teuthology.orchestra.run.smithi002.stdout: "storage_class": { 2023-11-16T18:34:58.381 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "storage_class", 2023-11-16T18:34:58.381 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.382 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.382 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.382 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "local", 2023-11-16T18:34:58.382 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.382 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.382 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.382 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-11-16T18:34:58.382 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.383 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.383 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.383 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.383 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.383 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.383 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.383 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "selftest", 2023-11-16T18:34:58.383 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.384 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.384 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.384 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.384 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.384 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.384 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.384 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.384 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.385 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.385 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.385 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.385 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.385 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.385 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.385 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.385 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.385 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.386 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.386 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.386 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.386 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.386 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.386 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.386 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.386 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.387 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.387 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.387 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.387 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.387 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.387 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.387 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.387 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.387 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.388 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.388 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.388 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.388 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.388 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.388 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.388 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.388 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.389 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.389 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.389 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.389 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.389 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.389 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.389 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.389 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.390 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.390 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.390 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.390 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.390 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.390 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.390 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.390 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.391 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.391 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.391 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.391 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.391 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.391 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.391 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.391 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.391 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.392 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.392 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.392 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.392 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.392 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.392 INFO:teuthology.orchestra.run.smithi002.stdout: "roption1": { 2023-11-16T18:34:58.392 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption1", 2023-11-16T18:34:58.392 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.393 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.393 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.393 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.393 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.393 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.393 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.393 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.393 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.394 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.394 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.394 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.394 INFO:teuthology.orchestra.run.smithi002.stdout: "roption2": { 2023-11-16T18:34:58.394 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption2", 2023-11-16T18:34:58.394 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.394 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.394 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.394 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "xyz", 2023-11-16T18:34:58.395 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.395 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.395 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.395 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.395 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.395 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.395 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.395 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.396 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption1": { 2023-11-16T18:34:58.396 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption1", 2023-11-16T18:34:58.396 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.396 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.396 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.396 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.396 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.396 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.396 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.397 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.397 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.397 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.397 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.397 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.397 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption2": { 2023-11-16T18:34:58.397 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption2", 2023-11-16T18:34:58.397 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.398 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.398 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.398 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.398 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.398 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.398 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.398 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.398 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.399 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.399 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.399 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.399 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption3": { 2023-11-16T18:34:58.399 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption3", 2023-11-16T18:34:58.399 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:58.399 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.399 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.400 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.400 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.400 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.400 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.400 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.400 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.400 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.400 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.401 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.401 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption4": { 2023-11-16T18:34:58.401 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption4", 2023-11-16T18:34:58.401 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.401 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.401 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.401 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.401 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.402 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.402 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.402 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.402 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.402 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.402 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.402 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.402 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption5": { 2023-11-16T18:34:58.403 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption5", 2023-11-16T18:34:58.403 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.403 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.403 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.403 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.403 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.403 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.403 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.404 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.404 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.404 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.404 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.404 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.404 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption6": { 2023-11-16T18:34:58.404 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption6", 2023-11-16T18:34:58.404 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.405 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.405 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.405 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.405 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.405 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.405 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.405 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.405 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.406 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.406 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.406 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.406 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption7": { 2023-11-16T18:34:58.406 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption7", 2023-11-16T18:34:58.406 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.406 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.406 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.407 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.407 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T18:34:58.407 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "42", 2023-11-16T18:34:58.407 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.407 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.407 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.407 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.407 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.407 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.408 INFO:teuthology.orchestra.run.smithi002.stdout: "testkey": { 2023-11-16T18:34:58.408 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testkey", 2023-11-16T18:34:58.408 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.408 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.408 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.408 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.408 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.408 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.409 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.409 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.409 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.409 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.409 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.409 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.409 INFO:teuthology.orchestra.run.smithi002.stdout: "testlkey": { 2023-11-16T18:34:58.409 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testlkey", 2023-11-16T18:34:58.410 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.410 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.410 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.410 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.410 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.410 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.410 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.410 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.411 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.411 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.411 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.411 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.411 INFO:teuthology.orchestra.run.smithi002.stdout: "testnewline": { 2023-11-16T18:34:58.411 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testnewline", 2023-11-16T18:34:58.411 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.411 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.412 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.412 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.412 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.412 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.412 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.412 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.412 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.412 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.413 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.413 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.413 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.413 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.413 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.413 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2023-11-16T18:34:58.413 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.413 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.413 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.414 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_m_granularity": { 2023-11-16T18:34:58.414 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_m_granularity", 2023-11-16T18:34:58.414 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.414 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.414 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.414 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.414 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.414 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.415 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.415 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow minute scheduled snapshots", 2023-11-16T18:34:58.415 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.415 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.415 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.415 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.415 INFO:teuthology.orchestra.run.smithi002.stdout: "dump_on_update": { 2023-11-16T18:34:58.415 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dump_on_update", 2023-11-16T18:34:58.416 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.416 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.416 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.416 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.416 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.416 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.416 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "dump database to debug log on update", 2023-11-16T18:34:58.416 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.417 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.417 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.417 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.417 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.417 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.417 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.417 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.417 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.418 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.418 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.418 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.418 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.418 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.418 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.418 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.418 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.419 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.419 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.419 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.419 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.419 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.419 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.419 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.419 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.419 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.420 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.420 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.420 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.420 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.420 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.420 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.420 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.420 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.421 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.421 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.421 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.421 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.421 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.421 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.421 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.421 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.421 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.422 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.422 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.422 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.422 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.422 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.422 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.422 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.422 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.423 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.423 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.423 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.423 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.423 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.423 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.423 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.423 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.424 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.424 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.424 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.424 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.424 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.424 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.424 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.424 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.425 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.425 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.425 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.425 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.425 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.425 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.425 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.425 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.426 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.426 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.426 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stats", 2023-11-16T18:34:58.426 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.426 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.426 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.426 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.426 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.426 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.427 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.427 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.427 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.427 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.427 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.427 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.427 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.427 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.428 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.428 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.428 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.428 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.428 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.428 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.428 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.428 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.429 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.429 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.429 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.429 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.429 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.429 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.429 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.429 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.430 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.430 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.430 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.430 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.430 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.430 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.430 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.430 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.431 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.431 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.431 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.431 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.431 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.431 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.431 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.431 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.432 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.432 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.432 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.432 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.432 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.432 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.432 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.432 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.433 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.433 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.433 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.433 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.433 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.433 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.433 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.433 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.434 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.434 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.434 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.434 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.434 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.434 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.434 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.434 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.435 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.435 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.435 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.435 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.435 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.435 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.435 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "status", 2023-11-16T18:34:58.435 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.435 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.436 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.436 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.436 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.436 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.436 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.436 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.436 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.437 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.437 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.437 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.437 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.437 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.437 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.437 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.437 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.438 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.438 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.438 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.438 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.438 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.438 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.438 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.438 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.438 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.439 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.439 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.439 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.439 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.439 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.439 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.439 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.439 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.440 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.440 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.440 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.440 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.440 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.440 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.440 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.440 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.440 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.441 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.441 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.441 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.441 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.441 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.441 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.441 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.441 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.442 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.442 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.442 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.442 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.442 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.442 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.442 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.442 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.443 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.443 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.443 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.443 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.443 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.443 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.443 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.443 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.443 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.444 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.444 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.444 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.444 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.444 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.444 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.444 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.444 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telegraf", 2023-11-16T18:34:58.444 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.445 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.445 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.445 INFO:teuthology.orchestra.run.smithi002.stdout: "address": { 2023-11-16T18:34:58.445 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "address", 2023-11-16T18:34:58.445 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.445 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.445 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.445 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-11-16T18:34:58.446 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.446 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.446 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.446 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.446 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.446 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.446 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.446 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.447 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:34:58.447 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:34:58.447 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.447 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.447 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.447 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15", 2023-11-16T18:34:58.447 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.447 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.447 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.448 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.448 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.448 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.448 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.448 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.448 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.448 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.448 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.449 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.449 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.449 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.449 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.449 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.449 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.449 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.449 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.450 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.450 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.450 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.450 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.450 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.450 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.450 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.450 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.450 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.451 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.451 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.451 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.451 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.451 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.451 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.451 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.451 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.452 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.452 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.452 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.452 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.452 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.452 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.452 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.452 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.452 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.453 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.453 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.453 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.453 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.453 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.453 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.453 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.453 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.454 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.454 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.454 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.454 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.454 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.454 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.454 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.454 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.454 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.455 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.455 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.455 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.455 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.455 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.455 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.455 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.455 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.456 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.456 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.456 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.456 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.456 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.456 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.457 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.457 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telemetry", 2023-11-16T18:34:58.457 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.457 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.457 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.457 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_basic": { 2023-11-16T18:34:58.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_basic", 2023-11-16T18:34:58.458 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.458 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.458 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.458 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.458 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.458 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.458 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.458 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share basic cluster information (size, version)", 2023-11-16T18:34:58.458 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.459 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.459 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.459 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.459 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_crash": { 2023-11-16T18:34:58.459 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_crash", 2023-11-16T18:34:58.459 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.459 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.459 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.460 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.460 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.460 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.460 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.460 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-11-16T18:34:58.460 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.460 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.460 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.461 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.461 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_device": { 2023-11-16T18:34:58.461 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_device", 2023-11-16T18:34:58.461 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.461 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.461 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.461 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.461 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.461 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.462 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.462 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-11-16T18:34:58.462 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.462 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.462 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.462 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.462 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_ident": { 2023-11-16T18:34:58.462 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_ident", 2023-11-16T18:34:58.463 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.463 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.463 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.463 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.463 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.463 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.463 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.463 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-11-16T18:34:58.463 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.464 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.464 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.464 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.464 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_perf": { 2023-11-16T18:34:58.464 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_perf", 2023-11-16T18:34:58.464 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.464 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.464 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.465 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.465 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.465 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.465 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.465 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share various performance metrics of a cluster", 2023-11-16T18:34:58.465 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.465 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.465 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.465 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.466 INFO:teuthology.orchestra.run.smithi002.stdout: "contact": { 2023-11-16T18:34:58.466 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "contact", 2023-11-16T18:34:58.466 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.466 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.466 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.466 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.466 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.466 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.467 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.467 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.467 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.467 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.467 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.467 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.467 INFO:teuthology.orchestra.run.smithi002.stdout: "description": { 2023-11-16T18:34:58.467 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "description", 2023-11-16T18:34:58.467 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.468 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.468 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.468 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.468 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.468 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.468 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.468 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.468 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.469 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.469 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.469 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.469 INFO:teuthology.orchestra.run.smithi002.stdout: "device_url": { 2023-11-16T18:34:58.469 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_url", 2023-11-16T18:34:58.469 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.469 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.469 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.469 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-11-16T18:34:58.470 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.470 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.470 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.470 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.470 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.470 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.470 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.470 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.471 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2023-11-16T18:34:58.471 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2023-11-16T18:34:58.471 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.471 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.471 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.471 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.471 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.471 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.472 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.472 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.472 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.472 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.472 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.472 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.472 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:34:58.472 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:34:58.473 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.473 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.473 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.473 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "24", 2023-11-16T18:34:58.473 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", 2023-11-16T18:34:58.473 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.473 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.473 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.473 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.474 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.474 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.474 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.474 INFO:teuthology.orchestra.run.smithi002.stdout: "last_opt_revision": { 2023-11-16T18:34:58.474 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "last_opt_revision", 2023-11-16T18:34:58.474 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.474 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.474 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.475 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1", 2023-11-16T18:34:58.475 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.475 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.475 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.475 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.475 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.475 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.475 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.475 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.476 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard": { 2023-11-16T18:34:58.476 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard", 2023-11-16T18:34:58.476 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.476 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.476 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.476 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.476 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.476 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.477 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.477 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.477 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.477 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.477 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.477 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard_description": { 2023-11-16T18:34:58.477 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard_description", 2023-11-16T18:34:58.477 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.478 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.478 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.478 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.478 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.478 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.478 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.478 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.478 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.479 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.479 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.479 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.479 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.479 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.479 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.479 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.479 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.479 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.480 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.480 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.480 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.480 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.480 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.480 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.480 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.480 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.481 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.481 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.481 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.481 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.481 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.481 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.481 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.481 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.481 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.482 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.482 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.482 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.482 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.482 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.482 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.482 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.482 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.483 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.483 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.483 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.483 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.483 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.483 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.483 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.483 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.483 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.484 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.484 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.484 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.484 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.484 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.484 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.484 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.484 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.485 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.485 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.485 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.485 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.486 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.486 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.486 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.486 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.486 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.486 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.486 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.486 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.487 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.487 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.487 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.487 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.487 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.487 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.487 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.487 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.487 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.488 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.488 INFO:teuthology.orchestra.run.smithi002.stdout: "organization": { 2023-11-16T18:34:58.488 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "organization", 2023-11-16T18:34:58.488 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.488 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.488 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.488 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.488 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.489 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.489 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.489 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.489 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.489 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.489 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.489 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.489 INFO:teuthology.orchestra.run.smithi002.stdout: "proxy": { 2023-11-16T18:34:58.489 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "proxy", 2023-11-16T18:34:58.490 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.490 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.490 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.490 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.490 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.490 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.490 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.490 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.491 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.491 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.491 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.491 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.491 INFO:teuthology.orchestra.run.smithi002.stdout: "url": { 2023-11-16T18:34:58.491 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url", 2023-11-16T18:34:58.491 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.491 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.491 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.492 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-11-16T18:34:58.492 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.492 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.492 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.492 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.492 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.492 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.492 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.493 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.493 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.493 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.493 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.493 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "test_orchestrator", 2023-11-16T18:34:58.493 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.493 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.493 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.494 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.494 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.494 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.494 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.494 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.494 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.494 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.494 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.495 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.495 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.495 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.495 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.495 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.495 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.495 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.495 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.496 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.496 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.496 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.496 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.496 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.496 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.496 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.496 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.497 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.497 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.497 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.497 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.497 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.497 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.497 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.498 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.498 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.498 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.498 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.498 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.498 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.498 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.498 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.498 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.499 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.499 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.499 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.499 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.499 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.499 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.499 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.499 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.500 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.500 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.500 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.500 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.500 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.500 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.500 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.500 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.501 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.501 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.501 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.501 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.501 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.501 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.501 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.501 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.501 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.502 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.502 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.502 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.502 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.502 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.502 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.502 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "volumes", 2023-11-16T18:34:58.502 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.503 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.503 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.503 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.503 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.503 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.503 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.503 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.503 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.504 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.504 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.504 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.504 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.504 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.504 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.504 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.504 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.504 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.505 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.505 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.505 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.505 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.505 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.505 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.505 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.505 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.506 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.506 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.506 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.506 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.506 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.506 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.506 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.506 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.507 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.507 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.507 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.507 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.507 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.507 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.507 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.507 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.507 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.508 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.508 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.508 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.508 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.508 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.508 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.508 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.508 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.509 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.509 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.509 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.509 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.509 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.509 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.509 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.509 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.509 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.510 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.510 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.510 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.510 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.510 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.510 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.510 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.510 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.511 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.511 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.511 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.511 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.511 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.511 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_clones": { 2023-11-16T18:34:58.511 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_clones", 2023-11-16T18:34:58.511 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.511 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.512 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.512 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4", 2023-11-16T18:34:58.512 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.512 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.512 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.512 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Number of asynchronous cloner threads", 2023-11-16T18:34:58.512 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.512 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.513 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.513 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.513 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot_clone_delay": { 2023-11-16T18:34:58.513 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snapshot_clone_delay", 2023-11-16T18:34:58.513 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.513 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.513 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.513 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:34:58.513 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.514 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.514 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.514 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-11-16T18:34:58.514 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.514 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.514 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.514 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.514 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.515 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.515 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.515 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix", 2023-11-16T18:34:58.515 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.515 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.515 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.515 INFO:teuthology.orchestra.run.smithi002.stdout: "discovery_interval": { 2023-11-16T18:34:58.515 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "discovery_interval", 2023-11-16T18:34:58.515 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:58.516 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.516 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.516 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "100", 2023-11-16T18:34:58.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.516 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.516 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.517 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.517 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.517 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.517 INFO:teuthology.orchestra.run.smithi002.stdout: "identifier": { 2023-11-16T18:34:58.517 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "identifier", 2023-11-16T18:34:58.517 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.517 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.517 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.518 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.518 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.518 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.518 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.518 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.518 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.518 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.518 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.519 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.519 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:34:58.519 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:34:58.519 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.519 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.519 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:34:58.519 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.519 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.520 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.520 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.520 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.520 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.520 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.520 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.520 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.520 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.521 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.521 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.521 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.521 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.521 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.521 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.521 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.521 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.521 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.522 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.522 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.522 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.522 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.522 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.522 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.522 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.522 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.523 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.523 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.523 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.523 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.523 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.523 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.523 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.523 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.523 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.524 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.524 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.524 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.524 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.524 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.524 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.524 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.524 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.525 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.525 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.525 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.525 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.525 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.525 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.525 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.525 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.526 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.526 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.526 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.526 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.526 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.526 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.526 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.526 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.526 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.527 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.527 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.527 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.527 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.527 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.527 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.527 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.527 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.528 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.528 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.528 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.528 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.528 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.528 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.528 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.528 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.528 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_host": { 2023-11-16T18:34:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_host", 2023-11-16T18:34:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_port": { 2023-11-16T18:34:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_port", 2023-11-16T18:34:58.531 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.531 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.531 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.531 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10051", 2023-11-16T18:34:58.531 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.531 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.531 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.531 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.531 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.532 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.532 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.532 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.532 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_sender": { 2023-11-16T18:34:58.532 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_sender", 2023-11-16T18:34:58.532 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.532 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.532 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.533 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-11-16T18:34:58.533 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.533 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.533 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.533 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.533 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.533 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.533 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.533 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.534 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.534 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.534 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T18:34:58.534 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.534 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.534 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 25202, 2023-11-16T18:34:58.534 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "y", 2023-11-16T18:34:58.534 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_features": 4540138322906710015, 2023-11-16T18:34:58.535 INFO:teuthology.orchestra.run.smithi002.stdout: "available_modules": [ 2023-11-16T18:34:58.535 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.535 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alerts", 2023-11-16T18:34:58.535 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.535 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.535 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.535 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:34:58.535 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:34:58.535 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.536 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.536 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.536 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:34:58.536 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.536 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.536 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How frequently to reexamine health status", 2023-11-16T18:34:58.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.537 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.537 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.537 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.537 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.537 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.537 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.537 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.537 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.537 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.538 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.538 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.538 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.538 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.538 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.538 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.538 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.538 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.539 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.539 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.539 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.539 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.539 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.539 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.539 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.539 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.540 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.540 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.540 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.540 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.540 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.540 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.540 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.540 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.540 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.541 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.541 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.541 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.541 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.541 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.541 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.541 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.542 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.542 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.542 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.542 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.542 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.542 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.542 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.542 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.543 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.543 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.543 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.543 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.543 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.543 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.543 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.543 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.544 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.544 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.544 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.544 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.544 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.544 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.544 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.544 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.544 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.545 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.545 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.545 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.545 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.545 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.545 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.545 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_destination": { 2023-11-16T18:34:58.545 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", 2023-11-16T18:34:58.546 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.546 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.546 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.546 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.546 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.546 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.546 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.546 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email address to send alerts to", 2023-11-16T18:34:58.547 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.547 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.547 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.547 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.547 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_from_name": { 2023-11-16T18:34:58.547 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_from_name", 2023-11-16T18:34:58.547 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.547 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.547 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.548 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "Ceph", 2023-11-16T18:34:58.548 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.548 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.548 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.548 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email From: name", 2023-11-16T18:34:58.548 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.548 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.548 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.549 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.549 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_host": { 2023-11-16T18:34:58.549 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_host", 2023-11-16T18:34:58.549 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.549 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.549 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.549 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.549 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.549 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.550 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.550 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP server", 2023-11-16T18:34:58.550 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.550 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.550 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.550 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.550 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_password": { 2023-11-16T18:34:58.550 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_password", 2023-11-16T18:34:58.551 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.551 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.551 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.551 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.551 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.551 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.551 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.551 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Password to authenticate with", 2023-11-16T18:34:58.552 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.552 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.552 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.552 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.552 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_port": { 2023-11-16T18:34:58.552 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_port", 2023-11-16T18:34:58.552 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.552 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.552 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.553 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", 2023-11-16T18:34:58.553 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.553 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.553 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.553 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP port", 2023-11-16T18:34:58.553 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.553 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.553 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.554 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.554 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_sender": { 2023-11-16T18:34:58.554 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_sender", 2023-11-16T18:34:58.554 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.554 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.554 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.554 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.554 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.555 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.555 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.555 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP envelope sender", 2023-11-16T18:34:58.555 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.555 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.555 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.555 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.555 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_ssl": { 2023-11-16T18:34:58.555 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_ssl", 2023-11-16T18:34:58.556 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.556 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.556 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.556 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.556 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use SSL to connect to SMTP server", 2023-11-16T18:34:58.557 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.557 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.557 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.557 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.557 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_user": { 2023-11-16T18:34:58.557 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_user", 2023-11-16T18:34:58.557 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.557 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.558 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.558 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.558 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.558 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.558 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.558 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "User to authenticate as", 2023-11-16T18:34:58.558 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.558 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.559 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.559 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.559 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.559 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.559 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.559 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "balancer", 2023-11-16T18:34:58.559 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.559 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.559 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.560 INFO:teuthology.orchestra.run.smithi002.stdout: "active": { 2023-11-16T18:34:58.560 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "active", 2023-11-16T18:34:58.560 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.560 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.560 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.560 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.560 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.560 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.561 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.561 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically balance PGs across cluster", 2023-11-16T18:34:58.561 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.561 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.561 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.561 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.561 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_time": { 2023-11-16T18:34:58.561 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_time", 2023-11-16T18:34:58.561 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.562 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.562 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.562 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0000", 2023-11-16T18:34:58.562 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.562 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.562 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.562 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "beginning time of day to automatically balance", 2023-11-16T18:34:58.562 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T18:34:58.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.563 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.563 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.563 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_weekday": { 2023-11-16T18:34:58.563 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_weekday", 2023-11-16T18:34:58.563 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:58.563 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.563 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.564 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:34:58.564 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2023-11-16T18:34:58.564 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2023-11-16T18:34:58.564 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.564 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-11-16T18:34:58.564 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T18:34:58.564 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.564 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.565 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.565 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_max_iterations": { 2023-11-16T18:34:58.565 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", 2023-11-16T18:34:58.565 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:58.565 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.565 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.565 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "25", 2023-11-16T18:34:58.565 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T18:34:58.565 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "250", 2023-11-16T18:34:58.566 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.566 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-11-16T18:34:58.566 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.566 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.566 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.566 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.566 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_metrics": { 2023-11-16T18:34:58.566 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_metrics", 2023-11-16T18:34:58.567 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.567 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.567 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.567 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "pgs,objects,bytes", 2023-11-16T18:34:58.567 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.567 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.567 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.567 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-11-16T18:34:58.567 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-11-16T18:34:58.568 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.568 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.568 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.568 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_step": { 2023-11-16T18:34:58.568 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_step", 2023-11-16T18:34:58.568 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:58.568 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.568 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.569 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.5", 2023-11-16T18:34:58.569 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0.001", 2023-11-16T18:34:58.569 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "0.999", 2023-11-16T18:34:58.569 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.569 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "aggressiveness of optimization", 2023-11-16T18:34:58.569 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-11-16T18:34:58.569 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.569 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.569 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.570 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { 2023-11-16T18:34:58.570 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", 2023-11-16T18:34:58.570 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.570 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.570 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.570 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2359", 2023-11-16T18:34:58.570 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.570 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.571 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.571 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ending time of day to automatically balance", 2023-11-16T18:34:58.571 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T18:34:58.571 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.571 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.571 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.571 INFO:teuthology.orchestra.run.smithi002.stdout: "end_weekday": { 2023-11-16T18:34:58.571 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_weekday", 2023-11-16T18:34:58.572 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:58.572 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.572 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.572 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:34:58.572 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2023-11-16T18:34:58.572 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2023-11-16T18:34:58.572 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.572 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-11-16T18:34:58.572 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T18:34:58.573 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.573 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.573 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.573 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.573 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.573 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.573 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.573 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.574 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.574 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.574 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.574 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.574 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.574 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.574 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.574 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.574 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.575 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.575 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.575 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.575 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.575 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.575 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.575 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.575 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.576 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.576 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.576 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.576 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.576 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.576 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.576 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.576 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.577 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.577 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.577 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.577 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.577 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.577 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.577 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.577 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.577 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.578 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.578 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.578 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.578 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.578 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.578 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.578 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.578 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.579 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.579 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.579 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.579 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.579 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.579 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.579 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.579 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.580 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.580 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.580 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.580 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.580 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.580 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.580 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.580 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.580 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.581 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.581 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.581 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.581 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.581 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.581 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.581 INFO:teuthology.orchestra.run.smithi002.stdout: "min_score": { 2023-11-16T18:34:58.581 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_score", 2023-11-16T18:34:58.582 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:58.582 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.582 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.582 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:34:58.582 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.582 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.582 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.582 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-11-16T18:34:58.582 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.583 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.583 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.583 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.583 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2023-11-16T18:34:58.583 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2023-11-16T18:34:58.583 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.583 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.583 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.584 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "upmap", 2023-11-16T18:34:58.584 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.584 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.584 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.584 INFO:teuthology.orchestra.run.smithi002.stdout: "crush-compat", 2023-11-16T18:34:58.584 INFO:teuthology.orchestra.run.smithi002.stdout: "none", 2023-11-16T18:34:58.584 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap" 2023-11-16T18:34:58.584 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.585 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Balancer mode", 2023-11-16T18:34:58.585 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.585 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.585 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.585 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.585 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_ids": { 2023-11-16T18:34:58.585 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_ids", 2023-11-16T18:34:58.585 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.585 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.586 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.586 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.586 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.586 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.586 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.586 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-11-16T18:34:58.586 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.586 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.587 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.587 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.587 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:34:58.587 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:34:58.587 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.587 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.587 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.587 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:34:58.588 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.588 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.588 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.588 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-11-16T18:34:58.588 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.588 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.588 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.588 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.588 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_deviation": { 2023-11-16T18:34:58.589 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_deviation", 2023-11-16T18:34:58.589 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.589 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.589 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.589 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T18:34:58.589 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T18:34:58.589 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.589 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.590 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "deviation below which no optimization is attempted", 2023-11-16T18:34:58.590 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-11-16T18:34:58.590 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.590 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.590 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.590 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_optimizations": { 2023-11-16T18:34:58.590 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_optimizations", 2023-11-16T18:34:58.590 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:58.591 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.591 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.591 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:58.591 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.591 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.591 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.591 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-11-16T18:34:58.591 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.591 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.592 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.592 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.592 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.592 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.592 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.592 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephadm", 2023-11-16T18:34:58.592 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.592 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.593 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.593 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_down_multiplier": { 2023-11-16T18:34:58.593 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_down_multiplier", 2023-11-16T18:34:58.593 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:58.593 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.593 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.593 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2023-11-16T18:34:58.593 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.593 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.594 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.594 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-11-16T18:34:58.594 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.594 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.594 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.594 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.594 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_refresh_rate": { 2023-11-16T18:34:58.594 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_refresh_rate", 2023-11-16T18:34:58.595 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.595 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.595 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.595 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "20", 2023-11-16T18:34:58.595 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.595 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.595 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.595 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-11-16T18:34:58.595 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.596 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.596 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.638 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.638 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_starting_port": { 2023-11-16T18:34:58.638 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_starting_port", 2023-11-16T18:34:58.638 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.638 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.638 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.638 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4721", 2023-11-16T18:34:58.639 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.639 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.639 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.639 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-11-16T18:34:58.639 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.639 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.639 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.639 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.640 INFO:teuthology.orchestra.run.smithi002.stdout: "alertmanager_web_password": { 2023-11-16T18:34:58.640 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alertmanager_web_password", 2023-11-16T18:34:58.640 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.640 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.640 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.640 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:58.640 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.640 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.641 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.641 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Alertmanager web password", 2023-11-16T18:34:58.641 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.641 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.641 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.641 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.641 INFO:teuthology.orchestra.run.smithi002.stdout: "alertmanager_web_user": { 2023-11-16T18:34:58.641 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alertmanager_web_user", 2023-11-16T18:34:58.642 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.642 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.642 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.642 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:58.642 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.642 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.642 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.643 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Alertmanager web user", 2023-11-16T18:34:58.643 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.643 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.643 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.643 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.643 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_ptrace": { 2023-11-16T18:34:58.643 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_ptrace", 2023-11-16T18:34:58.643 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.644 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.644 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.644 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.644 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.644 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.644 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-11-16T18:34:58.644 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-11-16T18:34:58.645 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.645 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.645 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.645 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_interval": { 2023-11-16T18:34:58.645 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_interval", 2023-11-16T18:34:58.645 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.645 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.645 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.645 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:34:58.646 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.646 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.646 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.646 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to autotune daemon memory", 2023-11-16T18:34:58.646 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.646 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.646 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.646 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.647 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { 2023-11-16T18:34:58.647 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-11-16T18:34:58.647 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:58.647 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.647 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.647 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.7", 2023-11-16T18:34:58.647 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.648 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.648 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.648 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-11-16T18:34:58.648 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.648 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.648 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.648 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.648 INFO:teuthology.orchestra.run.smithi002.stdout: "cgroups_split": { 2023-11-16T18:34:58.648 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cgroups_split", 2023-11-16T18:34:58.649 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.649 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.649 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.649 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.649 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.649 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.649 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.649 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-11-16T18:34:58.650 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.650 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.650 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.650 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.650 INFO:teuthology.orchestra.run.smithi002.stdout: "config_checks_enabled": { 2023-11-16T18:34:58.650 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_checks_enabled", 2023-11-16T18:34:58.650 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.650 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.651 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.651 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.651 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.651 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.651 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.651 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-11-16T18:34:58.651 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.651 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.652 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.652 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.652 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-11-16T18:34:58.652 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_dashboard", 2023-11-16T18:34:58.652 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.652 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.652 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.652 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.652 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.653 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.653 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.653 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-11-16T18:34:58.653 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.653 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.653 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.653 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.653 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_alertmanager": { 2023-11-16T18:34:58.654 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_alertmanager", 2023-11-16T18:34:58.654 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.654 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.654 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.654 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-11-16T18:34:58.654 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.654 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.654 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.654 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T18:34:58.655 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.655 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.655 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.655 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.655 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_base": { 2023-11-16T18:34:58.655 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_base", 2023-11-16T18:34:58.655 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.655 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph", 2023-11-16T18:34:58.656 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.656 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.656 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.656 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Container image name, without the tag", 2023-11-16T18:34:58.656 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.656 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.656 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.657 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.657 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_elasticsearch": { 2023-11-16T18:34:58.657 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_elasticsearch", 2023-11-16T18:34:58.657 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.657 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.657 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.657 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-11-16T18:34:58.657 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.658 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.658 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.658 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "elasticsearch container image", 2023-11-16T18:34:58.658 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.658 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.658 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.658 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.658 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_grafana": { 2023-11-16T18:34:58.659 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_grafana", 2023-11-16T18:34:58.659 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.659 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.659 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.659 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-11-16T18:34:58.659 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.659 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.659 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.660 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T18:34:58.660 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.660 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.660 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.661 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.661 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_haproxy": { 2023-11-16T18:34:58.661 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_haproxy", 2023-11-16T18:34:58.661 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.661 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.661 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.661 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-11-16T18:34:58.661 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.662 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.662 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.662 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "HAproxy container image", 2023-11-16T18:34:58.662 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.662 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.662 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.662 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.662 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_agent": { 2023-11-16T18:34:58.663 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_agent", 2023-11-16T18:34:58.663 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.663 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.663 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.663 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-11-16T18:34:58.663 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.663 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.663 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.663 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger agent container image", 2023-11-16T18:34:58.664 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.664 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.664 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.664 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.664 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_collector": { 2023-11-16T18:34:58.664 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", 2023-11-16T18:34:58.664 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.665 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.665 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-11-16T18:34:58.665 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.665 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.665 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.665 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger collector container image", 2023-11-16T18:34:58.665 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.665 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.665 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.666 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.666 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_query": { 2023-11-16T18:34:58.666 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_query", 2023-11-16T18:34:58.666 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.666 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.666 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.666 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-11-16T18:34:58.666 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.667 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.667 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.667 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger query container image", 2023-11-16T18:34:58.667 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.667 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.667 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.667 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.667 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_keepalived": { 2023-11-16T18:34:58.667 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_keepalived", 2023-11-16T18:34:58.668 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.668 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.668 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.668 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-11-16T18:34:58.668 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.668 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.668 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.668 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Keepalived container image", 2023-11-16T18:34:58.669 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.669 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.669 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.669 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.669 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_loki": { 2023-11-16T18:34:58.669 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_loki", 2023-11-16T18:34:58.669 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.669 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.670 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.670 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-11-16T18:34:58.670 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.670 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.670 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.670 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Loki container image", 2023-11-16T18:34:58.670 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.670 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.670 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.671 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.671 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_node_exporter": { 2023-11-16T18:34:58.671 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_node_exporter", 2023-11-16T18:34:58.671 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.671 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.671 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.671 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-11-16T18:34:58.671 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.672 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.672 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.672 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T18:34:58.672 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.672 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.672 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.672 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.672 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_prometheus": { 2023-11-16T18:34:58.672 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_prometheus", 2023-11-16T18:34:58.673 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.673 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.673 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.673 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-11-16T18:34:58.673 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.673 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.673 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.673 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T18:34:58.674 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.674 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.674 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.674 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.674 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_promtail": { 2023-11-16T18:34:58.674 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_promtail", 2023-11-16T18:34:58.674 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.674 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.674 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.675 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-11-16T18:34:58.675 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.675 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.675 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.675 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Promtail container image", 2023-11-16T18:34:58.675 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.675 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.675 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.676 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.676 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_snmp_gateway": { 2023-11-16T18:34:58.676 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_snmp_gateway", 2023-11-16T18:34:58.676 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.676 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.676 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.676 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-11-16T18:34:58.676 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.677 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.677 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.677 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SNMP Gateway container image", 2023-11-16T18:34:58.677 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.677 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.677 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.677 INFO:teuthology.orchestra.run.smithi002.stdout: "container_init": { 2023-11-16T18:34:58.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_init", 2023-11-16T18:34:58.678 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.678 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.678 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.678 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.678 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.678 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.678 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.678 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Run podman/docker with `--init`", 2023-11-16T18:34:58.679 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.679 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.679 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.679 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.679 INFO:teuthology.orchestra.run.smithi002.stdout: "daemon_cache_timeout": { 2023-11-16T18:34:58.679 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "daemon_cache_timeout", 2023-11-16T18:34:58.679 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.679 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.679 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.680 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:34:58.680 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.680 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.680 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.680 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-11-16T18:34:58.680 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.680 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.680 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.681 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.681 INFO:teuthology.orchestra.run.smithi002.stdout: "default_cephadm_command_timeout": { 2023-11-16T18:34:58.681 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_cephadm_command_timeout", 2023-11-16T18:34:58.681 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.681 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.681 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.681 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "900", 2023-11-16T18:34:58.681 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.681 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.682 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.682 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-11-16T18:34:58.682 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.682 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.682 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.682 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.682 INFO:teuthology.orchestra.run.smithi002.stdout: "default_registry": { 2023-11-16T18:34:58.682 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_registry", 2023-11-16T18:34:58.683 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.683 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.683 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.683 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io", 2023-11-16T18:34:58.683 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.683 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.683 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.683 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-11-16T18:34:58.683 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.684 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.684 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.684 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.684 INFO:teuthology.orchestra.run.smithi002.stdout: "device_cache_timeout": { 2023-11-16T18:34:58.684 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_cache_timeout", 2023-11-16T18:34:58.684 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.684 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.684 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.685 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1800", 2023-11-16T18:34:58.685 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.685 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.685 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.685 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache device inventory", 2023-11-16T18:34:58.685 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.685 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.685 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.685 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.686 INFO:teuthology.orchestra.run.smithi002.stdout: "device_enhanced_scan": { 2023-11-16T18:34:58.686 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_enhanced_scan", 2023-11-16T18:34:58.686 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.686 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.686 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.686 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.686 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.686 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.687 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.687 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-11-16T18:34:58.689 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.690 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.690 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.690 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.690 INFO:teuthology.orchestra.run.smithi002.stdout: "facts_cache_timeout": { 2023-11-16T18:34:58.690 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "facts_cache_timeout", 2023-11-16T18:34:58.690 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.690 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.691 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.691 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:34:58.691 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.691 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.691 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.691 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache host facts data", 2023-11-16T18:34:58.691 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.691 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.692 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.692 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.692 INFO:teuthology.orchestra.run.smithi002.stdout: "host_check_interval": { 2023-11-16T18:34:58.692 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "host_check_interval", 2023-11-16T18:34:58.692 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.692 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.692 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.692 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:34:58.693 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.693 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.693 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.693 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to perform a host check", 2023-11-16T18:34:58.693 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.693 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.693 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.694 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.694 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.694 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.694 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.694 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.694 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.694 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.694 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.695 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.695 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.695 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.695 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.695 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.695 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.695 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.695 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.696 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.696 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.696 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.696 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.696 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.696 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.696 INFO:teuthology.orchestra.run.smithi002.stdout: "log_refresh_metadata": { 2023-11-16T18:34:58.696 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_refresh_metadata", 2023-11-16T18:34:58.697 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.697 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.697 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.697 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.697 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.697 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.697 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.697 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-11-16T18:34:58.698 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.698 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.698 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.698 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.698 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.698 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.698 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.698 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.699 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.699 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.699 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.699 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.699 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.699 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-11-16T18:34:58.699 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.699 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.700 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.700 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.700 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.700 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.700 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.700 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.700 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.700 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.701 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.701 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.701 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.701 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.701 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.701 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.701 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.701 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.702 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.702 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.702 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.702 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.702 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.702 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.702 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.702 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.703 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.703 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.703 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.703 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.703 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.703 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.703 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.703 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.704 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.704 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.704 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.704 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.704 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.705 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf": { 2023-11-16T18:34:58.705 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-11-16T18:34:58.705 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.705 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.705 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.706 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.706 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.706 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.706 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.706 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-11-16T18:34:58.707 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.707 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.707 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.707 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.708 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-11-16T18:34:58.708 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-11-16T18:34:58.708 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.708 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.708 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.709 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "*", 2023-11-16T18:34:58.709 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.709 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.709 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.709 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-11-16T18:34:58.710 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.710 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.710 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.710 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.710 INFO:teuthology.orchestra.run.smithi002.stdout: "max_count_per_host": { 2023-11-16T18:34:58.711 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_count_per_host", 2023-11-16T18:34:58.711 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.711 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.711 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.711 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:58.711 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.712 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.712 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.712 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of daemons per service per host", 2023-11-16T18:34:58.712 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.712 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.712 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.712 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.712 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { 2023-11-16T18:34:58.713 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", 2023-11-16T18:34:58.713 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.713 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.713 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.713 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:58.713 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.713 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.713 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.714 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-11-16T18:34:58.714 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.714 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.714 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.714 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.714 INFO:teuthology.orchestra.run.smithi002.stdout: "migration_current": { 2023-11-16T18:34:58.714 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "migration_current", 2023-11-16T18:34:58.714 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.715 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.715 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.715 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.715 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.715 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.715 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.715 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "internal - do not modify", 2023-11-16T18:34:58.715 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.716 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.716 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.716 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.716 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2023-11-16T18:34:58.716 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2023-11-16T18:34:58.716 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.716 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.716 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.717 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "root", 2023-11-16T18:34:58.717 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.717 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.717 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.717 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm-package", 2023-11-16T18:34:58.717 INFO:teuthology.orchestra.run.smithi002.stdout: "root" 2023-11-16T18:34:58.717 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.717 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "mode for remote execution of cephadm", 2023-11-16T18:34:58.718 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.718 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.718 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.718 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.718 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_alerts_path": { 2023-11-16T18:34:58.718 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_alerts_path", 2023-11-16T18:34:58.718 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.718 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.719 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.719 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-11-16T18:34:58.719 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.719 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.719 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.719 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-11-16T18:34:58.720 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.720 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.720 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.720 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.721 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_web_password": { 2023-11-16T18:34:58.721 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_web_password", 2023-11-16T18:34:58.721 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.721 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.721 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.721 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:58.721 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.721 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.722 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.722 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus web password", 2023-11-16T18:34:58.722 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.722 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.722 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.722 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.722 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_web_user": { 2023-11-16T18:34:58.722 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_web_user", 2023-11-16T18:34:58.723 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.723 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.723 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.723 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:58.723 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.723 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.723 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.723 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus web user", 2023-11-16T18:34:58.724 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.724 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.724 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.724 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.724 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_insecure": { 2023-11-16T18:34:58.724 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_insecure", 2023-11-16T18:34:58.724 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.724 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.725 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.725 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.725 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.725 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.725 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.725 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-11-16T18:34:58.725 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.725 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.726 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.726 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.726 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_password": { 2023-11-16T18:34:58.726 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_password", 2023-11-16T18:34:58.726 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.726 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.726 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.726 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.727 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.727 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.727 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.727 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-11-16T18:34:58.727 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.727 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.727 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.727 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.728 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_url": { 2023-11-16T18:34:58.728 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_url", 2023-11-16T18:34:58.729 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.730 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.730 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.730 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.730 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.730 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.731 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.731 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-11-16T18:34:58.731 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.731 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.731 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.731 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.732 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { 2023-11-16T18:34:58.732 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", 2023-11-16T18:34:58.732 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.732 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.732 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.732 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.732 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.732 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.733 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.733 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-11-16T18:34:58.733 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.733 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.733 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.733 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.733 INFO:teuthology.orchestra.run.smithi002.stdout: "secure_monitoring_stack": { 2023-11-16T18:34:58.733 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "secure_monitoring_stack", 2023-11-16T18:34:58.734 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.734 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.734 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.734 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.734 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.734 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.734 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.734 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-11-16T18:34:58.735 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.735 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.735 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.735 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.735 INFO:teuthology.orchestra.run.smithi002.stdout: "service_discovery_port": { 2023-11-16T18:34:58.735 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "service_discovery_port", 2023-11-16T18:34:58.735 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.735 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.736 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.736 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8765", 2023-11-16T18:34:58.736 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.736 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.736 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.736 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "cephadm service discovery port", 2023-11-16T18:34:58.736 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.736 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.737 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.737 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.737 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { 2023-11-16T18:34:58.737 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-11-16T18:34:58.737 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.737 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.737 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.737 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.737 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.738 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.738 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.738 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-11-16T18:34:58.738 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.738 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.738 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.738 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.738 INFO:teuthology.orchestra.run.smithi002.stdout: "use_agent": { 2023-11-16T18:34:58.739 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_agent", 2023-11-16T18:34:58.739 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.739 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.739 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.739 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.739 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.739 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.740 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.740 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-11-16T18:34:58.740 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.740 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.740 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.740 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.740 INFO:teuthology.orchestra.run.smithi002.stdout: "use_repo_digest": { 2023-11-16T18:34:58.740 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_repo_digest", 2023-11-16T18:34:58.741 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.741 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.741 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.741 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.741 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.741 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.742 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.742 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-11-16T18:34:58.742 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.742 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.742 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.742 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.742 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_failed_host_check": { 2023-11-16T18:34:58.742 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_failed_host_check", 2023-11-16T18:34:58.743 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.743 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.743 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.743 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.743 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.743 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.743 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if the host check fails", 2023-11-16T18:34:58.744 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.744 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.744 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.744 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.744 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_daemons": { 2023-11-16T18:34:58.744 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_daemons", 2023-11-16T18:34:58.744 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.744 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.745 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.745 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.745 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.745 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.745 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.745 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-11-16T18:34:58.745 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.745 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.745 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.746 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.746 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_hosts": { 2023-11-16T18:34:58.746 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_hosts", 2023-11-16T18:34:58.746 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.746 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.746 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.746 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.746 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.747 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.747 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-11-16T18:34:58.747 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.747 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.747 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.747 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.747 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.748 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.748 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.748 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crash", 2023-11-16T18:34:58.748 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.748 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.748 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.748 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:58.748 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:58.749 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.749 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.749 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.749 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.749 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.749 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.749 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.749 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.750 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.750 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.750 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.750 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.750 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.750 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.750 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.750 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.751 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.751 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.751 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.751 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:58.751 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:58.751 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.751 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.751 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.752 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.752 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.752 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.752 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.752 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.752 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.752 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.752 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.753 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.753 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:58.753 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:58.753 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.753 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.753 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.753 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:58.754 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.754 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.754 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:58.754 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:58.754 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:58.754 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:58.754 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:58.754 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:58.755 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:58.755 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:58.755 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.755 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.770 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.770 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.770 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.770 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:58.771 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:58.771 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.771 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.771 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.771 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.771 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.771 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.771 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.772 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.772 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.772 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.772 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.772 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.772 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { 2023-11-16T18:34:58.772 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-11-16T18:34:58.772 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.772 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.773 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.773 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "31536000", 2023-11-16T18:34:58.773 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.773 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.773 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.773 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain crashes before pruning them", 2023-11-16T18:34:58.773 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.773 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.774 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.774 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.774 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_recent_interval": { 2023-11-16T18:34:58.774 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_recent_interval", 2023-11-16T18:34:58.774 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:58.774 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.774 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:58.774 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1209600", 2023-11-16T18:34:58.775 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.775 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.775 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.775 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "time interval in which to warn about recent crashes", 2023-11-16T18:34:58.775 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.775 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.775 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.775 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.776 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:58.776 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.776 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:58.838 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dashboard", 2023-11-16T18:34:58.839 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:58.839 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:58.839 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:58.839 INFO:teuthology.orchestra.run.smithi002.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-11-16T18:34:58.839 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-11-16T18:34:58.839 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:58.839 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.840 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.840 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:58.840 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.840 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.840 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.840 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.840 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.840 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.841 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.841 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.841 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_HOST": { 2023-11-16T18:34:58.841 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_HOST", 2023-11-16T18:34:58.841 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.841 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.841 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.842 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.842 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.842 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.842 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.842 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.842 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.842 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.842 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.843 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.843 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-11-16T18:34:58.843 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-11-16T18:34:58.843 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.843 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.843 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.843 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.843 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.844 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.844 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.844 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.844 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.844 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.844 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.844 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.844 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_ENABLED": { 2023-11-16T18:34:58.845 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_ENABLED", 2023-11-16T18:34:58.845 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.871 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.872 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:58.872 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.872 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.872 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.872 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.872 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.872 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.872 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.873 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.873 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-11-16T18:34:58.873 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-11-16T18:34:58.873 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.873 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.873 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.873 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.873 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.874 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.874 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.874 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.874 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.874 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.874 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.874 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.875 INFO:teuthology.orchestra.run.smithi002.stdout: "ENABLE_BROWSABLE_API": { 2023-11-16T18:34:58.875 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ENABLE_BROWSABLE_API", 2023-11-16T18:34:58.875 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.875 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.875 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.875 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.875 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.875 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.875 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.876 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.876 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.876 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.876 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.876 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.876 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-11-16T18:34:58.876 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-11-16T18:34:58.876 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.877 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.877 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.877 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.877 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.985 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.985 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.985 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.986 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.986 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.986 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.986 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.986 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-11-16T18:34:58.986 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-11-16T18:34:58.986 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.987 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.987 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.987 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.987 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.987 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.987 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.987 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.988 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.988 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.988 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.988 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.988 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-11-16T18:34:58.988 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-11-16T18:34:58.988 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.988 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.989 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.989 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.989 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.989 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.989 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.989 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.989 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.989 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.989 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.990 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.990 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-11-16T18:34:58.990 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-11-16T18:34:58.990 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.990 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.990 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.990 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.991 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.991 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.991 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.991 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.991 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.991 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.991 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.991 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.992 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_NFS": { 2023-11-16T18:34:58.992 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-11-16T18:34:58.992 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.992 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.992 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.992 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.992 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.992 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.993 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.993 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.993 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.993 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.993 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.993 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.993 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RBD": { 2023-11-16T18:34:58.994 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-11-16T18:34:58.994 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.994 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.994 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.994 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.994 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.994 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.995 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.995 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.995 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.995 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.995 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.995 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-11-16T18:34:58.995 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-11-16T18:34:58.995 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:58.996 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.996 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.996 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:58.996 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.996 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.996 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.996 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.996 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.996 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.997 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.997 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.997 INFO:teuthology.orchestra.run.smithi002.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-11-16T18:34:58.997 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-11-16T18:34:58.997 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.997 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.997 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.997 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:58.998 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:58.998 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:58.998 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:58.998 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:58.998 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:58.998 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:58.998 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:58.999 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:58.999 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_PASSWORD": { 2023-11-16T18:34:58.999 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_PASSWORD", 2023-11-16T18:34:58.999 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:58.999 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:58.999 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:58.999 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:59.000 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.000 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.000 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.000 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.000 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.000 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.000 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.001 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-11-16T18:34:59.001 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-11-16T18:34:59.001 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.001 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.001 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.001 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.001 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.001 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.002 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.002 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.002 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.002 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.002 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.002 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.002 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_URL": { 2023-11-16T18:34:59.003 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_URL", 2023-11-16T18:34:59.003 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.003 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.003 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.003 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.003 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.003 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.003 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.004 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.004 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.004 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.004 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.004 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.004 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_USERNAME": { 2023-11-16T18:34:59.004 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_USERNAME", 2023-11-16T18:34:59.004 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.005 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.005 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.005 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:59.005 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.005 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.005 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.005 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.005 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.006 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.006 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.006 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.006 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-11-16T18:34:59.006 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-11-16T18:34:59.006 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.006 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.007 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.007 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.007 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.007 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.007 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.007 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.007 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.007 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.008 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.008 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.008 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-11-16T18:34:59.008 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-11-16T18:34:59.008 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.008 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.008 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.008 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.009 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.009 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.009 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.009 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.009 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.009 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.009 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.009 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.010 INFO:teuthology.orchestra.run.smithi002.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-11-16T18:34:59.010 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-11-16T18:34:59.010 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.010 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.010 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.010 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.010 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.010 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.011 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.011 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.011 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.011 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.011 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.011 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.011 INFO:teuthology.orchestra.run.smithi002.stdout: "ISSUE_TRACKER_API_KEY": { 2023-11-16T18:34:59.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-11-16T18:34:59.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.012 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.012 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.012 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.012 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.012 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.012 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { 2023-11-16T18:34:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", 2023-11-16T18:34:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-11-16T18:34:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-11-16T18:34:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-11-16T18:34:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-11-16T18:34:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-11-16T18:34:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-11-16T18:34:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-11-16T18:34:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-11-16T18:34:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-11-16T18:34:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-11-16T18:34:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-11-16T18:34:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-11-16T18:34:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-11-16T18:34:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-11-16T18:34:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-11-16T18:34:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-11-16T18:34:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_ENABLED": { 2023-11-16T18:34:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_ENABLED", 2023-11-16T18:34:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.031 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.031 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.031 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.031 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-11-16T18:34:59.031 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-11-16T18:34:59.031 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.032 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.032 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.032 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-11-16T18:34:59.032 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.032 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.032 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.032 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-11-16T18:34:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-11-16T18:34:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.034 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:59.034 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.034 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.034 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.034 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.035 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.035 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.035 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.035 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.035 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-11-16T18:34:59.035 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-11-16T18:34:59.035 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.035 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.036 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.036 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8", 2023-11-16T18:34:59.036 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.036 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.036 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.036 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.036 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.036 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.037 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.037 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.037 INFO:teuthology.orchestra.run.smithi002.stdout: "REST_REQUESTS_TIMEOUT": { 2023-11-16T18:34:59.037 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-11-16T18:34:59.037 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.037 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.037 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.037 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "45", 2023-11-16T18:34:59.038 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.038 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.038 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.038 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.038 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.038 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.038 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.039 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.039 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ACCESS_KEY": { 2023-11-16T18:34:59.039 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ACCESS_KEY", 2023-11-16T18:34:59.039 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.039 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.039 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.039 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.039 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.040 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.040 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.040 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.040 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.040 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.040 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.040 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.040 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-11-16T18:34:59.041 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-11-16T18:34:59.041 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.041 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.041 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.041 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:34:59.041 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.041 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.041 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.042 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.042 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.042 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.042 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.042 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.042 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SECRET_KEY": { 2023-11-16T18:34:59.042 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SECRET_KEY", 2023-11-16T18:34:59.042 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.043 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.043 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.043 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.043 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.043 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.043 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.043 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.044 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.044 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.044 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.075 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.075 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SSL_VERIFY": { 2023-11-16T18:34:59.075 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SSL_VERIFY", 2023-11-16T18:34:59.075 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.076 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.076 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.076 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.076 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.076 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.076 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.076 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.077 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.077 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.077 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.077 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.077 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-11-16T18:34:59.077 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-11-16T18:34:59.077 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.078 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.078 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.078 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:34:59.078 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.078 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.078 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.078 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.079 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.079 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.079 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.079 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.079 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-11-16T18:34:59.079 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-11-16T18:34:59.079 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.080 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.080 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.080 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:59.080 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.080 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.080 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.080 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.081 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.081 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.081 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.081 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.081 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-11-16T18:34:59.081 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-11-16T18:34:59.081 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.082 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.082 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.082 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T18:34:59.082 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.082 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.082 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.082 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.083 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.083 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.083 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.083 INFO:teuthology.orchestra.run.smithi002.stdout: "cross_origin_url": { 2023-11-16T18:34:59.083 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cross_origin_url", 2023-11-16T18:34:59.083 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.083 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.084 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.084 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.084 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.084 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.084 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.084 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.084 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.085 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.085 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.085 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.085 INFO:teuthology.orchestra.run.smithi002.stdout: "crt_file": { 2023-11-16T18:34:59.085 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crt_file", 2023-11-16T18:34:59.085 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.085 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.086 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.086 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.086 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.086 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.086 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.086 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.086 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.087 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.087 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.087 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.087 INFO:teuthology.orchestra.run.smithi002.stdout: "debug": { 2023-11-16T18:34:59.087 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "debug", 2023-11-16T18:34:59.087 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.087 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.088 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.088 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.088 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.088 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.088 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.088 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable/disable debug options", 2023-11-16T18:34:59.088 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.088 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.089 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.089 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.089 INFO:teuthology.orchestra.run.smithi002.stdout: "jwt_token_ttl": { 2023-11-16T18:34:59.110 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "jwt_token_ttl", 2023-11-16T18:34:59.110 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.110 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.110 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.110 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "28800", 2023-11-16T18:34:59.110 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.111 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.111 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.111 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.112 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.112 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.112 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.112 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.113 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2023-11-16T18:34:59.113 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2023-11-16T18:34:59.113 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.113 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.114 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.114 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.114 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.115 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.115 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.115 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.115 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.116 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.116 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.116 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.116 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.117 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.117 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.117 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.117 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.118 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.118 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.118 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.118 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.119 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.136 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.136 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.136 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.136 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.137 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.137 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.137 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.137 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.138 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.138 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.138 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.138 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.139 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.139 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.139 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.139 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.140 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.140 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.140 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.140 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.141 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.141 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.141 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.141 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.142 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.142 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.142 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.142 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.142 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.142 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.143 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.143 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.143 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.143 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.143 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.143 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.143 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.143 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.144 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.144 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.144 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.145 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.145 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.145 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.146 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.181 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.181 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.181 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.182 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.183 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.183 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.183 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.184 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.184 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.184 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.184 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.184 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.184 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.184 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.184 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.185 INFO:teuthology.orchestra.run.smithi002.stdout: "motd": { 2023-11-16T18:34:59.185 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "motd", 2023-11-16T18:34:59.185 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.185 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.185 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.185 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.185 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.186 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.186 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.186 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "The message of the day", 2023-11-16T18:34:59.186 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.186 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.186 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.186 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.187 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect_resolve_ip_addr": { 2023-11-16T18:34:59.187 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "redirect_resolve_ip_addr", 2023-11-16T18:34:59.187 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.187 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.187 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.187 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.187 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.188 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.188 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.188 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.188 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.188 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.189 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T18:34:59.189 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T18:34:59.189 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.189 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.189 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.189 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-11-16T18:34:59.189 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.190 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.190 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.190 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.190 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.190 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.190 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.190 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.190 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T18:34:59.191 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T18:34:59.191 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.191 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.191 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.191 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8080", 2023-11-16T18:34:59.191 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.191 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.192 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.192 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.192 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.192 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.192 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.192 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.192 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2023-11-16T18:34:59.192 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2023-11-16T18:34:59.193 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.193 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.193 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.193 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.193 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.193 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.193 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.193 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.202 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.202 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.202 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.202 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.203 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl_server_port": { 2023-11-16T18:34:59.203 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", 2023-11-16T18:34:59.203 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.203 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.203 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.203 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8443", 2023-11-16T18:34:59.203 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.204 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.204 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.204 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.204 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.204 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.204 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.204 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.204 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2023-11-16T18:34:59.205 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2023-11-16T18:34:59.205 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.205 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.205 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.205 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "redirect", 2023-11-16T18:34:59.205 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.205 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.206 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.206 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.206 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect" 2023-11-16T18:34:59.206 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.206 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.206 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.206 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.207 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.207 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.207 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2023-11-16T18:34:59.207 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2023-11-16T18:34:59.207 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.207 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.207 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.207 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2023-11-16T18:34:59.208 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-11-16T18:34:59.208 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2023-11-16T18:34:59.208 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.208 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.208 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.208 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.209 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.209 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.209 INFO:teuthology.orchestra.run.smithi002.stdout: "url_prefix": { 2023-11-16T18:34:59.209 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url_prefix", 2023-11-16T18:34:59.209 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.209 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.209 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.209 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.210 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.210 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.210 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.210 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.210 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.210 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.210 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.211 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.211 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.211 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.211 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.211 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", 2023-11-16T18:34:59.211 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.211 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.212 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.212 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_monitoring": { 2023-11-16T18:34:59.212 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_monitoring", 2023-11-16T18:34:59.212 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.212 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.212 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.212 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.212 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.213 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.213 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.213 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "monitor device health metrics", 2023-11-16T18:34:59.213 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.213 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.213 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.213 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.213 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.214 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.214 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.214 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.214 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.214 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.214 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.214 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.215 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.215 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.215 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.215 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.215 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.215 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.215 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.215 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.216 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.216 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.216 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.216 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.216 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.216 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.216 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.216 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.217 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.217 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.217 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.217 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.217 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.217 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.217 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.217 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.218 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.218 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.218 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.218 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.218 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.218 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.218 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.219 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.219 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.219 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.219 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.219 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.219 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.219 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.219 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.220 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.220 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.220 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.220 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.220 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.220 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.220 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.220 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.220 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.221 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.221 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.221 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.221 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.221 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.221 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.221 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.222 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.222 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.222 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.222 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.222 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.222 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.222 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.222 INFO:teuthology.orchestra.run.smithi002.stdout: "mark_out_threshold": { 2023-11-16T18:34:59.223 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mark_out_threshold", 2023-11-16T18:34:59.223 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:59.223 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.223 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.223 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2419200", 2023-11-16T18:34:59.223 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.223 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.224 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.224 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-11-16T18:34:59.224 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.224 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.224 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.224 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.224 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-11-16T18:34:59.224 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_name", 2023-11-16T18:34:59.225 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.225 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.225 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.225 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "device_health_metrics", 2023-11-16T18:34:59.225 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.225 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.225 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.225 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name of pool in which to store device health metrics", 2023-11-16T18:34:59.226 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.226 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.226 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.226 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.226 INFO:teuthology.orchestra.run.smithi002.stdout: "retention_period": { 2023-11-16T18:34:59.226 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retention_period", 2023-11-16T18:34:59.226 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:59.227 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.227 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.227 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15552000", 2023-11-16T18:34:59.227 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.227 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.227 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.227 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain device health metrics", 2023-11-16T18:34:59.227 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.228 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.228 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.228 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_frequency": { 2023-11-16T18:34:59.228 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_frequency", 2023-11-16T18:34:59.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:59.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.229 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.229 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2023-11-16T18:34:59.229 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.229 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.229 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.229 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to scrape device health metrics", 2023-11-16T18:34:59.229 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.229 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.229 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.230 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.230 INFO:teuthology.orchestra.run.smithi002.stdout: "self_heal": { 2023-11-16T18:34:59.230 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "self_heal", 2023-11-16T18:34:59.230 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.230 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.230 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.230 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.231 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.231 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.231 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.231 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-11-16T18:34:59.231 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.231 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.231 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.231 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.232 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:34:59.232 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:34:59.232 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:59.232 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.232 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.232 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:34:59.232 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.232 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.232 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.233 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and check device health", 2023-11-16T18:34:59.233 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.233 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.233 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.233 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.233 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_threshold": { 2023-11-16T18:34:59.233 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_threshold", 2023-11-16T18:34:59.234 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:59.234 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.234 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.234 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7257600", 2023-11-16T18:34:59.234 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.234 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.234 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.234 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-11-16T18:34:59.235 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.235 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.235 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.235 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.235 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.235 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.235 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.236 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "diskprediction_local", 2023-11-16T18:34:59.236 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.236 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.236 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.236 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.236 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.236 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.236 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.237 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.237 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.237 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.237 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.237 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.237 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.237 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.237 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.238 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.238 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.238 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.238 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.238 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.238 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.238 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.238 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.239 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.239 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.239 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.239 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.239 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.239 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.239 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.241 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.241 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.241 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.241 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.241 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.241 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.241 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.242 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.242 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.242 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.242 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.242 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.242 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.242 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.243 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.243 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.243 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.243 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.243 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.243 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.243 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.244 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.244 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.244 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.244 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.244 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.244 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.244 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.245 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.245 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.245 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.245 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.245 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.245 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.245 INFO:teuthology.orchestra.run.smithi002.stdout: "predict_interval": { 2023-11-16T18:34:59.246 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predict_interval", 2023-11-16T18:34:59.246 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.246 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.246 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.246 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2023-11-16T18:34:59.246 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.246 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.246 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.247 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.247 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.247 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.247 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.247 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.247 INFO:teuthology.orchestra.run.smithi002.stdout: "predictor_model": { 2023-11-16T18:34:59.247 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predictor_model", 2023-11-16T18:34:59.248 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.248 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.248 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.248 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "prophetstor", 2023-11-16T18:34:59.248 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.248 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.248 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.248 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.249 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.249 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.249 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.249 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.249 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:34:59.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:34:59.249 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.250 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.250 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.250 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:34:59.250 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.250 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.250 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.250 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.250 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.251 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.251 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.251 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.251 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.251 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.251 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.251 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "influx", 2023-11-16T18:34:59.252 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": false, 2023-11-16T18:34:59.252 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "influxdb python module not found", 2023-11-16T18:34:59.252 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.252 INFO:teuthology.orchestra.run.smithi002.stdout: "batch_size": { 2023-11-16T18:34:59.252 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "batch_size", 2023-11-16T18:34:59.252 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.252 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.252 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.253 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5000", 2023-11-16T18:34:59.253 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.253 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.253 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.253 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-11-16T18:34:59.253 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.253 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.253 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.254 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.254 INFO:teuthology.orchestra.run.smithi002.stdout: "database": { 2023-11-16T18:34:59.254 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "database", 2023-11-16T18:34:59.254 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.254 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.254 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.254 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "ceph", 2023-11-16T18:34:59.255 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.255 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.255 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.255 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-11-16T18:34:59.255 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.255 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.255 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.256 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.256 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": { 2023-11-16T18:34:59.256 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "hostname", 2023-11-16T18:34:59.256 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.256 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.256 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.256 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.257 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.257 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.257 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.257 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server hostname", 2023-11-16T18:34:59.257 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.257 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.257 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.257 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.258 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:34:59.258 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:34:59.258 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:59.258 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.258 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.258 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "30", 2023-11-16T18:34:59.258 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "5", 2023-11-16T18:34:59.258 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.259 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.259 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-11-16T18:34:59.259 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.259 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.259 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.259 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.259 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.259 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.260 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.260 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.260 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.260 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.260 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.260 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.260 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.261 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.261 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.261 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.261 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.261 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.261 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.261 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.261 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.262 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.262 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.262 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.262 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.262 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.262 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.262 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.263 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.263 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.263 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.263 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.263 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.263 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.263 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.263 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.264 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.264 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.264 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.264 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.264 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.264 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.264 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.265 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.265 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.265 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.265 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.265 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.265 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.265 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.265 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.266 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.266 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.266 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.266 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.266 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.266 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.266 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.267 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.267 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.267 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.267 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.267 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.267 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.267 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.267 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.268 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.268 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.268 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.268 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.268 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.268 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.268 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.268 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.269 INFO:teuthology.orchestra.run.smithi002.stdout: "password": { 2023-11-16T18:34:59.269 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "password", 2023-11-16T18:34:59.269 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.269 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.269 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.269 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.269 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.269 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.270 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.270 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "password of InfluxDB server user", 2023-11-16T18:34:59.270 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.270 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.270 INFO:teuthology.orchestra.run.smithi002.stdout: "port": { 2023-11-16T18:34:59.271 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "port", 2023-11-16T18:34:59.271 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.271 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.271 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.271 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8086", 2023-11-16T18:34:59.271 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.271 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.271 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.272 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server port", 2023-11-16T18:34:59.272 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.272 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.272 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.272 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.272 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2023-11-16T18:34:59.272 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2023-11-16T18:34:59.272 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.273 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.273 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.273 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "false", 2023-11-16T18:34:59.273 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.273 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.273 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.273 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T18:34:59.273 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.274 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.274 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.274 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.274 INFO:teuthology.orchestra.run.smithi002.stdout: "threads": { 2023-11-16T18:34:59.274 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threads", 2023-11-16T18:34:59.274 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.274 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.274 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.275 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T18:34:59.275 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T18:34:59.275 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "32", 2023-11-16T18:34:59.275 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.275 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-11-16T18:34:59.275 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.275 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.275 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.276 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.276 INFO:teuthology.orchestra.run.smithi002.stdout: "username": { 2023-11-16T18:34:59.276 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "username", 2023-11-16T18:34:59.276 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.276 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.276 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.276 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.276 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.277 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.277 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.277 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "username of InfluxDB server user", 2023-11-16T18:34:59.277 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.277 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.277 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.277 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.277 INFO:teuthology.orchestra.run.smithi002.stdout: "verify_ssl": { 2023-11-16T18:34:59.278 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "verify_ssl", 2023-11-16T18:34:59.278 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.278 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.278 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.278 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "true", 2023-11-16T18:34:59.278 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.278 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.279 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.279 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T18:34:59.279 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.279 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.279 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.279 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.279 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.280 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.280 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "insights", 2023-11-16T18:34:59.280 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.280 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.280 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.280 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.281 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.281 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.281 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.281 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.281 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.281 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.281 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.281 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.282 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.282 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.282 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.282 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.282 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.282 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.283 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.283 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.283 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.283 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.283 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.283 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.283 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.283 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.284 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.284 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.284 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.284 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.284 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.284 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.285 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.285 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.285 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.285 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.285 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.285 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.285 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.285 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.286 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.286 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.286 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.286 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.286 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.286 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.286 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.287 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.287 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.287 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.287 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.287 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.287 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.287 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.287 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.288 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.288 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.288 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.288 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.288 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.288 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.288 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.289 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.289 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.289 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.289 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.289 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.289 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.289 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.289 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.290 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.290 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.290 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.290 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.290 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.290 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.290 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "iostat", 2023-11-16T18:34:59.291 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.291 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.291 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.291 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.291 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.291 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.291 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.291 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.292 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.292 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.292 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.292 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.292 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.292 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.292 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.292 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.293 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.293 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.293 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.293 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.293 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.293 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.293 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.294 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.294 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.294 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.294 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.294 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.294 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.294 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.294 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.295 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.295 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.295 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.295 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.295 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.295 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.295 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.296 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.296 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.296 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.296 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.296 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.296 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.296 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.297 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.297 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.297 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.297 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.297 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.297 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.297 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.298 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.298 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.298 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.298 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.298 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.298 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.299 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.301 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.301 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.301 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.301 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.301 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.301 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.302 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.302 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.302 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.302 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.302 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.303 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.303 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.303 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.303 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.303 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "k8sevents", 2023-11-16T18:34:59.303 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.303 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.303 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.304 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_event_retention_days": { 2023-11-16T18:34:59.304 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ceph_event_retention_days", 2023-11-16T18:34:59.304 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.304 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.304 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.304 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7", 2023-11-16T18:34:59.304 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.305 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.305 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.305 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Days to hold ceph event information within local cache", 2023-11-16T18:34:59.305 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.305 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.305 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.305 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.306 INFO:teuthology.orchestra.run.smithi002.stdout: "config_check_secs": { 2023-11-16T18:34:59.306 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_check_secs", 2023-11-16T18:34:59.306 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.306 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.306 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.306 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:59.306 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "10", 2023-11-16T18:34:59.307 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.307 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.307 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-11-16T18:34:59.307 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.307 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.307 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.307 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.307 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.308 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.308 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.308 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.308 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.308 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.308 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.308 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.309 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.309 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.309 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.309 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.309 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.309 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.309 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.310 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.310 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.310 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.310 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.310 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.310 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.310 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.310 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.311 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.311 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.311 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.311 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.311 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.311 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.311 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.311 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.312 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.312 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.312 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.312 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.312 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.312 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.312 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.313 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.313 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.313 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.313 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.313 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.313 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.313 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.313 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.314 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.314 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.314 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.314 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.314 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.314 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.314 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.315 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.315 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.315 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.315 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.315 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.315 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.315 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.315 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.316 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.316 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.316 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.316 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.316 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.316 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.316 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.317 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.317 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.317 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.317 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.317 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.317 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "localpool", 2023-11-16T18:34:59.317 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.317 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.318 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.318 INFO:teuthology.orchestra.run.smithi002.stdout: "failure_domain": { 2023-11-16T18:34:59.318 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "failure_domain", 2023-11-16T18:34:59.318 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.318 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.318 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.318 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "host", 2023-11-16T18:34:59.319 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.319 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.319 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.319 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "failure domain for any created local pool", 2023-11-16T18:34:59.319 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-11-16T18:34:59.319 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.319 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.319 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.320 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.320 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.320 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.320 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.320 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.320 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.320 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.320 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.321 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.321 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.321 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.321 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.321 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.321 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.321 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.321 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.322 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.322 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.322 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.322 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.322 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.322 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.322 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.323 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.323 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.323 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.323 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.323 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.323 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.323 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.323 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.324 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.324 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.324 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.324 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.324 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.324 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.324 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.325 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.325 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.325 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.325 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.325 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.325 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.325 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.326 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.326 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.326 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.326 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.326 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.326 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.326 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.326 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.327 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.327 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.327 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.327 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.327 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.327 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.327 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.327 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.328 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.328 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.328 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.328 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.328 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.328 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.328 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.328 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.329 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.329 INFO:teuthology.orchestra.run.smithi002.stdout: "min_size": { 2023-11-16T18:34:59.329 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_size", 2023-11-16T18:34:59.329 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.329 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.329 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.329 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.330 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.330 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.330 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.330 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default min_size for any created local pool", 2023-11-16T18:34:59.330 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-11-16T18:34:59.330 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.330 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.331 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.331 INFO:teuthology.orchestra.run.smithi002.stdout: "num_rep": { 2023-11-16T18:34:59.331 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "num_rep", 2023-11-16T18:34:59.331 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.331 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.331 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.331 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3", 2023-11-16T18:34:59.332 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.332 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.332 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.332 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default replica count for any created local pool", 2023-11-16T18:34:59.332 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.332 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.332 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.332 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.333 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num": { 2023-11-16T18:34:59.333 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_num", 2023-11-16T18:34:59.333 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.333 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.333 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.333 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "128", 2023-11-16T18:34:59.333 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.334 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.334 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.334 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default pg_num for any created local pool", 2023-11-16T18:34:59.334 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.334 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.334 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.334 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.335 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-11-16T18:34:59.335 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prefix", 2023-11-16T18:34:59.335 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.335 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.335 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.335 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.335 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.336 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.336 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.336 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name prefix for any created local pool", 2023-11-16T18:34:59.336 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.336 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.336 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.336 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.336 INFO:teuthology.orchestra.run.smithi002.stdout: "subtree": { 2023-11-16T18:34:59.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subtree", 2023-11-16T18:34:59.337 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.337 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.337 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.337 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "rack", 2023-11-16T18:34:59.337 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.337 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.338 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.338 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "CRUSH level for which to create a local pool", 2023-11-16T18:34:59.338 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-11-16T18:34:59.338 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.338 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.338 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.338 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.339 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.339 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.339 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mds_autoscaler", 2023-11-16T18:34:59.339 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.339 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.339 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.339 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.340 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.340 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.340 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.340 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.340 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.340 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.340 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.341 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.341 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.341 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.341 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.341 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.341 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.341 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.341 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.342 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.342 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.342 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.342 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.342 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.342 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.343 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.343 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.343 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.343 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.343 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.343 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.343 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.343 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.344 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.344 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.344 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.344 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.344 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.344 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.344 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.345 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.345 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.345 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.345 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.345 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.345 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.345 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.346 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.346 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.346 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.346 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.346 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.346 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.346 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.347 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.347 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.347 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.347 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.347 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.347 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.347 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.348 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.348 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.348 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.348 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.348 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.348 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.348 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.348 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.349 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.349 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.349 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.349 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.349 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.350 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.350 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.350 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirroring", 2023-11-16T18:34:59.350 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.350 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.350 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.350 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.351 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.351 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.351 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.351 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.351 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.351 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.352 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.352 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.352 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.352 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.352 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.352 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.352 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.352 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.353 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.353 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.353 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.353 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.353 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.353 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.353 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.354 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.354 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.354 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.354 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.354 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.354 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.354 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.354 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.355 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.355 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.355 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.355 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.355 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.355 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.355 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.356 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.356 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.356 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.356 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.356 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.356 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.356 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.356 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.357 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.357 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.357 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.357 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.357 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.357 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.357 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.358 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.358 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.358 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.358 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.358 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.358 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.359 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.359 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.359 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.359 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.359 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.359 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.359 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.359 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.360 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.360 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.360 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.360 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.360 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.360 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "nfs", 2023-11-16T18:34:59.360 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.360 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.361 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.361 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.361 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.361 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.361 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.361 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.361 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.361 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.362 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.362 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.362 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.362 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.362 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.362 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.362 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.363 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.363 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.363 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.363 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.363 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.363 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.363 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.363 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.364 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.364 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.364 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.364 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.364 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.364 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.364 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.364 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.365 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.365 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.365 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.365 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.365 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.365 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.365 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.366 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.366 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.366 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.366 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.366 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.366 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.366 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.367 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.367 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.367 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.367 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.367 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.367 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.367 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.367 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.368 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.368 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.368 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.368 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.368 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.368 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.368 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.368 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.369 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.369 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.369 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.369 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.369 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.369 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.369 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.370 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.370 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.370 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.370 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.370 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.370 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.370 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2023-11-16T18:34:59.370 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.371 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.371 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.371 INFO:teuthology.orchestra.run.smithi002.stdout: "fail_fs": { 2023-11-16T18:34:59.371 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "fail_fs", 2023-11-16T18:34:59.371 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.371 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.371 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.372 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.372 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.372 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.372 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.372 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-11-16T18:34:59.372 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.372 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.373 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.373 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.373 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.373 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.373 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.373 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.373 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.373 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.374 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.374 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.374 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.374 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.374 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.374 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.374 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.375 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.375 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.375 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.375 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.375 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.375 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.375 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.375 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.376 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.376 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.376 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.376 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.376 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.376 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.376 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.377 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.377 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.377 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.377 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.377 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.377 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.378 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.378 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.378 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.378 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.378 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.378 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.378 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.379 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.379 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.379 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.379 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.379 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.379 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.379 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.380 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.380 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.380 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.380 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.380 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.380 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.380 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.380 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.381 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.381 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.381 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.381 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.381 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.381 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.381 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.382 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.382 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.382 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.382 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.382 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.382 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.382 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator": { 2023-11-16T18:34:59.383 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2023-11-16T18:34:59.383 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.383 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.383 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.383 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.383 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.383 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.384 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.384 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm", 2023-11-16T18:34:59.384 INFO:teuthology.orchestra.run.smithi002.stdout: "rook", 2023-11-16T18:34:59.384 INFO:teuthology.orchestra.run.smithi002.stdout: "test_orchestrator" 2023-11-16T18:34:59.384 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.384 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Orchestrator backend", 2023-11-16T18:34:59.385 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.385 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.385 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.385 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.385 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.385 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.386 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.386 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_perf_query", 2023-11-16T18:34:59.386 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.386 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.387 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.387 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.387 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.387 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.388 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.388 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.388 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.388 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.388 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.389 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.389 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.389 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.389 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.389 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.390 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.390 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.390 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.390 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.391 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.391 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.391 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.391 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.391 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.392 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.392 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.392 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.392 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.392 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.393 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.393 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.393 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.393 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.393 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.393 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.393 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.394 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.394 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.394 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.394 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.394 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.394 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.395 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.395 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.395 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.395 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.395 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.395 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.395 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.396 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.396 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.396 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.396 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.396 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.396 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.397 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.397 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.397 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.397 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.397 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.397 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.397 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.397 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.398 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.398 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.398 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.398 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.398 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.398 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.398 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.399 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.399 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.399 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.399 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.399 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_support", 2023-11-16T18:34:59.399 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.399 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.401 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.401 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.401 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.401 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.401 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.401 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.401 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.402 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.402 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.402 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.402 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.402 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.402 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.402 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.402 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.403 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.403 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.403 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.403 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.403 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.403 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.403 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.404 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.404 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.404 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.404 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.404 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.404 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.404 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.404 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.405 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.405 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.405 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.405 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.405 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.405 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.405 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.406 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.406 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.406 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.406 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.406 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.406 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.406 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.407 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.407 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.407 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.407 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.407 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.407 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.407 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.407 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.408 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.408 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.408 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.408 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.408 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.408 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.409 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.409 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.409 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.409 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.409 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.409 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.409 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.409 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.410 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_autoscaler", 2023-11-16T18:34:59.410 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.410 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.410 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.410 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.410 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.410 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.410 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.411 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.411 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.411 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.411 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.411 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.411 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.411 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.411 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.412 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.412 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.412 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.412 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.412 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.412 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.412 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.413 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.413 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.413 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.413 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.413 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.413 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.413 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.414 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.414 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.414 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.414 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.414 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.414 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.414 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.415 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.415 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.415 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.415 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.415 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.415 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.415 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.416 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.416 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.416 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.416 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.416 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.416 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.416 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.417 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.417 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.417 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.417 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.417 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.417 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.417 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.418 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.418 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.418 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.418 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.418 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.418 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.418 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.418 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.419 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.419 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.419 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.419 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.419 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.419 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.419 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.419 INFO:teuthology.orchestra.run.smithi002.stdout: "noautoscale": { 2023-11-16T18:34:59.420 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "noautoscale", 2023-11-16T18:34:59.420 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.420 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.420 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.420 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.420 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.420 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.421 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.421 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "global autoscale flag", 2023-11-16T18:34:59.421 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-11-16T18:34:59.421 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.421 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.421 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.421 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:34:59.421 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:34:59.422 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:59.422 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.422 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.422 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:34:59.422 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.422 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.422 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.423 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.423 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.423 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.423 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.423 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.423 INFO:teuthology.orchestra.run.smithi002.stdout: "threshold": { 2023-11-16T18:34:59.423 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threshold", 2023-11-16T18:34:59.424 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:59.424 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.424 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.424 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2023-11-16T18:34:59.424 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1.0", 2023-11-16T18:34:59.424 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.424 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.424 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "scaling threshold", 2023-11-16T18:34:59.425 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-11-16T18:34:59.425 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.425 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.425 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.425 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.425 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.425 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.426 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "progress", 2023-11-16T18:34:59.426 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.426 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.426 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.426 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_pg_recovery_event": { 2023-11-16T18:34:59.426 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_pg_recovery_event", 2023-11-16T18:34:59.426 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.426 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.427 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.427 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.427 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.427 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.427 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.427 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow the module to show pg recovery progress", 2023-11-16T18:34:59.427 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.428 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.428 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.428 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.428 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2023-11-16T18:34:59.428 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2023-11-16T18:34:59.428 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.428 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.429 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.429 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.429 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.429 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.429 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.429 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.429 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.430 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.430 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.430 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.430 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.430 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.430 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.430 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.431 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.431 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.431 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.431 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.431 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.431 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.431 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.432 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.432 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.432 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.432 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.432 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.432 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.432 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.433 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.433 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.433 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.433 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.433 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.433 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.433 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.434 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.434 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.434 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.434 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.434 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.434 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.434 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.435 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.435 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.435 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.435 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.435 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.435 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.436 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.436 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.436 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.436 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.436 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.436 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.436 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.437 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.437 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.437 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.437 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.437 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.437 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.437 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.438 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.438 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.438 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.438 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.438 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.438 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.438 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.439 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.439 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.439 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.439 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.439 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.439 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.439 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.440 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.440 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.440 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.440 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.440 INFO:teuthology.orchestra.run.smithi002.stdout: "max_completed_events": { 2023-11-16T18:34:59.440 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_completed_events", 2023-11-16T18:34:59.440 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.441 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.441 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.441 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "50", 2023-11-16T18:34:59.441 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.441 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.441 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.441 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "number of past completed events to remember", 2023-11-16T18:34:59.442 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.442 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.442 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.442 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.442 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:34:59.442 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:34:59.442 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:59.443 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.443 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.443 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T18:34:59.443 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.443 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.443 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.443 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long the module is going to sleep", 2023-11-16T18:34:59.444 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.444 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.444 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.444 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.444 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.444 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.444 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.444 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus", 2023-11-16T18:34:59.445 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.445 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.445 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.445 INFO:teuthology.orchestra.run.smithi002.stdout: "cache": { 2023-11-16T18:34:59.445 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", 2023-11-16T18:34:59.445 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.445 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.446 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.446 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.446 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.446 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.446 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.446 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.446 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.446 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.447 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.447 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.447 INFO:teuthology.orchestra.run.smithi002.stdout: "exclude_perf_counters": { 2023-11-16T18:34:59.447 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "exclude_perf_counters", 2023-11-16T18:34:59.447 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.447 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.447 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.448 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.448 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.448 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.448 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.448 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-11-16T18:34:59.448 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-11-16T18:34:59.448 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.449 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.449 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.449 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.449 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.449 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.449 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.449 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.449 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.450 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.450 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.450 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.450 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.450 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.450 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.450 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.451 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.451 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.451 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.451 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.451 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.451 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.451 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.452 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.452 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.452 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.452 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.452 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.452 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.452 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.452 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.453 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.453 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.453 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.453 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.453 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.453 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.453 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.453 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.454 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.454 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.454 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.454 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.454 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.454 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.454 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.454 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.455 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.455 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.455 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.455 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.455 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.455 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.455 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.456 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.456 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.456 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.456 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.456 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.456 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.457 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.457 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.457 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.457 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.457 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.457 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.458 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.458 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.458 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.458 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.458 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools": { 2023-11-16T18:34:59.458 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools", 2023-11-16T18:34:59.458 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.458 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.459 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.459 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.459 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.459 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.459 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.459 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.459 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.460 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.460 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.460 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.460 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools_refresh_interval": { 2023-11-16T18:34:59.460 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-11-16T18:34:59.460 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.460 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.460 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.461 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300", 2023-11-16T18:34:59.461 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.461 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.461 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.461 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.461 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.461 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.462 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.462 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.462 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_interval": { 2023-11-16T18:34:59.462 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_interval", 2023-11-16T18:34:59.462 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:59.462 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.462 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.463 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15.0", 2023-11-16T18:34:59.463 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.463 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.463 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.463 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.463 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.463 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.463 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.464 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.464 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T18:34:59.464 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T18:34:59.464 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.464 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.464 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.464 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-11-16T18:34:59.465 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.465 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.465 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.465 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-11-16T18:34:59.465 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.465 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.465 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.465 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.466 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T18:34:59.466 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T18:34:59.466 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.466 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.466 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.466 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "9283", 2023-11-16T18:34:59.466 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.466 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.467 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.467 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-11-16T18:34:59.467 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.467 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.467 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.467 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.467 INFO:teuthology.orchestra.run.smithi002.stdout: "stale_cache_strategy": { 2023-11-16T18:34:59.468 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stale_cache_strategy", 2023-11-16T18:34:59.468 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.468 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.468 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.468 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "log", 2023-11-16T18:34:59.468 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.468 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.468 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.469 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.469 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.469 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.469 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.469 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.469 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2023-11-16T18:34:59.469 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2023-11-16T18:34:59.469 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.470 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.470 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.470 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "default", 2023-11-16T18:34:59.470 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.470 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.470 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.470 INFO:teuthology.orchestra.run.smithi002.stdout: "default", 2023-11-16T18:34:59.470 INFO:teuthology.orchestra.run.smithi002.stdout: "error" 2023-11-16T18:34:59.471 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.471 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.471 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.471 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.471 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.471 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.471 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2023-11-16T18:34:59.472 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2023-11-16T18:34:59.472 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.472 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.472 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.472 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2023-11-16T18:34:59.472 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-11-16T18:34:59.472 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2023-11-16T18:34:59.472 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.473 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.473 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.473 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.473 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.473 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.473 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.473 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.473 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.474 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_support", 2023-11-16T18:34:59.474 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.474 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.474 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.474 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.474 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.474 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.475 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.475 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.475 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.475 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.475 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.475 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.475 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.475 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.476 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.476 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.476 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.476 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.476 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.476 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.476 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.476 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.477 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.477 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.477 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.477 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.477 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.477 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.477 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.477 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.478 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.478 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.478 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.478 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.478 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.478 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.478 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.479 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.479 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.479 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.479 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.479 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.479 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.479 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.480 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.480 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.480 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.480 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.480 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.480 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.480 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.480 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.481 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.481 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.481 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.481 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.481 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.481 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.481 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.481 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.482 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.482 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.482 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.482 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.482 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.482 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.482 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.483 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.483 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.483 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.483 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.483 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.483 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_snap_create": { 2023-11-16T18:34:59.483 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_snap_create", 2023-11-16T18:34:59.484 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.484 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.484 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.484 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:59.484 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.484 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.485 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.485 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.485 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.485 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.485 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.485 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.485 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { 2023-11-16T18:34:59.485 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", 2023-11-16T18:34:59.486 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.486 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.486 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.486 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.486 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.486 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.486 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.487 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.487 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.487 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.487 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.487 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.487 INFO:teuthology.orchestra.run.smithi002.stdout: "trash_purge_schedule": { 2023-11-16T18:34:59.487 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "trash_purge_schedule", 2023-11-16T18:34:59.487 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.488 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.488 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.488 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.488 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.488 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.488 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.488 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.489 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.489 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.489 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.489 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.489 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.489 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.489 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.490 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "restful", 2023-11-16T18:34:59.490 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.490 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.490 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.490 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_auth": { 2023-11-16T18:34:59.490 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_auth", 2023-11-16T18:34:59.490 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.490 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.491 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.491 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.491 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.491 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.491 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.491 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.491 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.491 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.492 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.492 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.492 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2023-11-16T18:34:59.492 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2023-11-16T18:34:59.492 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.492 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.492 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.492 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.493 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.493 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.493 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.493 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.493 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.493 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.493 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.493 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.494 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.494 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.494 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.494 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.494 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.494 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.494 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.495 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.495 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.495 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.495 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.495 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.495 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.495 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.495 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.496 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.496 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.496 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.496 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.496 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.496 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.496 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.496 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.497 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.497 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.497 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.497 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.497 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.497 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.498 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.498 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.498 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.498 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.498 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.498 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.498 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.498 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.499 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.499 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.499 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.499 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.499 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.499 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.499 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.500 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.500 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.500 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.500 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.500 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.500 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.500 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.500 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.501 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.501 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.501 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.501 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.501 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.501 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.501 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.502 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.502 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.502 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.502 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.502 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.502 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.502 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.503 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.503 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.503 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.503 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T18:34:59.503 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T18:34:59.503 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.503 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.503 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.504 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.504 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.504 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.504 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.504 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.504 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.504 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.504 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.505 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.505 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T18:34:59.505 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T18:34:59.505 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.505 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.505 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.505 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.505 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.506 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.506 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.506 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.506 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.506 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.506 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.506 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.507 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.507 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.507 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.507 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rgw", 2023-11-16T18:34:59.507 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.507 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.507 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.508 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.508 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.508 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.508 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.508 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.508 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.508 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.509 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.509 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.509 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.509 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.509 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.509 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.509 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.509 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.510 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.510 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.510 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.510 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.510 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.510 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.510 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.511 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.511 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.511 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.511 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.511 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.511 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.511 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.511 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.512 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.512 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.512 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.512 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.512 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.512 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.512 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.513 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.513 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.513 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.513 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.513 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.513 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.513 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.513 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.514 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.514 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.514 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.514 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.514 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.514 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.514 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.515 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.515 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.515 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.515 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.515 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.515 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.515 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.516 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.516 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.516 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.516 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.516 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.517 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.517 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.517 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.517 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.517 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.517 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.517 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rook", 2023-11-16T18:34:59.518 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.518 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.518 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.518 INFO:teuthology.orchestra.run.smithi002.stdout: "drive_group_interval": { 2023-11-16T18:34:59.518 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "drive_group_interval", 2023-11-16T18:34:59.518 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:59.518 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.519 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300.0", 2023-11-16T18:34:59.519 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.519 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.519 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.519 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-11-16T18:34:59.519 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.519 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.520 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.520 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.520 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.520 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.520 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.520 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.520 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.520 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.521 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.521 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.521 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.521 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.521 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.521 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.521 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.522 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.522 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.522 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.522 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.522 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.522 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.522 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.522 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.523 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.523 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.523 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.523 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.523 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.523 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.523 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.524 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.524 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.524 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.524 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.524 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.524 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.524 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.524 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.525 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.525 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.525 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.525 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.525 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.525 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.525 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.525 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.526 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.526 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.526 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.526 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.526 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.526 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.526 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.527 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.527 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.527 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.527 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.527 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.527 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.527 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.527 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.528 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.528 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.528 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.528 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.528 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.528 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.528 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.528 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.529 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.529 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.529 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.529 INFO:teuthology.orchestra.run.smithi002.stdout: "storage_class": { 2023-11-16T18:34:59.529 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "storage_class", 2023-11-16T18:34:59.529 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.529 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.530 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.530 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "local", 2023-11-16T18:34:59.530 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.530 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.530 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.530 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-11-16T18:34:59.530 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.530 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.531 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.531 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.531 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.531 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.531 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.531 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "selftest", 2023-11-16T18:34:59.531 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.532 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.532 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.532 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.532 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.532 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.532 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.532 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.533 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.533 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.533 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.533 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.533 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.533 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.533 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.533 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.534 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.534 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.534 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.534 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.534 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.534 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.534 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.535 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.535 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.535 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.535 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.535 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.535 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.535 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.536 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.536 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.536 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.536 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.536 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.536 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.537 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.537 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.537 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.537 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.537 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.537 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.537 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.537 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.538 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.538 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.538 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.538 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.538 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.538 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.538 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.539 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.539 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.539 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.539 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.539 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.539 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.539 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.539 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.539 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.540 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.540 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.540 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.540 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.540 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.540 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.540 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.541 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.541 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.541 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.541 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.541 INFO:teuthology.orchestra.run.smithi002.stdout: "roption1": { 2023-11-16T18:34:59.541 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption1", 2023-11-16T18:34:59.541 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.541 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.542 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.542 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.542 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.542 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.542 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.542 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.542 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.543 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.543 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.543 INFO:teuthology.orchestra.run.smithi002.stdout: "roption2": { 2023-11-16T18:34:59.543 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption2", 2023-11-16T18:34:59.543 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.543 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.543 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.544 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "xyz", 2023-11-16T18:34:59.544 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.544 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.544 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.544 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.544 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.544 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.545 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.545 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.545 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption1": { 2023-11-16T18:34:59.545 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption1", 2023-11-16T18:34:59.545 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.545 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.545 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.546 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.546 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.546 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.546 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.546 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.546 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.546 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.547 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.547 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.547 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption2": { 2023-11-16T18:34:59.547 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption2", 2023-11-16T18:34:59.547 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.547 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.547 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.547 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.548 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.548 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.548 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.548 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.548 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.548 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.548 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.548 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.549 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption3": { 2023-11-16T18:34:59.549 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption3", 2023-11-16T18:34:59.549 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:59.549 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.549 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.549 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.549 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.550 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.550 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.550 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.550 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.550 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.550 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.550 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.550 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption4": { 2023-11-16T18:34:59.551 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption4", 2023-11-16T18:34:59.551 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.551 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.551 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.551 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.551 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.551 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.551 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption5": { 2023-11-16T18:34:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption5", 2023-11-16T18:34:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.553 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.553 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.553 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.553 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.553 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.553 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.553 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.554 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.554 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.554 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.554 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption6": { 2023-11-16T18:34:59.554 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption6", 2023-11-16T18:34:59.554 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.554 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.555 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.555 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.555 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.555 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.555 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.555 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.555 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.555 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.556 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.556 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.556 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption7": { 2023-11-16T18:34:59.556 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption7", 2023-11-16T18:34:59.556 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.556 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.556 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.557 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.557 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T18:34:59.557 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "42", 2023-11-16T18:34:59.557 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.557 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.557 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.557 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.557 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.557 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.558 INFO:teuthology.orchestra.run.smithi002.stdout: "testkey": { 2023-11-16T18:34:59.558 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testkey", 2023-11-16T18:34:59.558 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.558 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.558 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.558 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.558 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.558 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.559 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.559 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.559 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.559 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.559 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.559 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.559 INFO:teuthology.orchestra.run.smithi002.stdout: "testlkey": { 2023-11-16T18:34:59.560 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testlkey", 2023-11-16T18:34:59.560 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.560 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.560 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.560 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.560 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.560 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.561 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.561 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.561 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.561 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.561 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.561 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.561 INFO:teuthology.orchestra.run.smithi002.stdout: "testnewline": { 2023-11-16T18:34:59.561 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testnewline", 2023-11-16T18:34:59.562 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.562 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.562 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.562 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.562 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.562 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.562 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.563 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.563 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.563 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.563 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.563 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.564 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2023-11-16T18:34:59.564 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.564 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.564 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.564 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_m_granularity": { 2023-11-16T18:34:59.564 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_m_granularity", 2023-11-16T18:34:59.564 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.564 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.565 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.565 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.565 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.565 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.565 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.565 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow minute scheduled snapshots", 2023-11-16T18:34:59.565 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.566 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.566 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.566 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.566 INFO:teuthology.orchestra.run.smithi002.stdout: "dump_on_update": { 2023-11-16T18:34:59.566 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dump_on_update", 2023-11-16T18:34:59.566 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.566 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.566 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.567 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.567 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.567 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.567 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.567 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "dump database to debug log on update", 2023-11-16T18:34:59.567 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.567 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.567 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.568 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.568 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.568 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.568 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.568 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.568 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.568 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.568 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.569 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.569 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.569 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.569 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.569 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.569 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.569 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.569 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.570 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.570 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.570 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.570 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.570 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.570 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.570 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.571 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.571 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.571 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.571 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.571 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.571 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.571 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.571 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.572 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.572 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.572 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.572 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.572 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.572 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.572 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.572 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.572 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.573 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.573 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.573 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.573 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.573 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.573 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.573 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.573 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.574 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.574 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.574 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.574 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.574 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.574 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.574 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.575 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.575 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.575 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.575 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.575 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.575 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.575 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.575 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.576 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.576 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.576 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.576 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.576 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.576 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.576 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.577 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.577 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.577 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.577 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.577 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stats", 2023-11-16T18:34:59.577 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.577 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.577 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.578 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.578 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.578 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.578 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.578 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.578 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.578 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.578 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.579 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.579 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.579 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.579 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.579 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.579 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.579 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.579 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.580 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.581 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.582 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.582 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.582 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.582 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.582 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.582 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.582 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.583 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.583 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.583 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.583 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.583 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.583 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.583 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.583 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.584 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.584 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.584 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.584 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.584 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.584 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.584 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.585 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.585 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.585 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.585 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.585 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.585 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.585 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.586 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.586 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.586 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.586 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.586 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.586 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.586 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.586 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.587 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.587 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.587 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.587 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.587 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "status", 2023-11-16T18:34:59.587 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.587 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.588 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.588 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.588 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.588 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.588 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.588 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.589 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.589 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.589 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.589 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.589 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.589 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.590 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.590 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.590 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.591 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.591 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.591 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.591 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.591 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.591 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.591 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.592 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.592 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.592 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.592 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.592 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.592 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.592 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.592 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.593 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.593 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.593 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.593 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.593 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.593 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.593 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.594 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.594 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.594 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.594 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.594 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.594 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.594 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.594 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.595 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.595 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.595 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.595 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.595 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.595 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.595 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.595 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.596 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.596 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.596 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.596 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.596 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.596 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.596 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.596 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.597 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.597 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.597 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.597 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.597 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.597 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.597 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.597 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.598 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.598 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.598 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.598 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.598 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.598 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telegraf", 2023-11-16T18:34:59.598 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.599 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.599 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.599 INFO:teuthology.orchestra.run.smithi002.stdout: "address": { 2023-11-16T18:34:59.599 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "address", 2023-11-16T18:34:59.599 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.599 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.599 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.600 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-11-16T18:34:59.600 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.600 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.600 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.600 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.600 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.600 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.601 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.601 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.601 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:34:59.601 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:34:59.601 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:59.601 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.601 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.602 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15", 2023-11-16T18:34:59.602 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.602 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.602 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.602 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.602 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.602 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.603 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.603 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.603 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.603 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.603 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.603 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.603 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.604 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.604 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.604 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.604 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.604 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.604 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.604 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.604 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.605 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.605 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.605 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.605 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.605 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.605 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.605 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.605 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.606 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.606 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.606 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.606 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.606 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.606 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.606 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.607 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.607 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.607 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.607 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.607 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.607 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.607 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.607 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.608 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.608 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.608 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.608 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.608 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.608 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.608 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.609 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.609 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.609 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.609 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.609 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.609 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.609 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.609 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.609 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.610 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.610 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.610 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.610 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.610 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.610 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.610 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.611 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.611 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.611 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.611 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.611 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.611 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.611 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.611 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.612 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.612 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.612 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.612 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.612 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.612 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.612 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telemetry", 2023-11-16T18:34:59.612 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.613 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.613 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.613 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_basic": { 2023-11-16T18:34:59.613 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_basic", 2023-11-16T18:34:59.613 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.613 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.613 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.613 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.614 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.614 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.614 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.614 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share basic cluster information (size, version)", 2023-11-16T18:34:59.614 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.614 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.614 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.614 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.615 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_crash": { 2023-11-16T18:34:59.615 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_crash", 2023-11-16T18:34:59.615 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.615 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.615 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.615 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.615 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.616 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.616 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.616 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-11-16T18:34:59.616 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.616 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.616 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.616 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.616 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_device": { 2023-11-16T18:34:59.617 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_device", 2023-11-16T18:34:59.617 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.617 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.617 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.617 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.617 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.617 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.617 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.618 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-11-16T18:34:59.618 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.618 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.618 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.618 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.618 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_ident": { 2023-11-16T18:34:59.618 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_ident", 2023-11-16T18:34:59.619 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.619 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.619 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.619 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.619 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.619 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.619 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.620 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-11-16T18:34:59.620 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.620 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.620 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.620 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.620 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_perf": { 2023-11-16T18:34:59.620 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_perf", 2023-11-16T18:34:59.620 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.621 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.621 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.621 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.621 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.621 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.621 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.621 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share various performance metrics of a cluster", 2023-11-16T18:34:59.621 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.622 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.622 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.622 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.622 INFO:teuthology.orchestra.run.smithi002.stdout: "contact": { 2023-11-16T18:34:59.622 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "contact", 2023-11-16T18:34:59.622 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.622 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.623 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.623 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.623 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.623 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.623 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.623 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.624 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.624 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.624 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.624 INFO:teuthology.orchestra.run.smithi002.stdout: "description": { 2023-11-16T18:34:59.624 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "description", 2023-11-16T18:34:59.624 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.624 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.624 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.625 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.625 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.625 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.625 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.625 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.625 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.625 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.625 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.626 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.626 INFO:teuthology.orchestra.run.smithi002.stdout: "device_url": { 2023-11-16T18:34:59.626 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_url", 2023-11-16T18:34:59.626 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.626 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.626 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.626 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-11-16T18:34:59.627 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.627 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.627 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.627 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.627 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.627 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.627 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.627 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.628 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2023-11-16T18:34:59.628 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2023-11-16T18:34:59.628 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.628 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.628 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.628 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.628 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.629 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.629 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.629 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.629 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.629 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.629 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.629 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.630 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:34:59.630 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:34:59.630 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.630 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.630 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.630 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "24", 2023-11-16T18:34:59.630 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", 2023-11-16T18:34:59.631 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.631 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.631 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.631 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.631 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.631 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.631 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.632 INFO:teuthology.orchestra.run.smithi002.stdout: "last_opt_revision": { 2023-11-16T18:34:59.632 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "last_opt_revision", 2023-11-16T18:34:59.632 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.632 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.632 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.632 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1", 2023-11-16T18:34:59.632 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.633 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.633 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.633 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.633 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.633 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.633 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.633 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.633 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard": { 2023-11-16T18:34:59.634 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard", 2023-11-16T18:34:59.634 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.634 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.634 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.634 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.634 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.634 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.634 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.635 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.635 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.635 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.635 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.635 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.635 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard_description": { 2023-11-16T18:34:59.635 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard_description", 2023-11-16T18:34:59.636 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.636 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.636 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.636 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.636 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.636 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.636 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.636 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.637 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.637 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.637 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.637 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.637 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.637 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.637 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.637 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.638 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.638 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.638 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.638 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.638 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.638 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.638 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.639 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.639 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.639 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.639 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.639 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.639 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.639 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.639 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.640 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.640 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.640 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.640 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.640 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.640 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.640 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.641 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.641 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.641 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.641 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.641 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.641 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.641 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.642 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.642 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.642 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.642 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.642 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.642 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.642 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.643 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.643 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.643 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.643 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.643 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.643 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.643 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.644 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.644 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.644 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.644 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.644 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.644 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.644 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.645 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.645 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.645 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.645 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.645 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.645 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.645 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.646 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.646 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.646 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.646 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.646 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.646 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.646 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.647 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.647 INFO:teuthology.orchestra.run.smithi002.stdout: "organization": { 2023-11-16T18:34:59.647 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "organization", 2023-11-16T18:34:59.647 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.647 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.647 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.647 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.647 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.648 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.648 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.648 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.648 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.648 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.648 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.648 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.649 INFO:teuthology.orchestra.run.smithi002.stdout: "proxy": { 2023-11-16T18:34:59.649 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "proxy", 2023-11-16T18:34:59.649 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.649 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.649 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.649 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.649 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.649 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.650 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.650 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.650 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.650 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.650 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.650 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.650 INFO:teuthology.orchestra.run.smithi002.stdout: "url": { 2023-11-16T18:34:59.651 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url", 2023-11-16T18:34:59.651 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.651 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.651 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.651 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-11-16T18:34:59.651 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.651 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.651 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.652 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.652 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.652 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.652 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.652 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.652 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.652 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.653 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.653 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "test_orchestrator", 2023-11-16T18:34:59.653 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.653 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.653 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.653 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.653 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.654 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.654 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.654 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.654 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.654 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.654 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.654 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.655 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.655 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.655 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.655 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.655 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.655 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.655 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.655 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.656 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.656 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.656 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.656 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.656 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.656 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.656 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.657 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.657 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.657 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.657 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.657 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.657 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.657 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.657 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.658 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.658 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.658 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.658 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.658 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.658 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.658 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.658 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.659 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.659 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.659 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.659 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.659 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.659 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.659 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.659 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.660 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.660 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.660 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.660 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.660 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.660 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.660 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.660 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.661 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.661 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.661 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.661 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.661 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.661 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.661 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.661 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.662 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.662 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.662 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.662 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.662 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.662 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.662 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.663 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.663 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "volumes", 2023-11-16T18:34:59.663 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.663 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.663 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.663 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.663 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.663 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.664 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.664 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.664 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.664 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.665 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.665 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.665 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.665 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.665 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.665 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.665 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.665 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.666 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.666 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.666 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.666 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.666 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.666 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.666 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.666 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.667 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.667 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.667 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.667 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.667 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.667 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.667 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.668 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.668 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.668 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.668 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.668 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.668 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.668 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.668 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.669 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.669 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.669 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.669 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.669 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.669 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.669 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.670 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.670 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.670 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.670 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.670 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.670 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.670 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.670 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.671 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.671 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.671 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.671 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.671 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.671 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.671 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.672 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.672 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.672 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.672 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.672 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.672 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.672 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.672 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_clones": { 2023-11-16T18:34:59.673 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_clones", 2023-11-16T18:34:59.673 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.673 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.673 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.673 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4", 2023-11-16T18:34:59.673 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.673 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.673 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.674 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Number of asynchronous cloner threads", 2023-11-16T18:34:59.674 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.674 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.674 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.674 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.674 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot_clone_delay": { 2023-11-16T18:34:59.674 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snapshot_clone_delay", 2023-11-16T18:34:59.675 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.675 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.675 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.675 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:34:59.675 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.675 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.675 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.675 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-11-16T18:34:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.676 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix", 2023-11-16T18:34:59.677 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.677 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.677 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.677 INFO:teuthology.orchestra.run.smithi002.stdout: "discovery_interval": { 2023-11-16T18:34:59.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "discovery_interval", 2023-11-16T18:34:59.677 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:59.677 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.677 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.677 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "100", 2023-11-16T18:34:59.678 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.678 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.678 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.678 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.678 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.678 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.678 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.679 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.679 INFO:teuthology.orchestra.run.smithi002.stdout: "identifier": { 2023-11-16T18:34:59.679 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "identifier", 2023-11-16T18:34:59.679 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.679 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.679 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.679 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.679 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.680 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.680 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.680 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.680 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.680 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.680 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.680 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.680 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:34:59.681 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:34:59.681 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:59.681 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.681 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.681 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:34:59.681 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.681 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.681 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.682 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.682 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.682 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.682 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.682 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.682 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.682 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.683 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.683 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.683 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.683 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.683 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.683 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.683 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.684 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.684 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.684 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.684 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.684 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.684 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.684 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.685 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.685 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.685 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.685 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.685 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.710 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.710 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.710 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.710 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.711 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.711 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.711 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.711 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.711 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.711 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.711 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.711 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.712 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.712 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.712 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.712 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.712 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.712 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.712 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.712 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.713 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.713 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.713 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.713 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.713 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.713 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.713 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.714 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.766 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.766 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.767 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.767 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.767 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.767 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.768 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.768 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.768 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.769 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.769 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.769 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.770 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.770 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.770 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.770 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.771 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.771 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.771 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.793 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_host": { 2023-11-16T18:34:59.794 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_host", 2023-11-16T18:34:59.794 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.794 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.794 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.794 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.794 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.795 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.795 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.795 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.795 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.795 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.795 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.795 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.795 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_port": { 2023-11-16T18:34:59.796 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_port", 2023-11-16T18:34:59.796 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.796 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.796 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.796 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10051", 2023-11-16T18:34:59.796 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.796 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.796 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.797 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.797 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.797 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.797 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.797 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.797 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_sender": { 2023-11-16T18:34:59.797 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_sender", 2023-11-16T18:34:59.797 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.798 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.798 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.798 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-11-16T18:34:59.798 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.798 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.798 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.798 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.799 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.799 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.799 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.799 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.799 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.799 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.799 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T18:34:59.799 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.799 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.800 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2023-11-16T18:34:59.800 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm", 2023-11-16T18:34:59.800 INFO:teuthology.orchestra.run.smithi002.stdout: "dashboard", 2023-11-16T18:34:59.800 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2023-11-16T18:34:59.800 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2023-11-16T18:34:59.800 INFO:teuthology.orchestra.run.smithi002.stdout: "restful", 2023-11-16T18:34:59.800 INFO:teuthology.orchestra.run.smithi002.stdout: "snap_schedule" 2023-11-16T18:34:59.800 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.801 INFO:teuthology.orchestra.run.smithi002.stdout: "available_modules": [ 2023-11-16T18:34:59.801 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.801 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alerts", 2023-11-16T18:34:59.801 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.801 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.801 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.801 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:34:59.801 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:34:59.802 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:59.802 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.802 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.802 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:34:59.802 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.803 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.803 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.803 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How frequently to reexamine health status", 2023-11-16T18:34:59.804 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.804 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.804 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.804 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.805 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.805 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.805 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.805 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.806 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.806 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.807 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.807 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.807 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.807 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.808 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.808 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.808 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.808 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.809 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.809 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.809 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.809 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.810 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.810 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.810 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.810 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.811 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.811 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.811 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.812 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.812 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.812 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.812 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.813 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.813 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.813 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.813 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.814 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.814 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.814 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.814 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.815 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.815 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.815 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.815 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.816 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.816 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.816 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.816 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.817 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.817 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.817 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.818 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.818 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.818 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.818 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.819 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.819 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.819 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.819 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.820 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.820 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.820 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.820 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.821 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.821 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.821 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.821 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.822 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.822 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.822 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.823 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.823 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.823 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_destination": { 2023-11-16T18:34:59.823 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", 2023-11-16T18:34:59.824 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.824 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.824 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.824 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.825 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.825 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.825 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.825 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email address to send alerts to", 2023-11-16T18:34:59.826 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.826 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.826 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.826 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.827 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_from_name": { 2023-11-16T18:34:59.827 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_from_name", 2023-11-16T18:34:59.827 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.827 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.828 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.828 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "Ceph", 2023-11-16T18:34:59.828 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.828 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.829 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.829 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email From: name", 2023-11-16T18:34:59.829 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.829 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.830 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.830 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.830 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_host": { 2023-11-16T18:34:59.831 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_host", 2023-11-16T18:34:59.831 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.831 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.831 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.832 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.832 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.832 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.832 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.833 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP server", 2023-11-16T18:34:59.833 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.833 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.833 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.834 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.834 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_password": { 2023-11-16T18:34:59.834 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_password", 2023-11-16T18:34:59.834 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.835 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.835 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.835 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.856 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.856 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.857 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.857 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Password to authenticate with", 2023-11-16T18:34:59.857 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.857 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.857 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.857 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.858 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_port": { 2023-11-16T18:34:59.858 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_port", 2023-11-16T18:34:59.858 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.858 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.858 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.858 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", 2023-11-16T18:34:59.858 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.858 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.859 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.859 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP port", 2023-11-16T18:34:59.859 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.859 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.859 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.859 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.859 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_sender": { 2023-11-16T18:34:59.860 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_sender", 2023-11-16T18:34:59.860 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.860 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.860 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.860 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.860 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.860 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.860 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.861 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP envelope sender", 2023-11-16T18:34:59.861 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.861 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.861 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.861 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.861 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_ssl": { 2023-11-16T18:34:59.861 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_ssl", 2023-11-16T18:34:59.861 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.862 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.862 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.862 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.862 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.862 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.862 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.862 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use SSL to connect to SMTP server", 2023-11-16T18:34:59.863 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.863 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.863 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.863 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.863 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_user": { 2023-11-16T18:34:59.863 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_user", 2023-11-16T18:34:59.863 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.864 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.864 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.864 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.864 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.865 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.865 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.865 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "User to authenticate as", 2023-11-16T18:34:59.865 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.865 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.865 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.865 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.866 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.866 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.866 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.867 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "balancer", 2023-11-16T18:34:59.867 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.867 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.867 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.867 INFO:teuthology.orchestra.run.smithi002.stdout: "active": { 2023-11-16T18:34:59.867 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "active", 2023-11-16T18:34:59.868 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.868 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.868 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.868 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:34:59.868 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.868 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.868 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.869 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically balance PGs across cluster", 2023-11-16T18:34:59.869 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.869 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.869 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.869 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.869 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_time": { 2023-11-16T18:34:59.869 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_time", 2023-11-16T18:34:59.870 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.870 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.870 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.870 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0000", 2023-11-16T18:34:59.870 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.870 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.870 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.895 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "beginning time of day to automatically balance", 2023-11-16T18:34:59.895 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T18:34:59.895 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.895 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.895 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.896 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_weekday": { 2023-11-16T18:34:59.896 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_weekday", 2023-11-16T18:34:59.896 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:59.896 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.896 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.896 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:34:59.896 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2023-11-16T18:34:59.896 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2023-11-16T18:34:59.897 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.897 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-11-16T18:34:59.897 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T18:34:59.897 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.897 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.897 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.897 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_max_iterations": { 2023-11-16T18:34:59.943 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", 2023-11-16T18:34:59.944 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:59.944 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.945 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.945 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "25", 2023-11-16T18:34:59.945 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T18:34:59.945 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "250", 2023-11-16T18:34:59.945 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.946 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-11-16T18:34:59.946 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.946 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.946 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.946 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.946 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_metrics": { 2023-11-16T18:34:59.946 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_metrics", 2023-11-16T18:34:59.947 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.947 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.947 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.947 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "pgs,objects,bytes", 2023-11-16T18:34:59.947 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.947 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.947 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.947 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-11-16T18:34:59.948 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-11-16T18:34:59.948 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.948 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.948 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.948 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_step": { 2023-11-16T18:34:59.948 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_step", 2023-11-16T18:34:59.948 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:59.948 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.949 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.949 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.5", 2023-11-16T18:34:59.949 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0.001", 2023-11-16T18:34:59.949 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "0.999", 2023-11-16T18:34:59.949 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.949 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "aggressiveness of optimization", 2023-11-16T18:34:59.949 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-11-16T18:34:59.949 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.950 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.950 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.950 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { 2023-11-16T18:34:59.950 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", 2023-11-16T18:34:59.950 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.950 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.950 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.951 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2359", 2023-11-16T18:34:59.951 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.951 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.951 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.951 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ending time of day to automatically balance", 2023-11-16T18:34:59.951 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T18:34:59.951 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.951 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.952 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.952 INFO:teuthology.orchestra.run.smithi002.stdout: "end_weekday": { 2023-11-16T18:34:59.953 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_weekday", 2023-11-16T18:34:59.953 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:59.953 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.953 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.953 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:34:59.954 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2023-11-16T18:34:59.954 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2023-11-16T18:34:59.954 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.954 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-11-16T18:34:59.954 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T18:34:59.954 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.954 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.955 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.955 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:34:59.955 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:34:59.955 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.955 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.955 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.956 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.956 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.956 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.956 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.956 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.957 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.957 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.957 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.957 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.957 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.958 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.958 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.958 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.958 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.958 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.959 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.959 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:34:59.959 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:34:59.959 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.959 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.960 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.960 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.960 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.960 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.960 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.961 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.961 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.961 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.961 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.961 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.962 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:34:59.962 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:34:59.962 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.962 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.963 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.963 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:34:59.963 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.963 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.963 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.964 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:34:59.964 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:34:59.964 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:34:59.964 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:34:59.964 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:34:59.965 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:34:59.965 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.965 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.965 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.965 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.965 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.965 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.966 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:34:59.966 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:34:59.966 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:34:59.966 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.966 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.966 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:34:59.966 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.966 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.967 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.967 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:34:59.967 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.967 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.967 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.967 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.967 INFO:teuthology.orchestra.run.smithi002.stdout: "min_score": { 2023-11-16T18:34:59.968 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_score", 2023-11-16T18:34:59.968 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:59.968 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.968 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.968 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:34:59.968 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.968 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.969 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.969 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-11-16T18:34:59.969 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.969 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.969 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.969 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.969 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2023-11-16T18:34:59.969 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2023-11-16T18:34:59.970 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.970 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.970 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.970 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "upmap", 2023-11-16T18:34:59.970 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.970 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.970 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:34:59.970 INFO:teuthology.orchestra.run.smithi002.stdout: "crush-compat", 2023-11-16T18:34:59.971 INFO:teuthology.orchestra.run.smithi002.stdout: "none", 2023-11-16T18:34:59.971 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap" 2023-11-16T18:34:59.971 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:34:59.971 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Balancer mode", 2023-11-16T18:34:59.971 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.971 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.971 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.971 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.972 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_ids": { 2023-11-16T18:34:59.972 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_ids", 2023-11-16T18:34:59.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:34:59.972 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.972 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.972 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:34:59.972 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.972 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.973 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.973 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-11-16T18:34:59.973 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.973 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.973 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.973 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.973 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:34:59.973 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:34:59.974 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:34:59.974 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.974 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.974 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:34:59.974 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.974 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.974 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.975 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-11-16T18:34:59.975 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.975 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.975 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.975 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.975 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_deviation": { 2023-11-16T18:34:59.975 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_deviation", 2023-11-16T18:34:59.975 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:34:59.976 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.976 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.976 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T18:34:59.976 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T18:34:59.976 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.976 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.976 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "deviation below which no optimization is attempted", 2023-11-16T18:34:59.976 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-11-16T18:34:59.977 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.977 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.977 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.977 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_optimizations": { 2023-11-16T18:34:59.977 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_optimizations", 2023-11-16T18:34:59.977 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:34:59.977 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.977 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:34:59.978 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:34:59.978 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.978 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:34:59.978 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:34:59.978 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-11-16T18:34:59.978 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:34:59.978 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:34:59.979 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:34:59.979 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.979 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:34:59.979 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:34:59.979 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:34:59.979 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephadm", 2023-11-16T18:34:59.979 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:34:59.979 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:34:59.980 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:34:59.980 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_down_multiplier": { 2023-11-16T18:34:59.980 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_down_multiplier", 2023-11-16T18:34:59.980 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:34:59.980 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:34:59.980 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:34:59.980 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2023-11-16T18:34:59.981 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:34:59.981 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.053 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-11-16T18:35:00.053 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.053 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.053 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.053 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.053 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_refresh_rate": { 2023-11-16T18:35:00.054 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_refresh_rate", 2023-11-16T18:35:00.054 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.054 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.054 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.054 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "20", 2023-11-16T18:35:00.054 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.054 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.054 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.055 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-11-16T18:35:00.055 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.055 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.055 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.055 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.055 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_starting_port": { 2023-11-16T18:35:00.055 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_starting_port", 2023-11-16T18:35:00.055 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.056 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.056 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.056 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4721", 2023-11-16T18:35:00.056 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.056 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.056 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.056 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-11-16T18:35:00.056 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.057 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.057 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.058 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.058 INFO:teuthology.orchestra.run.smithi002.stdout: "alertmanager_web_password": { 2023-11-16T18:35:00.058 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alertmanager_web_password", 2023-11-16T18:35:00.058 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.058 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.058 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.058 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:35:00.058 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.059 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.059 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.059 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Alertmanager web password", 2023-11-16T18:35:00.059 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.059 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.059 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.059 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.059 INFO:teuthology.orchestra.run.smithi002.stdout: "alertmanager_web_user": { 2023-11-16T18:35:00.060 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alertmanager_web_user", 2023-11-16T18:35:00.166 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.166 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.166 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.166 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:35:00.166 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.166 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.167 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Alertmanager web user", 2023-11-16T18:35:00.167 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.167 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.167 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.167 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.167 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_ptrace": { 2023-11-16T18:35:00.167 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_ptrace", 2023-11-16T18:35:00.168 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.168 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.168 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.168 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.168 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.168 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.168 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-11-16T18:35:00.169 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-11-16T18:35:00.169 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.169 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.169 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.169 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_interval": { 2023-11-16T18:35:00.169 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_interval", 2023-11-16T18:35:00.169 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.169 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.170 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.170 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:35:00.186 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.186 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.186 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.186 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to autotune daemon memory", 2023-11-16T18:35:00.186 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.187 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.187 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.187 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.187 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { 2023-11-16T18:35:00.187 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-11-16T18:35:00.187 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:35:00.187 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.187 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.188 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.7", 2023-11-16T18:35:00.188 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.188 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.191 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.255 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-11-16T18:35:00.255 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.255 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.255 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.256 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.256 INFO:teuthology.orchestra.run.smithi002.stdout: "cgroups_split": { 2023-11-16T18:35:00.256 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cgroups_split", 2023-11-16T18:35:00.256 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.256 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.256 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.256 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.256 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.257 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.257 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.257 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-11-16T18:35:00.257 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.257 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.257 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.257 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.257 INFO:teuthology.orchestra.run.smithi002.stdout: "config_checks_enabled": { 2023-11-16T18:35:00.258 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_checks_enabled", 2023-11-16T18:35:00.258 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.258 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.260 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.261 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.261 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.261 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.261 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.261 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-11-16T18:35:00.261 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-11-16T18:35:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_dashboard", 2023-11-16T18:35:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-11-16T18:35:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.264 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.264 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_alertmanager": { 2023-11-16T18:35:00.264 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_alertmanager", 2023-11-16T18:35:00.264 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.264 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.264 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.264 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-11-16T18:35:00.264 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.265 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.265 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.265 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T18:35:00.265 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.265 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.265 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.265 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.265 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_base": { 2023-11-16T18:35:00.265 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_base", 2023-11-16T18:35:00.266 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.266 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.266 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.266 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph", 2023-11-16T18:35:00.266 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.266 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.266 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.266 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Container image name, without the tag", 2023-11-16T18:35:00.267 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.267 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.267 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.267 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.267 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_elasticsearch": { 2023-11-16T18:35:00.267 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_elasticsearch", 2023-11-16T18:35:00.267 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.267 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.268 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.268 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-11-16T18:35:00.268 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.268 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.268 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.268 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "elasticsearch container image", 2023-11-16T18:35:00.268 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.268 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.269 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.269 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.269 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_grafana": { 2023-11-16T18:35:00.269 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_grafana", 2023-11-16T18:35:00.269 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.269 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.269 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.269 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-11-16T18:35:00.270 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.270 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.270 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.270 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T18:35:00.270 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.270 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.271 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_haproxy": { 2023-11-16T18:35:00.271 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_haproxy", 2023-11-16T18:35:00.271 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.271 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.271 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.271 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-11-16T18:35:00.271 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.271 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.272 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.272 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "HAproxy container image", 2023-11-16T18:35:00.272 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.272 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.272 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.272 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.272 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_agent": { 2023-11-16T18:35:00.272 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_agent", 2023-11-16T18:35:00.272 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.273 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.273 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.273 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-11-16T18:35:00.273 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.273 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.273 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.273 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger agent container image", 2023-11-16T18:35:00.273 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.274 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.274 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.274 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.274 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_collector": { 2023-11-16T18:35:00.274 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", 2023-11-16T18:35:00.274 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.274 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.274 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.275 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-11-16T18:35:00.275 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.275 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.275 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.275 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger collector container image", 2023-11-16T18:35:00.275 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.275 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.275 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.276 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.276 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_query": { 2023-11-16T18:35:00.276 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_query", 2023-11-16T18:35:00.276 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.276 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.276 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.276 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-11-16T18:35:00.276 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.276 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.277 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.277 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger query container image", 2023-11-16T18:35:00.277 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.277 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.277 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.277 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.277 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_keepalived": { 2023-11-16T18:35:00.277 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_keepalived", 2023-11-16T18:35:00.278 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.278 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.278 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.278 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-11-16T18:35:00.278 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.278 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.278 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.278 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Keepalived container image", 2023-11-16T18:35:00.279 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.279 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.279 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.279 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.279 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_loki": { 2023-11-16T18:35:00.279 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_loki", 2023-11-16T18:35:00.279 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.279 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.279 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.280 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-11-16T18:35:00.280 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.280 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.280 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.280 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Loki container image", 2023-11-16T18:35:00.280 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.280 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.281 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.281 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.281 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_node_exporter": { 2023-11-16T18:35:00.281 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_node_exporter", 2023-11-16T18:35:00.281 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.281 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.281 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.281 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-11-16T18:35:00.282 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.282 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.282 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.282 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T18:35:00.282 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.282 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.282 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.282 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.283 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_prometheus": { 2023-11-16T18:35:00.283 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_prometheus", 2023-11-16T18:35:00.283 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.283 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.283 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.283 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-11-16T18:35:00.283 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.283 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.283 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.284 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T18:35:00.284 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.284 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.284 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.284 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.284 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_promtail": { 2023-11-16T18:35:00.284 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_promtail", 2023-11-16T18:35:00.284 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.285 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.285 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.285 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-11-16T18:35:00.285 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.285 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.285 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.285 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Promtail container image", 2023-11-16T18:35:00.285 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.286 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.286 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.286 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.286 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_snmp_gateway": { 2023-11-16T18:35:00.286 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_snmp_gateway", 2023-11-16T18:35:00.286 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.286 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.287 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.287 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-11-16T18:35:00.287 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.287 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.287 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.287 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SNMP Gateway container image", 2023-11-16T18:35:00.287 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.287 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.288 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.288 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.288 INFO:teuthology.orchestra.run.smithi002.stdout: "container_init": { 2023-11-16T18:35:00.288 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_init", 2023-11-16T18:35:00.288 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.288 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.288 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.288 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.289 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.289 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.289 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.289 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Run podman/docker with `--init`", 2023-11-16T18:35:00.289 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.289 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.289 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.289 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.290 INFO:teuthology.orchestra.run.smithi002.stdout: "daemon_cache_timeout": { 2023-11-16T18:35:00.290 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "daemon_cache_timeout", 2023-11-16T18:35:00.290 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.290 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.290 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.290 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:35:00.290 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.290 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.290 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.291 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-11-16T18:35:00.291 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.291 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.291 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.291 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.291 INFO:teuthology.orchestra.run.smithi002.stdout: "default_cephadm_command_timeout": { 2023-11-16T18:35:00.291 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_cephadm_command_timeout", 2023-11-16T18:35:00.291 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.292 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.292 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.292 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "900", 2023-11-16T18:35:00.292 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.292 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.292 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.292 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-11-16T18:35:00.292 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.293 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.293 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.293 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.293 INFO:teuthology.orchestra.run.smithi002.stdout: "default_registry": { 2023-11-16T18:35:00.293 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_registry", 2023-11-16T18:35:00.294 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.294 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.294 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.294 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io", 2023-11-16T18:35:00.294 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.294 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.294 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.294 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-11-16T18:35:00.295 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.295 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.295 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.295 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.295 INFO:teuthology.orchestra.run.smithi002.stdout: "device_cache_timeout": { 2023-11-16T18:35:00.295 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_cache_timeout", 2023-11-16T18:35:00.295 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.295 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.296 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.296 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1800", 2023-11-16T18:35:00.296 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.296 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.296 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.296 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache device inventory", 2023-11-16T18:35:00.296 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.296 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.296 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.297 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.297 INFO:teuthology.orchestra.run.smithi002.stdout: "device_enhanced_scan": { 2023-11-16T18:35:00.297 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_enhanced_scan", 2023-11-16T18:35:00.297 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.297 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.297 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.297 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.297 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.298 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.298 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-11-16T18:35:00.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.298 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.298 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.298 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.298 INFO:teuthology.orchestra.run.smithi002.stdout: "facts_cache_timeout": { 2023-11-16T18:35:00.299 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "facts_cache_timeout", 2023-11-16T18:35:00.299 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.299 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.299 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.299 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:35:00.299 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.299 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.299 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.299 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache host facts data", 2023-11-16T18:35:00.300 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.300 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.300 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.300 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.300 INFO:teuthology.orchestra.run.smithi002.stdout: "host_check_interval": { 2023-11-16T18:35:00.300 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "host_check_interval", 2023-11-16T18:35:00.301 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.301 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.301 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.301 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:35:00.301 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.301 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.302 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.302 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to perform a host check", 2023-11-16T18:35:00.302 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.302 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.302 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.302 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.302 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:00.302 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:00.303 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.303 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.303 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.303 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.303 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.303 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.303 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.303 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.303 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.304 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.304 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.304 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.304 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.304 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.304 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.304 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.304 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.305 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.305 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.305 INFO:teuthology.orchestra.run.smithi002.stdout: "log_refresh_metadata": { 2023-11-16T18:35:00.305 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_refresh_metadata", 2023-11-16T18:35:00.305 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.305 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.305 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.305 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.306 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.306 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.306 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.306 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-11-16T18:35:00.306 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.306 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.306 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.306 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.307 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:00.307 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:00.307 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.307 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.307 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.307 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.307 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.307 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.308 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.308 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-11-16T18:35:00.308 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.308 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.308 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.308 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.308 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:00.308 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:00.308 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.309 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.309 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.309 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:00.309 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.309 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.309 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.309 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.309 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.310 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.310 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.310 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.310 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.310 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.310 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.310 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.310 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.311 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.311 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.311 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:00.311 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:00.311 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.311 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.312 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.312 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.312 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.312 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.312 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.312 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.312 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.312 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.312 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.313 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.313 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf": { 2023-11-16T18:35:00.313 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-11-16T18:35:00.313 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.313 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.313 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.313 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.313 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.314 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.314 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.314 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-11-16T18:35:00.314 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.314 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.314 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.314 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.314 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-11-16T18:35:00.315 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-11-16T18:35:00.315 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.315 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.315 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.315 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "*", 2023-11-16T18:35:00.315 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.315 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.315 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.316 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-11-16T18:35:00.316 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.316 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.316 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.316 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.316 INFO:teuthology.orchestra.run.smithi002.stdout: "max_count_per_host": { 2023-11-16T18:35:00.317 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_count_per_host", 2023-11-16T18:35:00.317 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.317 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.317 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.317 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:35:00.317 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.317 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.317 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.318 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of daemons per service per host", 2023-11-16T18:35:00.318 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.318 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.318 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { 2023-11-16T18:35:00.318 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", 2023-11-16T18:35:00.318 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.319 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.319 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.319 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:35:00.319 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.319 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.319 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.319 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-11-16T18:35:00.319 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.320 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.320 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.320 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.320 INFO:teuthology.orchestra.run.smithi002.stdout: "migration_current": { 2023-11-16T18:35:00.320 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "migration_current", 2023-11-16T18:35:00.320 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.320 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.320 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.321 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.321 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.321 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.321 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.321 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "internal - do not modify", 2023-11-16T18:35:00.321 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.321 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.321 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.322 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.322 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2023-11-16T18:35:00.322 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2023-11-16T18:35:00.322 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.322 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.322 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.322 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "root", 2023-11-16T18:35:00.325 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.325 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.325 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.325 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm-package", 2023-11-16T18:35:00.325 INFO:teuthology.orchestra.run.smithi002.stdout: "root" 2023-11-16T18:35:00.325 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.326 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "mode for remote execution of cephadm", 2023-11-16T18:35:00.326 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.326 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.326 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.326 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.326 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_alerts_path": { 2023-11-16T18:35:00.326 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_alerts_path", 2023-11-16T18:35:00.327 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.327 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.327 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.327 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-11-16T18:35:00.327 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.327 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.327 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.327 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-11-16T18:35:00.327 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.328 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.328 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.328 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.328 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_web_password": { 2023-11-16T18:35:00.328 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_web_password", 2023-11-16T18:35:00.328 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.328 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.328 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.329 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:35:00.329 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.329 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.329 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.329 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus web password", 2023-11-16T18:35:00.329 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.329 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.329 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.330 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.330 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_web_user": { 2023-11-16T18:35:00.330 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_web_user", 2023-11-16T18:35:00.330 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.330 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.330 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.330 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:35:00.330 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.331 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.331 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.331 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus web user", 2023-11-16T18:35:00.331 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.331 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.331 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.331 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.331 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_insecure": { 2023-11-16T18:35:00.331 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_insecure", 2023-11-16T18:35:00.332 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.332 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.332 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.332 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.332 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.332 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.332 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.332 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-11-16T18:35:00.333 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.333 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.333 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.333 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.333 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_password": { 2023-11-16T18:35:00.333 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_password", 2023-11-16T18:35:00.333 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.333 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.334 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.334 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.334 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.334 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.334 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.334 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-11-16T18:35:00.334 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.334 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.335 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.335 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.335 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_url": { 2023-11-16T18:35:00.335 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_url", 2023-11-16T18:35:00.335 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.335 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.335 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.335 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.336 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.336 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.336 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.336 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-11-16T18:35:00.336 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.336 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.336 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.336 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.337 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { 2023-11-16T18:35:00.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", 2023-11-16T18:35:00.337 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.337 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.337 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.337 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.337 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.337 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.338 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.338 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-11-16T18:35:00.338 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.338 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.338 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.338 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.338 INFO:teuthology.orchestra.run.smithi002.stdout: "secure_monitoring_stack": { 2023-11-16T18:35:00.338 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "secure_monitoring_stack", 2023-11-16T18:35:00.339 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.339 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.339 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.339 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.339 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.339 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.339 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.339 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-11-16T18:35:00.340 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.340 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.340 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.340 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.340 INFO:teuthology.orchestra.run.smithi002.stdout: "service_discovery_port": { 2023-11-16T18:35:00.340 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "service_discovery_port", 2023-11-16T18:35:00.340 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.340 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.341 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.341 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8765", 2023-11-16T18:35:00.341 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.341 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.341 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.341 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "cephadm service discovery port", 2023-11-16T18:35:00.341 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.341 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.342 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.342 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.342 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { 2023-11-16T18:35:00.342 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-11-16T18:35:00.342 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.342 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.342 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.342 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.342 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.343 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.343 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.343 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-11-16T18:35:00.343 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.343 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.343 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.343 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.343 INFO:teuthology.orchestra.run.smithi002.stdout: "use_agent": { 2023-11-16T18:35:00.344 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_agent", 2023-11-16T18:35:00.344 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.344 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.344 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.344 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.344 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.344 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.344 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.345 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-11-16T18:35:00.345 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.345 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.345 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.345 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.345 INFO:teuthology.orchestra.run.smithi002.stdout: "use_repo_digest": { 2023-11-16T18:35:00.428 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_repo_digest", 2023-11-16T18:35:00.428 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.428 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.429 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.429 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.429 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.429 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.430 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.430 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-11-16T18:35:00.430 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.430 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.431 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.431 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_failed_host_check": { 2023-11-16T18:35:00.432 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_failed_host_check", 2023-11-16T18:35:00.432 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.432 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.432 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.433 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.433 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.433 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.433 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.434 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if the host check fails", 2023-11-16T18:35:00.434 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.435 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.435 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.435 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.435 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_daemons": { 2023-11-16T18:35:00.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_daemons", 2023-11-16T18:35:00.436 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.436 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.436 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.437 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.437 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.437 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.437 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.438 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-11-16T18:35:00.438 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.441 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.441 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.441 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.441 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_hosts": { 2023-11-16T18:35:00.442 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_hosts", 2023-11-16T18:35:00.442 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.442 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.442 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.442 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.442 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.442 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.443 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.443 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-11-16T18:35:00.443 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.443 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.443 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.443 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.443 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.443 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.443 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:00.444 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crash", 2023-11-16T18:35:00.444 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:00.444 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:00.444 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:00.444 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:00.444 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:00.444 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.444 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.445 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.445 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.445 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.445 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.445 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.445 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.445 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.445 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.446 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.446 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.446 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.446 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.446 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.446 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.446 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.446 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.447 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.447 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:00.447 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:00.447 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.447 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.447 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.447 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.447 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.448 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.448 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.448 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.448 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.448 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.448 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.448 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.448 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:00.448 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:00.449 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.449 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.449 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.449 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:00.449 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.449 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.449 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.449 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.450 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.450 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.450 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.450 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.450 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.450 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.450 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.450 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.451 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.451 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.451 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.451 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:00.451 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:00.451 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.451 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.451 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.452 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.452 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.452 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.452 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.452 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.452 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.452 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.452 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.452 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.453 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { 2023-11-16T18:35:00.453 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-11-16T18:35:00.453 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.453 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.453 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.453 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "31536000", 2023-11-16T18:35:00.453 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.453 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.457 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain crashes before pruning them", 2023-11-16T18:35:00.457 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.457 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.458 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.458 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.458 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_recent_interval": { 2023-11-16T18:35:00.458 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_recent_interval", 2023-11-16T18:35:00.459 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.459 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.459 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.459 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1209600", 2023-11-16T18:35:00.460 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.460 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.460 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.460 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "time interval in which to warn about recent crashes", 2023-11-16T18:35:00.461 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.461 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.461 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.461 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.461 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.462 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.462 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:00.462 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dashboard", 2023-11-16T18:35:00.462 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:00.463 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:00.463 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:00.463 INFO:teuthology.orchestra.run.smithi002.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-11-16T18:35:00.463 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-11-16T18:35:00.464 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.464 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.464 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.464 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:35:00.465 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.465 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.465 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.465 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.466 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.466 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.466 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.466 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.467 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_HOST": { 2023-11-16T18:35:00.467 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_HOST", 2023-11-16T18:35:00.467 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.467 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.468 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.468 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.468 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.468 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.469 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.469 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.470 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.470 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.470 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.470 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.471 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-11-16T18:35:00.471 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-11-16T18:35:00.471 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.471 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.472 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.472 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.472 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.472 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.473 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.473 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.473 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.473 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.474 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.474 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.474 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_ENABLED": { 2023-11-16T18:35:00.474 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_ENABLED", 2023-11-16T18:35:00.475 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.475 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.475 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.475 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.476 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.476 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.476 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.476 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.477 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.477 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.477 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.478 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-11-16T18:35:00.478 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-11-16T18:35:00.478 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.478 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.479 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.479 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.479 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.479 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.480 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.480 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.480 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.480 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.481 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.481 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.481 INFO:teuthology.orchestra.run.smithi002.stdout: "ENABLE_BROWSABLE_API": { 2023-11-16T18:35:00.481 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ENABLE_BROWSABLE_API", 2023-11-16T18:35:00.482 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.482 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.482 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.482 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.483 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.483 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.483 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.483 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.484 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.484 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.484 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.484 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.485 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-11-16T18:35:00.485 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-11-16T18:35:00.485 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.485 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.486 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.486 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.486 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.486 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.487 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.487 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.487 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.487 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.487 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.488 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.488 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-11-16T18:35:00.488 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-11-16T18:35:00.488 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.489 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.489 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.490 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.490 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.491 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.491 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.491 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.491 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.492 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.492 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.492 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.492 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-11-16T18:35:00.492 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-11-16T18:35:00.493 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.493 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.493 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.493 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.494 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.494 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.494 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.494 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.495 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.495 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.495 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.496 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.496 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-11-16T18:35:00.496 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-11-16T18:35:00.496 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.497 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.497 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.497 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.498 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.498 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.499 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.576 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.577 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.577 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.577 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_NFS": { 2023-11-16T18:35:00.577 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-11-16T18:35:00.577 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.578 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.586 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.587 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.587 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.587 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.587 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.587 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.587 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.587 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.587 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.588 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.588 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RBD": { 2023-11-16T18:35:00.588 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-11-16T18:35:00.588 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.588 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.588 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.588 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.588 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.589 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.589 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.589 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.589 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.589 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.589 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.589 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.589 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-11-16T18:35:00.590 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-11-16T18:35:00.590 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.590 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.590 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.590 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.590 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.590 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.590 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.591 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.591 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.591 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.591 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.591 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.591 INFO:teuthology.orchestra.run.smithi002.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-11-16T18:35:00.591 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-11-16T18:35:00.591 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.592 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.592 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.592 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.592 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.592 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.592 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.592 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.592 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.593 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.680 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.680 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.680 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_PASSWORD": { 2023-11-16T18:35:00.681 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_PASSWORD", 2023-11-16T18:35:00.681 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.681 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.681 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.681 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:35:00.681 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.681 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.681 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.682 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.682 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.682 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.682 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.682 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.682 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-11-16T18:35:00.682 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-11-16T18:35:00.682 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.683 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.683 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.683 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.683 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.683 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.683 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.683 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.683 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.684 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.684 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.684 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.684 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_URL": { 2023-11-16T18:35:00.684 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_URL", 2023-11-16T18:35:00.684 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.684 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.684 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.685 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.685 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.685 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.685 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.685 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.685 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.685 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.685 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.686 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.686 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_USERNAME": { 2023-11-16T18:35:00.686 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_USERNAME", 2023-11-16T18:35:00.688 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.688 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.688 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.688 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:35:00.689 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.689 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.689 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.689 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.689 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.690 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.690 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.690 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.690 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-11-16T18:35:00.690 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-11-16T18:35:00.691 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.691 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.691 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.691 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.691 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.691 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.691 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.691 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.692 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.692 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.692 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.692 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.692 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-11-16T18:35:00.692 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-11-16T18:35:00.692 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.692 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.693 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.693 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.693 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.693 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.753 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.753 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.753 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.753 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.754 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.754 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.754 INFO:teuthology.orchestra.run.smithi002.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-11-16T18:35:00.754 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-11-16T18:35:00.754 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.754 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.754 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.754 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.755 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.755 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.755 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.755 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.755 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.755 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.755 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.756 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.756 INFO:teuthology.orchestra.run.smithi002.stdout: "ISSUE_TRACKER_API_KEY": { 2023-11-16T18:35:00.756 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-11-16T18:35:00.756 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.756 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.756 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.756 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.757 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.758 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.758 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.758 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.758 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.758 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.758 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.758 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.758 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { 2023-11-16T18:35:00.759 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", 2023-11-16T18:35:00.759 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.759 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.759 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.802 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.802 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.802 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.802 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.802 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.802 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.802 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.803 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.803 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.803 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-11-16T18:35:00.803 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-11-16T18:35:00.803 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.803 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.803 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.803 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.803 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.804 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.804 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.804 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.804 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.804 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.804 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.804 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.804 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-11-16T18:35:00.805 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-11-16T18:35:00.805 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.805 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.805 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.805 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.805 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.805 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.805 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.806 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.806 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.806 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.806 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.806 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.806 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-11-16T18:35:00.806 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-11-16T18:35:00.806 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.807 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.807 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.807 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.807 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.807 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.807 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.807 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.808 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.810 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.810 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.810 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-11-16T18:35:00.811 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-11-16T18:35:00.811 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.811 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.811 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.811 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.811 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.811 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.811 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.812 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.812 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.812 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.812 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.812 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.812 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-11-16T18:35:00.812 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-11-16T18:35:00.812 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.813 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.813 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.815 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.815 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.815 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.816 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.817 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.817 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.817 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.817 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.817 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.818 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-11-16T18:35:00.818 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-11-16T18:35:00.818 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.818 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.818 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.818 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.818 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.818 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.819 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.819 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.819 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.819 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.819 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.819 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.819 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-11-16T18:35:00.819 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-11-16T18:35:00.820 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.820 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.820 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.820 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.820 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.820 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.820 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.821 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.821 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.821 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.821 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.821 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.821 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-11-16T18:35:00.821 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-11-16T18:35:00.821 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.822 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.822 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.823 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.823 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.823 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.823 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.823 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.823 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.823 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.824 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_ENABLED": { 2023-11-16T18:35:00.824 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_ENABLED", 2023-11-16T18:35:00.824 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.824 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.824 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.824 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.824 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.824 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.825 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.825 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.825 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.825 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.825 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.825 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.825 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-11-16T18:35:00.825 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-11-16T18:35:00.826 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.826 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.826 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.826 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-11-16T18:35:00.826 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.826 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.826 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.826 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.827 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.827 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.827 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.827 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.827 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-11-16T18:35:00.827 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-11-16T18:35:00.827 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.827 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.828 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.828 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:35:00.828 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.828 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.839 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.839 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.839 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.839 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.839 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.839 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.840 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-11-16T18:35:00.840 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-11-16T18:35:00.840 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.840 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.840 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.843 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8", 2023-11-16T18:35:00.843 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.843 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.843 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.843 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.843 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.844 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.844 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.844 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.852 INFO:teuthology.orchestra.run.smithi002.stdout: "REST_REQUESTS_TIMEOUT": { 2023-11-16T18:35:00.852 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-11-16T18:35:00.853 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.853 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.853 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.853 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "45", 2023-11-16T18:35:00.853 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.853 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.853 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.853 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.854 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.854 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.854 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.854 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.854 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ACCESS_KEY": { 2023-11-16T18:35:00.854 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ACCESS_KEY", 2023-11-16T18:35:00.854 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.854 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.855 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.855 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.855 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.855 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.855 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.855 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.855 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.856 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.856 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.856 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.856 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-11-16T18:35:00.856 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-11-16T18:35:00.856 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.856 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.856 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.856 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T18:35:00.857 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.857 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.857 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.857 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.857 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.857 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.857 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.857 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.858 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SECRET_KEY": { 2023-11-16T18:35:00.858 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SECRET_KEY", 2023-11-16T18:35:00.858 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.858 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.858 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.858 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.858 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.858 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.859 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.859 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.859 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.859 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.859 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.859 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.859 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SSL_VERIFY": { 2023-11-16T18:35:00.859 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SSL_VERIFY", 2023-11-16T18:35:00.860 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.860 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.860 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.860 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.860 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.860 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.860 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.860 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.861 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.861 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.861 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.861 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.861 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-11-16T18:35:00.861 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-11-16T18:35:00.861 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.861 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.862 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.862 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:35:00.862 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.862 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.862 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.862 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.862 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.862 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.863 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.863 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.863 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-11-16T18:35:00.863 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-11-16T18:35:00.863 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.863 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.863 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.863 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:35:00.864 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.864 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.864 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.864 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.864 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.864 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.864 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.864 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.865 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-11-16T18:35:00.865 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-11-16T18:35:00.865 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.865 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.865 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.865 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T18:35:00.865 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.865 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.865 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.866 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.866 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.866 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.866 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.866 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.866 INFO:teuthology.orchestra.run.smithi002.stdout: "cross_origin_url": { 2023-11-16T18:35:00.866 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cross_origin_url", 2023-11-16T18:35:00.866 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.867 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.867 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.867 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.867 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.867 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.867 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.867 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.868 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.868 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.868 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.868 INFO:teuthology.orchestra.run.smithi002.stdout: "crt_file": { 2023-11-16T18:35:00.868 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crt_file", 2023-11-16T18:35:00.868 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.868 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.868 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.869 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.869 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.869 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.869 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.869 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.869 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.869 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.869 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.870 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.870 INFO:teuthology.orchestra.run.smithi002.stdout: "debug": { 2023-11-16T18:35:00.870 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "debug", 2023-11-16T18:35:00.870 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.870 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.870 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.870 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.870 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.870 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.871 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.871 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable/disable debug options", 2023-11-16T18:35:00.871 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.871 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.871 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.871 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.871 INFO:teuthology.orchestra.run.smithi002.stdout: "jwt_token_ttl": { 2023-11-16T18:35:00.871 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "jwt_token_ttl", 2023-11-16T18:35:00.872 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.872 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.872 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.872 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "28800", 2023-11-16T18:35:00.872 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.872 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.872 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.872 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.873 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.873 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.873 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.873 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.873 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2023-11-16T18:35:00.873 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2023-11-16T18:35:00.873 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.873 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.874 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.874 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.874 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.874 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.874 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.874 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.874 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.874 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.875 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.875 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:00.875 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:00.875 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.875 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.875 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.875 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.876 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.876 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.876 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.876 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.876 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.876 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.876 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.876 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.876 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.877 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.877 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.877 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.877 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.877 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.877 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.877 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:00.877 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:00.878 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.878 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.878 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.878 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.878 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.878 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.878 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.878 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.879 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.879 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.879 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.879 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.879 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:00.879 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:00.879 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.879 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.880 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.880 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:00.880 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.880 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.880 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.880 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.880 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.880 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.881 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.881 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.881 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.881 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.881 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.881 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.881 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.881 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.882 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.882 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:00.882 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:00.882 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.882 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.882 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.882 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.882 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.882 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.883 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.883 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.883 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.883 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.883 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.883 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.883 INFO:teuthology.orchestra.run.smithi002.stdout: "motd": { 2023-11-16T18:35:00.883 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "motd", 2023-11-16T18:35:00.884 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.884 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.884 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.884 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.884 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.884 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.884 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.884 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "The message of the day", 2023-11-16T18:35:00.885 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.885 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.885 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.885 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.885 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect_resolve_ip_addr": { 2023-11-16T18:35:00.885 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "redirect_resolve_ip_addr", 2023-11-16T18:35:00.885 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.885 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.885 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.886 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.886 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.886 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.886 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.886 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.886 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.886 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.887 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.887 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.887 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T18:35:00.887 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T18:35:00.887 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.887 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.887 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.887 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-11-16T18:35:00.887 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.888 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.888 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.888 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.888 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.888 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.888 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.888 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.888 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T18:35:00.889 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T18:35:00.889 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.889 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.889 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.889 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8080", 2023-11-16T18:35:00.889 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.889 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.889 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.890 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.890 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.890 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.890 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.890 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.890 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2023-11-16T18:35:00.890 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2023-11-16T18:35:00.890 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.891 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.891 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.891 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.891 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.891 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.891 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.891 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.891 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.892 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.892 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.892 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl_server_port": { 2023-11-16T18:35:00.892 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", 2023-11-16T18:35:00.892 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.892 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.892 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.892 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8443", 2023-11-16T18:35:00.893 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.893 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.893 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.893 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.893 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.893 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.893 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.893 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.894 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2023-11-16T18:35:00.894 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2023-11-16T18:35:00.894 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.894 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.894 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.894 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "redirect", 2023-11-16T18:35:00.894 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.894 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.895 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.895 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.895 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect" 2023-11-16T18:35:00.895 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.895 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.895 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.895 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.896 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.896 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.896 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2023-11-16T18:35:00.896 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2023-11-16T18:35:00.896 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.896 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.896 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.896 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2023-11-16T18:35:00.896 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-11-16T18:35:00.897 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2023-11-16T18:35:00.897 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.897 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.898 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.898 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.898 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.898 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.898 INFO:teuthology.orchestra.run.smithi002.stdout: "url_prefix": { 2023-11-16T18:35:00.898 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url_prefix", 2023-11-16T18:35:00.898 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.898 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.899 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.899 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.899 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.899 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.899 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.899 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.899 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.899 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.900 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.900 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.900 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.900 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.900 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:00.900 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", 2023-11-16T18:35:00.900 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:00.900 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:00.900 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:00.901 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_monitoring": { 2023-11-16T18:35:00.901 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_monitoring", 2023-11-16T18:35:00.901 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.901 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.901 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.901 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.901 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.901 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.902 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.902 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "monitor device health metrics", 2023-11-16T18:35:00.902 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.902 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.902 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.902 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.903 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:00.903 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:00.903 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.903 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.903 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.903 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.903 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.903 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.903 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.904 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.904 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.904 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.904 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.904 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.904 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.904 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.904 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.905 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.905 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.905 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.905 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.905 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:00.905 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:00.905 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.905 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.906 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.906 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.906 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.906 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.906 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.906 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.906 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.906 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.907 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.907 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.907 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:00.907 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:00.907 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.907 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.907 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.907 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:00.908 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.908 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.908 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.908 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.908 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.908 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.908 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.908 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.909 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.909 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.909 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.909 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.909 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.909 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.909 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.909 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:00.910 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:00.910 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.910 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.910 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.910 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.910 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.910 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.910 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.911 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.911 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.911 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.911 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.911 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.911 INFO:teuthology.orchestra.run.smithi002.stdout: "mark_out_threshold": { 2023-11-16T18:35:00.911 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mark_out_threshold", 2023-11-16T18:35:00.911 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.912 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.912 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.912 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2419200", 2023-11-16T18:35:00.912 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.912 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.912 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.912 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-11-16T18:35:00.912 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.912 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.913 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.913 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.913 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-11-16T18:35:00.913 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_name", 2023-11-16T18:35:00.913 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.913 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.913 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.913 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "device_health_metrics", 2023-11-16T18:35:00.914 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.914 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.914 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.914 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name of pool in which to store device health metrics", 2023-11-16T18:35:00.914 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.914 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.914 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.914 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.915 INFO:teuthology.orchestra.run.smithi002.stdout: "retention_period": { 2023-11-16T18:35:00.915 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retention_period", 2023-11-16T18:35:00.915 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.915 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.915 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.915 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15552000", 2023-11-16T18:35:00.915 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.915 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.916 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.916 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain device health metrics", 2023-11-16T18:35:00.916 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.916 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.916 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.916 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.916 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_frequency": { 2023-11-16T18:35:00.916 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_frequency", 2023-11-16T18:35:00.916 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.917 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.917 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.917 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2023-11-16T18:35:00.917 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.917 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.917 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.917 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to scrape device health metrics", 2023-11-16T18:35:00.917 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.918 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.918 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.918 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.918 INFO:teuthology.orchestra.run.smithi002.stdout: "self_heal": { 2023-11-16T18:35:00.918 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "self_heal", 2023-11-16T18:35:00.918 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.918 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.918 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.919 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:00.919 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.919 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.919 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.919 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-11-16T18:35:00.919 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.919 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.919 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.920 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.920 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:35:00.920 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:35:00.920 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.920 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.920 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.920 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:35:00.920 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.920 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.921 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.921 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and check device health", 2023-11-16T18:35:00.921 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.921 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.921 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.921 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.921 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_threshold": { 2023-11-16T18:35:00.921 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_threshold", 2023-11-16T18:35:00.922 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.922 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.922 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.922 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7257600", 2023-11-16T18:35:00.922 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.922 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.922 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.922 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-11-16T18:35:00.923 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.923 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.923 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.923 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.923 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.923 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.923 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:00.923 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "diskprediction_local", 2023-11-16T18:35:00.923 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:00.924 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:00.924 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:00.924 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:00.924 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:00.924 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.924 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.924 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.924 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.925 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.925 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.925 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.925 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.925 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.925 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.925 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.925 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.926 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.926 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.926 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.926 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.926 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.926 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.926 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.926 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:00.927 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:00.927 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.927 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.927 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.927 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.927 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.927 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.927 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.927 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.928 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.928 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.928 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.928 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.928 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:00.928 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:00.928 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.928 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.929 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.929 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:00.929 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.929 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.929 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.929 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.929 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.929 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.930 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.930 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.930 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.930 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.930 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.930 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.930 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.930 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.931 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.931 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:00.931 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:00.931 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.931 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.931 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.931 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.931 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.932 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.932 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.932 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.932 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.932 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.932 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.932 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.932 INFO:teuthology.orchestra.run.smithi002.stdout: "predict_interval": { 2023-11-16T18:35:00.933 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predict_interval", 2023-11-16T18:35:00.933 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.933 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2023-11-16T18:35:00.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.933 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.933 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.933 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.934 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.934 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.934 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.934 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.934 INFO:teuthology.orchestra.run.smithi002.stdout: "predictor_model": { 2023-11-16T18:35:00.934 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predictor_model", 2023-11-16T18:35:00.934 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.934 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.935 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.935 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "prophetstor", 2023-11-16T18:35:00.935 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.935 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.935 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.935 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.935 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.937 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.937 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.937 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.937 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:35:00.937 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:35:00.937 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.938 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.938 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.938 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T18:35:00.938 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.938 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.938 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.938 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.938 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.939 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.939 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.939 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.939 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.939 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.939 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:00.939 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "influx", 2023-11-16T18:35:00.939 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": false, 2023-11-16T18:35:00.940 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "influxdb python module not found", 2023-11-16T18:35:00.940 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:00.940 INFO:teuthology.orchestra.run.smithi002.stdout: "batch_size": { 2023-11-16T18:35:00.941 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "batch_size", 2023-11-16T18:35:00.941 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.941 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.941 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.941 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5000", 2023-11-16T18:35:00.941 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.942 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.942 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.942 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-11-16T18:35:00.942 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.942 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.944 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.944 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.944 INFO:teuthology.orchestra.run.smithi002.stdout: "database": { 2023-11-16T18:35:00.944 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "database", 2023-11-16T18:35:00.944 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.944 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.944 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.945 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "ceph", 2023-11-16T18:35:00.945 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.945 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.945 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.945 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-11-16T18:35:00.945 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.945 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.945 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.946 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.946 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": { 2023-11-16T18:35:00.946 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "hostname", 2023-11-16T18:35:00.946 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.946 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.946 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.946 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.946 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.947 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.947 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.947 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server hostname", 2023-11-16T18:35:00.947 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.947 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.947 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.947 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.947 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:35:00.948 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:35:00.948 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:00.948 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.948 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.948 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "30", 2023-11-16T18:35:00.948 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "5", 2023-11-16T18:35:00.948 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.948 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.949 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-11-16T18:35:00.949 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.949 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.949 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.949 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.949 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:00.949 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:00.949 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.950 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.950 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.950 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.950 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.950 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.950 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.950 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.950 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.951 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.951 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.951 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.951 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.951 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.951 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.951 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.951 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.952 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.952 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.952 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:00.952 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:00.952 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.952 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.952 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.952 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.952 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.953 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.953 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.953 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.953 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.953 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.953 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.953 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.953 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:00.954 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:00.954 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.954 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.954 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.954 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:00.954 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.954 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.954 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.955 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.955 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.955 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.955 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.955 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.955 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.955 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.955 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.956 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.956 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.956 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.956 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.956 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:00.956 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:00.956 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.956 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.957 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.957 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.957 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.957 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.957 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.957 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.957 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.957 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.957 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.958 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.958 INFO:teuthology.orchestra.run.smithi002.stdout: "password": { 2023-11-16T18:35:00.958 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "password", 2023-11-16T18:35:00.958 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.958 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.958 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.958 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.958 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.959 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.959 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.959 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "password of InfluxDB server user", 2023-11-16T18:35:00.959 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.959 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.959 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.959 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.959 INFO:teuthology.orchestra.run.smithi002.stdout: "port": { 2023-11-16T18:35:00.960 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "port", 2023-11-16T18:35:00.960 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.960 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.960 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.960 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8086", 2023-11-16T18:35:00.960 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.961 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.961 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.961 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server port", 2023-11-16T18:35:00.961 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.961 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.961 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.961 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.961 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2023-11-16T18:35:00.962 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2023-11-16T18:35:00.962 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.962 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.962 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.962 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "false", 2023-11-16T18:35:00.962 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.962 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.962 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.963 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T18:35:00.963 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.963 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.963 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.963 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.963 INFO:teuthology.orchestra.run.smithi002.stdout: "threads": { 2023-11-16T18:35:00.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threads", 2023-11-16T18:35:00.963 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.964 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.964 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.964 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T18:35:00.964 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T18:35:00.964 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "32", 2023-11-16T18:35:00.964 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.964 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-11-16T18:35:00.964 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.965 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.965 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.965 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.965 INFO:teuthology.orchestra.run.smithi002.stdout: "username": { 2023-11-16T18:35:00.965 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "username", 2023-11-16T18:35:00.965 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.965 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.965 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.965 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.966 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.966 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.966 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.966 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "username of InfluxDB server user", 2023-11-16T18:35:00.966 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.966 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.966 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.966 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.967 INFO:teuthology.orchestra.run.smithi002.stdout: "verify_ssl": { 2023-11-16T18:35:00.967 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "verify_ssl", 2023-11-16T18:35:00.967 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.967 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.967 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.967 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "true", 2023-11-16T18:35:00.967 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.967 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.968 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.968 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T18:35:00.968 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.968 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.968 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.968 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.968 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.968 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.968 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:00.969 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "insights", 2023-11-16T18:35:00.969 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:00.969 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:00.969 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:00.969 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:00.969 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:00.969 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.969 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.970 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.970 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.970 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.970 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.970 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.970 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.970 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.970 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.971 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.971 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.971 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.971 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.971 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.971 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.971 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.971 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.972 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.972 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:00.972 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:00.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.972 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.972 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.972 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.972 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.973 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.973 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.973 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.973 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.973 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.973 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.973 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.973 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:00.973 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:00.974 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.974 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.974 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.974 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:00.974 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.974 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.974 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.974 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.975 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.975 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.975 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.975 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.975 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.975 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.975 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.975 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.976 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.976 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.976 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.976 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:00.976 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:00.976 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.976 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.976 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.977 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.977 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.977 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.977 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.977 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.977 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.977 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.977 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.978 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.978 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.978 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.978 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:00.978 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "iostat", 2023-11-16T18:35:00.978 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:00.978 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:00.978 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:00.979 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:00.979 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:00.979 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.979 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.979 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.979 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.979 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.979 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.980 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.980 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.980 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.980 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.980 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.980 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.980 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.980 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.980 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.981 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.981 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.981 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.981 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.981 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:00.981 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:00.981 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.981 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.982 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.982 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.982 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.982 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.982 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.982 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.982 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.982 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.983 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.983 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.983 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:00.983 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:00.983 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.983 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.983 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.983 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:00.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.984 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.984 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.984 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.984 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.984 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.984 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.984 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.985 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.985 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.985 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.985 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.985 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.985 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.985 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.985 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:00.985 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:00.986 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.986 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.986 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.986 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.986 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.986 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.986 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.986 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.987 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.987 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.987 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.987 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.987 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:00.987 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.987 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:00.987 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "k8sevents", 2023-11-16T18:35:00.988 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:00.988 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:00.988 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:00.988 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_event_retention_days": { 2023-11-16T18:35:00.988 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ceph_event_retention_days", 2023-11-16T18:35:00.988 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.988 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.988 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.989 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7", 2023-11-16T18:35:00.989 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.989 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.989 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.989 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Days to hold ceph event information within local cache", 2023-11-16T18:35:00.989 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.989 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.989 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.990 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.990 INFO:teuthology.orchestra.run.smithi002.stdout: "config_check_secs": { 2023-11-16T18:35:00.990 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_check_secs", 2023-11-16T18:35:00.990 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:00.990 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.990 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:00.990 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:35:00.990 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "10", 2023-11-16T18:35:00.991 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.991 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.991 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-11-16T18:35:00.991 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.991 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.991 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.991 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.991 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:00.991 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:00.992 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.992 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.992 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.992 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:00.992 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.992 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.992 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.992 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.993 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.993 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.993 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.993 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.993 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.993 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.993 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.993 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.994 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.994 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.994 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.994 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:00.994 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:00.994 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.994 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.994 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.995 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.995 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.995 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.995 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.995 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.995 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.995 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.996 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.996 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:00.996 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:00.996 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:00.996 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.996 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.996 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:00.996 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.996 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.997 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:00.997 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:00.997 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:00.997 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:00.997 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:00.997 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:00.997 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:00.997 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:00.998 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.998 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.998 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:00.998 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:00.998 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:00.998 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:00.998 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:00.998 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:00.999 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:00.999 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:00.999 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:00.999 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:00.999 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:00.999 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:00.999 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:00.999 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:00.999 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.000 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.000 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.000 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.000 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.000 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "localpool", 2023-11-16T18:35:01.000 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.000 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.001 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.001 INFO:teuthology.orchestra.run.smithi002.stdout: "failure_domain": { 2023-11-16T18:35:01.001 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "failure_domain", 2023-11-16T18:35:01.001 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.001 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.001 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.001 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "host", 2023-11-16T18:35:01.001 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.002 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.002 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.002 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "failure domain for any created local pool", 2023-11-16T18:35:01.002 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-11-16T18:35:01.002 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.002 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.002 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.002 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.003 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.003 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.003 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.003 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.003 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.003 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.003 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.003 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.004 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.004 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.004 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.004 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.004 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.004 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.004 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.004 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.005 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.005 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.005 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.005 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.005 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.005 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.005 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.005 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.006 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.006 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.006 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.006 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.006 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.006 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.006 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.006 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.007 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.007 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.007 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.007 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.007 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.007 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.007 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.007 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.007 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.008 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.008 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.008 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.008 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.008 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.008 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.008 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.008 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.009 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.009 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.009 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.009 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.009 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.009 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.009 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.009 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.010 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.010 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.010 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.010 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.010 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.010 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.010 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.010 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.011 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.011 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.011 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.011 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.011 INFO:teuthology.orchestra.run.smithi002.stdout: "min_size": { 2023-11-16T18:35:01.011 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_size", 2023-11-16T18:35:01.011 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:01.011 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.012 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.012 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.012 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.012 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.012 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.012 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default min_size for any created local pool", 2023-11-16T18:35:01.012 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-11-16T18:35:01.012 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.012 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.013 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.013 INFO:teuthology.orchestra.run.smithi002.stdout: "num_rep": { 2023-11-16T18:35:01.013 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "num_rep", 2023-11-16T18:35:01.013 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:01.013 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.013 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3", 2023-11-16T18:35:01.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.014 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.014 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.014 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default replica count for any created local pool", 2023-11-16T18:35:01.014 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.014 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.014 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.014 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.015 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num": { 2023-11-16T18:35:01.015 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_num", 2023-11-16T18:35:01.015 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:01.015 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.015 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.015 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "128", 2023-11-16T18:35:01.015 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.015 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.015 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.016 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default pg_num for any created local pool", 2023-11-16T18:35:01.016 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.016 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.016 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.016 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.016 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-11-16T18:35:01.016 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prefix", 2023-11-16T18:35:01.016 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.017 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.017 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.017 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.017 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.017 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.017 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.017 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name prefix for any created local pool", 2023-11-16T18:35:01.017 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.018 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.018 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.018 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.018 INFO:teuthology.orchestra.run.smithi002.stdout: "subtree": { 2023-11-16T18:35:01.018 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subtree", 2023-11-16T18:35:01.018 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.018 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.018 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.019 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "rack", 2023-11-16T18:35:01.019 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.019 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.019 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.019 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "CRUSH level for which to create a local pool", 2023-11-16T18:35:01.019 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-11-16T18:35:01.019 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.019 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.020 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.020 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.020 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.020 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.020 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mds_autoscaler", 2023-11-16T18:35:01.020 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.020 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.020 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.021 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.021 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.021 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.021 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.021 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.021 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.021 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.021 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.022 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.022 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.022 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.022 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.022 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.022 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.022 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.022 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.023 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.023 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.023 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.023 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.023 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.023 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.023 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.023 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.023 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.024 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.024 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.024 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.024 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.024 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.024 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.024 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.024 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.025 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.025 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.025 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.025 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.025 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.025 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.025 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.025 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.026 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.026 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.026 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.026 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.026 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.026 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.026 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.026 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.027 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.027 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.027 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.027 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.027 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.027 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.027 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.027 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.028 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.028 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.028 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.028 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.028 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.028 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.028 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.028 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.029 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.029 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.029 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.029 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.029 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.029 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.029 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.029 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.029 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirroring", 2023-11-16T18:35:01.030 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.030 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.030 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.030 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.030 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.030 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.030 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.030 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.031 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.031 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.031 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.031 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.031 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.031 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.031 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.031 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.032 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.032 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.032 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.032 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.032 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.032 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.032 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.032 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.032 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.033 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.033 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.033 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.033 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.033 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.033 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.034 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.034 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.034 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.034 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.034 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.034 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.034 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.034 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.035 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.035 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.035 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.035 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.035 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.035 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.035 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.035 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.035 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.036 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.036 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.036 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.036 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.036 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.036 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.036 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.036 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.037 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.037 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.037 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.037 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.037 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.037 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.037 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.037 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.037 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.038 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.038 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.038 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.038 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.038 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.038 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.038 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.038 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.039 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.039 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.039 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "nfs", 2023-11-16T18:35:01.039 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.039 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.039 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.039 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.039 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.040 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.040 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.040 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.040 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.040 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.040 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.040 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.040 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.040 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.041 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.041 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.041 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.041 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.041 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.041 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.041 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.041 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.042 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.042 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.042 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.042 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.042 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.042 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.042 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.042 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.043 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.043 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.043 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.043 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.043 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.043 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.043 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.043 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.044 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.044 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.044 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.044 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.044 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.044 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.044 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.044 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.045 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.045 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.045 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.045 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.045 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.045 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.045 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.045 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.045 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.046 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.046 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.046 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.046 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.046 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.046 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.046 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.046 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.047 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.047 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.047 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.047 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.047 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.047 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.047 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.047 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.048 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.048 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.048 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.048 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.048 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.048 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2023-11-16T18:35:01.048 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.048 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.049 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.049 INFO:teuthology.orchestra.run.smithi002.stdout: "fail_fs": { 2023-11-16T18:35:01.049 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "fail_fs", 2023-11-16T18:35:01.049 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.049 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.049 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.049 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.049 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.049 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.050 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.050 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-11-16T18:35:01.050 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.050 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.050 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.050 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.050 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.050 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.051 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.051 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.051 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.051 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.051 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.051 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.051 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.051 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.052 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.052 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.052 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.052 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.052 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.052 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.052 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.052 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.053 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.053 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.053 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.053 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.053 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.053 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.053 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.053 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.053 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.054 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.054 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.054 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.054 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.054 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.054 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.054 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.087 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.087 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.087 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.087 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.087 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.087 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.087 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.087 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.088 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.088 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.088 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.088 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.088 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.088 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.088 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.088 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.089 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.089 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.089 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.089 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.089 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.089 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.089 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.089 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.090 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.090 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.090 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.090 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.090 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.090 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.090 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.090 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.091 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.091 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.091 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.091 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator": { 2023-11-16T18:35:01.091 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2023-11-16T18:35:01.091 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.091 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.091 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.091 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.092 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.092 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.092 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.092 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm", 2023-11-16T18:35:01.122 INFO:teuthology.orchestra.run.smithi002.stdout: "rook", 2023-11-16T18:35:01.122 INFO:teuthology.orchestra.run.smithi002.stdout: "test_orchestrator" 2023-11-16T18:35:01.122 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.122 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Orchestrator backend", 2023-11-16T18:35:01.122 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.122 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.122 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.123 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.123 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.123 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.123 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.123 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_perf_query", 2023-11-16T18:35:01.123 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.123 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.123 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.123 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.124 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.124 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.124 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.124 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.124 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.124 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.124 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.124 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.125 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.125 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.125 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.125 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.125 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.125 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.125 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.125 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.126 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.126 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.126 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.126 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.126 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.126 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.126 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.127 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.127 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.127 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.127 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.127 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.127 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.127 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.127 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.128 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.128 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.128 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.128 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.128 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.128 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.128 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.128 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.129 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.129 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.129 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.129 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.129 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.129 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_support", 2023-11-16T18:35:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_autoscaler", 2023-11-16T18:35:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.237 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.237 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.237 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.237 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.237 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "noautoscale": { 2023-11-16T18:35:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "noautoscale", 2023-11-16T18:35:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "global autoscale flag", 2023-11-16T18:35:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-11-16T18:35:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.250 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.250 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.250 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:35:01.250 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:35:01.250 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:01.267 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.267 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.267 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:35:01.267 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "threshold": { 2023-11-16T18:35:01.269 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threshold", 2023-11-16T18:35:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:35:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2023-11-16T18:35:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1.0", 2023-11-16T18:35:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "scaling threshold", 2023-11-16T18:35:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-11-16T18:35:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "progress", 2023-11-16T18:35:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_pg_recovery_event": { 2023-11-16T18:35:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_pg_recovery_event", 2023-11-16T18:35:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.353 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow the module to show pg recovery progress", 2023-11-16T18:35:01.353 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.353 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2023-11-16T18:35:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2023-11-16T18:35:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.420 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.420 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.420 INFO:teuthology.orchestra.run.smithi002.stdout: "max_completed_events": { 2023-11-16T18:35:01.420 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_completed_events", 2023-11-16T18:35:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "50", 2023-11-16T18:35:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "number of past completed events to remember", 2023-11-16T18:35:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T18:35:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T18:35:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T18:35:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long the module is going to sleep", 2023-11-16T18:35:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.466 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.466 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.466 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.466 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.467 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.467 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus", 2023-11-16T18:35:01.467 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.467 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "cache": { 2023-11-16T18:35:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", 2023-11-16T18:35:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "exclude_perf_counters": { 2023-11-16T18:35:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "exclude_perf_counters", 2023-11-16T18:35:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-11-16T18:35:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-11-16T18:35:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.479 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.479 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.479 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.479 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools": { 2023-11-16T18:35:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools", 2023-11-16T18:35:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools_refresh_interval": { 2023-11-16T18:35:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-11-16T18:35:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300", 2023-11-16T18:35:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_interval": { 2023-11-16T18:35:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_interval", 2023-11-16T18:35:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:35:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15.0", 2023-11-16T18:35:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T18:35:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T18:35:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-11-16T18:35:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-11-16T18:35:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T18:35:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T18:35:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "9283", 2023-11-16T18:35:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-11-16T18:35:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: "stale_cache_strategy": { 2023-11-16T18:35:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stale_cache_strategy", 2023-11-16T18:35:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "log", 2023-11-16T18:35:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2023-11-16T18:35:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2023-11-16T18:35:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "default", 2023-11-16T18:35:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "default", 2023-11-16T18:35:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "error" 2023-11-16T18:35:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2023-11-16T18:35:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2023-11-16T18:35:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2023-11-16T18:35:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-11-16T18:35:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2023-11-16T18:35:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_support", 2023-11-16T18:35:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.503 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.503 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.503 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.503 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.503 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_snap_create": { 2023-11-16T18:35:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_snap_create", 2023-11-16T18:35:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T18:35:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { 2023-11-16T18:35:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", 2023-11-16T18:35:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "trash_purge_schedule": { 2023-11-16T18:35:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "trash_purge_schedule", 2023-11-16T18:35:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "restful", 2023-11-16T18:35:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_auth": { 2023-11-16T18:35:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_auth", 2023-11-16T18:35:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2023-11-16T18:35:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2023-11-16T18:35:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T18:35:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T18:35:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T18:35:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T18:35:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.527 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.527 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.553 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.553 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.553 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.553 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rgw", 2023-11-16T18:35:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.645 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.645 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rook", 2023-11-16T18:35:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: "drive_group_interval": { 2023-11-16T18:35:01.669 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "drive_group_interval", 2023-11-16T18:35:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:35:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300.0", 2023-11-16T18:35:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-11-16T18:35:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.674 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.674 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.700 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.700 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.700 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.743 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "storage_class": { 2023-11-16T18:35:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "storage_class", 2023-11-16T18:35:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "local", 2023-11-16T18:35:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-11-16T18:35:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "selftest", 2023-11-16T18:35:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.759 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.759 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.759 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "roption1": { 2023-11-16T18:35:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption1", 2023-11-16T18:35:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "roption2": { 2023-11-16T18:35:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption2", 2023-11-16T18:35:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "xyz", 2023-11-16T18:35:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption1": { 2023-11-16T18:35:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption1", 2023-11-16T18:35:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption2": { 2023-11-16T18:35:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption2", 2023-11-16T18:35:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption3": { 2023-11-16T18:35:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption3", 2023-11-16T18:35:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T18:35:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption4": { 2023-11-16T18:35:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption4", 2023-11-16T18:35:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption5": { 2023-11-16T18:35:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption5", 2023-11-16T18:35:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption6": { 2023-11-16T18:35:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption6", 2023-11-16T18:35:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption7": { 2023-11-16T18:35:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption7", 2023-11-16T18:35:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T18:35:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "42", 2023-11-16T18:35:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "testkey": { 2023-11-16T18:35:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testkey", 2023-11-16T18:35:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.778 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.778 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.778 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.778 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "testlkey": { 2023-11-16T18:35:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testlkey", 2023-11-16T18:35:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "testnewline": { 2023-11-16T18:35:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testnewline", 2023-11-16T18:35:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2023-11-16T18:35:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_m_granularity": { 2023-11-16T18:35:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_m_granularity", 2023-11-16T18:35:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow minute scheduled snapshots", 2023-11-16T18:35:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "dump_on_update": { 2023-11-16T18:35:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dump_on_update", 2023-11-16T18:35:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "dump database to debug log on update", 2023-11-16T18:35:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.792 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.792 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.792 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stats", 2023-11-16T18:35:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.804 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.804 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.804 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.804 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.839 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.844 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.844 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.845 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.845 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "status", 2023-11-16T18:35:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.900 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.900 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.900 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telegraf", 2023-11-16T18:35:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: "address": { 2023-11-16T18:35:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "address", 2023-11-16T18:35:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-11-16T18:35:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:35:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:35:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15", 2023-11-16T18:35:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.908 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.909 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.909 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.909 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.910 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.910 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:01.910 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:01.910 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.912 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.912 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telemetry", 2023-11-16T18:35:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_basic": { 2023-11-16T18:35:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_basic", 2023-11-16T18:35:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share basic cluster information (size, version)", 2023-11-16T18:35:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_crash": { 2023-11-16T18:35:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_crash", 2023-11-16T18:35:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-11-16T18:35:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_device": { 2023-11-16T18:35:01.926 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_device", 2023-11-16T18:35:01.926 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T18:35:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-11-16T18:35:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_ident": { 2023-11-16T18:35:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_ident", 2023-11-16T18:35:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-11-16T18:35:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_perf": { 2023-11-16T18:35:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_perf", 2023-11-16T18:35:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share various performance metrics of a cluster", 2023-11-16T18:35:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "contact": { 2023-11-16T18:35:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "contact", 2023-11-16T18:35:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.978 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.978 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.979 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.979 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.979 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.979 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.980 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.980 INFO:teuthology.orchestra.run.smithi002.stdout: "description": { 2023-11-16T18:35:01.980 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "description", 2023-11-16T18:35:01.980 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.980 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.981 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.981 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: "device_url": { 2023-11-16T18:35:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_url", 2023-11-16T18:35:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-11-16T18:35:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:01.998 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:01.998 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:01.998 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2023-11-16T18:35:01.998 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2023-11-16T18:35:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.000 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.000 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.000 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.000 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.000 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.100 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.100 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:35:02.100 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:35:02.100 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "24", 2023-11-16T18:35:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", 2023-11-16T18:35:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: "last_opt_revision": { 2023-11-16T18:35:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "last_opt_revision", 2023-11-16T18:35:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1", 2023-11-16T18:35:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.104 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.104 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.104 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.104 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard": { 2023-11-16T18:35:02.105 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard", 2023-11-16T18:35:02.105 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:02.105 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard_description": { 2023-11-16T18:35:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard_description", 2023-11-16T18:35:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.125 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.126 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.128 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.132 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.133 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.133 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.134 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:02.135 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:02.136 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.137 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.137 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.137 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:02.138 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.138 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.138 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:02.142 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:02.142 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:02.143 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.144 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.144 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.145 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.145 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.146 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:02.146 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:02.147 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:02.148 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.148 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.149 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:02.150 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.150 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.151 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.151 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.152 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.153 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.154 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.154 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.155 INFO:teuthology.orchestra.run.smithi002.stdout: "organization": { 2023-11-16T18:35:02.155 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "organization", 2023-11-16T18:35:02.156 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.156 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.157 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.157 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:02.158 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.161 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.162 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.162 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.163 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: "proxy": { 2023-11-16T18:35:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "proxy", 2023-11-16T18:35:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: "url": { 2023-11-16T18:35:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url", 2023-11-16T18:35:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-11-16T18:35:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "test_orchestrator", 2023-11-16T18:35:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:02.188 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.188 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.188 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.188 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "volumes", 2023-11-16T18:35:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_clones": { 2023-11-16T18:35:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_clones", 2023-11-16T18:35:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4", 2023-11-16T18:35:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Number of asynchronous cloner threads", 2023-11-16T18:35:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot_clone_delay": { 2023-11-16T18:35:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snapshot_clone_delay", 2023-11-16T18:35:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T18:35:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-11-16T18:35:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix", 2023-11-16T18:35:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T18:35:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T18:35:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T18:35:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "discovery_interval": { 2023-11-16T18:35:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "discovery_interval", 2023-11-16T18:35:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T18:35:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "100", 2023-11-16T18:35:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "identifier": { 2023-11-16T18:35:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "identifier", 2023-11-16T18:35:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T18:35:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T18:35:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T18:35:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T18:35:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T18:35:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T18:35:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T18:35:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T18:35:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T18:35:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T18:35:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T18:35:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T18:35:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T18:35:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T18:35:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T18:35:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T18:35:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T18:35:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T18:35:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T18:35:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T18:35:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T18:35:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T18:35:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T18:35:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_host": { 2023-11-16T18:35:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_host", 2023-11-16T18:35:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T18:35:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_port": { 2023-11-16T18:35:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_port", 2023-11-16T18:35:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T18:35:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10051", 2023-11-16T18:35:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_sender": { 2023-11-16T18:35:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_sender", 2023-11-16T18:35:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T18:35:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T18:35:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T18:35:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-11-16T18:35:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T18:35:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T18:35:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T18:35:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T18:35:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T18:35:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T18:35:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T18:35:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: "services": { 2023-11-16T18:35:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: "dashboard": "https://172.21.15.2:8443/" 2023-11-16T18:35:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: "always_on_modules": { 2023-11-16T18:35:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "octopus": [ 2023-11-16T18:35:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", 2023-11-16T18:35:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-11-16T18:35:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "devicehealth", 2023-11-16T18:35:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator", 2023-11-16T18:35:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscaler", 2023-11-16T18:35:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "progress", 2023-11-16T18:35:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_support", 2023-11-16T18:35:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "status", 2023-11-16T18:35:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "telemetry", 2023-11-16T18:35:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "volumes" 2023-11-16T18:35:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "pacific": [ 2023-11-16T18:35:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", 2023-11-16T18:35:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-11-16T18:35:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "devicehealth", 2023-11-16T18:35:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator", 2023-11-16T18:35:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscaler", 2023-11-16T18:35:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "progress", 2023-11-16T18:35:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_support", 2023-11-16T18:35:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "status", 2023-11-16T18:35:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "telemetry", 2023-11-16T18:35:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: "volumes" 2023-11-16T18:35:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: "quincy": [ 2023-11-16T18:35:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", 2023-11-16T18:35:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-11-16T18:35:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: "devicehealth", 2023-11-16T18:35:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator", 2023-11-16T18:35:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscaler", 2023-11-16T18:35:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "progress", 2023-11-16T18:35:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_support", 2023-11-16T18:35:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "status", 2023-11-16T18:35:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "telemetry", 2023-11-16T18:35:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "volumes" 2023-11-16T18:35:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T18:35:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "reef": [ 2023-11-16T18:35:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", 2023-11-16T18:35:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-11-16T18:35:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "devicehealth", 2023-11-16T18:35:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator", 2023-11-16T18:35:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscaler", 2023-11-16T18:35:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "progress", 2023-11-16T18:35:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_support", 2023-11-16T18:35:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "status", 2023-11-16T18:35:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "telemetry", 2023-11-16T18:35:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "volumes" 2023-11-16T18:35:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T18:35:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "last_failure_osd_epoch": 91, 2023-11-16T18:35:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "active_clients": [ 2023-11-16T18:35:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", 2023-11-16T18:35:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T18:35:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T18:35:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T18:35:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3976091825 2023-11-16T18:35:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T18:35:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "libcephsqlite", 2023-11-16T18:35:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T18:35:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T18:35:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T18:35:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3574922251 2023-11-16T18:35:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T18:35:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_support", 2023-11-16T18:35:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T18:35:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T18:35:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T18:35:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3250155691 2023-11-16T18:35:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T18:35:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2023-11-16T18:35:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T18:35:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T18:35:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T18:35:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1958669134 2023-11-16T18:35:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T18:35:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2023-11-16T18:35:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T18:35:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T18:35:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T18:35:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3700793067 2023-11-16T18:35:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T18:35:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "volumes", 2023-11-16T18:35:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T18:35:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T18:35:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T18:35:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T18:35:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3601316131 2023-11-16T18:35:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T18:35:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:02.238 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T18:35:02.238 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:35:02.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:56 smithi040 ceph-mon[100251]: pgmap v314: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:35:02.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:56 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1799784171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:35:02.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:56 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1092344516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:35:02.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:56 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2787757545' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-11-16T18:35:02.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:57 smithi040 ceph-mon[100251]: pgmap v315: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:35:02.240 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:59 smithi040 ceph-mon[100251]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:35:02.241 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:34:59 smithi040 ceph-mon[100251]: pgmap v316: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 11 op/s 2023-11-16T18:35:02.241 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:56 smithi188 ceph-mon[100639]: pgmap v314: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:35:02.241 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:56 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1799784171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:35:02.241 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:56 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1092344516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:35:02.241 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:56 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2787757545' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-11-16T18:35:02.241 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:57 smithi188 ceph-mon[100639]: pgmap v315: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:35:02.241 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:59 smithi188 ceph-mon[100639]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:35:02.241 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:34:59 smithi188 ceph-mon[100639]: pgmap v316: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 11 op/s 2023-11-16T18:35:02.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.275+0000 7fdafe463700 1 -- 172.21.15.2:0/2787757545 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdae40a12c0 msgr2=0x7fdae40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:02.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.275+0000 7fdafe463700 1 --2- 172.21.15.2:0/2787757545 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdae40a12c0 0x7fdae40a3770 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdae0005950 tx=0x7fdae00058e0 comp rx=0 tx=0).stop 2023-11-16T18:35:02.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.275+0000 7fdafe463700 1 -- 172.21.15.2:0/2787757545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaf8072e10 msgr2=0x7fdaf8198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:02.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.275+0000 7fdafe463700 1 --2- 172.21.15.2:0/2787757545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaf8072e10 0x7fdaf8198700 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fdae800b550 tx=0x7fdae800b910 comp rx=0 tx=0).stop 2023-11-16T18:35:02.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.276+0000 7fdafe463700 1 -- 172.21.15.2:0/2787757545 shutdown_connections 2023-11-16T18:35:02.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.276+0000 7fdafe463700 1 --2- 172.21.15.2:0/2787757545 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fdae40a12c0 0x7fdae40a3770 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:02.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.276+0000 7fdafe463700 1 --2- 172.21.15.2:0/2787757545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf810cb90 0x7fdaf819bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:02.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.276+0000 7fdafe463700 1 --2- 172.21.15.2:0/2787757545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaf8072e10 0x7fdaf8198700 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:02.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.276+0000 7fdafe463700 1 --2- 172.21.15.2:0/2787757545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaf8073720 0x7fdaf8198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:02.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.276+0000 7fdafe463700 1 -- 172.21.15.2:0/2787757545 >> 172.21.15.2:0/2787757545 conn(0x7fdaf8077590 msgr2=0x7fdaf81045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:02.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.276+0000 7fdafe463700 1 -- 172.21.15.2:0/2787757545 shutdown_connections 2023-11-16T18:35:02.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:34:56.276+0000 7fdafe463700 1 -- 172.21.15.2:0/2787757545 wait complete. 2023-11-16T18:35:02.251 DEBUG:tasks.check_counter:Getting stats from a 2023-11-16T18:35:02.252 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.a perf dump 2023-11-16T18:35:02.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:56 smithi002 ceph-mon[98807]: pgmap v314: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:35:02.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:56 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1799784171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:35:02.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:56 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1092344516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:35:02.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:56 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2787757545' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-11-16T18:35:02.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:57 smithi002 ceph-mon[98807]: pgmap v315: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:35:02.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:59 smithi002 ceph-mon[98807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T18:35:02.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:34:59 smithi002 ceph-mon[98807]: pgmap v316: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 11 op/s 2023-11-16T18:35:02.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:02 smithi040 ceph-mon[100251]: pgmap v317: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2023-11-16T18:35:02.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:02 smithi188 ceph-mon[100639]: pgmap v317: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2023-11-16T18:35:02.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:02 smithi002 ceph-mon[98807]: pgmap v317: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2023-11-16T18:35:02.926 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:35:04.462 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:04 smithi002 ceph-mon[98807]: pgmap v318: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 26 op/s 2023-11-16T18:35:04.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:04 smithi040 ceph-mon[100251]: pgmap v318: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 26 op/s 2023-11-16T18:35:04.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:04 smithi188 ceph-mon[100639]: pgmap v318: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 26 op/s 2023-11-16T18:35:05.448 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:35:05.448 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T18:35:05.448 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 215, 2023-11-16T18:35:05.448 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 203, 2023-11-16T18:35:05.448 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1421712, 2023-11-16T18:35:05.448 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 114864, 2023-11-16T18:35:05.449 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2023-11-16T18:35:05.449 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2023-11-16T18:35:05.449 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.039969605, 2023-11-16T18:35:05.449 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.011975434, 2023-11-16T18:35:05.449 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.030352711, 2023-11-16T18:35:05.449 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.004767138, 2023-11-16T18:35:05.450 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:05.450 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 203, 2023-11-16T18:35:05.450 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.013558229, 2023-11-16T18:35:05.450 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000066789 2023-11-16T18:35:05.450 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:05.450 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:05.450 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:05.450 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:05.451 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:05.451 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:05.451 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 1421712, 2023-11-16T18:35:05.451 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 114864 2023-11-16T18:35:05.451 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:05.451 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T18:35:05.451 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 2, 2023-11-16T18:35:05.452 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 153, 2023-11-16T18:35:05.452 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 320, 2023-11-16T18:35:05.452 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 178624, 2023-11-16T18:35:05.452 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 5, 2023-11-16T18:35:05.452 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 3, 2023-11-16T18:35:05.452 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.012061761, 2023-11-16T18:35:05.452 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.007838559, 2023-11-16T18:35:05.453 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.002033695, 2023-11-16T18:35:05.453 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T18:35:05.454 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:05.454 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 150, 2023-11-16T18:35:05.455 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.013758628, 2023-11-16T18:35:05.455 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000091724 2023-11-16T18:35:05.455 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:05.455 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:05.456 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:05.457 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:05.457 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:05.457 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:05.457 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T18:35:05.457 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 178624 2023-11-16T18:35:05.457 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:05.457 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T18:35:05.457 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 0, 2023-11-16T18:35:05.458 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2, 2023-11-16T18:35:05.458 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 0, 2023-11-16T18:35:05.458 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 0, 2023-11-16T18:35:05.458 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2023-11-16T18:35:05.458 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2023-11-16T18:35:05.458 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.000462894, 2023-11-16T18:35:05.458 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T18:35:05.458 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.000377323, 2023-11-16T18:35:05.459 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T18:35:05.459 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:05.459 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:05.459 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:05.459 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:05.459 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:05.459 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:05.459 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:05.460 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:05.460 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:05.460 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:05.461 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T18:35:05.461 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T18:35:05.461 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:05.461 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2023-11-16T18:35:05.461 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2023-11-16T18:35:05.461 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2023-11-16T18:35:05.461 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:05.461 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2023-11-16T18:35:05.462 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2023-11-16T18:35:05.462 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2023-11-16T18:35:05.462 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T18:35:05.462 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2023-11-16T18:35:05.462 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T18:35:05.462 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2023-11-16T18:35:05.462 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T18:35:05.462 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T18:35:05.463 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T18:35:05.463 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T18:35:05.463 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T18:35:05.463 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2023-11-16T18:35:05.463 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T18:35:05.463 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T18:35:05.463 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_bytes": 0, 2023-11-16T18:35:05.463 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_items": 0, 2023-11-16T18:35:05.463 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_bytes": 0, 2023-11-16T18:35:05.464 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_items": 0, 2023-11-16T18:35:05.464 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T18:35:05.464 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T18:35:05.464 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T18:35:05.464 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T18:35:05.464 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T18:35:05.464 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2023-11-16T18:35:05.464 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2023-11-16T18:35:05.465 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2023-11-16T18:35:05.465 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T18:35:05.465 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T18:35:05.465 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2023-11-16T18:35:05.465 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2023-11-16T18:35:05.465 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2023-11-16T18:35:05.465 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2023-11-16T18:35:05.466 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T18:35:05.466 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2023-11-16T18:35:05.466 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T18:35:05.466 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2023-11-16T18:35:05.466 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 149057, 2023-11-16T18:35:05.466 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 32, 2023-11-16T18:35:05.466 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2023-11-16T18:35:05.466 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2023-11-16T18:35:05.467 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2023-11-16T18:35:05.467 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2023-11-16T18:35:05.467 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2023-11-16T18:35:05.467 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2023-11-16T18:35:05.467 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2023-11-16T18:35:05.467 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-11-16T18:35:05.467 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2023-11-16T18:35:05.468 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2023-11-16T18:35:05.468 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T18:35:05.468 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2023-11-16T18:35:05.469 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2023-11-16T18:35:05.469 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2023-11-16T18:35:05.469 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2023-11-16T18:35:05.469 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2023-11-16T18:35:05.469 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2023-11-16T18:35:05.469 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2023-11-16T18:35:05.469 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2023-11-16T18:35:05.470 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2023-11-16T18:35:05.470 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:05.471 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T18:35:05.471 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2023-11-16T18:35:05.471 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2023-11-16T18:35:05.471 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2023-11-16T18:35:05.471 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 213, 2023-11-16T18:35:05.471 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 1388939, 2023-11-16T18:35:05.471 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:05.472 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 213, 2023-11-16T18:35:05.472 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2023-11-16T18:35:05.472 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2023-11-16T18:35:05.472 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 213, 2023-11-16T18:35:05.472 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 1388939, 2023-11-16T18:35:05.472 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2023-11-16T18:35:05.472 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:05.472 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:05.473 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:05.473 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:05.473 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:05.473 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:35:05.881 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2023-11-16T18:35:05.881 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2023-11-16T18:35:05.882 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2023-11-16T18:35:05.882 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2023-11-16T18:35:05.882 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2023-11-16T18:35:05.882 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2023-11-16T18:35:05.882 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2023-11-16T18:35:05.882 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2023-11-16T18:35:05.882 DEBUG:tasks.check_counter:Getting stats from d 2023-11-16T18:35:05.883 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.d perf dump 2023-11-16T18:35:06.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:06 smithi040 ceph-mon[100251]: pgmap v319: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-11-16T18:35:06.536 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:35:06.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:06 smithi188 ceph-mon[100639]: pgmap v319: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-11-16T18:35:06.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:06 smithi002 ceph-mon[98807]: pgmap v319: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-11-16T18:35:08.129 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:35:08.130 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T18:35:08.130 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 4, 2023-11-16T18:35:08.130 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 4, 2023-11-16T18:35:08.130 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 2992, 2023-11-16T18:35:08.130 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 448, 2023-11-16T18:35:08.130 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2023-11-16T18:35:08.130 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2023-11-16T18:35:08.130 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.001878606, 2023-11-16T18:35:08.131 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000046386, 2023-11-16T18:35:08.131 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.000780179, 2023-11-16T18:35:08.131 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T18:35:08.131 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:08.131 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 3, 2023-11-16T18:35:08.131 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000809910, 2023-11-16T18:35:08.131 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000269970 2023-11-16T18:35:08.131 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:08.132 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:08.132 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:08.132 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:08.132 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:08.132 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:08.132 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-11-16T18:35:08.132 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 448 2023-11-16T18:35:08.132 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:08.132 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T18:35:08.133 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 216, 2023-11-16T18:35:08.133 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 206, 2023-11-16T18:35:08.133 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1420080, 2023-11-16T18:35:08.133 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 117376, 2023-11-16T18:35:08.133 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2023-11-16T18:35:08.133 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2023-11-16T18:35:08.133 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.040149966, 2023-11-16T18:35:08.133 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.013154649, 2023-11-16T18:35:08.134 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.030282194, 2023-11-16T18:35:08.134 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.004760662, 2023-11-16T18:35:08.134 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:08.134 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 205, 2023-11-16T18:35:08.134 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.018564415, 2023-11-16T18:35:08.134 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000090558 2023-11-16T18:35:08.134 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:08.134 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:08.135 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:08.135 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:08.135 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:08.135 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:08.135 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 1420080, 2023-11-16T18:35:08.135 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 117376 2023-11-16T18:35:08.135 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:08.135 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T18:35:08.135 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 2, 2023-11-16T18:35:08.136 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 157, 2023-11-16T18:35:08.136 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 320, 2023-11-16T18:35:08.136 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 179328, 2023-11-16T18:35:08.136 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 5, 2023-11-16T18:35:08.136 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 3, 2023-11-16T18:35:08.136 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.013137897, 2023-11-16T18:35:08.136 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.008752593, 2023-11-16T18:35:08.136 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.001939343, 2023-11-16T18:35:08.136 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T18:35:08.137 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:08.137 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 154, 2023-11-16T18:35:08.137 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.011666646, 2023-11-16T18:35:08.137 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000075757 2023-11-16T18:35:08.137 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:08.137 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:08.137 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:08.137 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:08.138 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:08.138 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:08.138 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T18:35:08.138 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 179328 2023-11-16T18:35:08.138 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:08.138 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2023-11-16T18:35:08.138 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2023-11-16T18:35:08.138 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2023-11-16T18:35:08.139 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:08.139 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2023-11-16T18:35:08.139 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2023-11-16T18:35:08.139 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2023-11-16T18:35:08.139 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T18:35:08.139 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2023-11-16T18:35:08.139 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T18:35:08.139 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2023-11-16T18:35:08.140 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T18:35:08.140 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T18:35:08.140 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T18:35:08.140 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T18:35:08.140 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T18:35:08.140 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2023-11-16T18:35:08.140 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T18:35:08.140 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T18:35:08.140 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_bytes": 0, 2023-11-16T18:35:08.141 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_items": 0, 2023-11-16T18:35:08.141 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_bytes": 0, 2023-11-16T18:35:08.141 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_items": 0, 2023-11-16T18:35:08.141 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T18:35:08.141 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T18:35:08.141 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T18:35:08.141 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T18:35:08.141 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T18:35:08.141 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2023-11-16T18:35:08.142 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2023-11-16T18:35:08.142 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2023-11-16T18:35:08.142 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T18:35:08.142 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T18:35:08.142 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2023-11-16T18:35:08.142 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2023-11-16T18:35:08.142 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2023-11-16T18:35:08.142 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2023-11-16T18:35:08.143 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T18:35:08.143 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2023-11-16T18:35:08.143 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T18:35:08.143 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2023-11-16T18:35:08.143 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 149057, 2023-11-16T18:35:08.143 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 32, 2023-11-16T18:35:08.143 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2023-11-16T18:35:08.143 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2023-11-16T18:35:08.143 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2023-11-16T18:35:08.144 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2023-11-16T18:35:08.144 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2023-11-16T18:35:08.144 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2023-11-16T18:35:08.144 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2023-11-16T18:35:08.144 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-11-16T18:35:08.144 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2023-11-16T18:35:08.144 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2023-11-16T18:35:08.144 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T18:35:08.145 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2023-11-16T18:35:08.145 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2023-11-16T18:35:08.145 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2023-11-16T18:35:08.145 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2023-11-16T18:35:08.145 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2023-11-16T18:35:08.145 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2023-11-16T18:35:08.145 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2023-11-16T18:35:08.145 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2023-11-16T18:35:08.145 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2023-11-16T18:35:08.146 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:08.146 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T18:35:08.146 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2023-11-16T18:35:08.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2023-11-16T18:35:08.146 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2023-11-16T18:35:08.146 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 218, 2023-11-16T18:35:08.146 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 1389589, 2023-11-16T18:35:08.146 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:08.147 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 218, 2023-11-16T18:35:08.147 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2023-11-16T18:35:08.147 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2023-11-16T18:35:08.147 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 218, 2023-11-16T18:35:08.147 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 1389589, 2023-11-16T18:35:08.147 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2023-11-16T18:35:08.147 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:08.147 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:08.147 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:08.148 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:08.148 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:08.148 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:35:08.517 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2023-11-16T18:35:08.517 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2023-11-16T18:35:08.517 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2023-11-16T18:35:08.517 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2023-11-16T18:35:08.517 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2023-11-16T18:35:08.517 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2023-11-16T18:35:08.518 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2023-11-16T18:35:08.518 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2023-11-16T18:35:08.518 DEBUG:tasks.check_counter:Getting stats from g 2023-11-16T18:35:08.518 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.g perf dump 2023-11-16T18:35:08.520 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:08 smithi002 ceph-mon[98807]: pgmap v320: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-11-16T18:35:08.520 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:08 smithi002 ceph-mon[98807]: osdmap e101: 12 total, 12 up, 12 in 2023-11-16T18:35:08.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:08 smithi188 ceph-mon[100639]: pgmap v320: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-11-16T18:35:08.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:08 smithi188 ceph-mon[100639]: osdmap e101: 12 total, 12 up, 12 in 2023-11-16T18:35:08.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:08 smithi040 ceph-mon[100251]: pgmap v320: 129 pgs: 129 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-11-16T18:35:08.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:08 smithi040 ceph-mon[100251]: osdmap e101: 12 total, 12 up, 12 in 2023-11-16T18:35:09.571 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:35:10.461 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:10 smithi002 ceph-mon[98807]: pgmap v322: 129 pgs: 57 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 54 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:35:10.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:10 smithi188 ceph-mon[100639]: pgmap v322: 129 pgs: 57 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 54 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:35:10.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:10 smithi040 ceph-mon[100251]: pgmap v322: 129 pgs: 57 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 54 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:35:11.260 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:35:11.261 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T18:35:11.261 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 0, 2023-11-16T18:35:11.261 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2, 2023-11-16T18:35:11.261 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 0, 2023-11-16T18:35:11.261 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 0, 2023-11-16T18:35:11.261 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2023-11-16T18:35:11.261 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2023-11-16T18:35:11.262 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.001500785, 2023-11-16T18:35:11.262 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T18:35:11.262 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.000172436, 2023-11-16T18:35:11.262 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T18:35:11.262 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:11.262 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:11.262 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:11.262 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:11.263 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:11.263 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:11.263 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:11.263 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:11.263 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:11.263 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:11.263 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T18:35:11.263 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T18:35:11.263 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:11.264 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T18:35:11.264 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 213, 2023-11-16T18:35:11.264 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 203, 2023-11-16T18:35:11.264 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1419264, 2023-11-16T18:35:11.264 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 115600, 2023-11-16T18:35:11.264 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2023-11-16T18:35:11.264 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2023-11-16T18:35:11.264 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.039818392, 2023-11-16T18:35:11.265 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.012708775, 2023-11-16T18:35:11.265 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.029639714, 2023-11-16T18:35:11.265 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.005283848, 2023-11-16T18:35:11.265 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:11.265 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 202, 2023-11-16T18:35:11.265 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.010827923, 2023-11-16T18:35:11.265 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000053603 2023-11-16T18:35:11.265 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:11.266 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:11.266 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:11.266 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:11.266 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:11.266 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:11.287 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 1419264, 2023-11-16T18:35:11.287 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 115600 2023-11-16T18:35:11.287 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:11.287 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T18:35:11.287 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 6, 2023-11-16T18:35:11.288 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 157, 2023-11-16T18:35:11.288 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 3312, 2023-11-16T18:35:11.288 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 179424, 2023-11-16T18:35:11.288 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 5, 2023-11-16T18:35:11.288 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 3, 2023-11-16T18:35:11.288 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.011765715, 2023-11-16T18:35:11.288 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.007837462, 2023-11-16T18:35:11.288 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.001819644, 2023-11-16T18:35:11.289 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T18:35:11.289 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:11.289 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 155, 2023-11-16T18:35:11.289 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.017440382, 2023-11-16T18:35:11.289 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000112518 2023-11-16T18:35:11.289 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:11.289 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:11.289 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:11.290 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:11.290 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:11.290 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:11.290 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-11-16T18:35:11.290 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 179424 2023-11-16T18:35:11.290 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:11.290 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2023-11-16T18:35:11.290 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2023-11-16T18:35:11.291 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2023-11-16T18:35:11.325 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:11.325 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2023-11-16T18:35:11.326 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2023-11-16T18:35:11.326 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2023-11-16T18:35:11.326 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T18:35:11.326 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2023-11-16T18:35:11.326 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T18:35:11.326 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2023-11-16T18:35:11.326 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T18:35:11.327 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T18:35:11.327 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T18:35:11.327 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T18:35:11.327 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T18:35:11.327 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2023-11-16T18:35:11.327 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T18:35:11.327 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T18:35:11.327 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_bytes": 0, 2023-11-16T18:35:11.328 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_items": 0, 2023-11-16T18:35:11.328 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_bytes": 0, 2023-11-16T18:35:11.328 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_items": 0, 2023-11-16T18:35:11.328 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T18:35:11.328 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T18:35:11.328 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T18:35:11.328 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T18:35:11.328 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T18:35:11.329 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2023-11-16T18:35:11.329 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2023-11-16T18:35:11.329 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2023-11-16T18:35:11.329 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T18:35:11.329 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T18:35:11.329 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2023-11-16T18:35:11.329 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2023-11-16T18:35:11.329 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2023-11-16T18:35:11.329 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2023-11-16T18:35:11.330 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T18:35:11.330 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2023-11-16T18:35:11.330 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T18:35:11.330 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2023-11-16T18:35:11.330 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 148955, 2023-11-16T18:35:11.330 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 30, 2023-11-16T18:35:11.330 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2023-11-16T18:35:11.330 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2023-11-16T18:35:11.331 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2023-11-16T18:35:11.331 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2023-11-16T18:35:11.331 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2023-11-16T18:35:11.331 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2023-11-16T18:35:11.331 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2023-11-16T18:35:11.331 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-11-16T18:35:11.331 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2023-11-16T18:35:11.331 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2023-11-16T18:35:11.332 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T18:35:11.332 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2023-11-16T18:35:11.332 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2023-11-16T18:35:11.332 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2023-11-16T18:35:11.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2023-11-16T18:35:11.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2023-11-16T18:35:11.332 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2023-11-16T18:35:11.332 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2023-11-16T18:35:11.333 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2023-11-16T18:35:11.333 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2023-11-16T18:35:11.333 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:11.333 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T18:35:11.333 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2023-11-16T18:35:11.333 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2023-11-16T18:35:11.344 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2023-11-16T18:35:11.344 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 215, 2023-11-16T18:35:11.345 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 1389199, 2023-11-16T18:35:11.434 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:11.434 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 215, 2023-11-16T18:35:11.434 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2023-11-16T18:35:11.435 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2023-11-16T18:35:11.435 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 215, 2023-11-16T18:35:11.435 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 1389199, 2023-11-16T18:35:11.435 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2023-11-16T18:35:11.435 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:11.435 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:11.435 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:11.435 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:11.436 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:11.436 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:35:11.754 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2023-11-16T18:35:11.754 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2023-11-16T18:35:11.754 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2023-11-16T18:35:11.754 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2023-11-16T18:35:11.755 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2023-11-16T18:35:11.755 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2023-11-16T18:35:11.755 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2023-11-16T18:35:11.755 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2023-11-16T18:35:11.755 DEBUG:tasks.check_counter:Getting stats from j 2023-11-16T18:35:11.755 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.j perf dump 2023-11-16T18:35:12.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:12 smithi188 ceph-mon[100639]: pgmap v323: 129 pgs: 57 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 54 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:35:12.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:12 smithi002 ceph-mon[98807]: pgmap v323: 129 pgs: 57 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 54 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:35:12.640 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T18:35:12.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:12 smithi040 ceph-mon[100251]: pgmap v323: 129 pgs: 57 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 54 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-11-16T18:35:14.292 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:14 smithi002 ceph-mon[98807]: pgmap v324: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:35:14.346 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T18:35:14.346 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T18:35:14.346 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 0, 2023-11-16T18:35:14.346 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2, 2023-11-16T18:35:14.347 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 0, 2023-11-16T18:35:14.347 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 0, 2023-11-16T18:35:14.347 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2023-11-16T18:35:14.347 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2023-11-16T18:35:14.347 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.001843051, 2023-11-16T18:35:14.347 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T18:35:14.347 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.000315068, 2023-11-16T18:35:14.347 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T18:35:14.348 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:14.348 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:14.348 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:14.348 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:14.348 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:14.348 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:14.348 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:14.349 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:14.349 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:14.349 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:14.349 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T18:35:14.349 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T18:35:14.349 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:14.349 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T18:35:14.349 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 218, 2023-11-16T18:35:14.350 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 208, 2023-11-16T18:35:14.350 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1420624, 2023-11-16T18:35:14.350 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 118560, 2023-11-16T18:35:14.350 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2023-11-16T18:35:14.350 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2023-11-16T18:35:14.350 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.040385559, 2023-11-16T18:35:14.350 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.012803566, 2023-11-16T18:35:14.351 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.031621146, 2023-11-16T18:35:14.351 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.004715952, 2023-11-16T18:35:14.351 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:14.351 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 207, 2023-11-16T18:35:14.351 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.011517953, 2023-11-16T18:35:14.351 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000055642 2023-11-16T18:35:14.351 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:14.351 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:14.352 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:14.352 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:14.352 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:14.352 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:14.352 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 1420624, 2023-11-16T18:35:14.352 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 118560 2023-11-16T18:35:14.352 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:14.353 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T18:35:14.353 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 6, 2023-11-16T18:35:14.353 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 161, 2023-11-16T18:35:14.353 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 3312, 2023-11-16T18:35:14.353 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 180128, 2023-11-16T18:35:14.353 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 5, 2023-11-16T18:35:14.353 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 3, 2023-11-16T18:35:14.353 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.012737007, 2023-11-16T18:35:14.354 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.008684308, 2023-11-16T18:35:14.354 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.001840325, 2023-11-16T18:35:14.354 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T18:35:14.354 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:14.354 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 159, 2023-11-16T18:35:14.354 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.017131789, 2023-11-16T18:35:14.354 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000107747 2023-11-16T18:35:14.355 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:14.355 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:14.355 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:14.355 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:14.355 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:14.355 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:14.355 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-11-16T18:35:14.355 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 180128 2023-11-16T18:35:14.356 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:14.356 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2023-11-16T18:35:14.356 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2023-11-16T18:35:14.356 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2023-11-16T18:35:14.356 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:14.356 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2023-11-16T18:35:14.356 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2023-11-16T18:35:14.356 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2023-11-16T18:35:14.386 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T18:35:14.386 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2023-11-16T18:35:14.386 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T18:35:14.386 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2023-11-16T18:35:14.387 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T18:35:14.387 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T18:35:14.387 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T18:35:14.387 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T18:35:14.387 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T18:35:14.387 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2023-11-16T18:35:14.387 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T18:35:14.388 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T18:35:14.388 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_bytes": 0, 2023-11-16T18:35:14.388 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_items": 0, 2023-11-16T18:35:14.388 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_bytes": 0, 2023-11-16T18:35:14.388 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_items": 0, 2023-11-16T18:35:14.388 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T18:35:14.388 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T18:35:14.388 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T18:35:14.389 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T18:35:14.389 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T18:35:14.389 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2023-11-16T18:35:14.389 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2023-11-16T18:35:14.389 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2023-11-16T18:35:14.389 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T18:35:14.389 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T18:35:14.389 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2023-11-16T18:35:14.390 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2023-11-16T18:35:14.390 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2023-11-16T18:35:14.390 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2023-11-16T18:35:14.390 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T18:35:14.390 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2023-11-16T18:35:14.390 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T18:35:14.390 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2023-11-16T18:35:14.391 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 148977, 2023-11-16T18:35:14.391 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 30, 2023-11-16T18:35:14.391 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2023-11-16T18:35:14.391 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2023-11-16T18:35:14.391 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2023-11-16T18:35:14.391 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2023-11-16T18:35:14.391 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2023-11-16T18:35:14.391 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2023-11-16T18:35:14.392 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2023-11-16T18:35:14.392 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-11-16T18:35:14.392 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2023-11-16T18:35:14.392 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2023-11-16T18:35:14.392 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T18:35:14.392 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2023-11-16T18:35:14.392 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2023-11-16T18:35:14.392 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2023-11-16T18:35:14.393 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2023-11-16T18:35:14.393 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2023-11-16T18:35:14.393 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2023-11-16T18:35:14.393 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2023-11-16T18:35:14.393 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2023-11-16T18:35:14.393 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2023-11-16T18:35:14.393 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T18:35:14.393 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T18:35:14.394 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2023-11-16T18:35:14.394 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2023-11-16T18:35:14.394 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2023-11-16T18:35:14.394 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 220, 2023-11-16T18:35:14.394 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 1389849, 2023-11-16T18:35:14.394 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:14.394 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 220, 2023-11-16T18:35:14.395 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2023-11-16T18:35:14.395 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2023-11-16T18:35:14.395 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 220, 2023-11-16T18:35:14.395 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 1389849, 2023-11-16T18:35:14.395 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2023-11-16T18:35:14.395 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T18:35:14.395 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T18:35:14.395 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T18:35:14.395 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:14.396 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T18:35:14.396 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T18:35:14.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:14 smithi188 ceph-mon[100639]: pgmap v324: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:35:14.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:14 smithi040 ceph-mon[100251]: pgmap v324: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:35:14.863 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2023-11-16T18:35:14.863 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2023-11-16T18:35:14.863 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2023-11-16T18:35:14.863 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2023-11-16T18:35:14.863 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2023-11-16T18:35:14.863 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2023-11-16T18:35:14.864 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2023-11-16T18:35:14.864 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2023-11-16T18:35:14.864 DEBUG:tasks.check_counter:Getting stats from b 2023-11-16T18:35:14.864 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.b perf dump 2023-11-16T18:35:15.863 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T18:35:16.470 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:16 smithi040 ceph-mon[100251]: pgmap v325: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:35:16.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:16 smithi188 ceph-mon[100639]: pgmap v325: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:35:16.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:16 smithi002 ceph-mon[98807]: pgmap v325: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:35:17.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:17 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:35:17.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:17 smithi040 ceph-mon[100251]: pgmap v326: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:35:17.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:17 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:35:17.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:17 smithi188 ceph-mon[100639]: pgmap v326: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:35:17.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:17 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T18:35:17.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:17 smithi002 ceph-mon[98807]: pgmap v326: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-11-16T18:35:19.028 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-11-16T18:35:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T18:35:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 36131, 2023-11-16T18:35:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 35973, 2023-11-16T18:35:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 17429042, 2023-11-16T18:35:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 162727197, 2023-11-16T18:35:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 124, 2023-11-16T18:35:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 8, 2023-11-16T18:35:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 2.691545326, 2023-11-16T18:35:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.802694887, 2023-11-16T18:35:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.450972450, 2023-11-16T18:35:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 1.333607807, 2023-11-16T18:35:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 35969, 2023-11-16T18:35:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 68.252586071, 2023-11-16T18:35:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001897539 2023-11-16T18:35:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 420, 2023-11-16T18:35:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000084720, 2023-11-16T18:35:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000201 2023-11-16T18:35:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 101952, 2023-11-16T18:35:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 34672 2023-11-16T18:35:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T18:35:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 96951, 2023-11-16T18:35:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 85015, 2023-11-16T18:35:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 56149178, 2023-11-16T18:35:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 223769563, 2023-11-16T18:35:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 13, 2023-11-16T18:35:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2023-11-16T18:35:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 6.333977812, 2023-11-16T18:35:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 2.550432000, 2023-11-16T18:35:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 3.067342901, 2023-11-16T18:35:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 1.266955854, 2023-11-16T18:35:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 85014, 2023-11-16T18:35:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 68.822130287, 2023-11-16T18:35:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000809538 2023-11-16T18:35:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 115668, 2023-11-16T18:35:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.021055361, 2023-11-16T18:35:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000182 2023-11-16T18:35:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 43424, 2023-11-16T18:35:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 439120 2023-11-16T18:35:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T18:35:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 111436, 2023-11-16T18:35:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 172443, 2023-11-16T18:35:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 169433652, 2023-11-16T18:35:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 145891847, 2023-11-16T18:35:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 15, 2023-11-16T18:35:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 7, 2023-11-16T18:35:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 7.314648388, 2023-11-16T18:35:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 2.733096982, 2023-11-16T18:35:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 143.333056194, 2023-11-16T18:35:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 1.020758770, 2023-11-16T18:35:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 172443, 2023-11-16T18:35:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 160.699816685, 2023-11-16T18:35:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000931901 2023-11-16T18:35:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 215676, 2023-11-16T18:35:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.045695599, 2023-11-16T18:35:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000211 2023-11-16T18:35:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1854832, 2023-11-16T18:35:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 232624 2023-11-16T18:35:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2023-11-16T18:35:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2023-11-16T18:35:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2023-11-16T18:35:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2023-11-16T18:35:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2023-11-16T18:35:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2023-11-16T18:35:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 12200, 2023-11-16T18:35:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 120.882063185, 2023-11-16T18:35:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.009908365 2023-11-16T18:35:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2023-11-16T18:35:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2023-11-16T18:35:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2023-11-16T18:35:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 16643, 2023-11-16T18:35:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 13.809405936, 2023-11-16T18:35:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000829742 2023-11-16T18:35:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2023-11-16T18:35:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 46907, 2023-11-16T18:35:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 38163, 2023-11-16T18:35:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2023-11-16T18:35:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 38163, 2023-11-16T18:35:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 317.563811165, 2023-11-16T18:35:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.008321248 2023-11-16T18:35:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2023-11-16T18:35:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 8744, 2023-11-16T18:35:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 892, 2023-11-16T18:35:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 1, 2023-11-16T18:35:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 15159, 2023-11-16T18:35:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 142, 2023-11-16T18:35:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 86, 2023-11-16T18:35:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 13001, 2023-11-16T18:35:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 9084, 2023-11-16T18:35:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 3902, 2023-11-16T18:35:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 15, 2023-11-16T18:35:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 23, 2023-11-16T18:35:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 74817, 2023-11-16T18:35:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 5, 2023-11-16T18:35:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 5, 2023-11-16T18:35:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 4, 2023-11-16T18:35:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 58671, 2023-11-16T18:35:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 49548, 2023-11-16T18:35:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 7096, 2023-11-16T18:35:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 0, 2023-11-16T18:35:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 85, 2023-11-16T18:35:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2023-11-16T18:35:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2023-11-16T18:35:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2023-11-16T18:35:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2023-11-16T18:35:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 1133, 2023-11-16T18:35:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 54754, 2023-11-16T18:35:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 1321, 2023-11-16T18:35:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 0, 2023-11-16T18:35:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2023-11-16T18:35:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 11759, 2023-11-16T18:35:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2023-11-16T18:35:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 4, 2023-11-16T18:35:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 246, 2023-11-16T18:35:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 6, 2023-11-16T18:35:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 11759, 2023-11-16T18:35:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2023-11-16T18:35:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T18:35:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2023-11-16T18:35:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 58, 2023-11-16T18:35:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 6, 2023-11-16T18:35:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 52, 2023-11-16T18:35:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 11701, 2023-11-16T18:35:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 6, 2023-11-16T18:35:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 356, 2023-11-16T18:35:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 24955, 2023-11-16T18:35:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 1, 2023-11-16T18:35:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 4, 2023-11-16T18:35:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T18:35:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 22910, 2023-11-16T18:35:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 22548, 2023-11-16T18:35:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 237, 2023-11-16T18:35:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 23461 2023-11-16T18:35:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2023-11-16T18:35:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 978, 2023-11-16T18:35:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2023-11-16T18:35:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2023-11-16T18:35:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 0, 2023-11-16T18:35:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 1254, 2023-11-16T18:35:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 6, 2023-11-16T18:35:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2023-11-16T18:35:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2023-11-16T18:35:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 10914, 2023-11-16T18:35:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 10908, 2023-11-16T18:35:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2023-11-16T18:35:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2023-11-16T18:35:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2023-11-16T18:35:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2023-11-16T18:35:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2023-11-16T18:35:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2023-11-16T18:35:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2023-11-16T18:35:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 6, 2023-11-16T18:35:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 1137, 2023-11-16T18:35:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2023-11-16T18:35:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 250, 2023-11-16T18:35:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2023-11-16T18:35:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2023-11-16T18:35:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2023-11-16T18:35:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 63440, 2023-11-16T18:35:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 63377, 2023-11-16T18:35:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 63377, 2023-11-16T18:35:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 63, 2023-11-16T18:35:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2023-11-16T18:35:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 23, 2023-11-16T18:35:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 134, 2023-11-16T18:35:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 133, 2023-11-16T18:35:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 133, 2023-11-16T18:35:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 2, 2023-11-16T18:35:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2023-11-16T18:35:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 1, 2023-11-16T18:35:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 331355302, 2023-11-16T18:35:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 331551230, 2023-11-16T18:35:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 0, 2023-11-16T18:35:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2023-11-16T18:35:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3844, 2023-11-16T18:35:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 376.689161526, 2023-11-16T18:35:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.097994058 2023-11-16T18:35:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 0 2023-11-16T18:35:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2023-11-16T18:35:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 12350, 2023-11-16T18:35:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 74774, 2023-11-16T18:35:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 62424, 2023-11-16T18:35:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 522, 2023-11-16T18:35:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 3415, 2023-11-16T18:35:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 2893, 2023-11-16T18:35:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 13001, 2023-11-16T18:35:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 103539, 2023-11-16T18:35:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 90538, 2023-11-16T18:35:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 5, 2023-11-16T18:35:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 34191, 2023-11-16T18:35:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 34186, 2023-11-16T18:35:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 238644, 2023-11-16T18:35:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 207156 2023-11-16T18:35:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2023-11-16T18:35:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 47814, 2023-11-16T18:35:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 778, 2023-11-16T18:35:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 46907, 2023-11-16T18:35:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 171, 2023-11-16T18:35:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 1607, 2023-11-16T18:35:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2023-11-16T18:35:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 13953, 2023-11-16T18:35:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 128.003465610, 2023-11-16T18:35:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.009173902 2023-11-16T18:35:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2023-11-16T18:35:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 329, 2023-11-16T18:35:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 1.121070740, 2023-11-16T18:35:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.003407509 2023-11-16T18:35:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2023-11-16T18:35:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2023-11-16T18:35:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2023-11-16T18:35:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 11200, 2023-11-16T18:35:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 64.728315192, 2023-11-16T18:35:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.005779313 2023-11-16T18:35:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2023-11-16T18:35:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2023-11-16T18:35:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2023-11-16T18:35:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2023-11-16T18:35:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2023-11-16T18:35:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2023-11-16T18:35:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 11, 2023-11-16T18:35:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.008587433, 2023-11-16T18:35:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000780675 2023-11-16T18:35:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2023-11-16T18:35:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 125, 2023-11-16T18:35:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.181075730, 2023-11-16T18:35:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001448605 2023-11-16T18:35:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2023-11-16T18:35:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2023-11-16T18:35:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 11, 2023-11-16T18:35:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 14.210780404, 2023-11-16T18:35:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 1.291889127 2023-11-16T18:35:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2023-11-16T18:35:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2023-11-16T18:35:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1568, 2023-11-16T18:35:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 6.562737620, 2023-11-16T18:35:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.004185419 2023-11-16T18:35:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2023-11-16T18:35:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2023-11-16T18:35:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.003711023, 2023-11-16T18:35:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001855511 2023-11-16T18:35:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2023-11-16T18:35:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2023-11-16T18:35:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2023-11-16T18:35:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 112, 2023-11-16T18:35:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.527750306, 2023-11-16T18:35:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.004712056 2023-11-16T18:35:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2023-11-16T18:35:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 5, 2023-11-16T18:35:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 3.168265474, 2023-11-16T18:35:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.633653094 2023-11-16T18:35:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2023-11-16T18:35:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3, 2023-11-16T18:35:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.002672757, 2023-11-16T18:35:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000890919 2023-11-16T18:35:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2023-11-16T18:35:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 4, 2023-11-16T18:35:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.018618976, 2023-11-16T18:35:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.004654744 2023-11-16T18:35:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2023-11-16T18:35:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2023-11-16T18:35:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 22, 2023-11-16T18:35:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 10.020242436, 2023-11-16T18:35:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.455465565 2023-11-16T18:35:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2023-11-16T18:35:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2023-11-16T18:35:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 16, 2023-11-16T18:35:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.039468121, 2023-11-16T18:35:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002466757 2023-11-16T18:35:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2023-11-16T18:35:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2023-11-16T18:35:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 10802, 2023-11-16T18:35:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 88.967049343, 2023-11-16T18:35:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.008236164 2023-11-16T18:35:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2023-11-16T18:35:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2023-11-16T18:35:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2023-11-16T18:35:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2023-11-16T18:35:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 2, 2023-11-16T18:35:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 6, 2023-11-16T18:35:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 4, 2023-11-16T18:35:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 2, 2023-11-16T18:35:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2023-11-16T18:35:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 920, 2023-11-16T18:35:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 460, 2023-11-16T18:35:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 629 2023-11-16T18:35:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2023-11-16T18:35:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 600, 2023-11-16T18:35:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 600, 2023-11-16T18:35:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T18:35:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2023-11-16T18:35:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T18:35:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2023-11-16T18:35:19.081 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T18:35:19.081 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T18:35:19.081 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T18:35:19.081 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T18:35:19.081 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T18:35:19.081 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2023-11-16T18:35:19.081 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T18:35:19.081 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T18:35:19.082 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2023-11-16T18:35:19.082 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2023-11-16T18:35:19.082 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2023-11-16T18:35:19.082 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2023-11-16T18:35:19.082 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T18:35:19.082 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T18:35:19.082 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T18:35:19.082 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T18:35:19.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T18:35:19.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2023-11-16T18:35:19.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2023-11-16T18:35:19.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2023-11-16T18:35:19.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T18:35:19.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T18:35:19.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2023-11-16T18:35:19.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2023-11-16T18:35:19.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2023-11-16T18:35:19.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2023-11-16T18:35:19.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T18:35:19.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2023-11-16T18:35:19.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T18:35:19.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2023-11-16T18:35:19.084 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 729861, 2023-11-16T18:35:19.084 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 12417, 2023-11-16T18:35:19.085 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 176, 2023-11-16T18:35:19.085 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 2, 2023-11-16T18:35:19.085 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2023-11-16T18:35:19.085 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2023-11-16T18:35:19.085 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2023-11-16T18:35:19.085 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2023-11-16T18:35:19.085 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2023-11-16T18:35:19.085 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2023-11-16T18:35:19.086 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 9944, 2023-11-16T18:35:19.086 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 192, 2023-11-16T18:35:19.086 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T18:35:19.086 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2023-11-16T18:35:19.086 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2023-11-16T18:35:19.086 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2023-11-16T18:35:19.086 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 37659493, 2023-11-16T18:35:19.086 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 330162, 2023-11-16T18:35:19.087 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2023-11-16T18:35:19.087 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2023-11-16T18:35:19.087 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2023-11-16T18:35:19.087 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2023-11-16T18:35:19.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.087 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2023-11-16T18:35:19.087 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2023-11-16T18:35:19.087 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2023-11-16T18:35:19.088 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 92792, 2023-11-16T18:35:19.088 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 404410259, 2023-11-16T18:35:19.088 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 31, 2023-11-16T18:35:19.088 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 92792, 2023-11-16T18:35:19.088 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2023-11-16T18:35:19.088 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 92792, 2023-11-16T18:35:19.088 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 27998.170556991, 2023-11-16T18:35:19.088 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.301730435 2023-11-16T18:35:19.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.089 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2023-11-16T18:35:19.089 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2023-11-16T18:35:19.089 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 92792, 2023-11-16T18:35:19.089 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 170592 2023-11-16T18:35:19.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.089 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 92792, 2023-11-16T18:35:19.089 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 12928, 2023-11-16T18:35:19.089 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 79864, 2023-11-16T18:35:19.090 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2023-11-16T18:35:19.090 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2023-11-16T18:35:19.090 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 13901, 2023-11-16T18:35:19.090 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 37346, 2023-11-16T18:35:19.090 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 270, 2023-11-16T18:35:19.090 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 4108, 2023-11-16T18:35:19.090 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 249, 2023-11-16T18:35:19.090 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2023-11-16T18:35:19.091 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2023-11-16T18:35:19.091 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2023-11-16T18:35:19.091 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 5, 2023-11-16T18:35:19.091 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 22857, 2023-11-16T18:35:19.091 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2023-11-16T18:35:19.091 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2023-11-16T18:35:19.091 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2023-11-16T18:35:19.091 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 11770, 2023-11-16T18:35:19.092 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 56536, 2023-11-16T18:35:19.092 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2023-11-16T18:35:19.092 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2023-11-16T18:35:19.092 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2023-11-16T18:35:19.092 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2023-11-16T18:35:19.092 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2023-11-16T18:35:19.092 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2023-11-16T18:35:19.092 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T18:35:19.093 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2023-11-16T18:35:19.093 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2023-11-16T18:35:19.093 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 1, 2023-11-16T18:35:19.093 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2023-11-16T18:35:19.093 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2023-11-16T18:35:19.093 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2023-11-16T18:35:19.093 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2023-11-16T18:35:19.093 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2023-11-16T18:35:19.094 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2023-11-16T18:35:19.094 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2023-11-16T18:35:19.094 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2023-11-16T18:35:19.094 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2023-11-16T18:35:19.094 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2023-11-16T18:35:19.094 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2023-11-16T18:35:19.094 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2023-11-16T18:35:19.094 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2023-11-16T18:35:19.095 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2023-11-16T18:35:19.095 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2023-11-16T18:35:19.095 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2023-11-16T18:35:19.095 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 101, 2023-11-16T18:35:19.095 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2023-11-16T18:35:19.095 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 20, 2023-11-16T18:35:19.095 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 12, 2023-11-16T18:35:19.095 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 12, 2023-11-16T18:35:19.096 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2023-11-16T18:35:19.096 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2023-11-16T18:35:19.096 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 20779, 2023-11-16T18:35:19.096 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 1786, 2023-11-16T18:35:19.096 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 984 2023-11-16T18:35:19.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.096 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2023-11-16T18:35:19.096 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 1, 2023-11-16T18:35:19.097 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2023-11-16T18:35:19.097 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2023-11-16T18:35:19.097 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2023-11-16T18:35:19.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.097 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2023-11-16T18:35:19.097 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 49086 2023-11-16T18:35:19.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.097 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2023-11-16T18:35:19.098 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2023-11-16T18:35:19.098 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 16, 2023-11-16T18:35:19.098 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2023-11-16T18:35:19.098 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 16, 2023-11-16T18:35:19.098 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 10908, 2023-11-16T18:35:19.098 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2023-11-16T18:35:19.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.098 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T18:35:19.099 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:19.099 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2023-11-16T18:35:19.099 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:19.099 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 244514, 2023-11-16T18:35:19.099 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 220938263, 2023-11-16T18:35:19.099 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:19.099 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 244514, 2023-11-16T18:35:19.099 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2023-11-16T18:35:19.100 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2023-11-16T18:35:19.100 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 244514, 2023-11-16T18:35:19.100 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 220938263, 2023-11-16T18:35:19.100 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:19.100 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.100 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.100 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.100 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:19.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.101 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2023-11-16T18:35:19.101 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:19.101 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2023-11-16T18:35:19.101 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:19.101 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2023-11-16T18:35:19.101 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2023-11-16T18:35:19.101 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:19.102 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:19.102 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 92792, 2023-11-16T18:35:19.102 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 405790579, 2023-11-16T18:35:19.102 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 92792, 2023-11-16T18:35:19.102 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 405790579, 2023-11-16T18:35:19.102 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:19.102 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.103 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.103 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.103 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:19.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.103 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2023-11-16T18:35:19.103 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:19.103 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2023-11-16T18:35:19.103 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:19.104 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2023-11-16T18:35:19.104 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2023-11-16T18:35:19.104 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:19.104 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:19.104 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 92792, 2023-11-16T18:35:19.104 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 92792, 2023-11-16T18:35:19.104 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 92792, 2023-11-16T18:35:19.104 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 92792, 2023-11-16T18:35:19.105 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:19.105 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.105 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.105 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.105 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:19.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.105 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2023-11-16T18:35:19.106 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:19.106 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2023-11-16T18:35:19.106 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:19.106 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 10908, 2023-11-16T18:35:19.106 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 1394596, 2023-11-16T18:35:19.106 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:19.106 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 10908, 2023-11-16T18:35:19.106 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2023-11-16T18:35:19.107 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2023-11-16T18:35:19.107 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 264, 2023-11-16T18:35:19.107 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1394596, 2023-11-16T18:35:19.107 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:19.107 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.107 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.107 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.107 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:19.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:19.108 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x556fc04d9cc0": { 2023-11-16T18:35:19.108 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:19.108 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2023-11-16T18:35:19.108 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:19.108 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 63440, 2023-11-16T18:35:19.108 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 327356926, 2023-11-16T18:35:19.108 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:19.109 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 63440, 2023-11-16T18:35:19.109 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2023-11-16T18:35:19.109 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2023-11-16T18:35:19.109 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 3844, 2023-11-16T18:35:19.109 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 327356926, 2023-11-16T18:35:19.109 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:19.109 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:19.109 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:19.109 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:19.110 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:19.110 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:19.110 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-11-16T18:35:19.743 INFO:tasks.check_counter:Daemon mds.b mds.exported=1133 2023-11-16T18:35:19.743 INFO:tasks.check_counter:Daemon mds.b mds.imported=1321 2023-11-16T18:35:19.743 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=978 2023-11-16T18:35:19.744 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2023-11-16T18:35:19.744 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2023-11-16T18:35:19.744 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2023-11-16T18:35:19.744 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2023-11-16T18:35:19.744 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=142 2023-11-16T18:35:19.744 DEBUG:tasks.check_counter:Getting stats from e 2023-11-16T18:35:19.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.e perf dump 2023-11-16T18:35:20.236 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:20 smithi040 ceph-mon[100251]: pgmap v327: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 171 B/s wr, 9 op/s 2023-11-16T18:35:20.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:20 smithi188 ceph-mon[100639]: pgmap v327: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 171 B/s wr, 9 op/s 2023-11-16T18:35:20.607 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:20 smithi002 ceph-mon[98807]: pgmap v327: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 171 B/s wr, 9 op/s 2023-11-16T18:35:21.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:21 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:21.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:21 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:21.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:21 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:21.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:21 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:21.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:21 smithi188 ceph-mon[100639]: pgmap v328: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:35:21.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:21 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:21.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:21 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:21.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:21 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:21.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:21 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:21.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:21 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:21.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:21 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:21.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:21 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:21.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:21 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:21.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:21 smithi002 ceph-mon[98807]: pgmap v328: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:35:21.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:21 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:21.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:21 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:21.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:21 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:21.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:21 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:21.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:21 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:21.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:21 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:21.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:21 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:21.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:21 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:21.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:21 smithi040 ceph-mon[100251]: pgmap v328: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T18:35:21.986 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:21 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:21.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:21 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:21.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:21 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:21.987 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:21 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:22.983 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:22 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi188 to 196.2M 2023-11-16T18:35:22.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:22 smithi040 ceph-mon[100251]: Unable to set osd_memory_target on smithi188 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T18:35:22.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:22 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:22.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:22 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:22.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:22 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:22.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:22 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:22.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:22 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi002 to 197.2M 2023-11-16T18:35:22.985 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:22 smithi040 ceph-mon[100251]: Unable to set osd_memory_target on smithi002 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-11-16T18:35:23.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:22 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi188 to 196.2M 2023-11-16T18:35:23.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:22 smithi188 ceph-mon[100639]: Unable to set osd_memory_target on smithi188 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T18:35:23.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:22 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:23.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:22 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:23.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:22 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:23.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:22 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:23.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:22 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi002 to 197.2M 2023-11-16T18:35:23.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:22 smithi188 ceph-mon[100639]: Unable to set osd_memory_target on smithi002 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-11-16T18:35:23.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:22 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi188 to 196.2M 2023-11-16T18:35:23.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:22 smithi002 ceph-mon[98807]: Unable to set osd_memory_target on smithi188 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T18:35:23.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:22 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:23.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:22 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:23.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:22 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:23.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:22 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:23.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:22 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi002 to 197.2M 2023-11-16T18:35:23.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:22 smithi002 ceph-mon[98807]: Unable to set osd_memory_target on smithi002 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-11-16T18:35:23.984 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:23 smithi040 ceph-mon[100251]: pgmap v329: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:35:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T18:35:31.947 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:26 smithi040 ceph-mon[100251]: pgmap v330: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-11-16T18:35:32.002 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:27 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:32.002 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:27 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:32.002 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:27 smithi040 ceph-mon[100251]: pgmap v331: 129 pgs: 60 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 50 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-11-16T18:35:32.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:23 smithi002 ceph-mon[98807]: pgmap v329: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:35:32.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:26 smithi002 ceph-mon[98807]: pgmap v330: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-11-16T18:35:32.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:27 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:32.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:27 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:32.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:27 smithi002 ceph-mon[98807]: pgmap v331: 129 pgs: 60 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 50 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-11-16T18:35:32.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:28 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:32.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:28 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:32.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:28 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:32.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:28 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:32.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:28 smithi002 ceph-mon[98807]: Adjusting osd_memory_target on smithi040 to 197.2M 2023-11-16T18:35:32.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:28 smithi002 ceph-mon[98807]: Unable to set osd_memory_target on smithi040 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-11-16T18:35:32.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:28 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:35:32.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:28 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:35:32.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:28 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:35:32.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:28 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:32.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:28 smithi002 ceph-mon[98807]: Service mds.all has no related daemon already placed. 2023-11-16T18:35:32.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:28 smithi002 ceph-mon[98807]: Service mgr has no related daemon already placed. 2023-11-16T18:35:32.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:28 smithi002 ceph-mon[98807]: Service mon has no related daemon already placed. 2023-11-16T18:35:32.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:29 smithi002 ceph-mon[98807]: pgmap v332: 129 pgs: 48 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 66 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:35:32.112 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:23 smithi188 ceph-mon[100639]: pgmap v329: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-11-16T18:35:32.112 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:26 smithi188 ceph-mon[100639]: pgmap v330: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-11-16T18:35:32.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:27 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:32.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:27 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:32.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:27 smithi188 ceph-mon[100639]: pgmap v331: 129 pgs: 60 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 50 active+clean; 719 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-11-16T18:35:32.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:28 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:32.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:28 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:32.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:28 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:32.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:28 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:32.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:28 smithi188 ceph-mon[100639]: Adjusting osd_memory_target on smithi040 to 197.2M 2023-11-16T18:35:32.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:28 smithi188 ceph-mon[100639]: Unable to set osd_memory_target on smithi040 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-11-16T18:35:32.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:28 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:35:32.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:28 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:35:32.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:28 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:35:32.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:28 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:32.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:28 smithi188 ceph-mon[100639]: Service mds.all has no related daemon already placed. 2023-11-16T18:35:32.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:28 smithi188 ceph-mon[100639]: Service mgr has no related daemon already placed. 2023-11-16T18:35:32.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:28 smithi188 ceph-mon[100639]: Service mon has no related daemon already placed. 2023-11-16T18:35:32.115 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:29 smithi188 ceph-mon[100639]: pgmap v332: 129 pgs: 48 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 66 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:35:32.115 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-11-16T18:35:32.115 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T18:35:32.115 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 868, 2023-11-16T18:35:32.116 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 870, 2023-11-16T18:35:32.116 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 113480340, 2023-11-16T18:35:32.139 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 270012, 2023-11-16T18:35:32.139 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 8, 2023-11-16T18:35:32.139 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2023-11-16T18:35:32.139 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.136627010, 2023-11-16T18:35:32.140 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.026683998, 2023-11-16T18:35:32.140 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.074518192, 2023-11-16T18:35:32.140 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.026970846, 2023-11-16T18:35:32.140 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:32.140 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 868, 2023-11-16T18:35:32.140 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.058337397, 2023-11-16T18:35:32.140 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000067208 2023-11-16T18:35:32.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.141 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:32.141 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:32.141 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:32.141 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:32.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.141 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T18:35:32.141 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T18:35:32.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.142 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T18:35:32.142 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 570, 2023-11-16T18:35:32.142 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 729, 2023-11-16T18:35:32.142 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 126011133, 2023-11-16T18:35:32.142 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 602429, 2023-11-16T18:35:32.142 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 9, 2023-11-16T18:35:32.142 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 7, 2023-11-16T18:35:32.142 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.131785669, 2023-11-16T18:35:32.143 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.016886359, 2023-11-16T18:35:32.143 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.090463950, 2023-11-16T18:35:32.143 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.017375240, 2023-11-16T18:35:32.143 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:32.168 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 726, 2023-11-16T18:35:32.168 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.067711907, 2023-11-16T18:35:32.168 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000093267 2023-11-16T18:35:32.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.168 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:32.168 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:32.168 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:32.169 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:32.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.169 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T18:35:32.169 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 425776 2023-11-16T18:35:32.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.169 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T18:35:32.169 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1498, 2023-11-16T18:35:32.170 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 1473, 2023-11-16T18:35:32.170 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 120174850, 2023-11-16T18:35:32.170 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 509615, 2023-11-16T18:35:32.170 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2023-11-16T18:35:32.170 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2023-11-16T18:35:32.170 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.229973948, 2023-11-16T18:35:32.170 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.046733483, 2023-11-16T18:35:32.170 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.129056487, 2023-11-16T18:35:32.171 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.051363161, 2023-11-16T18:35:32.171 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:32.171 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1473, 2023-11-16T18:35:32.171 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.183767863, 2023-11-16T18:35:32.171 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000124757 2023-11-16T18:35:32.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.171 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:32.171 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:32.172 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:32.172 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:32.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.172 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1505056, 2023-11-16T18:35:32.172 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 123664 2023-11-16T18:35:32.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.173 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2023-11-16T18:35:32.198 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2023-11-16T18:35:32.198 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2023-11-16T18:35:32.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.198 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2023-11-16T18:35:32.198 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2023-11-16T18:35:32.199 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2023-11-16T18:35:32.199 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 4241, 2023-11-16T18:35:32.199 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.129285556, 2023-11-16T18:35:32.199 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000030484 2023-11-16T18:35:32.199 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:32.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.201 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2023-11-16T18:35:32.201 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2023-11-16T18:35:32.201 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2023-11-16T18:35:32.201 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:32.202 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:32.202 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:32.202 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:32.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.202 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2023-11-16T18:35:32.202 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 0, 2023-11-16T18:35:32.202 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 0, 2023-11-16T18:35:32.203 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2023-11-16T18:35:32.203 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:32.203 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:32.203 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:32.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.203 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2023-11-16T18:35:32.203 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2023-11-16T18:35:32.203 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 0, 2023-11-16T18:35:32.204 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2023-11-16T18:35:32.204 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 0, 2023-11-16T18:35:32.204 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 0, 2023-11-16T18:35:32.204 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2023-11-16T18:35:32.204 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 16978, 2023-11-16T18:35:32.204 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 0, 2023-11-16T18:35:32.240 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 0, 2023-11-16T18:35:32.240 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 16978, 2023-11-16T18:35:32.241 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 16978, 2023-11-16T18:35:32.241 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 0, 2023-11-16T18:35:32.241 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 0, 2023-11-16T18:35:32.241 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 0, 2023-11-16T18:35:32.241 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 1, 2023-11-16T18:35:32.241 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 0, 2023-11-16T18:35:32.241 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 0, 2023-11-16T18:35:32.241 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2023-11-16T18:35:32.242 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 0, 2023-11-16T18:35:32.242 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2023-11-16T18:35:32.242 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2023-11-16T18:35:32.242 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2023-11-16T18:35:32.242 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2023-11-16T18:35:32.242 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2023-11-16T18:35:32.242 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 0, 2023-11-16T18:35:32.242 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 0, 2023-11-16T18:35:32.243 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 0, 2023-11-16T18:35:32.243 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 0, 2023-11-16T18:35:32.243 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2023-11-16T18:35:32.243 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2023-11-16T18:35:32.243 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2023-11-16T18:35:32.243 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 0, 2023-11-16T18:35:32.243 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 0, 2023-11-16T18:35:32.243 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 0, 2023-11-16T18:35:32.244 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2023-11-16T18:35:32.244 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2023-11-16T18:35:32.244 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T18:35:32.244 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2023-11-16T18:35:32.244 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2023-11-16T18:35:32.244 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2023-11-16T18:35:32.244 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2023-11-16T18:35:32.244 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2023-11-16T18:35:32.245 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2023-11-16T18:35:32.276 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 0, 2023-11-16T18:35:32.276 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 0, 2023-11-16T18:35:32.276 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2023-11-16T18:35:32.276 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-16T18:35:32.276 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T18:35:32.276 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 0, 2023-11-16T18:35:32.277 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 0, 2023-11-16T18:35:32.277 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 0, 2023-11-16T18:35:32.277 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 0 2023-11-16T18:35:32.277 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.277 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2023-11-16T18:35:32.277 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2023-11-16T18:35:32.277 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2023-11-16T18:35:32.277 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2023-11-16T18:35:32.278 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 0, 2023-11-16T18:35:32.278 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 0, 2023-11-16T18:35:32.278 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2023-11-16T18:35:32.278 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2023-11-16T18:35:32.278 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2023-11-16T18:35:32.629 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 0, 2023-11-16T18:35:32.629 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 0, 2023-11-16T18:35:32.630 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2023-11-16T18:35:32.630 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2023-11-16T18:35:32.630 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2023-11-16T18:35:32.630 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2023-11-16T18:35:32.630 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2023-11-16T18:35:32.630 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2023-11-16T18:35:32.630 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2023-11-16T18:35:32.631 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2023-11-16T18:35:32.631 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 0, 2023-11-16T18:35:32.631 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2023-11-16T18:35:32.631 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 0, 2023-11-16T18:35:32.631 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2023-11-16T18:35:32.631 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2023-11-16T18:35:32.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.631 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2023-11-16T18:35:32.632 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 0, 2023-11-16T18:35:32.632 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2023-11-16T18:35:32.632 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2023-11-16T18:35:32.632 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 0, 2023-11-16T18:35:32.632 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2023-11-16T18:35:32.632 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2023-11-16T18:35:32.632 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 0, 2023-11-16T18:35:32.979 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2023-11-16T18:35:32.979 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2023-11-16T18:35:32.979 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 128, 2023-11-16T18:35:32.979 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2023-11-16T18:35:32.980 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2023-11-16T18:35:32.980 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 29362240, 2023-11-16T18:35:32.980 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 0, 2023-11-16T18:35:32.980 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 356537114, 2023-11-16T18:35:32.980 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2023-11-16T18:35:32.980 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:32.980 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:32.980 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:32.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:32.981 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 90545 2023-11-16T18:35:32.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.079 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2023-11-16T18:35:33.079 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 13, 2023-11-16T18:35:33.079 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 78087, 2023-11-16T18:35:33.079 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 78074, 2023-11-16T18:35:33.079 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 11, 2023-11-16T18:35:33.079 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 2738, 2023-11-16T18:35:33.080 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 2727, 2023-11-16T18:35:33.080 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 16978, 2023-11-16T18:35:33.080 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 126982, 2023-11-16T18:35:33.080 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 110004, 2023-11-16T18:35:33.080 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 0, 2023-11-16T18:35:33.080 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 0, 2023-11-16T18:35:33.080 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 0, 2023-11-16T18:35:33.080 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 120792, 2023-11-16T18:35:33.081 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 207156 2023-11-16T18:35:33.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.081 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2023-11-16T18:35:33.081 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 0, 2023-11-16T18:35:33.081 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2023-11-16T18:35:33.081 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 0, 2023-11-16T18:35:33.081 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 0, 2023-11-16T18:35:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2023-11-16T18:35:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2023-11-16T18:35:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2023-11-16T18:35:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2023-11-16T18:35:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2023-11-16T18:35:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2023-11-16T18:35:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2023-11-16T18:35:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2023-11-16T18:35:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2023-11-16T18:35:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2023-11-16T18:35:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.470 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.471 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2023-11-16T18:35:33.471 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.471 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.471 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.471 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2023-11-16T18:35:33.471 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.471 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.472 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.472 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2023-11-16T18:35:33.472 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.472 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.472 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.472 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2023-11-16T18:35:33.473 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.473 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.473 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.473 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2023-11-16T18:35:33.487 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.488 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.488 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.488 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2023-11-16T18:35:33.488 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.488 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.488 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.488 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2023-11-16T18:35:33.489 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.489 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.489 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.489 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2023-11-16T18:35:33.489 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.490 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.490 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.490 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2023-11-16T18:35:33.490 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.490 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.490 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.491 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2023-11-16T18:35:33.491 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.491 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.491 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.491 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2023-11-16T18:35:33.503 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.503 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.503 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.503 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2023-11-16T18:35:33.503 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.503 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.504 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.504 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2023-11-16T18:35:33.504 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.504 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.504 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.504 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2023-11-16T18:35:33.505 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.505 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.505 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.505 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2023-11-16T18:35:33.505 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.505 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.505 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.506 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2023-11-16T18:35:33.506 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.507 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.507 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.507 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2023-11-16T18:35:33.507 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.536 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.536 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.536 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2023-11-16T18:35:33.536 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.536 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.536 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.537 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2023-11-16T18:35:33.537 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.537 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.537 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.537 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2023-11-16T18:35:33.537 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.538 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.538 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.540 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2023-11-16T18:35:33.553 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2023-11-16T18:35:33.553 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2023-11-16T18:35:33.553 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.553 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.553 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.553 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:33.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.554 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2023-11-16T18:35:33.554 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 2, 2023-11-16T18:35:33.554 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 3, 2023-11-16T18:35:33.554 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 1, 2023-11-16T18:35:33.554 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 2, 2023-11-16T18:35:33.554 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2023-11-16T18:35:33.554 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 0, 2023-11-16T18:35:33.555 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 0, 2023-11-16T18:35:33.555 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 2161 2023-11-16T18:35:33.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.555 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2023-11-16T18:35:33.555 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2023-11-16T18:35:33.555 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2023-11-16T18:35:33.555 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T18:35:33.555 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2023-11-16T18:35:33.556 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T18:35:33.556 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2023-11-16T18:35:33.556 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T18:35:33.556 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T18:35:33.556 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T18:35:33.556 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T18:35:33.556 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T18:35:33.556 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2023-11-16T18:35:33.556 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T18:35:33.557 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T18:35:33.557 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2023-11-16T18:35:33.557 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2023-11-16T18:35:33.557 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2023-11-16T18:35:33.557 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2023-11-16T18:35:33.557 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T18:35:33.557 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T18:35:33.557 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T18:35:33.558 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T18:35:33.558 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T18:35:33.558 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2023-11-16T18:35:33.558 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2023-11-16T18:35:33.558 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2023-11-16T18:35:33.558 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T18:35:33.558 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T18:35:33.559 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2023-11-16T18:35:33.559 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2023-11-16T18:35:33.559 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2023-11-16T18:35:33.559 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2023-11-16T18:35:33.559 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T18:35:33.559 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2023-11-16T18:35:33.559 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T18:35:33.559 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2023-11-16T18:35:33.559 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 239458, 2023-11-16T18:35:33.560 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 80, 2023-11-16T18:35:33.560 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2023-11-16T18:35:33.560 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2023-11-16T18:35:33.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2023-11-16T18:35:33.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2023-11-16T18:35:33.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2023-11-16T18:35:33.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2023-11-16T18:35:33.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2023-11-16T18:35:33.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2023-11-16T18:35:33.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 9944, 2023-11-16T18:35:33.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 192, 2023-11-16T18:35:33.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T18:35:33.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2023-11-16T18:35:33.561 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2023-11-16T18:35:33.561 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2023-11-16T18:35:33.561 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 12025344, 2023-11-16T18:35:33.561 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 68090, 2023-11-16T18:35:33.562 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2023-11-16T18:35:33.562 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2023-11-16T18:35:33.562 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2023-11-16T18:35:33.562 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2023-11-16T18:35:33.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.562 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2023-11-16T18:35:33.562 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2023-11-16T18:35:33.562 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2023-11-16T18:35:33.563 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 2677, 2023-11-16T18:35:33.563 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 21, 2023-11-16T18:35:33.563 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 3, 2023-11-16T18:35:33.563 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 2677, 2023-11-16T18:35:33.563 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2023-11-16T18:35:33.563 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2677, 2023-11-16T18:35:33.563 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 5.178931184, 2023-11-16T18:35:33.563 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001934602 2023-11-16T18:35:33.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.564 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2023-11-16T18:35:33.564 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2023-11-16T18:35:33.564 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2677, 2023-11-16T18:35:33.564 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 2678 2023-11-16T18:35:33.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.564 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 2677, 2023-11-16T18:35:33.564 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 2677, 2023-11-16T18:35:33.564 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 0, 2023-11-16T18:35:33.565 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2023-11-16T18:35:33.565 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2023-11-16T18:35:33.565 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 1674, 2023-11-16T18:35:33.565 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 0, 2023-11-16T18:35:33.565 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 1001, 2023-11-16T18:35:33.565 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 0, 2023-11-16T18:35:33.565 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 0, 2023-11-16T18:35:33.565 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2023-11-16T18:35:33.566 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2023-11-16T18:35:33.566 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2023-11-16T18:35:33.566 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2023-11-16T18:35:33.566 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 0, 2023-11-16T18:35:33.566 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2023-11-16T18:35:33.566 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2023-11-16T18:35:33.566 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2023-11-16T18:35:33.566 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 1, 2023-11-16T18:35:33.567 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 0, 2023-11-16T18:35:33.567 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2023-11-16T18:35:33.567 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2023-11-16T18:35:33.567 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2023-11-16T18:35:33.567 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2023-11-16T18:35:33.567 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2023-11-16T18:35:33.567 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2023-11-16T18:35:33.567 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T18:35:33.567 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2023-11-16T18:35:33.568 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2023-11-16T18:35:33.568 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2023-11-16T18:35:33.568 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2023-11-16T18:35:33.568 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2023-11-16T18:35:33.568 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2023-11-16T18:35:33.568 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2023-11-16T18:35:33.568 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2023-11-16T18:35:33.568 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2023-11-16T18:35:33.569 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2023-11-16T18:35:33.569 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2023-11-16T18:35:33.569 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2023-11-16T18:35:33.569 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2023-11-16T18:35:33.569 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2023-11-16T18:35:33.569 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2023-11-16T18:35:33.569 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2023-11-16T18:35:33.569 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2023-11-16T18:35:33.570 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2023-11-16T18:35:33.570 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2023-11-16T18:35:33.570 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 101, 2023-11-16T18:35:33.570 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2023-11-16T18:35:33.570 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 15, 2023-11-16T18:35:33.570 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 11, 2023-11-16T18:35:33.570 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 11, 2023-11-16T18:35:33.570 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2023-11-16T18:35:33.571 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2023-11-16T18:35:33.571 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 0, 2023-11-16T18:35:33.571 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 2, 2023-11-16T18:35:33.571 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 0 2023-11-16T18:35:33.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.571 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2023-11-16T18:35:33.571 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 0, 2023-11-16T18:35:33.571 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2023-11-16T18:35:33.572 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2023-11-16T18:35:33.572 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2023-11-16T18:35:33.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.572 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2023-11-16T18:35:33.572 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2023-11-16T18:35:33.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.572 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2023-11-16T18:35:33.572 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2023-11-16T18:35:33.572 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 0, 2023-11-16T18:35:33.573 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2023-11-16T18:35:33.573 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 0, 2023-11-16T18:35:33.573 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 0, 2023-11-16T18:35:33.573 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2023-11-16T18:35:33.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.573 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T18:35:33.573 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:33.573 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2023-11-16T18:35:33.574 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:33.574 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 2932, 2023-11-16T18:35:33.574 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 359386541, 2023-11-16T18:35:33.574 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:33.574 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 2932, 2023-11-16T18:35:33.574 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2023-11-16T18:35:33.574 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2023-11-16T18:35:33.574 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 2932, 2023-11-16T18:35:33.575 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 359386541, 2023-11-16T18:35:33.575 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:33.575 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.575 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.575 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.575 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:33.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.575 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2023-11-16T18:35:33.575 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:33.576 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2023-11-16T18:35:33.576 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:33.576 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2023-11-16T18:35:33.576 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2023-11-16T18:35:33.576 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:33.576 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:33.576 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 2677, 2023-11-16T18:35:33.576 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 357415493, 2023-11-16T18:35:33.577 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 2677, 2023-11-16T18:35:33.577 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 357415493, 2023-11-16T18:35:33.577 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:33.577 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.577 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.577 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.577 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:33.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.578 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2023-11-16T18:35:33.578 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:33.578 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2023-11-16T18:35:33.578 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:33.578 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2023-11-16T18:35:33.578 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2023-11-16T18:35:33.583 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:33.583 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:33.583 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 2677, 2023-11-16T18:35:33.583 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 2677, 2023-11-16T18:35:33.583 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 2677, 2023-11-16T18:35:33.583 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 2677, 2023-11-16T18:35:33.633 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:33.634 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.634 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.634 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.661 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:33.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.661 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2023-11-16T18:35:33.661 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:33.662 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2023-11-16T18:35:33.662 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:33.662 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2023-11-16T18:35:33.662 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2023-11-16T18:35:33.662 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:33.662 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:33.662 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2023-11-16T18:35:33.662 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2023-11-16T18:35:33.663 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2023-11-16T18:35:33.663 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2023-11-16T18:35:33.663 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:33.663 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.663 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.663 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.663 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:33.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:33.664 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x55f144728000": { 2023-11-16T18:35:33.664 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:33.664 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2023-11-16T18:35:33.664 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:33.664 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2023-11-16T18:35:33.664 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2023-11-16T18:35:33.664 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:33.664 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:33.665 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2023-11-16T18:35:33.665 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2023-11-16T18:35:33.665 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2023-11-16T18:35:33.672 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2023-11-16T18:35:33.672 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:33.672 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:33.672 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:33.672 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:33.673 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:33.673 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:33.673 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-11-16T18:35:33.673 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:28 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:33.674 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:28 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:33.674 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:28 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:33.674 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:28 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T18:35:33.674 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:28 smithi040 ceph-mon[100251]: Adjusting osd_memory_target on smithi040 to 197.2M 2023-11-16T18:35:33.674 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:28 smithi040 ceph-mon[100251]: Unable to set osd_memory_target on smithi040 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-11-16T18:35:33.674 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:28 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T18:35:33.674 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:28 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T18:35:33.674 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:28 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:35:33.675 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:28 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' 2023-11-16T18:35:33.675 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:28 smithi040 ceph-mon[100251]: Service mds.all has no related daemon already placed. 2023-11-16T18:35:33.675 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:28 smithi040 ceph-mon[100251]: Service mgr has no related daemon already placed. 2023-11-16T18:35:33.675 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:28 smithi040 ceph-mon[100251]: Service mon has no related daemon already placed. 2023-11-16T18:35:33.677 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2023-11-16T18:35:33.677 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2023-11-16T18:35:33.677 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2023-11-16T18:35:33.677 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2023-11-16T18:35:33.677 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2023-11-16T18:35:33.677 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2023-11-16T18:35:33.677 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2023-11-16T18:35:33.678 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2023-11-16T18:35:33.678 DEBUG:tasks.check_counter:Getting stats from h 2023-11-16T18:35:33.678 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.h perf dump 2023-11-16T18:35:33.678 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:32 smithi002 ceph-mon[98807]: pgmap v333: 129 pgs: 48 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 66 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:35:33.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:32 smithi188 ceph-mon[100639]: pgmap v333: 129 pgs: 48 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 66 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:35:33.679 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:29 smithi040 ceph-mon[100251]: pgmap v332: 129 pgs: 48 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 66 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:35:33.679 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:32 smithi040 ceph-mon[100251]: pgmap v333: 129 pgs: 48 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 66 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:35:34.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:34 smithi188 ceph-mon[100639]: pgmap v334: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:35:34.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:34 smithi002 ceph-mon[98807]: pgmap v334: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:35:34.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:34 smithi040 ceph-mon[100251]: pgmap v334: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:35:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T18:35:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-11-16T18:35:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T18:35:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 19539, 2023-11-16T18:35:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 19671, 2023-11-16T18:35:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 6297515, 2023-11-16T18:35:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 111505741, 2023-11-16T18:35:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 23, 2023-11-16T18:35:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2023-11-16T18:35:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 1.649692075, 2023-11-16T18:35:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.547692090, 2023-11-16T18:35:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.313640614, 2023-11-16T18:35:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.700216921, 2023-11-16T18:35:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 19668, 2023-11-16T18:35:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 8.086185577, 2023-11-16T18:35:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000411134 2023-11-16T18:35:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 21, 2023-11-16T18:35:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000003101, 2023-11-16T18:35:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000147 2023-11-16T18:35:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1696, 2023-11-16T18:35:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 311408 2023-11-16T18:35:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T18:35:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 195658, 2023-11-16T18:35:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 255549, 2023-11-16T18:35:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 196907027, 2023-11-16T18:35:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 280491445, 2023-11-16T18:35:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 11, 2023-11-16T18:35:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 8, 2023-11-16T18:35:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 10.987231900, 2023-11-16T18:35:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 4.337991361, 2023-11-16T18:35:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 85.658840733, 2023-11-16T18:35:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.670205887, 2023-11-16T18:35:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 255548, 2023-11-16T18:35:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 26.547443408, 2023-11-16T18:35:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000103884 2023-11-16T18:35:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 393230, 2023-11-16T18:35:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.067379207, 2023-11-16T18:35:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000171 2023-11-16T18:35:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 3152, 2023-11-16T18:35:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 119600 2023-11-16T18:35:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T18:35:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 64491, 2023-11-16T18:35:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 76494, 2023-11-16T18:35:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 45417334, 2023-11-16T18:35:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 103186039, 2023-11-16T18:35:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 12, 2023-11-16T18:35:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2023-11-16T18:35:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 4.756703876, 2023-11-16T18:35:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 2.084348977, 2023-11-16T18:35:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 2.735939971, 2023-11-16T18:35:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.401376688, 2023-11-16T18:35:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 76493, 2023-11-16T18:35:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 9.132626933, 2023-11-16T18:35:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000119391 2023-11-16T18:35:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 115010, 2023-11-16T18:35:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.020971101, 2023-11-16T18:35:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000182 2023-11-16T18:35:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1521360, 2023-11-16T18:35:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 527152 2023-11-16T18:35:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2023-11-16T18:35:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2023-11-16T18:35:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2023-11-16T18:35:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2023-11-16T18:35:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2023-11-16T18:35:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2023-11-16T18:35:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 9110, 2023-11-16T18:35:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 101.990441252, 2023-11-16T18:35:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.011195438 2023-11-16T18:35:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2023-11-16T18:35:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2023-11-16T18:35:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2023-11-16T18:35:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 25772, 2023-11-16T18:35:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 22.701615228, 2023-11-16T18:35:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000880863 2023-11-16T18:35:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2023-11-16T18:35:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 48245, 2023-11-16T18:35:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 42027, 2023-11-16T18:35:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2023-11-16T18:35:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 42027, 2023-11-16T18:35:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 303.655751619, 2023-11-16T18:35:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.007225254 2023-11-16T18:35:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2023-11-16T18:35:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 6218, 2023-11-16T18:35:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 162, 2023-11-16T18:35:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2023-11-16T18:35:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 707, 2023-11-16T18:35:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 87, 2023-11-16T18:35:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2023-11-16T18:35:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 20, 2023-11-16T18:35:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 2, 2023-11-16T18:35:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 8, 2023-11-16T18:35:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 10, 2023-11-16T18:35:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 15, 2023-11-16T18:35:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 108190, 2023-11-16T18:35:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 1, 2023-11-16T18:35:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 1, 2023-11-16T18:35:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 4, 2023-11-16T18:35:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 68237, 2023-11-16T18:35:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 60484, 2023-11-16T18:35:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 6113, 2023-11-16T18:35:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 11, 2023-11-16T18:35:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 159, 2023-11-16T18:35:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2023-11-16T18:35:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2023-11-16T18:35:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2023-11-16T18:35:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2023-11-16T18:35:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 2604, 2023-11-16T18:35:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 102337, 2023-11-16T18:35:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 1995, 2023-11-16T18:35:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 0, 2023-11-16T18:35:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2023-11-16T18:35:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 3092, 2023-11-16T18:35:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2023-11-16T18:35:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 6, 2023-11-16T18:35:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 246, 2023-11-16T18:35:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 6, 2023-11-16T18:35:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 3092, 2023-11-16T18:35:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2023-11-16T18:35:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T18:35:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2023-11-16T18:35:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2023-11-16T18:35:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2023-11-16T18:35:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2023-11-16T18:35:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 3092, 2023-11-16T18:35:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2023-11-16T18:35:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 60, 2023-11-16T18:35:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 24817, 2023-11-16T18:35:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2023-11-16T18:35:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-16T18:35:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T18:35:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 18811, 2023-11-16T18:35:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 18751, 2023-11-16T18:35:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 543, 2023-11-16T18:35:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 34268 2023-11-16T18:35:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2023-11-16T18:35:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 198, 2023-11-16T18:35:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 3490, 2023-11-16T18:35:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 1747, 2023-11-16T18:35:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 1757, 2023-11-16T18:35:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 198, 2023-11-16T18:35:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2023-11-16T18:35:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2023-11-16T18:35:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2023-11-16T18:35:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 16968, 2023-11-16T18:35:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 16968, 2023-11-16T18:35:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2023-11-16T18:35:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2023-11-16T18:35:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2023-11-16T18:35:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2023-11-16T18:35:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2023-11-16T18:35:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2023-11-16T18:35:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2023-11-16T18:35:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2023-11-16T18:35:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 3158, 2023-11-16T18:35:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2023-11-16T18:35:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 88, 2023-11-16T18:35:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2023-11-16T18:35:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2023-11-16T18:35:36.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.159 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2023-11-16T18:35:36.159 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 90545, 2023-11-16T18:35:36.159 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 11099, 2023-11-16T18:35:36.159 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 11099, 2023-11-16T18:35:36.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 79446, 2023-11-16T18:35:36.159 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2023-11-16T18:35:36.159 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 288, 2023-11-16T18:35:36.159 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 141, 2023-11-16T18:35:36.160 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 13, 2023-11-16T18:35:36.160 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 13, 2023-11-16T18:35:36.160 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 129, 2023-11-16T18:35:36.160 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2023-11-16T18:35:36.160 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 1, 2023-11-16T18:35:36.160 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 29362240, 2023-11-16T18:35:36.160 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 356540621, 2023-11-16T18:35:36.160 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 0, 2023-11-16T18:35:36.161 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2023-11-16T18:35:36.161 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 6370, 2023-11-16T18:35:36.161 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 544.817315778, 2023-11-16T18:35:36.161 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.085528621 2023-11-16T18:35:36.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.161 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 0 2023-11-16T18:35:36.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.161 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2023-11-16T18:35:36.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 25, 2023-11-16T18:35:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 79377, 2023-11-16T18:35:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 79352, 2023-11-16T18:35:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 23, 2023-11-16T18:35:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 3030, 2023-11-16T18:35:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 3007, 2023-11-16T18:35:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 20, 2023-11-16T18:35:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 125178, 2023-11-16T18:35:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 125158, 2023-11-16T18:35:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 1, 2023-11-16T18:35:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 37446, 2023-11-16T18:35:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 37445, 2023-11-16T18:35:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 187108, 2023-11-16T18:35:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 198964 2023-11-16T18:35:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2023-11-16T18:35:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 53439, 2023-11-16T18:35:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 1, 2023-11-16T18:35:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 48245, 2023-11-16T18:35:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 22, 2023-11-16T18:35:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 1373, 2023-11-16T18:35:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2023-11-16T18:35:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 7643, 2023-11-16T18:35:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 25.638165443, 2023-11-16T18:35:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.003354463 2023-11-16T18:35:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2023-11-16T18:35:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 23, 2023-11-16T18:35:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.052662731, 2023-11-16T18:35:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002289683 2023-11-16T18:35:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2023-11-16T18:35:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2023-11-16T18:35:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2023-11-16T18:35:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 16587, 2023-11-16T18:35:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 83.592530307, 2023-11-16T18:35:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.005039641 2023-11-16T18:35:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2023-11-16T18:35:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2023-11-16T18:35:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1, 2023-11-16T18:35:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000112537, 2023-11-16T18:35:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000112537 2023-11-16T18:35:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2023-11-16T18:35:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2023-11-16T18:35:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2023-11-16T18:35:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2023-11-16T18:35:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2023-11-16T18:35:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2023-11-16T18:35:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2023-11-16T18:35:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2023-11-16T18:35:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2023-11-16T18:35:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 805, 2023-11-16T18:35:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 3.486738285, 2023-11-16T18:35:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.004331351 2023-11-16T18:35:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2023-11-16T18:35:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2023-11-16T18:35:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2023-11-16T18:35:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2023-11-16T18:35:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2023-11-16T18:35:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2023-11-16T18:35:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2023-11-16T18:35:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2023-11-16T18:35:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2023-11-16T18:35:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2023-11-16T18:35:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2023-11-16T18:35:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2023-11-16T18:35:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2023-11-16T18:35:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 16968, 2023-11-16T18:35:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 190.885542316, 2023-11-16T18:35:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.011249737 2023-11-16T18:35:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2023-11-16T18:35:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2023-11-16T18:35:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2023-11-16T18:35:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2023-11-16T18:35:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 2, 2023-11-16T18:35:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 3, 2023-11-16T18:35:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 1, 2023-11-16T18:35:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 2, 2023-11-16T18:35:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2023-11-16T18:35:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 9845, 2023-11-16T18:35:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 4922, 2023-11-16T18:35:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 636 2023-11-16T18:35:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2023-11-16T18:35:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2023-11-16T18:35:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2023-11-16T18:35:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T18:35:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2023-11-16T18:35:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T18:35:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2023-11-16T18:35:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T18:35:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T18:35:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T18:35:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T18:35:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T18:35:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2023-11-16T18:35:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T18:35:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T18:35:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2023-11-16T18:35:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2023-11-16T18:35:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2023-11-16T18:35:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2023-11-16T18:35:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T18:35:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T18:35:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T18:35:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T18:35:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T18:35:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2023-11-16T18:35:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2023-11-16T18:35:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2023-11-16T18:35:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T18:35:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T18:35:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2023-11-16T18:35:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2023-11-16T18:35:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2023-11-16T18:35:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2023-11-16T18:35:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T18:35:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2023-11-16T18:35:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T18:35:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2023-11-16T18:35:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 303837, 2023-11-16T18:35:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 112, 2023-11-16T18:35:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2023-11-16T18:35:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2023-11-16T18:35:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2023-11-16T18:35:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2023-11-16T18:35:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2023-11-16T18:35:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2023-11-16T18:35:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2023-11-16T18:35:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2023-11-16T18:35:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 9944, 2023-11-16T18:35:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 192, 2023-11-16T18:35:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T18:35:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2023-11-16T18:35:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2023-11-16T18:35:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2023-11-16T18:35:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 143056, 2023-11-16T18:35:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 653, 2023-11-16T18:35:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2023-11-16T18:35:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2023-11-16T18:35:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2023-11-16T18:35:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2023-11-16T18:35:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2023-11-16T18:35:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2023-11-16T18:35:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2023-11-16T18:35:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 46475, 2023-11-16T18:35:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 362896382, 2023-11-16T18:35:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 28, 2023-11-16T18:35:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 46475, 2023-11-16T18:35:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2023-11-16T18:35:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 46475, 2023-11-16T18:35:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 458.565821596, 2023-11-16T18:35:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.009866935 2023-11-16T18:35:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2023-11-16T18:35:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2023-11-16T18:35:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 46475, 2023-11-16T18:35:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 48613 2023-11-16T18:35:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 46475, 2023-11-16T18:35:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 3742, 2023-11-16T18:35:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 42733, 2023-11-16T18:35:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2023-11-16T18:35:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2023-11-16T18:35:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 0, 2023-11-16T18:35:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 784, 2023-11-16T18:35:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 488, 2023-11-16T18:35:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 6858, 2023-11-16T18:35:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 198, 2023-11-16T18:35:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2023-11-16T18:35:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2023-11-16T18:35:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2023-11-16T18:35:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2023-11-16T18:35:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 34125, 2023-11-16T18:35:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2023-11-16T18:35:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2023-11-16T18:35:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2023-11-16T18:35:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 3092, 2023-11-16T18:35:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 1181, 2023-11-16T18:35:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2023-11-16T18:35:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2023-11-16T18:35:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2023-11-16T18:35:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2023-11-16T18:35:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2023-11-16T18:35:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2023-11-16T18:35:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T18:35:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2023-11-16T18:35:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2023-11-16T18:35:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2023-11-16T18:35:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2023-11-16T18:35:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2023-11-16T18:35:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2023-11-16T18:35:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2023-11-16T18:35:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2023-11-16T18:35:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2023-11-16T18:35:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2023-11-16T18:35:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2023-11-16T18:35:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2023-11-16T18:35:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2023-11-16T18:35:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2023-11-16T18:35:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2023-11-16T18:35:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2023-11-16T18:35:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2023-11-16T18:35:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2023-11-16T18:35:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2023-11-16T18:35:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 101, 2023-11-16T18:35:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2023-11-16T18:35:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 15, 2023-11-16T18:35:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 12, 2023-11-16T18:35:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 12, 2023-11-16T18:35:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2023-11-16T18:35:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2023-11-16T18:35:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 1520, 2023-11-16T18:35:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 324, 2023-11-16T18:35:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 43 2023-11-16T18:35:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2023-11-16T18:35:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 1, 2023-11-16T18:35:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2023-11-16T18:35:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2023-11-16T18:35:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2023-11-16T18:35:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2023-11-16T18:35:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 51492 2023-11-16T18:35:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2023-11-16T18:35:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2023-11-16T18:35:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 16, 2023-11-16T18:35:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2023-11-16T18:35:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 8, 2023-11-16T18:35:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 16968, 2023-11-16T18:35:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2023-11-16T18:35:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T18:35:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2023-11-16T18:35:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 279684, 2023-11-16T18:35:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 223430219, 2023-11-16T18:35:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 279684, 2023-11-16T18:35:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2023-11-16T18:35:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2023-11-16T18:35:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 279684, 2023-11-16T18:35:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 223430219, 2023-11-16T18:35:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2023-11-16T18:35:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2023-11-16T18:35:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2023-11-16T18:35:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2023-11-16T18:35:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 46475, 2023-11-16T18:35:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 365020302, 2023-11-16T18:35:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 46475, 2023-11-16T18:35:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 365020302, 2023-11-16T18:35:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2023-11-16T18:35:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2023-11-16T18:35:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2023-11-16T18:35:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2023-11-16T18:35:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 46475, 2023-11-16T18:35:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 46475, 2023-11-16T18:35:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 46475, 2023-11-16T18:35:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 46475, 2023-11-16T18:35:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2023-11-16T18:35:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2023-11-16T18:35:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 16968, 2023-11-16T18:35:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 2126512, 2023-11-16T18:35:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 16968, 2023-11-16T18:35:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2023-11-16T18:35:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2023-11-16T18:35:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 488, 2023-11-16T18:35:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 2126512, 2023-11-16T18:35:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x55d213777cc0": { 2023-11-16T18:35:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2023-11-16T18:35:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 90545, 2023-11-16T18:35:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 352346317, 2023-11-16T18:35:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 90545, 2023-11-16T18:35:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2023-11-16T18:35:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2023-11-16T18:35:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 6370, 2023-11-16T18:35:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 352346317, 2023-11-16T18:35:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:36.215 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-11-16T18:35:36.471 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:36 smithi040 ceph-mon[100251]: pgmap v335: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:35:36.539 INFO:tasks.check_counter:Daemon mds.h mds.exported=2604 2023-11-16T18:35:36.539 INFO:tasks.check_counter:Daemon mds.h mds.imported=1995 2023-11-16T18:35:36.539 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=198 2023-11-16T18:35:36.540 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=3490 2023-11-16T18:35:36.540 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2023-11-16T18:35:36.540 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2023-11-16T18:35:36.540 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2023-11-16T18:35:36.540 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=87 2023-11-16T18:35:36.540 DEBUG:tasks.check_counter:Getting stats from k 2023-11-16T18:35:36.540 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.k perf dump 2023-11-16T18:35:36.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:36 smithi188 ceph-mon[100639]: pgmap v335: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:35:36.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:36 smithi002 ceph-mon[98807]: pgmap v335: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-16T18:35:37.199 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T18:35:37.734 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:37 smithi040 ceph-mon[100251]: osdmap e102: 12 total, 12 up, 12 in 2023-11-16T18:35:37.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:37 smithi040 ceph-mon[100251]: pgmap v337: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:37.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:37 smithi188 ceph-mon[100639]: osdmap e102: 12 total, 12 up, 12 in 2023-11-16T18:35:37.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:37 smithi188 ceph-mon[100639]: pgmap v337: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:37.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:37 smithi002 ceph-mon[98807]: osdmap e102: 12 total, 12 up, 12 in 2023-11-16T18:35:37.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:37 smithi002 ceph-mon[98807]: pgmap v337: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 719 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:38.526 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-11-16T18:35:38.526 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T18:35:38.526 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 4, 2023-11-16T18:35:38.526 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 4, 2023-11-16T18:35:38.526 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 2992, 2023-11-16T18:35:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 448, 2023-11-16T18:35:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2023-11-16T18:35:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 0, 2023-11-16T18:35:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.001682022, 2023-11-16T18:35:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000050001, 2023-11-16T18:35:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.000518284, 2023-11-16T18:35:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T18:35:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3, 2023-11-16T18:35:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000735581, 2023-11-16T18:35:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000245193 2023-11-16T18:35:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:38.550 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:38.550 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-11-16T18:35:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 448 2023-11-16T18:35:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T18:35:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 224, 2023-11-16T18:35:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 214, 2023-11-16T18:35:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 1422256, 2023-11-16T18:35:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 122112, 2023-11-16T18:35:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 2, 2023-11-16T18:35:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2023-11-16T18:35:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.043462854, 2023-11-16T18:35:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.012114814, 2023-11-16T18:35:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.033512834, 2023-11-16T18:35:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.006593215, 2023-11-16T18:35:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 213, 2023-11-16T18:35:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.016612640, 2023-11-16T18:35:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000077993 2023-11-16T18:35:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:38.554 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:38.554 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:38.554 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:38.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:38.554 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1422256, 2023-11-16T18:35:38.554 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 122112 2023-11-16T18:35:38.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T18:35:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2, 2023-11-16T18:35:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 164, 2023-11-16T18:35:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 320, 2023-11-16T18:35:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 180560, 2023-11-16T18:35:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2023-11-16T18:35:38.586 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2023-11-16T18:35:38.586 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.010020978, 2023-11-16T18:35:38.586 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.006085675, 2023-11-16T18:35:38.586 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.001850811, 2023-11-16T18:35:38.586 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T18:35:38.586 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:38.586 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 161, 2023-11-16T18:35:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.019192472, 2023-11-16T18:35:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000119207 2023-11-16T18:35:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T18:35:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 180560 2023-11-16T18:35:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2023-11-16T18:35:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 0, 2023-11-16T18:35:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2023-11-16T18:35:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:38.589 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2023-11-16T18:35:38.589 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2023-11-16T18:35:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2023-11-16T18:35:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T18:35:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2023-11-16T18:35:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T18:35:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2023-11-16T18:35:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T18:35:38.611 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T18:35:38.611 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T18:35:38.611 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T18:35:38.611 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T18:35:38.612 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2023-11-16T18:35:38.612 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T18:35:38.612 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T18:35:38.612 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2023-11-16T18:35:38.612 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2023-11-16T18:35:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2023-11-16T18:35:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2023-11-16T18:35:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T18:35:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T18:35:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T18:35:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T18:35:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T18:35:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2023-11-16T18:35:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2023-11-16T18:35:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2023-11-16T18:35:38.701 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T18:35:38.702 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T18:35:38.702 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2023-11-16T18:35:38.702 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2023-11-16T18:35:38.702 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2023-11-16T18:35:38.702 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2023-11-16T18:35:38.702 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T18:35:38.702 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2023-11-16T18:35:38.703 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T18:35:38.703 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2023-11-16T18:35:38.703 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 148923, 2023-11-16T18:35:38.703 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 29, 2023-11-16T18:35:38.703 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2023-11-16T18:35:38.703 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2023-11-16T18:35:38.703 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2023-11-16T18:35:38.703 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2023-11-16T18:35:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2023-11-16T18:35:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2023-11-16T18:35:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2023-11-16T18:35:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2023-11-16T18:35:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 0, 2023-11-16T18:35:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 0, 2023-11-16T18:35:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T18:35:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2023-11-16T18:35:38.705 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2023-11-16T18:35:38.705 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2023-11-16T18:35:38.705 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 928, 2023-11-16T18:35:38.706 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2, 2023-11-16T18:35:38.706 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2023-11-16T18:35:38.706 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2023-11-16T18:35:38.706 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2023-11-16T18:35:38.706 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2023-11-16T18:35:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-11-16T18:35:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T18:35:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2023-11-16T18:35:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2023-11-16T18:35:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2023-11-16T18:35:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 226, 2023-11-16T18:35:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 1390629, 2023-11-16T18:35:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 226, 2023-11-16T18:35:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2023-11-16T18:35:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2023-11-16T18:35:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 226, 2023-11-16T18:35:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1390629, 2023-11-16T18:35:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2023-11-16T18:35:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2023-11-16T18:35:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2023-11-16T18:35:38.709 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2023-11-16T18:35:38.709 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:38.709 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-11-16T18:35:38.709 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-11-16T18:35:38.951 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2023-11-16T18:35:38.951 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2023-11-16T18:35:38.951 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2023-11-16T18:35:38.951 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2023-11-16T18:35:38.952 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2023-11-16T18:35:38.952 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2023-11-16T18:35:38.952 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2023-11-16T18:35:38.952 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2023-11-16T18:35:38.952 DEBUG:tasks.check_counter:Getting stats from c 2023-11-16T18:35:38.952 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.c perf dump 2023-11-16T18:35:39.614 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:35:40.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:40 smithi040 ceph-mon[100251]: pgmap v338: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:40.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:40 smithi188 ceph-mon[100639]: pgmap v338: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:40.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:40 smithi002 ceph-mon[98807]: pgmap v338: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:40.916 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2023-11-16T18:35:40.916 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T18:35:40.916 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 287, 2023-11-16T18:35:40.917 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 278, 2023-11-16T18:35:40.917 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 276830, 2023-11-16T18:35:40.917 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 98687, 2023-11-16T18:35:40.917 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 6, 2023-11-16T18:35:40.917 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 2, 2023-11-16T18:35:40.917 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.038823642, 2023-11-16T18:35:40.917 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.011010623, 2023-11-16T18:35:40.918 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.012939284, 2023-11-16T18:35:40.918 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.011910932, 2023-11-16T18:35:40.918 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:40.918 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 278, 2023-11-16T18:35:40.918 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.035138959, 2023-11-16T18:35:40.918 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000126399 2023-11-16T18:35:40.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.919 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:40.919 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.919 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.919 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.919 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 203376, 2023-11-16T18:35:40.920 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 34608 2023-11-16T18:35:40.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.920 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T18:35:40.920 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 225, 2023-11-16T18:35:40.920 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 268, 2023-11-16T18:35:40.920 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 255326, 2023-11-16T18:35:40.920 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 225657, 2023-11-16T18:35:40.921 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 5, 2023-11-16T18:35:40.921 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 4, 2023-11-16T18:35:40.921 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.026750313, 2023-11-16T18:35:40.921 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.008229170, 2023-11-16T18:35:40.921 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.007727424, 2023-11-16T18:35:40.921 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.008095567, 2023-11-16T18:35:40.921 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:40.922 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 266, 2023-11-16T18:35:40.922 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.034211343, 2023-11-16T18:35:40.922 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000128614 2023-11-16T18:35:40.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.922 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:40.922 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.922 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.923 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.923 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 160, 2023-11-16T18:35:40.923 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 155936 2023-11-16T18:35:40.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.923 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T18:35:40.924 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 206, 2023-11-16T18:35:40.924 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 208, 2023-11-16T18:35:40.924 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 51280, 2023-11-16T18:35:40.924 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 64073, 2023-11-16T18:35:40.924 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 5, 2023-11-16T18:35:40.924 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 4, 2023-11-16T18:35:40.924 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.023508451, 2023-11-16T18:35:40.925 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.005280234, 2023-11-16T18:35:40.925 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.006425457, 2023-11-16T18:35:40.926 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.009748897, 2023-11-16T18:35:40.926 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:40.927 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 206, 2023-11-16T18:35:40.927 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.016977617, 2023-11-16T18:35:40.927 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000082415 2023-11-16T18:35:40.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.927 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:40.928 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.928 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.928 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.928 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T18:35:40.928 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T18:35:40.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.929 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2023-11-16T18:35:40.929 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 1, 2023-11-16T18:35:40.929 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2023-11-16T18:35:40.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.931 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-MDSRank": { 2023-11-16T18:35:40.931 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2023-11-16T18:35:40.931 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2023-11-16T18:35:40.931 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1008, 2023-11-16T18:35:40.932 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.036819514, 2023-11-16T18:35:40.932 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000036527 2023-11-16T18:35:40.932 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:40.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.932 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-PurgeQueue": { 2023-11-16T18:35:40.932 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2023-11-16T18:35:40.932 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2023-11-16T18:35:40.932 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.933 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.933 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.933 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:40.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.933 INFO:teuthology.orchestra.run.smithi188.stdout: "mds": { 2023-11-16T18:35:40.933 INFO:teuthology.orchestra.run.smithi188.stdout: "request": 0, 2023-11-16T18:35:40.933 INFO:teuthology.orchestra.run.smithi188.stdout: "reply": 0, 2023-11-16T18:35:40.933 INFO:teuthology.orchestra.run.smithi188.stdout: "reply_latency": { 2023-11-16T18:35:40.934 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.934 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.934 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.934 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_reply": 0, 2023-11-16T18:35:40.934 INFO:teuthology.orchestra.run.smithi188.stdout: "forward": 0, 2023-11-16T18:35:40.934 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_complete": 0, 2023-11-16T18:35:40.934 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_keys": 0, 2023-11-16T18:35:40.935 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_commit": 0, 2023-11-16T18:35:40.935 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_split": 0, 2023-11-16T18:35:40.935 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_merge": 0, 2023-11-16T18:35:40.935 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes": 14, 2023-11-16T18:35:40.935 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_top": 0, 2023-11-16T18:35:40.935 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_bottom": 0, 2023-11-16T18:35:40.935 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pin_tail": 14, 2023-11-16T18:35:40.935 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pinned": 14, 2023-11-16T18:35:40.935 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_expired": 0, 2023-11-16T18:35:40.936 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_with_caps": 0, 2023-11-16T18:35:40.936 INFO:teuthology.orchestra.run.smithi188.stdout: "caps": 0, 2023-11-16T18:35:40.936 INFO:teuthology.orchestra.run.smithi188.stdout: "subtrees": 2, 2023-11-16T18:35:40.936 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse": 0, 2023-11-16T18:35:40.936 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_hit": 0, 2023-11-16T18:35:40.936 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_forward": 0, 2023-11-16T18:35:40.936 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_discover": 0, 2023-11-16T18:35:40.936 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_dir_fetch": 0, 2023-11-16T18:35:40.937 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_remote_ino": 0, 2023-11-16T18:35:40.937 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_lock": 0, 2023-11-16T18:35:40.937 INFO:teuthology.orchestra.run.smithi188.stdout: "load_cent": 0, 2023-11-16T18:35:40.937 INFO:teuthology.orchestra.run.smithi188.stdout: "q": 0, 2023-11-16T18:35:40.937 INFO:teuthology.orchestra.run.smithi188.stdout: "exported": 0, 2023-11-16T18:35:40.937 INFO:teuthology.orchestra.run.smithi188.stdout: "exported_inodes": 0, 2023-11-16T18:35:40.937 INFO:teuthology.orchestra.run.smithi188.stdout: "imported": 0, 2023-11-16T18:35:40.937 INFO:teuthology.orchestra.run.smithi188.stdout: "imported_inodes": 0, 2023-11-16T18:35:40.938 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_dir_fetch": 0, 2023-11-16T18:35:40.938 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_backtrace_fetch": 0, 2023-11-16T18:35:40.938 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_peer_discover": 0, 2023-11-16T18:35:40.938 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rfiles": 4, 2023-11-16T18:35:40.938 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rbytes": 246, 2023-11-16T18:35:40.938 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rsnaps": 6, 2023-11-16T18:35:40.938 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_fetch": 0, 2023-11-16T18:35:40.938 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_set_tag": 0, 2023-11-16T18:35:40.938 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T18:35:40.939 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_inotable_repaired": 0, 2023-11-16T18:35:40.939 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_inodes": 0, 2023-11-16T18:35:40.939 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_base_inodes": 0, 2023-11-16T18:35:40.939 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dirfrag_rstats": 0, 2023-11-16T18:35:40.939 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_file_inodes": 0, 2023-11-16T18:35:40.939 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_inode_file_caps": 0, 2023-11-16T18:35:40.939 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_revoke": 0, 2023-11-16T18:35:40.939 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_grant": 0, 2023-11-16T18:35:40.940 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_trunc": 0, 2023-11-16T18:35:40.940 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-16T18:35:40.940 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T18:35:40.940 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps": 0, 2023-11-16T18:35:40.940 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps_dirty": 0, 2023-11-16T18:35:40.940 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_cap_release": 0, 2023-11-16T18:35:40.940 INFO:teuthology.orchestra.run.smithi188.stdout: "process_request_cap_release": 0 2023-11-16T18:35:40.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.941 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_cache": { 2023-11-16T18:35:40.941 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update": 0, 2023-11-16T18:35:40.941 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update_receipt": 0, 2023-11-16T18:35:40.941 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_try_discover": 0, 2023-11-16T18:35:40.941 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_send_discover": 0, 2023-11-16T18:35:40.941 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_handle_discover": 0, 2023-11-16T18:35:40.941 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays": 0, 2023-11-16T18:35:40.941 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_delayed": 0, 2023-11-16T18:35:40.941 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_enqueuing": 0, 2023-11-16T18:35:40.942 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_created": 0, 2023-11-16T18:35:40.942 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_enqueued": 0, 2023-11-16T18:35:40.942 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_reintegrated": 0, 2023-11-16T18:35:40.942 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_migrated": 0, 2023-11-16T18:35:40.942 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_processing": 0, 2023-11-16T18:35:40.942 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_enqueued": 0, 2023-11-16T18:35:40.942 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_prioritized": 0, 2023-11-16T18:35:40.942 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_started": 0, 2023-11-16T18:35:40.943 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_completed": 0, 2023-11-16T18:35:40.943 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_enqueue_scrub": 0, 2023-11-16T18:35:40.943 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_exportdir": 0, 2023-11-16T18:35:40.943 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_flush": 0, 2023-11-16T18:35:40.943 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragmentdir": 0, 2023-11-16T18:35:40.943 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragstats": 0, 2023-11-16T18:35:40.943 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_inodestats": 0 2023-11-16T18:35:40.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.944 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_log": { 2023-11-16T18:35:40.944 INFO:teuthology.orchestra.run.smithi188.stdout: "evadd": 0, 2023-11-16T18:35:40.944 INFO:teuthology.orchestra.run.smithi188.stdout: "evex": 0, 2023-11-16T18:35:40.944 INFO:teuthology.orchestra.run.smithi188.stdout: "evtrm": 0, 2023-11-16T18:35:40.944 INFO:teuthology.orchestra.run.smithi188.stdout: "ev": 0, 2023-11-16T18:35:40.944 INFO:teuthology.orchestra.run.smithi188.stdout: "evexg": 0, 2023-11-16T18:35:40.944 INFO:teuthology.orchestra.run.smithi188.stdout: "evexd": 0, 2023-11-16T18:35:40.944 INFO:teuthology.orchestra.run.smithi188.stdout: "segadd": 0, 2023-11-16T18:35:40.944 INFO:teuthology.orchestra.run.smithi188.stdout: "segex": 0, 2023-11-16T18:35:40.945 INFO:teuthology.orchestra.run.smithi188.stdout: "segtrm": 0, 2023-11-16T18:35:40.945 INFO:teuthology.orchestra.run.smithi188.stdout: "seg": 1, 2023-11-16T18:35:40.945 INFO:teuthology.orchestra.run.smithi188.stdout: "segexg": 0, 2023-11-16T18:35:40.945 INFO:teuthology.orchestra.run.smithi188.stdout: "segexd": 0, 2023-11-16T18:35:40.945 INFO:teuthology.orchestra.run.smithi188.stdout: "expos": 331355302, 2023-11-16T18:35:40.945 INFO:teuthology.orchestra.run.smithi188.stdout: "wrpos": 0, 2023-11-16T18:35:40.945 INFO:teuthology.orchestra.run.smithi188.stdout: "rdpos": 331551094, 2023-11-16T18:35:40.945 INFO:teuthology.orchestra.run.smithi188.stdout: "jlat": { 2023-11-16T18:35:40.946 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.946 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.946 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.946 INFO:teuthology.orchestra.run.smithi188.stdout: "replayed": 63 2023-11-16T18:35:40.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.946 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_mem": { 2023-11-16T18:35:40.946 INFO:teuthology.orchestra.run.smithi188.stdout: "ino": 12, 2023-11-16T18:35:40.946 INFO:teuthology.orchestra.run.smithi188.stdout: "ino+": 14, 2023-11-16T18:35:40.947 INFO:teuthology.orchestra.run.smithi188.stdout: "ino-": 2, 2023-11-16T18:35:40.947 INFO:teuthology.orchestra.run.smithi188.stdout: "dir": 11, 2023-11-16T18:35:40.947 INFO:teuthology.orchestra.run.smithi188.stdout: "dir+": 12, 2023-11-16T18:35:40.947 INFO:teuthology.orchestra.run.smithi188.stdout: "dir-": 1, 2023-11-16T18:35:40.947 INFO:teuthology.orchestra.run.smithi188.stdout: "dn": 14, 2023-11-16T18:35:40.947 INFO:teuthology.orchestra.run.smithi188.stdout: "dn+": 15, 2023-11-16T18:35:40.947 INFO:teuthology.orchestra.run.smithi188.stdout: "dn-": 1, 2023-11-16T18:35:40.947 INFO:teuthology.orchestra.run.smithi188.stdout: "cap": 0, 2023-11-16T18:35:40.948 INFO:teuthology.orchestra.run.smithi188.stdout: "cap+": 0, 2023-11-16T18:35:40.948 INFO:teuthology.orchestra.run.smithi188.stdout: "cap-": 0, 2023-11-16T18:35:40.948 INFO:teuthology.orchestra.run.smithi188.stdout: "rss": 43784, 2023-11-16T18:35:40.948 INFO:teuthology.orchestra.run.smithi188.stdout: "heap": 207132 2023-11-16T18:35:40.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.948 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_server": { 2023-11-16T18:35:40.948 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_client_request": 0, 2023-11-16T18:35:40.948 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_server_request": 0, 2023-11-16T18:35:40.948 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_request": 0, 2023-11-16T18:35:40.949 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_session": 0, 2023-11-16T18:35:40.949 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_peer_request": 0, 2023-11-16T18:35:40.949 INFO:teuthology.orchestra.run.smithi188.stdout: "req_create_latency": { 2023-11-16T18:35:40.949 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.949 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.949 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.949 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getattr_latency": { 2023-11-16T18:35:40.950 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.950 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.950 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.950 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getfilelock_latency": { 2023-11-16T18:35:40.950 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.950 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.950 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.951 INFO:teuthology.orchestra.run.smithi188.stdout: "req_link_latency": { 2023-11-16T18:35:40.951 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.951 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.951 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.952 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookup_latency": { 2023-11-16T18:35:40.952 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.952 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.953 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.953 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookuphash_latency": { 2023-11-16T18:35:40.953 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.953 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.953 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.953 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupino_latency": { 2023-11-16T18:35:40.954 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.954 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.954 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.954 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupname_latency": { 2023-11-16T18:35:40.954 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.954 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.954 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.955 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupparent_latency": { 2023-11-16T18:35:40.955 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.955 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.955 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.956 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupsnap_latency": { 2023-11-16T18:35:40.956 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.956 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.956 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.956 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lssnap_latency": { 2023-11-16T18:35:40.956 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.957 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.957 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.957 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mkdir_latency": { 2023-11-16T18:35:40.957 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.957 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.957 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.958 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mknod_latency": { 2023-11-16T18:35:40.958 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.958 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.958 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.958 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mksnap_latency": { 2023-11-16T18:35:40.971 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.971 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.971 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.971 INFO:teuthology.orchestra.run.smithi188.stdout: "req_open_latency": { 2023-11-16T18:35:40.971 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.971 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.971 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.972 INFO:teuthology.orchestra.run.smithi188.stdout: "req_readdir_latency": { 2023-11-16T18:35:40.973 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.973 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.973 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.974 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rename_latency": { 2023-11-16T18:35:40.974 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.974 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.974 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.974 INFO:teuthology.orchestra.run.smithi188.stdout: "req_renamesnap_latency": { 2023-11-16T18:35:40.974 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.974 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.975 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.975 INFO:teuthology.orchestra.run.smithi188.stdout: "req_snapdiff_latency": { 2023-11-16T18:35:40.975 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.975 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.976 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.976 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmdir_latency": { 2023-11-16T18:35:40.976 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.976 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.977 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.977 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmsnap_latency": { 2023-11-16T18:35:40.977 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.977 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.977 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.977 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmxattr_latency": { 2023-11-16T18:35:40.978 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.978 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.978 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.978 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setattr_latency": { 2023-11-16T18:35:40.978 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.978 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.979 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.979 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setdirlayout_latency": { 2023-11-16T18:35:40.979 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.979 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.979 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.980 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setfilelock_latency": { 2023-11-16T18:35:40.980 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.980 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.980 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.980 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setlayout_latency": { 2023-11-16T18:35:40.980 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.980 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.981 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.981 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setxattr_latency": { 2023-11-16T18:35:40.981 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.981 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.981 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.981 INFO:teuthology.orchestra.run.smithi188.stdout: "req_symlink_latency": { 2023-11-16T18:35:40.982 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.982 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.982 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.982 INFO:teuthology.orchestra.run.smithi188.stdout: "req_unlink_latency": { 2023-11-16T18:35:40.982 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.982 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.983 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.983 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_revoke_eviction": 0, 2023-11-16T18:35:40.983 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_acquisition_throttle": 0, 2023-11-16T18:35:40.983 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getvxattr_latency": { 2023-11-16T18:35:40.983 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:40.983 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:40.983 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:40.984 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:40.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.984 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_sessions": { 2023-11-16T18:35:40.984 INFO:teuthology.orchestra.run.smithi188.stdout: "session_count": 2, 2023-11-16T18:35:40.984 INFO:teuthology.orchestra.run.smithi188.stdout: "session_add": 2, 2023-11-16T18:35:40.984 INFO:teuthology.orchestra.run.smithi188.stdout: "session_remove": 0, 2023-11-16T18:35:40.984 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_open": 0, 2023-11-16T18:35:40.985 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_stale": 0, 2023-11-16T18:35:40.985 INFO:teuthology.orchestra.run.smithi188.stdout: "total_load": 0, 2023-11-16T18:35:40.985 INFO:teuthology.orchestra.run.smithi188.stdout: "average_load": 0, 2023-11-16T18:35:40.985 INFO:teuthology.orchestra.run.smithi188.stdout: "avg_session_uptime": 2167 2023-11-16T18:35:40.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.985 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2023-11-16T18:35:40.985 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2023-11-16T18:35:40.986 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2023-11-16T18:35:40.986 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T18:35:40.986 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2023-11-16T18:35:40.986 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T18:35:40.986 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2023-11-16T18:35:40.986 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T18:35:40.986 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T18:35:40.987 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T18:35:40.987 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T18:35:40.987 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T18:35:40.987 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2023-11-16T18:35:40.987 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T18:35:40.987 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T18:35:40.988 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2023-11-16T18:35:40.988 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2023-11-16T18:35:40.988 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2023-11-16T18:35:40.988 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2023-11-16T18:35:40.988 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T18:35:40.988 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T18:35:40.989 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T18:35:40.989 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T18:35:40.989 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T18:35:40.989 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2023-11-16T18:35:40.989 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2023-11-16T18:35:40.989 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2023-11-16T18:35:40.990 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T18:35:40.990 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T18:35:40.990 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2023-11-16T18:35:40.990 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2023-11-16T18:35:40.990 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2023-11-16T18:35:40.990 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2023-11-16T18:35:40.990 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T18:35:40.991 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2023-11-16T18:35:40.991 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T18:35:40.991 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2023-11-16T18:35:40.991 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 205683, 2023-11-16T18:35:40.991 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 65, 2023-11-16T18:35:40.991 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 0, 2023-11-16T18:35:40.991 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 0, 2023-11-16T18:35:40.992 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2023-11-16T18:35:40.992 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2023-11-16T18:35:40.992 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2023-11-16T18:35:40.992 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2023-11-16T18:35:40.992 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2023-11-16T18:35:40.992 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2023-11-16T18:35:40.992 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 9704, 2023-11-16T18:35:40.992 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 186, 2023-11-16T18:35:40.993 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T18:35:40.993 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2023-11-16T18:35:40.993 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2023-11-16T18:35:40.993 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2023-11-16T18:35:40.993 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 78510, 2023-11-16T18:35:40.993 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 405, 2023-11-16T18:35:40.993 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2023-11-16T18:35:40.994 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2023-11-16T18:35:40.994 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2023-11-16T18:35:40.994 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2023-11-16T18:35:40.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.994 INFO:teuthology.orchestra.run.smithi188.stdout: "objecter": { 2023-11-16T18:35:40.994 INFO:teuthology.orchestra.run.smithi188.stdout: "op_active": 0, 2023-11-16T18:35:40.994 INFO:teuthology.orchestra.run.smithi188.stdout: "op_laggy": 0, 2023-11-16T18:35:40.995 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send": 636, 2023-11-16T18:35:40.995 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send_bytes": 26, 2023-11-16T18:35:40.997 INFO:teuthology.orchestra.run.smithi188.stdout: "op_resend": 4, 2023-11-16T18:35:40.997 INFO:teuthology.orchestra.run.smithi188.stdout: "op_reply": 636, 2023-11-16T18:35:40.997 INFO:teuthology.orchestra.run.smithi188.stdout: "op_latency": { 2023-11-16T18:35:40.997 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 636, 2023-11-16T18:35:40.997 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.725986815, 2023-11-16T18:35:40.998 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001141488 2023-11-16T18:35:40.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.998 INFO:teuthology.orchestra.run.smithi188.stdout: "op_inflight": 0, 2023-11-16T18:35:40.998 INFO:teuthology.orchestra.run.smithi188.stdout: "oplen_avg": { 2023-11-16T18:35:40.998 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 636, 2023-11-16T18:35:40.998 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 637 2023-11-16T18:35:40.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:40.999 INFO:teuthology.orchestra.run.smithi188.stdout: "op": 636, 2023-11-16T18:35:40.999 INFO:teuthology.orchestra.run.smithi188.stdout: "op_r": 636, 2023-11-16T18:35:40.999 INFO:teuthology.orchestra.run.smithi188.stdout: "op_w": 0, 2023-11-16T18:35:40.999 INFO:teuthology.orchestra.run.smithi188.stdout: "op_rmw": 0, 2023-11-16T18:35:40.999 INFO:teuthology.orchestra.run.smithi188.stdout: "op_pg": 0, 2023-11-16T18:35:40.999 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_stat": 406, 2023-11-16T18:35:40.999 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_create": 0, 2023-11-16T18:35:40.999 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_read": 227, 2023-11-16T18:35:41.000 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_write": 0, 2023-11-16T18:35:41.000 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writefull": 0, 2023-11-16T18:35:41.000 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writesame": 0, 2023-11-16T18:35:41.000 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_append": 0, 2023-11-16T18:35:41.000 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_zero": 0, 2023-11-16T18:35:41.000 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_truncate": 0, 2023-11-16T18:35:41.000 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_delete": 0, 2023-11-16T18:35:41.001 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_mapext": 0, 2023-11-16T18:35:41.001 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_sparse_read": 0, 2023-11-16T18:35:41.001 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_clonerange": 0, 2023-11-16T18:35:41.001 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_getxattr": 2, 2023-11-16T18:35:41.001 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_setxattr": 0, 2023-11-16T18:35:41.002 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_cmpxattr": 0, 2023-11-16T18:35:41.002 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_rmxattr": 0, 2023-11-16T18:35:41.002 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_resetxattrs": 0, 2023-11-16T18:35:41.003 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_call": 0, 2023-11-16T18:35:41.003 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_watch": 0, 2023-11-16T18:35:41.003 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_notify": 0, 2023-11-16T18:35:41.003 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T18:35:41.003 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls": 0, 2023-11-16T18:35:41.003 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls_filter": 0, 2023-11-16T18:35:41.003 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_other": 0, 2023-11-16T18:35:41.004 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_active": 0, 2023-11-16T18:35:41.004 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_send": 0, 2023-11-16T18:35:41.004 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_resend": 0, 2023-11-16T18:35:41.004 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_ping": 0, 2023-11-16T18:35:41.004 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_active": 0, 2023-11-16T18:35:41.004 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_send": 0, 2023-11-16T18:35:41.004 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_resend": 0, 2023-11-16T18:35:41.005 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_active": 0, 2023-11-16T18:35:41.005 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_send": 0, 2023-11-16T18:35:41.005 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_resend": 0, 2023-11-16T18:35:41.005 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_active": 0, 2023-11-16T18:35:41.005 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_send": 0, 2023-11-16T18:35:41.005 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_resend": 0, 2023-11-16T18:35:41.005 INFO:teuthology.orchestra.run.smithi188.stdout: "command_active": 0, 2023-11-16T18:35:41.006 INFO:teuthology.orchestra.run.smithi188.stdout: "command_send": 0, 2023-11-16T18:35:41.006 INFO:teuthology.orchestra.run.smithi188.stdout: "command_resend": 0, 2023-11-16T18:35:41.006 INFO:teuthology.orchestra.run.smithi188.stdout: "map_epoch": 102, 2023-11-16T18:35:41.006 INFO:teuthology.orchestra.run.smithi188.stdout: "map_full": 0, 2023-11-16T18:35:41.006 INFO:teuthology.orchestra.run.smithi188.stdout: "map_inc": 6, 2023-11-16T18:35:41.006 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_sessions": 7, 2023-11-16T18:35:41.006 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_open": 7, 2023-11-16T18:35:41.007 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_close": 0, 2023-11-16T18:35:41.007 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_laggy": 0, 2023-11-16T18:35:41.007 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_wr": 0, 2023-11-16T18:35:41.008 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_rd": 2, 2023-11-16T18:35:41.008 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_del": 0 2023-11-16T18:35:41.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:41.008 INFO:teuthology.orchestra.run.smithi188.stdout: "oft": { 2023-11-16T18:35:41.008 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_objs": 0, 2023-11-16T18:35:41.009 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_kv_pairs": 0, 2023-11-16T18:35:41.009 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_updates": 0, 2023-11-16T18:35:41.009 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_removes": 0 2023-11-16T18:35:41.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:41.009 INFO:teuthology.orchestra.run.smithi188.stdout: "osd-slow-ops": { 2023-11-16T18:35:41.009 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_ops_count": 0 2023-11-16T18:35:41.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:41.010 INFO:teuthology.orchestra.run.smithi188.stdout: "purge_queue": { 2023-11-16T18:35:41.010 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops": 0, 2023-11-16T18:35:41.010 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops_high_water": 0, 2023-11-16T18:35:41.010 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing": 0, 2023-11-16T18:35:41.010 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_high_water": 0, 2023-11-16T18:35:41.010 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed": 0, 2023-11-16T18:35:41.010 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_item_in_journal": 0 2023-11-16T18:35:41.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:41.011 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T18:35:41.011 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:41.011 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2023-11-16T18:35:41.011 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:41.011 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 714, 2023-11-16T18:35:41.011 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 512294, 2023-11-16T18:35:41.012 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:41.012 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 714, 2023-11-16T18:35:41.012 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2023-11-16T18:35:41.012 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2023-11-16T18:35:41.012 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 714, 2023-11-16T18:35:41.012 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 512294, 2023-11-16T18:35:41.012 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:41.013 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:41.013 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:41.013 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:41.013 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:41.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:41.013 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_bytes": { 2023-11-16T18:35:41.013 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:41.014 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2023-11-16T18:35:41.014 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:41.014 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2023-11-16T18:35:41.014 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2023-11-16T18:35:41.014 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:41.014 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:41.014 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 636, 2023-11-16T18:35:41.015 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 195938, 2023-11-16T18:35:41.015 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 636, 2023-11-16T18:35:41.015 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 195938, 2023-11-16T18:35:41.015 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:41.015 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:41.015 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:41.015 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:41.016 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:41.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:41.016 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_ops": { 2023-11-16T18:35:41.016 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:41.016 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 1024, 2023-11-16T18:35:41.016 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:41.016 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2023-11-16T18:35:41.017 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2023-11-16T18:35:41.017 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:41.017 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:41.017 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 636, 2023-11-16T18:35:41.017 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 636, 2023-11-16T18:35:41.017 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 636, 2023-11-16T18:35:41.017 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 636, 2023-11-16T18:35:41.018 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:41.018 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:41.018 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:41.018 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:41.018 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:41.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:41.019 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle": { 2023-11-16T18:35:41.019 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:41.019 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2023-11-16T18:35:41.019 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:41.019 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2023-11-16T18:35:41.020 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2023-11-16T18:35:41.020 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:41.021 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:41.021 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2023-11-16T18:35:41.021 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2023-11-16T18:35:41.021 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 0, 2023-11-16T18:35:41.021 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 0, 2023-11-16T18:35:41.021 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:41.021 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:41.022 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:41.022 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:41.022 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:41.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:41.022 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle-0x558018546000": { 2023-11-16T18:35:41.022 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:41.022 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2023-11-16T18:35:41.023 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:41.023 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2023-11-16T18:35:41.023 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2023-11-16T18:35:41.023 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:41.023 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:41.023 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2023-11-16T18:35:41.023 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2023-11-16T18:35:41.024 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 0, 2023-11-16T18:35:41.024 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 0, 2023-11-16T18:35:41.024 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:41.024 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:41.024 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:41.024 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:41.024 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:41.025 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:41.025 INFO:teuthology.orchestra.run.smithi188.stdout:} 2023-11-16T18:35:41.358 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2023-11-16T18:35:41.358 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2023-11-16T18:35:41.358 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2023-11-16T18:35:41.358 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2023-11-16T18:35:41.358 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2023-11-16T18:35:41.359 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2023-11-16T18:35:41.359 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2023-11-16T18:35:41.359 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2023-11-16T18:35:41.359 DEBUG:tasks.check_counter:Getting stats from f 2023-11-16T18:35:41.359 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.f perf dump 2023-11-16T18:35:42.002 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:35:42.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:42 smithi040 ceph-mon[100251]: pgmap v339: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:42.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:42 smithi040 ceph-mon[100251]: osdmap e103: 12 total, 12 up, 12 in 2023-11-16T18:35:42.538 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:42 smithi188 ceph-mon[100639]: pgmap v339: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:42.538 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:42 smithi188 ceph-mon[100639]: osdmap e103: 12 total, 12 up, 12 in 2023-11-16T18:35:42.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:42 smithi002 ceph-mon[98807]: pgmap v339: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:42.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:42 smithi002 ceph-mon[98807]: osdmap e103: 12 total, 12 up, 12 in 2023-11-16T18:35:43.358 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2023-11-16T18:35:43.358 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T18:35:43.358 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 1651, 2023-11-16T18:35:43.358 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 1652, 2023-11-16T18:35:43.358 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 181166197, 2023-11-16T18:35:43.358 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 513157, 2023-11-16T18:35:43.359 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 8, 2023-11-16T18:35:43.359 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 4, 2023-11-16T18:35:43.359 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.223727690, 2023-11-16T18:35:43.359 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.042150294, 2023-11-16T18:35:43.359 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.113823125, 2023-11-16T18:35:43.359 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.053954623, 2023-11-16T18:35:43.359 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:43.359 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1650, 2023-11-16T18:35:43.359 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.096768536, 2023-11-16T18:35:43.360 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000058647 2023-11-16T18:35:43.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.360 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:43.360 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.360 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.360 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.360 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T18:35:43.361 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T18:35:43.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.361 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T18:35:43.361 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 857, 2023-11-16T18:35:43.361 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 834, 2023-11-16T18:35:43.361 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 127505278, 2023-11-16T18:35:43.361 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 311695, 2023-11-16T18:35:43.361 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 6, 2023-11-16T18:35:43.362 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 5, 2023-11-16T18:35:43.362 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.158750789, 2023-11-16T18:35:43.362 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.024610733, 2023-11-16T18:35:43.362 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.112401068, 2023-11-16T18:35:43.362 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.025421022, 2023-11-16T18:35:43.362 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:43.362 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 833, 2023-11-16T18:35:43.362 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.046554025, 2023-11-16T18:35:43.363 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000055887 2023-11-16T18:35:43.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.363 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:43.363 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.363 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.363 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.363 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 1504640, 2023-11-16T18:35:43.364 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 125344 2023-11-16T18:35:43.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.364 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T18:35:43.364 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 781, 2023-11-16T18:35:43.364 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 940, 2023-11-16T18:35:43.364 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 181107431, 2023-11-16T18:35:43.364 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 672257, 2023-11-16T18:35:43.364 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 8, 2023-11-16T18:35:43.365 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 6, 2023-11-16T18:35:43.365 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.154726214, 2023-11-16T18:35:43.365 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.022591165, 2023-11-16T18:35:43.365 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.103012507, 2023-11-16T18:35:43.365 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.021251276, 2023-11-16T18:35:43.365 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:43.365 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 938, 2023-11-16T18:35:43.366 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.048461454, 2023-11-16T18:35:43.366 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000051664 2023-11-16T18:35:43.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.366 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:43.366 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.366 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.366 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.367 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-11-16T18:35:43.367 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 431232 2023-11-16T18:35:43.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.367 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2023-11-16T18:35:43.367 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 1, 2023-11-16T18:35:43.367 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2023-11-16T18:35:43.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.367 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-MDSRank": { 2023-11-16T18:35:43.367 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2023-11-16T18:35:43.368 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2023-11-16T18:35:43.368 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 4672, 2023-11-16T18:35:43.368 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.137107462, 2023-11-16T18:35:43.368 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000029346 2023-11-16T18:35:43.368 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:43.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.369 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-PurgeQueue": { 2023-11-16T18:35:43.369 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2023-11-16T18:35:43.369 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2023-11-16T18:35:43.369 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.369 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.369 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.369 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:43.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.369 INFO:teuthology.orchestra.run.smithi188.stdout: "mds": { 2023-11-16T18:35:43.370 INFO:teuthology.orchestra.run.smithi188.stdout: "request": 0, 2023-11-16T18:35:43.370 INFO:teuthology.orchestra.run.smithi188.stdout: "reply": 0, 2023-11-16T18:35:43.370 INFO:teuthology.orchestra.run.smithi188.stdout: "reply_latency": { 2023-11-16T18:35:43.370 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.370 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.370 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.370 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.370 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_reply": 0, 2023-11-16T18:35:43.371 INFO:teuthology.orchestra.run.smithi188.stdout: "forward": 0, 2023-11-16T18:35:43.371 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_complete": 0, 2023-11-16T18:35:43.371 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_keys": 0, 2023-11-16T18:35:43.371 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_commit": 0, 2023-11-16T18:35:43.371 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_split": 0, 2023-11-16T18:35:43.371 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_merge": 0, 2023-11-16T18:35:43.371 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes": 106001, 2023-11-16T18:35:43.372 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_top": 6597, 2023-11-16T18:35:43.372 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_bottom": 39559, 2023-11-16T18:35:43.372 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pin_tail": 59845, 2023-11-16T18:35:43.372 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pinned": 96576, 2023-11-16T18:35:43.372 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_expired": 0, 2023-11-16T18:35:43.372 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_with_caps": 0, 2023-11-16T18:35:43.372 INFO:teuthology.orchestra.run.smithi188.stdout: "caps": 0, 2023-11-16T18:35:43.373 INFO:teuthology.orchestra.run.smithi188.stdout: "subtrees": 1, 2023-11-16T18:35:43.373 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse": 0, 2023-11-16T18:35:43.373 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_hit": 0, 2023-11-16T18:35:43.373 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_forward": 0, 2023-11-16T18:35:43.373 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_discover": 0, 2023-11-16T18:35:43.374 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_dir_fetch": 0, 2023-11-16T18:35:43.374 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_remote_ino": 0, 2023-11-16T18:35:43.374 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_lock": 0, 2023-11-16T18:35:43.375 INFO:teuthology.orchestra.run.smithi188.stdout: "load_cent": 0, 2023-11-16T18:35:43.375 INFO:teuthology.orchestra.run.smithi188.stdout: "q": 0, 2023-11-16T18:35:43.375 INFO:teuthology.orchestra.run.smithi188.stdout: "exported": 0, 2023-11-16T18:35:43.375 INFO:teuthology.orchestra.run.smithi188.stdout: "exported_inodes": 0, 2023-11-16T18:35:43.375 INFO:teuthology.orchestra.run.smithi188.stdout: "imported": 0, 2023-11-16T18:35:43.375 INFO:teuthology.orchestra.run.smithi188.stdout: "imported_inodes": 0, 2023-11-16T18:35:43.375 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_dir_fetch": 0, 2023-11-16T18:35:43.375 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_backtrace_fetch": 0, 2023-11-16T18:35:43.376 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_peer_discover": 0, 2023-11-16T18:35:43.376 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rfiles": 0, 2023-11-16T18:35:43.376 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rbytes": 0, 2023-11-16T18:35:43.376 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rsnaps": 0, 2023-11-16T18:35:43.376 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_fetch": 0, 2023-11-16T18:35:43.376 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_set_tag": 0, 2023-11-16T18:35:43.376 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T18:35:43.376 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_inotable_repaired": 0, 2023-11-16T18:35:43.377 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_inodes": 0, 2023-11-16T18:35:43.377 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_base_inodes": 0, 2023-11-16T18:35:43.377 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dirfrag_rstats": 0, 2023-11-16T18:35:43.377 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_file_inodes": 0, 2023-11-16T18:35:43.377 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_inode_file_caps": 0, 2023-11-16T18:35:43.377 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_revoke": 0, 2023-11-16T18:35:43.377 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_grant": 0, 2023-11-16T18:35:43.377 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_trunc": 0, 2023-11-16T18:35:43.378 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-16T18:35:43.378 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T18:35:43.378 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps": 0, 2023-11-16T18:35:43.378 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps_dirty": 0, 2023-11-16T18:35:43.378 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_cap_release": 0, 2023-11-16T18:35:43.378 INFO:teuthology.orchestra.run.smithi188.stdout: "process_request_cap_release": 0 2023-11-16T18:35:43.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.378 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_cache": { 2023-11-16T18:35:43.379 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update": 0, 2023-11-16T18:35:43.379 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update_receipt": 0, 2023-11-16T18:35:43.379 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_try_discover": 0, 2023-11-16T18:35:43.379 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_send_discover": 0, 2023-11-16T18:35:43.379 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_handle_discover": 0, 2023-11-16T18:35:43.379 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays": 0, 2023-11-16T18:35:43.379 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_delayed": 0, 2023-11-16T18:35:43.379 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_enqueuing": 0, 2023-11-16T18:35:43.380 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_created": 0, 2023-11-16T18:35:43.380 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_enqueued": 0, 2023-11-16T18:35:43.380 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_reintegrated": 0, 2023-11-16T18:35:43.380 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_migrated": 0, 2023-11-16T18:35:43.380 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_processing": 0, 2023-11-16T18:35:43.380 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_enqueued": 0, 2023-11-16T18:35:43.380 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_prioritized": 0, 2023-11-16T18:35:43.380 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_started": 0, 2023-11-16T18:35:43.380 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_completed": 0, 2023-11-16T18:35:43.381 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_enqueue_scrub": 0, 2023-11-16T18:35:43.381 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_exportdir": 0, 2023-11-16T18:35:43.381 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_flush": 0, 2023-11-16T18:35:43.381 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragmentdir": 0, 2023-11-16T18:35:43.381 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragstats": 0, 2023-11-16T18:35:43.381 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_inodestats": 0 2023-11-16T18:35:43.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.381 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_log": { 2023-11-16T18:35:43.382 INFO:teuthology.orchestra.run.smithi188.stdout: "evadd": 0, 2023-11-16T18:35:43.382 INFO:teuthology.orchestra.run.smithi188.stdout: "evex": 0, 2023-11-16T18:35:43.382 INFO:teuthology.orchestra.run.smithi188.stdout: "evtrm": 0, 2023-11-16T18:35:43.382 INFO:teuthology.orchestra.run.smithi188.stdout: "ev": 0, 2023-11-16T18:35:43.382 INFO:teuthology.orchestra.run.smithi188.stdout: "evexg": 0, 2023-11-16T18:35:43.382 INFO:teuthology.orchestra.run.smithi188.stdout: "evexd": 0, 2023-11-16T18:35:43.382 INFO:teuthology.orchestra.run.smithi188.stdout: "segadd": 0, 2023-11-16T18:35:43.382 INFO:teuthology.orchestra.run.smithi188.stdout: "segex": 0, 2023-11-16T18:35:43.382 INFO:teuthology.orchestra.run.smithi188.stdout: "segtrm": 0, 2023-11-16T18:35:43.383 INFO:teuthology.orchestra.run.smithi188.stdout: "seg": 129, 2023-11-16T18:35:43.383 INFO:teuthology.orchestra.run.smithi188.stdout: "segexg": 0, 2023-11-16T18:35:43.383 INFO:teuthology.orchestra.run.smithi188.stdout: "segexd": 0, 2023-11-16T18:35:43.384 INFO:teuthology.orchestra.run.smithi188.stdout: "expos": 74481386, 2023-11-16T18:35:43.384 INFO:teuthology.orchestra.run.smithi188.stdout: "wrpos": 0, 2023-11-16T18:35:43.384 INFO:teuthology.orchestra.run.smithi188.stdout: "rdpos": 489923281, 2023-11-16T18:35:43.385 INFO:teuthology.orchestra.run.smithi188.stdout: "jlat": { 2023-11-16T18:35:43.385 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.385 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.385 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.385 INFO:teuthology.orchestra.run.smithi188.stdout: "replayed": 107525 2023-11-16T18:35:43.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.386 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_mem": { 2023-11-16T18:35:43.386 INFO:teuthology.orchestra.run.smithi188.stdout: "ino": 46237, 2023-11-16T18:35:43.386 INFO:teuthology.orchestra.run.smithi188.stdout: "ino+": 104838, 2023-11-16T18:35:43.386 INFO:teuthology.orchestra.run.smithi188.stdout: "ino-": 58601, 2023-11-16T18:35:43.386 INFO:teuthology.orchestra.run.smithi188.stdout: "dir": 2049, 2023-11-16T18:35:43.386 INFO:teuthology.orchestra.run.smithi188.stdout: "dir+": 2411, 2023-11-16T18:35:43.386 INFO:teuthology.orchestra.run.smithi188.stdout: "dir-": 362, 2023-11-16T18:35:43.387 INFO:teuthology.orchestra.run.smithi188.stdout: "dn": 106001, 2023-11-16T18:35:43.387 INFO:teuthology.orchestra.run.smithi188.stdout: "dn+": 161756, 2023-11-16T18:35:43.387 INFO:teuthology.orchestra.run.smithi188.stdout: "dn-": 55755, 2023-11-16T18:35:43.387 INFO:teuthology.orchestra.run.smithi188.stdout: "cap": 0, 2023-11-16T18:35:43.387 INFO:teuthology.orchestra.run.smithi188.stdout: "cap+": 0, 2023-11-16T18:35:43.387 INFO:teuthology.orchestra.run.smithi188.stdout: "cap-": 0, 2023-11-16T18:35:43.387 INFO:teuthology.orchestra.run.smithi188.stdout: "rss": 287860, 2023-11-16T18:35:43.387 INFO:teuthology.orchestra.run.smithi188.stdout: "heap": 207156 2023-11-16T18:35:43.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.388 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_server": { 2023-11-16T18:35:43.388 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_client_request": 0, 2023-11-16T18:35:43.388 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_server_request": 0, 2023-11-16T18:35:43.388 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_request": 0, 2023-11-16T18:35:43.388 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_session": 0, 2023-11-16T18:35:43.388 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_peer_request": 0, 2023-11-16T18:35:43.388 INFO:teuthology.orchestra.run.smithi188.stdout: "req_create_latency": { 2023-11-16T18:35:43.388 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.389 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.389 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.389 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getattr_latency": { 2023-11-16T18:35:43.389 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.389 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.389 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.390 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getfilelock_latency": { 2023-11-16T18:35:43.390 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.390 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.390 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.390 INFO:teuthology.orchestra.run.smithi188.stdout: "req_link_latency": { 2023-11-16T18:35:43.390 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.390 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.391 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.391 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookup_latency": { 2023-11-16T18:35:43.391 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.391 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.391 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.392 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookuphash_latency": { 2023-11-16T18:35:43.392 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.392 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.392 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.392 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupino_latency": { 2023-11-16T18:35:43.392 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.392 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.392 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.393 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupname_latency": { 2023-11-16T18:35:43.393 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.393 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.393 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.393 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupparent_latency": { 2023-11-16T18:35:43.393 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.394 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.394 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.394 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupsnap_latency": { 2023-11-16T18:35:43.394 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.394 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.394 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.395 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lssnap_latency": { 2023-11-16T18:35:43.395 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.395 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.395 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.395 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mkdir_latency": { 2023-11-16T18:35:43.395 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.396 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.396 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.396 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mknod_latency": { 2023-11-16T18:35:43.396 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.396 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.396 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.397 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mksnap_latency": { 2023-11-16T18:35:43.397 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.397 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.397 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.397 INFO:teuthology.orchestra.run.smithi188.stdout: "req_open_latency": { 2023-11-16T18:35:43.397 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.397 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.397 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.398 INFO:teuthology.orchestra.run.smithi188.stdout: "req_readdir_latency": { 2023-11-16T18:35:43.398 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.398 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.398 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.398 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rename_latency": { 2023-11-16T18:35:43.398 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.399 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.399 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.399 INFO:teuthology.orchestra.run.smithi188.stdout: "req_renamesnap_latency": { 2023-11-16T18:35:43.399 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.399 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.399 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.400 INFO:teuthology.orchestra.run.smithi188.stdout: "req_snapdiff_latency": { 2023-11-16T18:35:43.400 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.400 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.400 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.400 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmdir_latency": { 2023-11-16T18:35:43.400 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.400 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.400 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.401 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmsnap_latency": { 2023-11-16T18:35:43.401 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.401 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.405 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.406 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmxattr_latency": { 2023-11-16T18:35:43.406 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.406 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.406 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.406 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setattr_latency": { 2023-11-16T18:35:43.406 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.407 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.407 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.407 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setdirlayout_latency": { 2023-11-16T18:35:43.407 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.407 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.407 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.408 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setfilelock_latency": { 2023-11-16T18:35:43.408 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.408 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.408 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.408 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setlayout_latency": { 2023-11-16T18:35:43.408 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.408 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.408 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.409 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setxattr_latency": { 2023-11-16T18:35:43.409 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.409 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.409 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.409 INFO:teuthology.orchestra.run.smithi188.stdout: "req_symlink_latency": { 2023-11-16T18:35:43.409 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.468 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.468 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.469 INFO:teuthology.orchestra.run.smithi188.stdout: "req_unlink_latency": { 2023-11-16T18:35:43.469 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.469 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.469 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.469 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_revoke_eviction": 0, 2023-11-16T18:35:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_acquisition_throttle": 0, 2023-11-16T18:35:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getvxattr_latency": { 2023-11-16T18:35:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_sessions": { 2023-11-16T18:35:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: "session_count": 2, 2023-11-16T18:35:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: "session_add": 3, 2023-11-16T18:35:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: "session_remove": 1, 2023-11-16T18:35:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_open": 2, 2023-11-16T18:35:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_stale": 0, 2023-11-16T18:35:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: "total_load": 0, 2023-11-16T18:35:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: "average_load": 0, 2023-11-16T18:35:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: "avg_session_uptime": 3592 2023-11-16T18:35:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2023-11-16T18:35:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2023-11-16T18:35:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2023-11-16T18:35:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T18:35:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2023-11-16T18:35:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T18:35:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2023-11-16T18:35:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T18:35:43.473 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T18:35:43.473 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T18:35:43.473 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T18:35:43.473 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T18:35:43.473 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2023-11-16T18:35:43.473 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T18:35:43.473 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T18:35:43.473 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2023-11-16T18:35:43.474 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2023-11-16T18:35:43.474 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2023-11-16T18:35:43.474 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2023-11-16T18:35:43.474 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T18:35:43.474 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T18:35:43.474 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T18:35:43.474 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T18:35:43.474 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T18:35:43.475 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2023-11-16T18:35:43.475 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2023-11-16T18:35:43.475 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2023-11-16T18:35:43.475 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T18:35:43.475 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T18:35:43.475 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2023-11-16T18:35:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2023-11-16T18:35:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2023-11-16T18:35:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2023-11-16T18:35:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T18:35:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2023-11-16T18:35:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T18:35:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2023-11-16T18:35:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 1811055, 2023-11-16T18:35:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 46325, 2023-11-16T18:35:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 0, 2023-11-16T18:35:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 0, 2023-11-16T18:35:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2023-11-16T18:35:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2023-11-16T18:35:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2023-11-16T18:35:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2023-11-16T18:35:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2023-11-16T18:35:43.478 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2023-11-16T18:35:43.478 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 9584, 2023-11-16T18:35:43.478 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 183, 2023-11-16T18:35:43.478 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T18:35:43.479 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2023-11-16T18:35:43.479 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2023-11-16T18:35:43.479 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2023-11-16T18:35:43.479 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 181205264, 2023-11-16T18:35:43.479 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 1454088, 2023-11-16T18:35:43.479 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2023-11-16T18:35:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2023-11-16T18:35:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2023-11-16T18:35:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2023-11-16T18:35:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: "objecter": { 2023-11-16T18:35:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: "op_active": 0, 2023-11-16T18:35:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: "op_laggy": 0, 2023-11-16T18:35:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send": 3024, 2023-11-16T18:35:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send_bytes": 21, 2023-11-16T18:35:43.481 INFO:teuthology.orchestra.run.smithi188.stdout: "op_resend": 3, 2023-11-16T18:35:43.481 INFO:teuthology.orchestra.run.smithi188.stdout: "op_reply": 3024, 2023-11-16T18:35:43.481 INFO:teuthology.orchestra.run.smithi188.stdout: "op_latency": { 2023-11-16T18:35:43.481 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 3024, 2023-11-16T18:35:43.481 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 7.395865314, 2023-11-16T18:35:43.481 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.002445722 2023-11-16T18:35:43.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.482 INFO:teuthology.orchestra.run.smithi188.stdout: "op_inflight": 0, 2023-11-16T18:35:43.482 INFO:teuthology.orchestra.run.smithi188.stdout: "oplen_avg": { 2023-11-16T18:35:43.482 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 3024, 2023-11-16T18:35:43.482 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 3025 2023-11-16T18:35:43.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.482 INFO:teuthology.orchestra.run.smithi188.stdout: "op": 3024, 2023-11-16T18:35:43.482 INFO:teuthology.orchestra.run.smithi188.stdout: "op_r": 3024, 2023-11-16T18:35:43.483 INFO:teuthology.orchestra.run.smithi188.stdout: "op_w": 0, 2023-11-16T18:35:43.483 INFO:teuthology.orchestra.run.smithi188.stdout: "op_rmw": 0, 2023-11-16T18:35:43.483 INFO:teuthology.orchestra.run.smithi188.stdout: "op_pg": 0, 2023-11-16T18:35:43.483 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_stat": 1938, 2023-11-16T18:35:43.483 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_create": 0, 2023-11-16T18:35:43.483 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_read": 1084, 2023-11-16T18:35:43.483 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_write": 0, 2023-11-16T18:35:43.483 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writefull": 0, 2023-11-16T18:35:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writesame": 0, 2023-11-16T18:35:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_append": 0, 2023-11-16T18:35:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_zero": 0, 2023-11-16T18:35:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_truncate": 0, 2023-11-16T18:35:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_delete": 0, 2023-11-16T18:35:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_mapext": 0, 2023-11-16T18:35:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_sparse_read": 0, 2023-11-16T18:35:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_clonerange": 0, 2023-11-16T18:35:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_getxattr": 1, 2023-11-16T18:35:43.485 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_setxattr": 0, 2023-11-16T18:35:43.485 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_cmpxattr": 0, 2023-11-16T18:35:43.485 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_rmxattr": 0, 2023-11-16T18:35:43.485 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_resetxattrs": 0, 2023-11-16T18:35:43.486 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_call": 0, 2023-11-16T18:35:43.486 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_watch": 0, 2023-11-16T18:35:43.486 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_notify": 0, 2023-11-16T18:35:43.487 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T18:35:43.487 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls": 0, 2023-11-16T18:35:43.487 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls_filter": 0, 2023-11-16T18:35:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_other": 0, 2023-11-16T18:35:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_active": 0, 2023-11-16T18:35:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_send": 0, 2023-11-16T18:35:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_resend": 0, 2023-11-16T18:35:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_ping": 0, 2023-11-16T18:35:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_active": 0, 2023-11-16T18:35:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_send": 0, 2023-11-16T18:35:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_resend": 0, 2023-11-16T18:35:43.489 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_active": 0, 2023-11-16T18:35:43.489 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_send": 0, 2023-11-16T18:35:43.489 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_resend": 0, 2023-11-16T18:35:43.489 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_active": 0, 2023-11-16T18:35:43.489 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_send": 0, 2023-11-16T18:35:43.489 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_resend": 0, 2023-11-16T18:35:43.489 INFO:teuthology.orchestra.run.smithi188.stdout: "command_active": 0, 2023-11-16T18:35:43.489 INFO:teuthology.orchestra.run.smithi188.stdout: "command_send": 0, 2023-11-16T18:35:43.491 INFO:teuthology.orchestra.run.smithi188.stdout: "command_resend": 0, 2023-11-16T18:35:43.491 INFO:teuthology.orchestra.run.smithi188.stdout: "map_epoch": 103, 2023-11-16T18:35:43.491 INFO:teuthology.orchestra.run.smithi188.stdout: "map_full": 0, 2023-11-16T18:35:43.492 INFO:teuthology.orchestra.run.smithi188.stdout: "map_inc": 17, 2023-11-16T18:35:43.492 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_sessions": 11, 2023-11-16T18:35:43.492 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_open": 11, 2023-11-16T18:35:43.492 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_close": 0, 2023-11-16T18:35:43.492 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_laggy": 0, 2023-11-16T18:35:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_wr": 0, 2023-11-16T18:35:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_rd": 2, 2023-11-16T18:35:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_del": 0 2023-11-16T18:35:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: "oft": { 2023-11-16T18:35:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_objs": 0, 2023-11-16T18:35:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_kv_pairs": 0, 2023-11-16T18:35:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_updates": 0, 2023-11-16T18:35:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_removes": 0 2023-11-16T18:35:43.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.494 INFO:teuthology.orchestra.run.smithi188.stdout: "osd-slow-ops": { 2023-11-16T18:35:43.494 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_ops_count": 0 2023-11-16T18:35:43.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.494 INFO:teuthology.orchestra.run.smithi188.stdout: "purge_queue": { 2023-11-16T18:35:43.494 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops": 0, 2023-11-16T18:35:43.494 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops_high_water": 0, 2023-11-16T18:35:43.494 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing": 0, 2023-11-16T18:35:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_high_water": 0, 2023-11-16T18:35:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed": 0, 2023-11-16T18:35:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_item_in_journal": 0 2023-11-16T18:35:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T18:35:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2023-11-16T18:35:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:43.496 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 3285, 2023-11-16T18:35:43.496 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 489467094, 2023-11-16T18:35:43.496 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:43.496 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 3285, 2023-11-16T18:35:43.496 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2023-11-16T18:35:43.496 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2023-11-16T18:35:43.496 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 3285, 2023-11-16T18:35:43.496 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 489467094, 2023-11-16T18:35:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_bytes": { 2023-11-16T18:35:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2023-11-16T18:35:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2023-11-16T18:35:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2023-11-16T18:35:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 3024, 2023-11-16T18:35:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 487436832, 2023-11-16T18:35:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 3024, 2023-11-16T18:35:43.499 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 487436832, 2023-11-16T18:35:43.499 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:43.499 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.499 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.499 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.499 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:43.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.499 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_ops": { 2023-11-16T18:35:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 1024, 2023-11-16T18:35:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2023-11-16T18:35:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2023-11-16T18:35:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 3024, 2023-11-16T18:35:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 3024, 2023-11-16T18:35:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 3024, 2023-11-16T18:35:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 3024, 2023-11-16T18:35:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle": { 2023-11-16T18:35:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2023-11-16T18:35:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2023-11-16T18:35:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2023-11-16T18:35:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:43.503 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2023-11-16T18:35:43.503 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2023-11-16T18:35:43.503 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 0, 2023-11-16T18:35:43.503 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 0, 2023-11-16T18:35:43.503 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:43.503 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.503 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.503 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle-0x5571dfda2000": { 2023-11-16T18:35:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2023-11-16T18:35:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2023-11-16T18:35:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2023-11-16T18:35:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:43.505 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:43.505 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2023-11-16T18:35:43.505 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2023-11-16T18:35:43.505 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 0, 2023-11-16T18:35:43.505 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 0, 2023-11-16T18:35:43.505 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:43.505 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:43.505 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:43.506 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:43.506 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:43.507 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:43.507 INFO:teuthology.orchestra.run.smithi188.stdout:} 2023-11-16T18:35:43.748 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2023-11-16T18:35:43.748 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2023-11-16T18:35:43.749 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2023-11-16T18:35:43.749 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2023-11-16T18:35:43.749 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2023-11-16T18:35:43.749 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2023-11-16T18:35:43.749 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2023-11-16T18:35:43.749 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2023-11-16T18:35:43.750 DEBUG:tasks.check_counter:Getting stats from i 2023-11-16T18:35:43.750 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.i perf dump 2023-11-16T18:35:44.389 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:35:44.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:44 smithi188 ceph-mon[100639]: pgmap v341: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:35:44.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:44 smithi002 ceph-mon[98807]: pgmap v341: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:35:44.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:44 smithi040 ceph-mon[100251]: pgmap v341: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-11-16T18:35:45.678 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2023-11-16T18:35:45.679 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T18:35:45.679 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 149124, 2023-11-16T18:35:45.679 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 148352, 2023-11-16T18:35:45.679 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 84518776, 2023-11-16T18:35:45.679 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 429398951, 2023-11-16T18:35:45.679 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 12, 2023-11-16T18:35:45.680 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 4, 2023-11-16T18:35:45.680 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 6.919167766, 2023-11-16T18:35:45.680 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 2.627853150, 2023-11-16T18:35:45.680 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 4.147644656, 2023-11-16T18:35:45.680 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.828562613, 2023-11-16T18:35:45.680 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:45.680 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 148352, 2023-11-16T18:35:45.680 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 13.286002624, 2023-11-16T18:35:45.681 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000089557 2023-11-16T18:35:45.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.681 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:45.681 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 250682, 2023-11-16T18:35:45.681 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.031183130, 2023-11-16T18:35:45.681 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000124 2023-11-16T18:35:45.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.681 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T18:35:45.682 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T18:35:45.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.682 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T18:35:45.682 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 20141, 2023-11-16T18:35:45.682 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 20333, 2023-11-16T18:35:45.682 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 6475072, 2023-11-16T18:35:45.682 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 182039117, 2023-11-16T18:35:45.683 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 21, 2023-11-16T18:35:45.683 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 7, 2023-11-16T18:35:45.683 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 1.765938185, 2023-11-16T18:35:45.683 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.676879897, 2023-11-16T18:35:45.683 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.285731091, 2023-11-16T18:35:45.683 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.715904260, 2023-11-16T18:35:45.683 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:45.683 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 20330, 2023-11-16T18:35:45.684 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 6.690848673, 2023-11-16T18:35:45.684 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000329112 2023-11-16T18:35:45.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.684 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:45.684 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 75, 2023-11-16T18:35:45.684 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000010858, 2023-11-16T18:35:45.684 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000144 2023-11-16T18:35:45.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.685 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 4432, 2023-11-16T18:35:45.685 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 445456 2023-11-16T18:35:45.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.685 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T18:35:45.685 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 143847, 2023-11-16T18:35:45.685 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 260314, 2023-11-16T18:35:45.685 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 49632480, 2023-11-16T18:35:45.685 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 278932093, 2023-11-16T18:35:45.686 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 12, 2023-11-16T18:35:45.686 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 7, 2023-11-16T18:35:45.687 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 8.517990666, 2023-11-16T18:35:45.687 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 3.654870790, 2023-11-16T18:35:45.687 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 163.125810501, 2023-11-16T18:35:45.688 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 1.000160700, 2023-11-16T18:35:45.688 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:45.688 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 260312, 2023-11-16T18:35:45.688 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 19.264386623, 2023-11-16T18:35:45.688 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000074004 2023-11-16T18:35:45.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.690 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:45.690 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 305379, 2023-11-16T18:35:45.691 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.048004847, 2023-11-16T18:35:45.691 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000157 2023-11-16T18:35:45.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.692 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 1520096, 2023-11-16T18:35:45.692 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 130880 2023-11-16T18:35:45.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.692 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2023-11-16T18:35:45.692 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 1, 2023-11-16T18:35:45.692 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2023-11-16T18:35:45.694 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.694 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-MDSRank": { 2023-11-16T18:35:45.694 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2023-11-16T18:35:45.694 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2023-11-16T18:35:45.694 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 15369, 2023-11-16T18:35:45.695 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 99.805030985, 2023-11-16T18:35:45.695 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.006493918 2023-11-16T18:35:45.695 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:45.695 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.696 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-PurgeQueue": { 2023-11-16T18:35:45.696 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2023-11-16T18:35:45.696 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2023-11-16T18:35:45.697 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 36823, 2023-11-16T18:35:45.697 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 14.693403251, 2023-11-16T18:35:45.697 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000399027 2023-11-16T18:35:45.697 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:45.698 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.698 INFO:teuthology.orchestra.run.smithi188.stdout: "mds": { 2023-11-16T18:35:45.698 INFO:teuthology.orchestra.run.smithi188.stdout: "request": 82477, 2023-11-16T18:35:45.710 INFO:teuthology.orchestra.run.smithi188.stdout: "reply": 75519, 2023-11-16T18:35:45.710 INFO:teuthology.orchestra.run.smithi188.stdout: "reply_latency": { 2023-11-16T18:35:45.710 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 75519, 2023-11-16T18:35:45.710 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 263.347887152, 2023-11-16T18:35:45.710 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.003487173 2023-11-16T18:35:45.711 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.711 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_reply": 0, 2023-11-16T18:35:45.711 INFO:teuthology.orchestra.run.smithi188.stdout: "forward": 6958, 2023-11-16T18:35:45.711 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_complete": 236, 2023-11-16T18:35:45.711 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_keys": 0, 2023-11-16T18:35:45.711 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_commit": 1355, 2023-11-16T18:35:45.711 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_split": 168, 2023-11-16T18:35:45.712 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_merge": 0, 2023-11-16T18:35:45.712 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes": 83850, 2023-11-16T18:35:45.712 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_top": 6674, 2023-11-16T18:35:45.712 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_bottom": 8147, 2023-11-16T18:35:45.712 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pin_tail": 69029, 2023-11-16T18:35:45.713 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pinned": 74313, 2023-11-16T18:35:45.714 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_expired": 39266, 2023-11-16T18:35:45.714 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_with_caps": 2, 2023-11-16T18:35:45.714 INFO:teuthology.orchestra.run.smithi188.stdout: "caps": 2, 2023-11-16T18:35:45.714 INFO:teuthology.orchestra.run.smithi188.stdout: "subtrees": 3, 2023-11-16T18:35:45.714 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse": 105616, 2023-11-16T18:35:45.714 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_hit": 98257, 2023-11-16T18:35:45.715 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_forward": 6086, 2023-11-16T18:35:45.715 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_discover": 67, 2023-11-16T18:35:45.715 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_dir_fetch": 137, 2023-11-16T18:35:45.715 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_remote_ino": 0, 2023-11-16T18:35:45.715 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_lock": 0, 2023-11-16T18:35:45.715 INFO:teuthology.orchestra.run.smithi188.stdout: "load_cent": 1000, 2023-11-16T18:35:45.715 INFO:teuthology.orchestra.run.smithi188.stdout: "q": 1, 2023-11-16T18:35:45.716 INFO:teuthology.orchestra.run.smithi188.stdout: "exported": 1051, 2023-11-16T18:35:45.716 INFO:teuthology.orchestra.run.smithi188.stdout: "exported_inodes": 55788, 2023-11-16T18:35:45.716 INFO:teuthology.orchestra.run.smithi188.stdout: "imported": 1472, 2023-11-16T18:35:45.716 INFO:teuthology.orchestra.run.smithi188.stdout: "imported_inodes": 0, 2023-11-16T18:35:45.717 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_dir_fetch": 0, 2023-11-16T18:35:45.717 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_backtrace_fetch": 18702, 2023-11-16T18:35:45.717 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_peer_discover": 0, 2023-11-16T18:35:45.717 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rfiles": 6, 2023-11-16T18:35:45.718 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rbytes": 246, 2023-11-16T18:35:45.719 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rsnaps": 6, 2023-11-16T18:35:45.719 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_fetch": 18702, 2023-11-16T18:35:45.719 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_set_tag": 0, 2023-11-16T18:35:45.719 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T18:35:45.720 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_inotable_repaired": 0, 2023-11-16T18:35:45.720 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_inodes": 3, 2023-11-16T18:35:45.720 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_base_inodes": 0, 2023-11-16T18:35:45.720 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dirfrag_rstats": 3, 2023-11-16T18:35:45.721 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_file_inodes": 18699, 2023-11-16T18:35:45.721 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_inode_file_caps": 7, 2023-11-16T18:35:45.722 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_revoke": 58, 2023-11-16T18:35:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_grant": 50637, 2023-11-16T18:35:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_trunc": 0, 2023-11-16T18:35:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-16T18:35:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T18:35:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps": 29709, 2023-11-16T18:35:45.723 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps_dirty": 29650, 2023-11-16T18:35:45.724 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_cap_release": 295, 2023-11-16T18:35:45.724 INFO:teuthology.orchestra.run.smithi188.stdout: "process_request_cap_release": 37981 2023-11-16T18:35:45.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.732 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_cache": { 2023-11-16T18:35:45.732 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update": 1056, 2023-11-16T18:35:45.734 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update_receipt": 983, 2023-11-16T18:35:45.734 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_try_discover": 494, 2023-11-16T18:35:45.734 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_send_discover": 754, 2023-11-16T18:35:45.735 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_handle_discover": 1060, 2023-11-16T18:35:45.735 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays": 4, 2023-11-16T18:35:45.736 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_delayed": 0, 2023-11-16T18:35:45.736 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_enqueuing": 0, 2023-11-16T18:35:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_created": 22242, 2023-11-16T18:35:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_enqueued": 22238, 2023-11-16T18:35:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_reintegrated": 0, 2023-11-16T18:35:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_migrated": 0, 2023-11-16T18:35:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_processing": 0, 2023-11-16T18:35:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_enqueued": 0, 2023-11-16T18:35:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_prioritized": 0, 2023-11-16T18:35:45.737 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_started": 0, 2023-11-16T18:35:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_completed": 0, 2023-11-16T18:35:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_enqueue_scrub": 0, 2023-11-16T18:35:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_exportdir": 1096, 2023-11-16T18:35:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_flush": 0, 2023-11-16T18:35:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragmentdir": 174, 2023-11-16T18:35:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragstats": 0, 2023-11-16T18:35:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_inodestats": 0 2023-11-16T18:35:45.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_log": { 2023-11-16T18:35:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: "evadd": 107525, 2023-11-16T18:35:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: "evex": 25171, 2023-11-16T18:35:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: "evtrm": 25171, 2023-11-16T18:35:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: "ev": 82354, 2023-11-16T18:35:45.739 INFO:teuthology.orchestra.run.smithi188.stdout: "evexg": 0, 2023-11-16T18:35:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: "evexd": 1024, 2023-11-16T18:35:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: "segadd": 160, 2023-11-16T18:35:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: "segex": 32, 2023-11-16T18:35:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: "segtrm": 32, 2023-11-16T18:35:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: "seg": 129, 2023-11-16T18:35:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: "segexg": 0, 2023-11-16T18:35:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: "segexd": 1, 2023-11-16T18:35:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: "expos": 74481386, 2023-11-16T18:35:45.740 INFO:teuthology.orchestra.run.smithi188.stdout: "wrpos": 489926788, 2023-11-16T18:35:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: "rdpos": 0, 2023-11-16T18:35:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: "jlat": { 2023-11-16T18:35:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 3813, 2023-11-16T18:35:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 381.810591001, 2023-11-16T18:35:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.100133907 2023-11-16T18:35:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.741 INFO:teuthology.orchestra.run.smithi188.stdout: "replayed": 0 2023-11-16T18:35:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_mem": { 2023-11-16T18:35:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: "ino": 46405, 2023-11-16T18:35:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: "ino+": 105622, 2023-11-16T18:35:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: "ino-": 59217, 2023-11-16T18:35:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: "dir": 2059, 2023-11-16T18:35:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: "dir+": 3304, 2023-11-16T18:35:45.742 INFO:teuthology.orchestra.run.smithi188.stdout: "dir-": 1245, 2023-11-16T18:35:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: "dn": 83850, 2023-11-16T18:35:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: "dn+": 145372, 2023-11-16T18:35:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: "dn-": 61522, 2023-11-16T18:35:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: "cap": 2, 2023-11-16T18:35:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: "cap+": 52172, 2023-11-16T18:35:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: "cap-": 52170, 2023-11-16T18:35:45.743 INFO:teuthology.orchestra.run.smithi188.stdout: "rss": 359636, 2023-11-16T18:35:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: "heap": 198964 2023-11-16T18:35:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.744 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_server": { 2023-11-16T18:35:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_client_request": 85090, 2023-11-16T18:35:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_server_request": 668, 2023-11-16T18:35:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_request": 82477, 2023-11-16T18:35:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_session": 109, 2023-11-16T18:35:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_peer_request": 1363, 2023-11-16T18:35:45.745 INFO:teuthology.orchestra.run.smithi188.stdout: "req_create_latency": { 2023-11-16T18:35:45.746 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 28416, 2023-11-16T18:35:45.746 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 98.189550476, 2023-11-16T18:35:45.746 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.003455431 2023-11-16T18:35:45.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.746 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getattr_latency": { 2023-11-16T18:35:45.746 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 11, 2023-11-16T18:35:45.746 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.017446767, 2023-11-16T18:35:45.747 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001586069 2023-11-16T18:35:45.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.747 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getfilelock_latency": { 2023-11-16T18:35:45.747 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.747 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.747 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.748 INFO:teuthology.orchestra.run.smithi188.stdout: "req_link_latency": { 2023-11-16T18:35:45.748 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.748 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.748 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.748 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookup_latency": { 2023-11-16T18:35:45.748 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 22235, 2023-11-16T18:35:45.748 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 52.310326918, 2023-11-16T18:35:45.749 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.002352611 2023-11-16T18:35:45.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.749 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookuphash_latency": { 2023-11-16T18:35:45.749 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.749 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.749 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.749 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupino_latency": { 2023-11-16T18:35:45.750 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1, 2023-11-16T18:35:45.750 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000162816, 2023-11-16T18:35:45.750 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000162816 2023-11-16T18:35:45.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.750 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupname_latency": { 2023-11-16T18:35:45.751 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.751 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.752 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.752 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.752 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupparent_latency": { 2023-11-16T18:35:45.752 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.752 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.753 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.753 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupsnap_latency": { 2023-11-16T18:35:45.753 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.754 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.754 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.754 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lssnap_latency": { 2023-11-16T18:35:45.755 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.755 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.755 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.756 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mkdir_latency": { 2023-11-16T18:35:45.756 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.756 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.756 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.757 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mknod_latency": { 2023-11-16T18:35:45.757 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.757 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.757 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.757 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mksnap_latency": { 2023-11-16T18:35:45.757 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.758 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.758 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.758 INFO:teuthology.orchestra.run.smithi188.stdout: "req_open_latency": { 2023-11-16T18:35:45.758 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.758 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.758 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.759 INFO:teuthology.orchestra.run.smithi188.stdout: "req_readdir_latency": { 2023-11-16T18:35:45.759 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 2614, 2023-11-16T18:35:45.759 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 31.118199866, 2023-11-16T18:35:45.759 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.011904437 2023-11-16T18:35:45.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.759 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rename_latency": { 2023-11-16T18:35:45.759 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.759 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.760 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.760 INFO:teuthology.orchestra.run.smithi188.stdout: "req_renamesnap_latency": { 2023-11-16T18:35:45.760 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.760 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.760 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.760 INFO:teuthology.orchestra.run.smithi188.stdout: "req_snapdiff_latency": { 2023-11-16T18:35:45.761 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.761 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.761 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.761 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmdir_latency": { 2023-11-16T18:35:45.761 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 4, 2023-11-16T18:35:45.761 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.144388338, 2023-11-16T18:35:45.761 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.036097084 2023-11-16T18:35:45.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.762 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmsnap_latency": { 2023-11-16T18:35:45.762 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.762 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.762 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.762 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmxattr_latency": { 2023-11-16T18:35:45.762 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.763 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.763 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.763 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setattr_latency": { 2023-11-16T18:35:45.763 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.763 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.763 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.764 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setdirlayout_latency": { 2023-11-16T18:35:45.764 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.764 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.764 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.764 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setfilelock_latency": { 2023-11-16T18:35:45.764 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.765 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.765 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.765 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setlayout_latency": { 2023-11-16T18:35:45.765 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.765 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.765 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.766 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setxattr_latency": { 2023-11-16T18:35:45.766 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.766 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.766 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.767 INFO:teuthology.orchestra.run.smithi188.stdout: "req_symlink_latency": { 2023-11-16T18:35:45.767 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.767 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.767 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.768 INFO:teuthology.orchestra.run.smithi188.stdout: "req_unlink_latency": { 2023-11-16T18:35:45.768 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 22238, 2023-11-16T18:35:45.768 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 81.567811971, 2023-11-16T18:35:45.769 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.003667947 2023-11-16T18:35:45.769 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.769 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_revoke_eviction": 0, 2023-11-16T18:35:45.769 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_acquisition_throttle": 0, 2023-11-16T18:35:45.770 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getvxattr_latency": { 2023-11-16T18:35:45.770 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.770 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.770 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.771 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:45.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.771 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_sessions": { 2023-11-16T18:35:45.771 INFO:teuthology.orchestra.run.smithi188.stdout: "session_count": 2, 2023-11-16T18:35:45.772 INFO:teuthology.orchestra.run.smithi188.stdout: "session_add": 3, 2023-11-16T18:35:45.772 INFO:teuthology.orchestra.run.smithi188.stdout: "session_remove": 1, 2023-11-16T18:35:45.772 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_open": 2, 2023-11-16T18:35:45.772 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_stale": 0, 2023-11-16T18:35:45.773 INFO:teuthology.orchestra.run.smithi188.stdout: "total_load": 11663, 2023-11-16T18:35:45.773 INFO:teuthology.orchestra.run.smithi188.stdout: "average_load": 5831, 2023-11-16T18:35:45.773 INFO:teuthology.orchestra.run.smithi188.stdout: "avg_session_uptime": 657 2023-11-16T18:35:45.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.774 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2023-11-16T18:35:45.774 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2023-11-16T18:35:45.774 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2023-11-16T18:35:45.774 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T18:35:45.775 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2023-11-16T18:35:45.776 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T18:35:45.776 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2023-11-16T18:35:45.776 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T18:35:45.776 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T18:35:45.777 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T18:35:45.777 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T18:35:45.777 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T18:35:45.777 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2023-11-16T18:35:45.777 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T18:35:45.777 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T18:35:45.777 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2023-11-16T18:35:45.777 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2023-11-16T18:35:45.778 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2023-11-16T18:35:45.778 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2023-11-16T18:35:45.778 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T18:35:45.778 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T18:35:45.778 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T18:35:45.778 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T18:35:45.779 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T18:35:45.779 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2023-11-16T18:35:45.779 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2023-11-16T18:35:45.779 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2023-11-16T18:35:45.779 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T18:35:45.779 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T18:35:45.779 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2023-11-16T18:35:45.779 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2023-11-16T18:35:45.780 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2023-11-16T18:35:45.780 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2023-11-16T18:35:45.780 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T18:35:45.780 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2023-11-16T18:35:45.780 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T18:35:45.780 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2023-11-16T18:35:45.780 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 1796467, 2023-11-16T18:35:45.780 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 43291, 2023-11-16T18:35:45.781 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 440, 2023-11-16T18:35:45.781 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 5, 2023-11-16T18:35:45.781 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2023-11-16T18:35:45.781 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2023-11-16T18:35:45.781 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2023-11-16T18:35:45.781 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2023-11-16T18:35:45.781 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2023-11-16T18:35:45.781 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2023-11-16T18:35:45.782 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 9584, 2023-11-16T18:35:45.782 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 183, 2023-11-16T18:35:45.782 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T18:35:45.782 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2023-11-16T18:35:45.782 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2023-11-16T18:35:45.782 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2023-11-16T18:35:45.782 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 167451592, 2023-11-16T18:35:45.782 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 1572426, 2023-11-16T18:35:45.783 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2023-11-16T18:35:45.783 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2023-11-16T18:35:45.783 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2023-11-16T18:35:45.783 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2023-11-16T18:35:45.783 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.783 INFO:teuthology.orchestra.run.smithi188.stdout: "objecter": { 2023-11-16T18:35:45.783 INFO:teuthology.orchestra.run.smithi188.stdout: "op_active": 0, 2023-11-16T18:35:45.783 INFO:teuthology.orchestra.run.smithi188.stdout: "op_laggy": 0, 2023-11-16T18:35:45.784 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send": 72632, 2023-11-16T18:35:45.784 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send_bytes": 506194016, 2023-11-16T18:35:45.784 INFO:teuthology.orchestra.run.smithi188.stdout: "op_resend": 28, 2023-11-16T18:35:45.784 INFO:teuthology.orchestra.run.smithi188.stdout: "op_reply": 72632, 2023-11-16T18:35:45.784 INFO:teuthology.orchestra.run.smithi188.stdout: "op_latency": { 2023-11-16T18:35:45.784 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 72632, 2023-11-16T18:35:45.784 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 704.794131949, 2023-11-16T18:35:45.784 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.009703631 2023-11-16T18:35:45.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.785 INFO:teuthology.orchestra.run.smithi188.stdout: "op_inflight": 0, 2023-11-16T18:35:45.785 INFO:teuthology.orchestra.run.smithi188.stdout: "oplen_avg": { 2023-11-16T18:35:45.785 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 72632, 2023-11-16T18:35:45.785 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 78767 2023-11-16T18:35:45.785 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.785 INFO:teuthology.orchestra.run.smithi188.stdout: "op": 72632, 2023-11-16T18:35:45.785 INFO:teuthology.orchestra.run.smithi188.stdout: "op_r": 19220, 2023-11-16T18:35:45.785 INFO:teuthology.orchestra.run.smithi188.stdout: "op_w": 53412, 2023-11-16T18:35:45.786 INFO:teuthology.orchestra.run.smithi188.stdout: "op_rmw": 0, 2023-11-16T18:35:45.786 INFO:teuthology.orchestra.run.smithi188.stdout: "op_pg": 0, 2023-11-16T18:35:45.786 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_stat": 0, 2023-11-16T18:35:45.786 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_create": 2974, 2023-11-16T18:35:45.786 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_read": 282, 2023-11-16T18:35:45.786 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_write": 4095, 2023-11-16T18:35:45.786 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writefull": 135, 2023-11-16T18:35:45.786 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writesame": 0, 2023-11-16T18:35:45.787 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_append": 0, 2023-11-16T18:35:45.787 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_zero": 0, 2023-11-16T18:35:45.787 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_truncate": 0, 2023-11-16T18:35:45.787 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_delete": 44787, 2023-11-16T18:35:45.787 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_mapext": 0, 2023-11-16T18:35:45.787 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_sparse_read": 0, 2023-11-16T18:35:45.787 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_clonerange": 0, 2023-11-16T18:35:45.787 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_getxattr": 18702, 2023-11-16T18:35:45.788 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_setxattr": 4466, 2023-11-16T18:35:45.788 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_cmpxattr": 0, 2023-11-16T18:35:45.788 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_rmxattr": 0, 2023-11-16T18:35:45.788 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_resetxattrs": 0, 2023-11-16T18:35:45.788 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_call": 0, 2023-11-16T18:35:45.788 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_watch": 0, 2023-11-16T18:35:45.788 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_notify": 0, 2023-11-16T18:35:45.788 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T18:35:45.788 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls": 0, 2023-11-16T18:35:45.789 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls_filter": 0, 2023-11-16T18:35:45.789 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_other": 0, 2023-11-16T18:35:45.789 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_active": 0, 2023-11-16T18:35:45.789 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_send": 0, 2023-11-16T18:35:45.789 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_resend": 0, 2023-11-16T18:35:45.789 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_ping": 0, 2023-11-16T18:35:45.789 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_active": 0, 2023-11-16T18:35:45.789 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_send": 0, 2023-11-16T18:35:45.790 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_resend": 0, 2023-11-16T18:35:45.790 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_active": 0, 2023-11-16T18:35:45.791 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_send": 0, 2023-11-16T18:35:45.791 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_resend": 0, 2023-11-16T18:35:45.791 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_active": 0, 2023-11-16T18:35:45.791 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_send": 0, 2023-11-16T18:35:45.791 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_resend": 0, 2023-11-16T18:35:45.791 INFO:teuthology.orchestra.run.smithi188.stdout: "command_active": 0, 2023-11-16T18:35:45.791 INFO:teuthology.orchestra.run.smithi188.stdout: "command_send": 0, 2023-11-16T18:35:45.791 INFO:teuthology.orchestra.run.smithi188.stdout: "command_resend": 0, 2023-11-16T18:35:45.792 INFO:teuthology.orchestra.run.smithi188.stdout: "map_epoch": 103, 2023-11-16T18:35:45.792 INFO:teuthology.orchestra.run.smithi188.stdout: "map_full": 0, 2023-11-16T18:35:45.792 INFO:teuthology.orchestra.run.smithi188.stdout: "map_inc": 17, 2023-11-16T18:35:45.792 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_sessions": 12, 2023-11-16T18:35:45.792 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_open": 12, 2023-11-16T18:35:45.792 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_close": 0, 2023-11-16T18:35:45.792 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_laggy": 0, 2023-11-16T18:35:45.792 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_wr": 2813, 2023-11-16T18:35:45.793 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_rd": 472, 2023-11-16T18:35:45.793 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_del": 41 2023-11-16T18:35:45.793 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.793 INFO:teuthology.orchestra.run.smithi188.stdout: "oft": { 2023-11-16T18:35:45.793 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_objs": 1, 2023-11-16T18:35:45.793 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_kv_pairs": 0, 2023-11-16T18:35:45.793 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_updates": 0, 2023-11-16T18:35:45.793 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_removes": 0 2023-11-16T18:35:45.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.794 INFO:teuthology.orchestra.run.smithi188.stdout: "osd-slow-ops": { 2023-11-16T18:35:45.794 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_ops_count": 84417 2023-11-16T18:35:45.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.794 INFO:teuthology.orchestra.run.smithi188.stdout: "purge_queue": { 2023-11-16T18:35:45.794 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops": 0, 2023-11-16T18:35:45.794 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops_high_water": 16, 2023-11-16T18:35:45.794 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing": 0, 2023-11-16T18:35:45.795 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_high_water": 8, 2023-11-16T18:35:45.795 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed": 22238, 2023-11-16T18:35:45.795 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_item_in_journal": 0 2023-11-16T18:35:45.795 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.795 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T18:35:45.795 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:45.795 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2023-11-16T18:35:45.795 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:45.795 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 313108, 2023-11-16T18:35:45.796 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 112428266, 2023-11-16T18:35:45.796 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:45.796 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 313108, 2023-11-16T18:35:45.796 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2023-11-16T18:35:45.796 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2023-11-16T18:35:45.796 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 313108, 2023-11-16T18:35:45.796 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 112428266, 2023-11-16T18:35:45.796 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:45.797 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.797 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.797 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.797 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:45.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.797 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_bytes": { 2023-11-16T18:35:45.797 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:45.797 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2023-11-16T18:35:45.797 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:45.798 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2023-11-16T18:35:45.798 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2023-11-16T18:35:45.798 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:45.798 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:45.798 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 72632, 2023-11-16T18:35:45.798 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 508947998, 2023-11-16T18:35:45.798 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 72632, 2023-11-16T18:35:45.799 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 508947998, 2023-11-16T18:35:45.799 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:45.799 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.799 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.799 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.799 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:45.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.799 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_ops": { 2023-11-16T18:35:45.799 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:45.800 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 1024, 2023-11-16T18:35:45.800 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:45.801 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2023-11-16T18:35:45.801 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2023-11-16T18:35:45.801 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:45.801 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2023-11-16T18:35:45.801 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 72632, 2023-11-16T18:35:45.801 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 72632, 2023-11-16T18:35:45.801 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 72632, 2023-11-16T18:35:45.801 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 72632, 2023-11-16T18:35:45.802 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:45.802 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.802 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.802 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.802 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:45.802 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.802 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle": { 2023-11-16T18:35:45.802 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:45.803 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2023-11-16T18:35:45.803 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:45.803 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 22238, 2023-11-16T18:35:45.803 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 2757758, 2023-11-16T18:35:45.803 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:45.803 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 22238, 2023-11-16T18:35:45.803 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2023-11-16T18:35:45.803 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2023-11-16T18:35:45.804 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 282, 2023-11-16T18:35:45.804 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 2757758, 2023-11-16T18:35:45.804 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:45.804 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.804 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.804 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.804 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:45.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:45.805 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle-0x55ab45edfcc0": { 2023-11-16T18:35:45.805 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:45.805 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2023-11-16T18:35:45.805 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:45.805 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 107525, 2023-11-16T18:35:45.805 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 485732484, 2023-11-16T18:35:45.805 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:45.805 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 107525, 2023-11-16T18:35:45.805 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2023-11-16T18:35:45.806 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2023-11-16T18:35:45.806 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 3813, 2023-11-16T18:35:45.806 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 485732484, 2023-11-16T18:35:45.806 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:45.806 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:45.806 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:45.806 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:45.806 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:45.807 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:45.807 INFO:teuthology.orchestra.run.smithi188.stdout:} 2023-11-16T18:35:46.134 INFO:tasks.check_counter:Daemon mds.i mds.exported=1051 2023-11-16T18:35:46.134 INFO:tasks.check_counter:Daemon mds.i mds.imported=1472 2023-11-16T18:35:46.134 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=1056 2023-11-16T18:35:46.134 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=983 2023-11-16T18:35:46.134 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2023-11-16T18:35:46.134 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2023-11-16T18:35:46.135 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2023-11-16T18:35:46.135 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=168 2023-11-16T18:35:46.135 DEBUG:tasks.check_counter:Getting stats from l 2023-11-16T18:35:46.135 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.l perf dump 2023-11-16T18:35:46.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:46 smithi188 ceph-mon[100639]: pgmap v342: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2023-11-16T18:35:46.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:46 smithi002 ceph-mon[98807]: pgmap v342: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2023-11-16T18:35:46.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:46 smithi040 ceph-mon[100251]: pgmap v342: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2023-11-16T18:35:46.775 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T18:35:47.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:47 smithi188 ceph-mon[100639]: pgmap v343: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:47.733 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:47 smithi040 ceph-mon[100251]: pgmap v343: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:47.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:47 smithi002 ceph-mon[98807]: pgmap v343: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:48.022 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2023-11-16T18:35:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T18:35:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 0, 2023-11-16T18:35:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 2, 2023-11-16T18:35:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 0, 2023-11-16T18:35:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 0, 2023-11-16T18:35:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 4, 2023-11-16T18:35:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 0, 2023-11-16T18:35:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.001216075, 2023-11-16T18:35:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T18:35:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.000366245, 2023-11-16T18:35:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T18:35:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T18:35:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T18:35:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T18:35:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 227, 2023-11-16T18:35:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 217, 2023-11-16T18:35:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 1423072, 2023-11-16T18:35:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 123888, 2023-11-16T18:35:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 2, 2023-11-16T18:35:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 1, 2023-11-16T18:35:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.042722145, 2023-11-16T18:35:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.013240366, 2023-11-16T18:35:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.031724496, 2023-11-16T18:35:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.005823627, 2023-11-16T18:35:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 216, 2023-11-16T18:35:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.012992241, 2023-11-16T18:35:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000060149 2023-11-16T18:35:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 1423072, 2023-11-16T18:35:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 123888 2023-11-16T18:35:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T18:35:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 6, 2023-11-16T18:35:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 169, 2023-11-16T18:35:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 3312, 2023-11-16T18:35:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 181536, 2023-11-16T18:35:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 5, 2023-11-16T18:35:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 3, 2023-11-16T18:35:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.010130556, 2023-11-16T18:35:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.006179229, 2023-11-16T18:35:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.001865955, 2023-11-16T18:35:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T18:35:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T18:35:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 167, 2023-11-16T18:35:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.020244347, 2023-11-16T18:35:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000121223 2023-11-16T18:35:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2023-11-16T18:35:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-11-16T18:35:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 181536 2023-11-16T18:35:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2023-11-16T18:35:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 0, 2023-11-16T18:35:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2023-11-16T18:35:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2023-11-16T18:35:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2023-11-16T18:35:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2023-11-16T18:35:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T18:35:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2023-11-16T18:35:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T18:35:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2023-11-16T18:35:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T18:35:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T18:35:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T18:35:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T18:35:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T18:35:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2023-11-16T18:35:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T18:35:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T18:35:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2023-11-16T18:35:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2023-11-16T18:35:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2023-11-16T18:35:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2023-11-16T18:35:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T18:35:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T18:35:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T18:35:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T18:35:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T18:35:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2023-11-16T18:35:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2023-11-16T18:35:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2023-11-16T18:35:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T18:35:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T18:35:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2023-11-16T18:35:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2023-11-16T18:35:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2023-11-16T18:35:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2023-11-16T18:35:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T18:35:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2023-11-16T18:35:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T18:35:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2023-11-16T18:35:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 149034, 2023-11-16T18:35:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 32, 2023-11-16T18:35:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 0, 2023-11-16T18:35:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 0, 2023-11-16T18:35:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2023-11-16T18:35:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2023-11-16T18:35:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2023-11-16T18:35:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2023-11-16T18:35:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2023-11-16T18:35:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2023-11-16T18:35:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 0, 2023-11-16T18:35:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 0, 2023-11-16T18:35:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T18:35:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2023-11-16T18:35:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2023-11-16T18:35:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2023-11-16T18:35:48.039 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 928, 2023-11-16T18:35:48.039 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 2, 2023-11-16T18:35:48.039 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2023-11-16T18:35:48.039 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2023-11-16T18:35:48.040 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2023-11-16T18:35:48.040 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2023-11-16T18:35:48.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2023-11-16T18:35:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T18:35:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2023-11-16T18:35:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2023-11-16T18:35:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2023-11-16T18:35:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 229, 2023-11-16T18:35:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 1391019, 2023-11-16T18:35:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2023-11-16T18:35:48.042 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 229, 2023-11-16T18:35:48.042 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2023-11-16T18:35:48.042 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2023-11-16T18:35:48.042 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 229, 2023-11-16T18:35:48.042 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 1391019, 2023-11-16T18:35:48.043 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2023-11-16T18:35:48.043 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2023-11-16T18:35:48.043 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2023-11-16T18:35:48.043 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2023-11-16T18:35:48.044 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:48.044 INFO:teuthology.orchestra.run.smithi188.stdout: } 2023-11-16T18:35:48.044 INFO:teuthology.orchestra.run.smithi188.stdout:} 2023-11-16T18:35:48.537 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2023-11-16T18:35:48.548 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2023-11-16T18:35:48.548 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2023-11-16T18:35:48.549 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2023-11-16T18:35:48.549 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2023-11-16T18:35:48.549 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2023-11-16T18:35:48.549 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2023-11-16T18:35:48.549 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2023-11-16T18:35:48.550 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-11-16T18:35:48.568 INFO:tasks.kclient:Unmounting kernel clients... 2023-11-16T18:35:48.568 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:35:48.569 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T18:35:48.601 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:35:48.602 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T18:35:48.665 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:35:48.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.856+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/3291835276 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e68100a50 msgr2=0x7f4e68100ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:48.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.856+0000 7f4e6d6e5700 1 --2- 172.21.15.2:0/3291835276 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e68100a50 0x7f4e68100ea0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c009990 tx=0x7f4e5c009ca0 comp rx=0 tx=0).stop 2023-11-16T18:35:48.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.856+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/3291835276 shutdown_connections 2023-11-16T18:35:48.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.856+0000 7f4e6d6e5700 1 --2- 172.21.15.2:0/3291835276 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e681013e0 0x7f4e6810d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:48.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.856+0000 7f4e6d6e5700 1 --2- 172.21.15.2:0/3291835276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e681062d0 0x7f4e681066a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:48.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.856+0000 7f4e6d6e5700 1 --2- 172.21.15.2:0/3291835276 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e68100a50 0x7f4e68100ea0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:48.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.856+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/3291835276 >> 172.21.15.2:0/3291835276 conn(0x7f4e680774a0 msgr2=0x7f4e680778a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:48.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.856+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/3291835276 shutdown_connections 2023-11-16T18:35:48.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.856+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/3291835276 wait complete. 2023-11-16T18:35:48.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.856+0000 7f4e6d6e5700 1 Processor -- start 2023-11-16T18:35:48.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.856+0000 7f4e6d6e5700 1 -- start start 2023-11-16T18:35:48.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e6d6e5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e68100a50 0x7f4e681963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:48.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e6d6e5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e681013e0 0x7f4e68196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:48.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e6d6e5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e681062d0 0x7f4e6819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:48.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e6d6e5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6819b360 con 0x7f4e681013e0 2023-11-16T18:35:48.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e6d6e5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6819b4d0 con 0x7f4e68100a50 2023-11-16T18:35:48.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e6d6e5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6819b690 con 0x7f4e681062d0 2023-11-16T18:35:48.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e667fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e681013e0 0x7f4e68196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:48.872 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e667fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e681013e0 0x7f4e68196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53404/0 (socket says 172.21.15.2:53404) 2023-11-16T18:35:48.872 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e667fc700 1 -- 172.21.15.2:0/40666966 learned_addr learned my addr 172.21.15.2:0/40666966 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:48.872 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e677fe700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e681062d0 0x7f4e6819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:48.872 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e66ffd700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e68100a50 0x7f4e681963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:48.872 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e667fc700 1 -- 172.21.15.2:0/40666966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e681062d0 msgr2=0x7f4e6819ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:48.872 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e667fc700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e681062d0 0x7f4e6819ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:48.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e667fc700 1 -- 172.21.15.2:0/40666966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e68100a50 msgr2=0x7f4e681963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:48.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e667fc700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e68100a50 0x7f4e681963d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:48.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e667fc700 1 -- 172.21.15.2:0/40666966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c0095f0 con 0x7f4e681013e0 2023-11-16T18:35:48.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e677fe700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e681062d0 0x7f4e6819ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:35:48.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e667fc700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e681013e0 0x7f4e68196910 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c000c00 tx=0x7f4e5c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:48.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.857+0000 7f4e4ffff700 1 -- 172.21.15.2:0/40666966 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c01d070 con 0x7f4e681013e0 2023-11-16T18:35:48.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.858+0000 7f4e4ffff700 1 -- 172.21.15.2:0/40666966 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e5c00fc30 con 0x7f4e681013e0 2023-11-16T18:35:48.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.858+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/40666966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e681102e0 con 0x7f4e681013e0 2023-11-16T18:35:48.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.858+0000 7f4e4ffff700 1 -- 172.21.15.2:0/40666966 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c017770 con 0x7f4e681013e0 2023-11-16T18:35:48.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.858+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/40666966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e68110150 con 0x7f4e681013e0 2023-11-16T18:35:48.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.859+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/40666966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e68110470 con 0x7f4e681013e0 2023-11-16T18:35:48.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.860+0000 7f4e4ffff700 1 -- 172.21.15.2:0/40666966 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f4e5c022a30 con 0x7f4e681013e0 2023-11-16T18:35:48.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.860+0000 7f4e4ffff700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f4e500a12c0 0x7f4e500a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:48.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.860+0000 7f4e4ffff700 1 -- 172.21.15.2:0/40666966 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9438+0+0 (secure 0 0 0) 0x7f4e5c0ce760 con 0x7f4e681013e0 2023-11-16T18:35:48.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.860+0000 7f4e66ffd700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f4e500a12c0 0x7f4e500a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:48.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.861+0000 7f4e66ffd700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f4e500a12c0 0x7f4e500a3770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4e58005950 tx=0x7f4e58005c70 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:48.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:48.864+0000 7f4e4ffff700 1 -- 172.21.15.2:0/40666966 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4e5c097870 con 0x7f4e681013e0 2023-11-16T18:35:49.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.067+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/40666966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e68067f40 con 0x7f4e681013e0 2023-11-16T18:35:49.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.068+0000 7f4e4ffff700 1 -- 172.21.15.2:0/40666966 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+25060 (secure 0 0 0) 0x7f4e5c09ae90 con 0x7f4e681013e0 2023-11-16T18:35:49.071 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:35:49.071 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":103,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:35:41.412881+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6800/3369431713":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/1472878779":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/1347603606":"2023-11-17T18:24:50.949710+0000","172.21.15.2:6801/3369431713":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/3335793429":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/3821858662":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:35:49.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.070+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/40666966 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f4e500a12c0 msgr2=0x7f4e500a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.070+0000 7f4e6d6e5700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f4e500a12c0 0x7f4e500a3770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4e58005950 tx=0x7f4e58005c70 comp rx=0 tx=0).stop 2023-11-16T18:35:49.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.070+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/40666966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e681013e0 msgr2=0x7f4e68196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.070+0000 7f4e6d6e5700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e681013e0 0x7f4e68196910 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c000c00 tx=0x7f4e5c00f790 comp rx=0 tx=0).stop 2023-11-16T18:35:49.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.070+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/40666966 shutdown_connections 2023-11-16T18:35:49.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.070+0000 7f4e6d6e5700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f4e500a12c0 0x7f4e500a3770 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.070+0000 7f4e6d6e5700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e681062d0 0x7f4e6819ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.070+0000 7f4e6d6e5700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e68100a50 0x7f4e681963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.070+0000 7f4e6d6e5700 1 --2- 172.21.15.2:0/40666966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e681013e0 0x7f4e68196910 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.070+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/40666966 >> 172.21.15.2:0/40666966 conn(0x7f4e680774a0 msgr2=0x7f4e680ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:49.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.071+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/40666966 shutdown_connections 2023-11-16T18:35:49.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.071+0000 7f4e6d6e5700 1 -- 172.21.15.2:0/40666966 wait complete. 2023-11-16T18:35:49.085 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T18:18:46.259592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T18:22:44.902889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T18:22:46.187275+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T18:22:49.539148+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T18:35:49.086 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T18:35:49.217 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.213+0000 7f43feb4d700 1 -- 172.21.15.2:0/1141064865 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43f8067dc0 msgr2=0x7f43f8068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.213+0000 7f43feb4d700 1 --2- 172.21.15.2:0/1141064865 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43f8067dc0 0x7f43f8068230 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f43e40099c0 tx=0x7f43e4009cd0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.213+0000 7f43feb4d700 1 -- 172.21.15.2:0/1141064865 shutdown_connections 2023-11-16T18:35:49.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.213+0000 7f43feb4d700 1 --2- 172.21.15.2:0/1141064865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f8068770 0x7f43f810d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.213+0000 7f43feb4d700 1 --2- 172.21.15.2:0/1141064865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f81052a0 0x7f43f8105670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.213+0000 7f43feb4d700 1 --2- 172.21.15.2:0/1141064865 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43f8067dc0 0x7f43f8068230 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.213+0000 7f43feb4d700 1 -- 172.21.15.2:0/1141064865 >> 172.21.15.2:0/1141064865 conn(0x7f43f80763a0 msgr2=0x7f43f80767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:49.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.213+0000 7f43feb4d700 1 -- 172.21.15.2:0/1141064865 shutdown_connections 2023-11-16T18:35:49.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.213+0000 7f43feb4d700 1 -- 172.21.15.2:0/1141064865 wait complete. 2023-11-16T18:35:49.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43feb4d700 1 Processor -- start 2023-11-16T18:35:49.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43feb4d700 1 -- start start 2023-11-16T18:35:49.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43feb4d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f8067dc0 0x7f43f8198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:49.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43feb4d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f8068770 0x7f43f8198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:49.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43feb4d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43f81052a0 0x7f43f819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:49.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43feb4d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f819d560 con 0x7f43f81052a0 2023-11-16T18:35:49.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43feb4d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f819d6d0 con 0x7f43f8068770 2023-11-16T18:35:49.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43feb4d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f819d890 con 0x7f43f8067dc0 2023-11-16T18:35:49.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43fd0ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43f81052a0 0x7f43f819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:49.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43fd0ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43f81052a0 0x7f43f819cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53420/0 (socket says 172.21.15.2:53420) 2023-11-16T18:35:49.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43fd0ea700 1 -- 172.21.15.2:0/3644153321 learned_addr learned my addr 172.21.15.2:0/3644153321 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:49.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43fc8e9700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f8067dc0 0x7f43f8198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:49.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43f7fff700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f8068770 0x7f43f8198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:49.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43fd0ea700 1 -- 172.21.15.2:0/3644153321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f8067dc0 msgr2=0x7f43f8198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43fd0ea700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f8067dc0 0x7f43f8198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43fd0ea700 1 -- 172.21.15.2:0/3644153321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f8068770 msgr2=0x7f43f8198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43fd0ea700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f8068770 0x7f43f8198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43fd0ea700 1 -- 172.21.15.2:0/3644153321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e40095f0 con 0x7f43f81052a0 2023-11-16T18:35:49.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.214+0000 7f43f7fff700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f8068770 0x7f43f8198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:35:49.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.215+0000 7f43fd0ea700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43f81052a0 0x7f43f819cf20 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f43f0007ee0 tx=0x7f43f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:49.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.215+0000 7f43f5ffb700 1 -- 172.21.15.2:0/3644153321 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0017070 con 0x7f43f81052a0 2023-11-16T18:35:49.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.215+0000 7f43f5ffb700 1 -- 172.21.15.2:0/3644153321 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43f000f040 con 0x7f43f81052a0 2023-11-16T18:35:49.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.215+0000 7f43feb4d700 1 -- 172.21.15.2:0/3644153321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f8110390 con 0x7f43f81052a0 2023-11-16T18:35:49.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.215+0000 7f43f5ffb700 1 -- 172.21.15.2:0/3644153321 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0005740 con 0x7f43f81052a0 2023-11-16T18:35:49.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.215+0000 7f43feb4d700 1 -- 172.21.15.2:0/3644153321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f8110200 con 0x7f43f81052a0 2023-11-16T18:35:49.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.216+0000 7f43feb4d700 1 -- 172.21.15.2:0/3644153321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f8110520 con 0x7f43f81052a0 2023-11-16T18:35:49.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.217+0000 7f43f5ffb700 1 -- 172.21.15.2:0/3644153321 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f43f000dc60 con 0x7f43f81052a0 2023-11-16T18:35:49.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.218+0000 7f43f5ffb700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f43e80a12c0 0x7f43e80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:49.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.218+0000 7f43fc8e9700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f43e80a12c0 0x7f43e80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:49.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.218+0000 7f43f5ffb700 1 -- 172.21.15.2:0/3644153321 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9438+0+0 (secure 0 0 0) 0x7f43f00cc8c0 con 0x7f43f81052a0 2023-11-16T18:35:49.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.218+0000 7f43fc8e9700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f43e80a12c0 0x7f43e80a3770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f43ec005950 tx=0x7f43ec00a300 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:49.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.222+0000 7f43f5ffb700 1 -- 172.21.15.2:0/3644153321 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f43f00958e0 con 0x7f43f81052a0 2023-11-16T18:35:49.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:49 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/40666966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:35:49.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.421+0000 7f43feb4d700 1 -- 172.21.15.2:0/3644153321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f43f81a5650 con 0x7f43f81052a0 2023-11-16T18:35:49.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.421+0000 7f43f5ffb700 1 -- 172.21.15.2:0/3644153321 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7f43f0098f00 con 0x7f43f81052a0 2023-11-16T18:35:49.426 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T18:35:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.423+0000 7f43feb4d700 1 -- 172.21.15.2:0/3644153321 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f43e80a12c0 msgr2=0x7f43e80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.423+0000 7f43feb4d700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f43e80a12c0 0x7f43e80a3770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f43ec005950 tx=0x7f43ec00a300 comp rx=0 tx=0).stop 2023-11-16T18:35:49.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.423+0000 7f43feb4d700 1 -- 172.21.15.2:0/3644153321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43f81052a0 msgr2=0x7f43f819cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.423+0000 7f43feb4d700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43f81052a0 0x7f43f819cf20 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f43f0007ee0 tx=0x7f43f000d3b0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.423+0000 7f43feb4d700 1 -- 172.21.15.2:0/3644153321 shutdown_connections 2023-11-16T18:35:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.423+0000 7f43feb4d700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f43e80a12c0 0x7f43e80a3770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.423+0000 7f43feb4d700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f8067dc0 0x7f43f8198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.423+0000 7f43feb4d700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43f81052a0 0x7f43f819cf20 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.423+0000 7f43feb4d700 1 --2- 172.21.15.2:0/3644153321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f8068770 0x7f43f8198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.423+0000 7f43feb4d700 1 -- 172.21.15.2:0/3644153321 >> 172.21.15.2:0/3644153321 conn(0x7f43f80763a0 msgr2=0x7f43f810b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.424+0000 7f43feb4d700 1 -- 172.21.15.2:0/3644153321 shutdown_connections 2023-11-16T18:35:49.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.424+0000 7f43feb4d700 1 -- 172.21.15.2:0/3644153321 wait complete. 2023-11-16T18:35:49.438 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T18:35:49.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:49 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/40666966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:35:49.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:49 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/40666966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:35:49.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.564+0000 7fd96bfa7700 1 -- 172.21.15.2:0/433097541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd964100420 msgr2=0x7fd96410bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.564+0000 7fd96bfa7700 1 --2- 172.21.15.2:0/433097541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd964100420 0x7fd96410bfd0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00b0a0 tx=0x7fd95c00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.564+0000 7fd96bfa7700 1 -- 172.21.15.2:0/433097541 shutdown_connections 2023-11-16T18:35:49.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.564+0000 7fd96bfa7700 1 --2- 172.21.15.2:0/433097541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd964100420 0x7fd96410bfd0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.564+0000 7fd96bfa7700 1 --2- 172.21.15.2:0/433097541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd964105250 0x7fd964105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.564+0000 7fd96bfa7700 1 --2- 172.21.15.2:0/433097541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9640ffa70 0x7fd9640ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.564+0000 7fd96bfa7700 1 -- 172.21.15.2:0/433097541 >> 172.21.15.2:0/433097541 conn(0x7fd9640763a0 msgr2=0x7fd9640767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:49.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.564+0000 7fd96bfa7700 1 -- 172.21.15.2:0/433097541 shutdown_connections 2023-11-16T18:35:49.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.564+0000 7fd96bfa7700 1 -- 172.21.15.2:0/433097541 wait complete. 2023-11-16T18:35:49.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd96bfa7700 1 Processor -- start 2023-11-16T18:35:49.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd96bfa7700 1 -- start start 2023-11-16T18:35:49.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd96bfa7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9640ffa70 0x7fd9641972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:49.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd96bfa7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd964100420 0x7fd964197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:49.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd96bfa7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd964105250 0x7fd9641a2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:49.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd96bfa7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd964198150 con 0x7fd9640ffa70 2023-11-16T18:35:49.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd96bfa7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9641a2890 con 0x7fd964105250 2023-11-16T18:35:49.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd96bfa7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9641a2a50 con 0x7fd964100420 2023-11-16T18:35:49.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd969d43700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9640ffa70 0x7fd9641972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:49.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd969d43700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9640ffa70 0x7fd9641972d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53430/0 (socket says 172.21.15.2:53430) 2023-11-16T18:35:49.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd969d43700 1 -- 172.21.15.2:0/973085819 learned_addr learned my addr 172.21.15.2:0/973085819 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:49.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd96a544700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd964105250 0x7fd9641a2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:49.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd969542700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd964100420 0x7fd964197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:49.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd969d43700 1 -- 172.21.15.2:0/973085819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd964100420 msgr2=0x7fd964197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd969d43700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd964100420 0x7fd964197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd969d43700 1 -- 172.21.15.2:0/973085819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd964105250 msgr2=0x7fd9641a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd969d43700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd964105250 0x7fd9641a2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.565+0000 7fd969d43700 1 -- 172.21.15.2:0/973085819 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c009d00 con 0x7fd9640ffa70 2023-11-16T18:35:49.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.566+0000 7fd969d43700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9640ffa70 0x7fd9641972d0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd95400e9a0 tx=0x7fd95400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:49.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.566+0000 7fd95affd700 1 -- 172.21.15.2:0/973085819 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954009790 con 0x7fd9640ffa70 2023-11-16T18:35:49.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.566+0000 7fd95affd700 1 -- 172.21.15.2:0/973085819 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd95400cbd0 con 0x7fd9640ffa70 2023-11-16T18:35:49.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.566+0000 7fd96bfa7700 1 -- 172.21.15.2:0/973085819 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96410f1e0 con 0x7fd9640ffa70 2023-11-16T18:35:49.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.566+0000 7fd95affd700 1 -- 172.21.15.2:0/973085819 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954018680 con 0x7fd9640ffa70 2023-11-16T18:35:49.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.566+0000 7fd96bfa7700 1 -- 172.21.15.2:0/973085819 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96410f050 con 0x7fd9640ffa70 2023-11-16T18:35:49.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.567+0000 7fd96bfa7700 1 -- 172.21.15.2:0/973085819 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96410f370 con 0x7fd9640ffa70 2023-11-16T18:35:49.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.568+0000 7fd95affd700 1 -- 172.21.15.2:0/973085819 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fd95400cd40 con 0x7fd9640ffa70 2023-11-16T18:35:49.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.569+0000 7fd95affd700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd9500a12c0 0x7fd9500a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:49.574 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.569+0000 7fd969542700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd9500a12c0 0x7fd9500a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:49.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.569+0000 7fd95affd700 1 -- 172.21.15.2:0/973085819 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9438+0+0 (secure 0 0 0) 0x7fd954014070 con 0x7fd9640ffa70 2023-11-16T18:35:49.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.569+0000 7fd969542700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd9500a12c0 0x7fd9500a3770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd960005950 tx=0x7fd9600058e0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:49.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.573+0000 7fd95affd700 1 -- 172.21.15.2:0/973085819 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd9540965b0 con 0x7fd9640ffa70 2023-11-16T18:35:49.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.772+0000 7fd96bfa7700 1 -- 172.21.15.2:0/973085819 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd9641a34d0 con 0x7fd9640ffa70 2023-11-16T18:35:49.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.773+0000 7fd95affd700 1 -- 172.21.15.2:0/973085819 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd954099bd0 con 0x7fd9640ffa70 2023-11-16T18:35:49.777 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:35:49.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.775+0000 7fd96bfa7700 1 -- 172.21.15.2:0/973085819 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd9500a12c0 msgr2=0x7fd9500a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.775+0000 7fd96bfa7700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd9500a12c0 0x7fd9500a3770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd960005950 tx=0x7fd9600058e0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.775+0000 7fd96bfa7700 1 -- 172.21.15.2:0/973085819 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9640ffa70 msgr2=0x7fd9641972d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.775+0000 7fd96bfa7700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9640ffa70 0x7fd9641972d0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd95400e9a0 tx=0x7fd95400ecb0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.775+0000 7fd96bfa7700 1 -- 172.21.15.2:0/973085819 shutdown_connections 2023-11-16T18:35:49.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.775+0000 7fd96bfa7700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd9500a12c0 0x7fd9500a3770 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.775+0000 7fd96bfa7700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd964105250 0x7fd9641a2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.775+0000 7fd96bfa7700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9640ffa70 0x7fd9641972d0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.775+0000 7fd96bfa7700 1 --2- 172.21.15.2:0/973085819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd964100420 0x7fd964197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.775+0000 7fd96bfa7700 1 -- 172.21.15.2:0/973085819 >> 172.21.15.2:0/973085819 conn(0x7fd9640763a0 msgr2=0x7fd9640fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:49.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.775+0000 7fd96bfa7700 1 -- 172.21.15.2:0/973085819 shutdown_connections 2023-11-16T18:35:49.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.775+0000 7fd96bfa7700 1 -- 172.21.15.2:0/973085819 wait complete. 2023-11-16T18:35:49.789 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T18:35:49.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.916+0000 7f37dc771700 1 -- 172.21.15.2:0/3941790917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37d40fea50 msgr2=0x7f37d40fee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.916+0000 7f37dc771700 1 --2- 172.21.15.2:0/3941790917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37d40fea50 0x7f37d40fee20 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f37c80099c0 tx=0x7f37c8009cd0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.916+0000 7f37dc771700 1 -- 172.21.15.2:0/3941790917 shutdown_connections 2023-11-16T18:35:49.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.916+0000 7f37dc771700 1 --2- 172.21.15.2:0/3941790917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37d41055f0 0x7f37d410d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.916+0000 7f37dc771700 1 --2- 172.21.15.2:0/3941790917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37d40fea50 0x7f37d40fee20 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.916+0000 7f37dc771700 1 --2- 172.21.15.2:0/3941790917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d40ff3f0 0x7f37d4104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.916+0000 7f37dc771700 1 -- 172.21.15.2:0/3941790917 >> 172.21.15.2:0/3941790917 conn(0x7f37d4075880 msgr2=0x7f37d4075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:49.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.916+0000 7f37dc771700 1 -- 172.21.15.2:0/3941790917 shutdown_connections 2023-11-16T18:35:49.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.916+0000 7f37dc771700 1 -- 172.21.15.2:0/3941790917 wait complete. 2023-11-16T18:35:49.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dc771700 1 Processor -- start 2023-11-16T18:35:49.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dc771700 1 -- start start 2023-11-16T18:35:49.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dc771700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37d40fea50 0x7f37d40ff9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:49.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dc771700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d40ff3f0 0x7f37d40fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:49.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dc771700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37d41055f0 0x7f37d41a4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:49.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dc771700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d41007a0 con 0x7f37d41055f0 2023-11-16T18:35:49.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dc771700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d41a4aa0 con 0x7f37d40ff3f0 2023-11-16T18:35:49.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dc771700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d41a4be0 con 0x7f37d40fea50 2023-11-16T18:35:49.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dad0e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37d41055f0 0x7f37d41a4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:49.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dad0e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37d41055f0 0x7f37d41a4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53448/0 (socket says 172.21.15.2:53448) 2023-11-16T18:35:49.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dad0e700 1 -- 172.21.15.2:0/957845597 learned_addr learned my addr 172.21.15.2:0/957845597 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:49.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37d9d0c700 1 --2- 172.21.15.2:0/957845597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d40ff3f0 0x7f37d40fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:49.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dad0e700 1 -- 172.21.15.2:0/957845597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37d40fea50 msgr2=0x7f37d40ff9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dad0e700 1 --2- 172.21.15.2:0/957845597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37d40fea50 0x7f37d40ff9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dad0e700 1 -- 172.21.15.2:0/957845597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d40ff3f0 msgr2=0x7f37d40fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:49.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dad0e700 1 --2- 172.21.15.2:0/957845597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d40ff3f0 0x7f37d40fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:49.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.917+0000 7f37dad0e700 1 -- 172.21.15.2:0/957845597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c80095f0 con 0x7f37d41055f0 2023-11-16T18:35:49.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.918+0000 7f37dad0e700 1 --2- 172.21.15.2:0/957845597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37d41055f0 0x7f37d41a4560 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f37cc00b7b0 tx=0x7f37cc00bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:49.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.918+0000 7f37c77fe700 1 -- 172.21.15.2:0/957845597 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc00f840 con 0x7f37d41055f0 2023-11-16T18:35:49.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.918+0000 7f37c77fe700 1 -- 172.21.15.2:0/957845597 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37cc00fe60 con 0x7f37d41055f0 2023-11-16T18:35:49.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.918+0000 7f37dc771700 1 -- 172.21.15.2:0/957845597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d4110380 con 0x7f37d41055f0 2023-11-16T18:35:49.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.918+0000 7f37c77fe700 1 -- 172.21.15.2:0/957845597 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc00d3d0 con 0x7f37d41055f0 2023-11-16T18:35:49.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.918+0000 7f37dc771700 1 -- 172.21.15.2:0/957845597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d41101f0 con 0x7f37d41055f0 2023-11-16T18:35:49.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.919+0000 7f37dc771700 1 -- 172.21.15.2:0/957845597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d4110510 con 0x7f37d41055f0 2023-11-16T18:35:49.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.920+0000 7f37c77fe700 1 -- 172.21.15.2:0/957845597 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f37cc00f9a0 con 0x7f37d41055f0 2023-11-16T18:35:49.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.921+0000 7f37c77fe700 1 --2- 172.21.15.2:0/957845597 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f37c00a1340 0x7f37c00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:49.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.921+0000 7f37da50d700 1 --2- 172.21.15.2:0/957845597 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f37c00a1340 0x7f37c00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:49.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.921+0000 7f37c77fe700 1 -- 172.21.15.2:0/957845597 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9438+0+0 (secure 0 0 0) 0x7f37cc0cd420 con 0x7f37d41055f0 2023-11-16T18:35:49.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.921+0000 7f37da50d700 1 --2- 172.21.15.2:0/957845597 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f37c00a1340 0x7f37c00a37f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f37c80098e0 tx=0x7f37c8000bc0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:49.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:49.924+0000 7f37c77fe700 1 -- 172.21.15.2:0/957845597 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f37cc0964d0 con 0x7f37d41055f0 2023-11-16T18:35:50.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.129+0000 7f37dc771700 1 -- 172.21.15.2:0/957845597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f37d4100ef0 con 0x7f37d41055f0 2023-11-16T18:35:50.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.130+0000 7f37c77fe700 1 -- 172.21.15.2:0/957845597 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) v1 ==== 100+0+11 (secure 0 0 0) 0x7f37cc015300 con 0x7f37d41055f0 2023-11-16T18:35:50.132 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T18:35:50.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.132+0000 7f37dc771700 1 -- 172.21.15.2:0/957845597 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f37c00a1340 msgr2=0x7f37c00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:50.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.132+0000 7f37dc771700 1 --2- 172.21.15.2:0/957845597 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f37c00a1340 0x7f37c00a37f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f37c80098e0 tx=0x7f37c8000bc0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.132+0000 7f37dc771700 1 -- 172.21.15.2:0/957845597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37d41055f0 msgr2=0x7f37d41a4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:50.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.132+0000 7f37dc771700 1 --2- 172.21.15.2:0/957845597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37d41055f0 0x7f37d41a4560 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f37cc00b7b0 tx=0x7f37cc00bb70 comp rx=0 tx=0).stop 2023-11-16T18:35:50.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.132+0000 7f37dc771700 1 -- 172.21.15.2:0/957845597 shutdown_connections 2023-11-16T18:35:50.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.132+0000 7f37dc771700 1 --2- 172.21.15.2:0/957845597 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f37c00a1340 0x7f37c00a37f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.132+0000 7f37dc771700 1 --2- 172.21.15.2:0/957845597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37d40fea50 0x7f37d40ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.132+0000 7f37dc771700 1 --2- 172.21.15.2:0/957845597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37d41055f0 0x7f37d41a4560 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.132+0000 7f37dc771700 1 --2- 172.21.15.2:0/957845597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d40ff3f0 0x7f37d40fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.132+0000 7f37dc771700 1 -- 172.21.15.2:0/957845597 >> 172.21.15.2:0/957845597 conn(0x7f37d4075880 msgr2=0x7f37d40fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:50.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.132+0000 7f37dc771700 1 -- 172.21.15.2:0/957845597 shutdown_connections 2023-11-16T18:35:50.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.132+0000 7f37dc771700 1 -- 172.21.15.2:0/957845597 wait complete. 2023-11-16T18:35:50.147 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T18:35:50.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.272+0000 7fcfa8675700 1 -- 172.21.15.2:0/4163813935 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfa01055c0 msgr2=0x7fcfa010d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:50.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.272+0000 7fcfa8675700 1 --2- 172.21.15.2:0/4163813935 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfa01055c0 0x7fcfa010d140 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800b0d0 tx=0x7fcf9800b3e0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.272+0000 7fcfa8675700 1 -- 172.21.15.2:0/4163813935 shutdown_connections 2023-11-16T18:35:50.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.272+0000 7fcfa8675700 1 --2- 172.21.15.2:0/4163813935 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfa01055c0 0x7fcfa010d140 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.272+0000 7fcfa8675700 1 --2- 172.21.15.2:0/4163813935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfa0100cc0 0x7fcfa0101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.272+0000 7fcfa8675700 1 --2- 172.21.15.2:0/4163813935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfa0101660 0x7fcfa0104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.272+0000 7fcfa8675700 1 -- 172.21.15.2:0/4163813935 >> 172.21.15.2:0/4163813935 conn(0x7fcfa0077560 msgr2=0x7fcfa0077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:50.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.272+0000 7fcfa8675700 1 -- 172.21.15.2:0/4163813935 shutdown_connections 2023-11-16T18:35:50.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.272+0000 7fcfa8675700 1 -- 172.21.15.2:0/4163813935 wait complete. 2023-11-16T18:35:50.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.273+0000 7fcfa8675700 1 Processor -- start 2023-11-16T18:35:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.273+0000 7fcfa8675700 1 -- start start 2023-11-16T18:35:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa8675700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfa0100cc0 0x7fcfa0196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa8675700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfa0101660 0x7fcfa0196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa8675700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfa01055c0 0x7fcfa019aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa8675700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa019b500 con 0x7fcfa0101660 2023-11-16T18:35:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa8675700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa019b670 con 0x7fcfa01055c0 2023-11-16T18:35:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa8675700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa019b830 con 0x7fcfa0100cc0 2023-11-16T18:35:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa5c10700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfa0101660 0x7fcfa0196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa5c10700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfa0101660 0x7fcfa0196a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53460/0 (socket says 172.21.15.2:53460) 2023-11-16T18:35:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa5c10700 1 -- 172.21.15.2:0/1114271029 learned_addr learned my addr 172.21.15.2:0/1114271029 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa6c12700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfa01055c0 0x7fcfa019aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa6411700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfa0100cc0 0x7fcfa0196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa5c10700 1 -- 172.21.15.2:0/1114271029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfa0100cc0 msgr2=0x7fcfa0196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa5c10700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfa0100cc0 0x7fcfa0196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa5c10700 1 -- 172.21.15.2:0/1114271029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfa01055c0 msgr2=0x7fcfa019aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:50.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa5c10700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfa01055c0 0x7fcfa019aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa5c10700 1 -- 172.21.15.2:0/1114271029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf98009d00 con 0x7fcfa0101660 2023-11-16T18:35:50.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa5c10700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfa0101660 0x7fcfa0196a60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400c890 tx=0x7fcf9400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:50.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcf937fe700 1 -- 172.21.15.2:0/1114271029 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94004220 con 0x7fcfa0101660 2023-11-16T18:35:50.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcf937fe700 1 -- 172.21.15.2:0/1114271029 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf94004cf0 con 0x7fcfa0101660 2023-11-16T18:35:50.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcfa8675700 1 -- 172.21.15.2:0/1114271029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa0110340 con 0x7fcfa0101660 2023-11-16T18:35:50.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.274+0000 7fcf937fe700 1 -- 172.21.15.2:0/1114271029 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94005710 con 0x7fcfa0101660 2023-11-16T18:35:50.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.275+0000 7fcfa8675700 1 -- 172.21.15.2:0/1114271029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa01101b0 con 0x7fcfa0101660 2023-11-16T18:35:50.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.275+0000 7fcfa8675700 1 -- 172.21.15.2:0/1114271029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa01104d0 con 0x7fcfa0101660 2023-11-16T18:35:50.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.277+0000 7fcf937fe700 1 -- 172.21.15.2:0/1114271029 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fcf94004e60 con 0x7fcfa0101660 2023-11-16T18:35:50.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.277+0000 7fcf937fe700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fcf8c0a12c0 0x7fcf8c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:50.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.277+0000 7fcfa6411700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fcf8c0a12c0 0x7fcf8c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:50.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.277+0000 7fcf937fe700 1 -- 172.21.15.2:0/1114271029 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9438+0+0 (secure 0 0 0) 0x7fcf940ccec0 con 0x7fcfa0101660 2023-11-16T18:35:50.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.278+0000 7fcfa6411700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fcf8c0a12c0 0x7fcf8c0a3770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c005fd0 tx=0x7fcf9c00a560 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:50.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.281+0000 7fcf937fe700 1 -- 172.21.15.2:0/1114271029 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcf94095f40 con 0x7fcfa0101660 2023-11-16T18:35:50.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:50 smithi040 ceph-mon[100251]: pgmap v344: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:50.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:50 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3644153321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:35:50.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:50 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/973085819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:35:50.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.483+0000 7fcfa8675700 1 -- 172.21.15.2:0/1114271029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fcfa01a3650 con 0x7fcfa0101660 2023-11-16T18:35:50.486 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.484+0000 7fcf937fe700 1 -- 172.21.15.2:0/1114271029 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v103) v1 ==== 103+0+11 (secure 0 0 0) 0x7fcf9401d0a0 con 0x7fcfa0101660 2023-11-16T18:35:50.486 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T18:35:50.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.486+0000 7fcfa8675700 1 -- 172.21.15.2:0/1114271029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fcf8c0a12c0 msgr2=0x7fcf8c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:50.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.486+0000 7fcfa8675700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fcf8c0a12c0 0x7fcf8c0a3770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c005fd0 tx=0x7fcf9c00a560 comp rx=0 tx=0).stop 2023-11-16T18:35:50.488 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.486+0000 7fcfa8675700 1 -- 172.21.15.2:0/1114271029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfa0101660 msgr2=0x7fcfa0196a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:50.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.486+0000 7fcfa8675700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfa0101660 0x7fcfa0196a60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400c890 tx=0x7fcf9400cba0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.486+0000 7fcfa8675700 1 -- 172.21.15.2:0/1114271029 shutdown_connections 2023-11-16T18:35:50.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.486+0000 7fcfa8675700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fcf8c0a12c0 0x7fcf8c0a3770 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.486+0000 7fcfa8675700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfa0100cc0 0x7fcfa0196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.486+0000 7fcfa8675700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfa01055c0 0x7fcfa019aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.486+0000 7fcfa8675700 1 --2- 172.21.15.2:0/1114271029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfa0101660 0x7fcfa0196a60 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.486+0000 7fcfa8675700 1 -- 172.21.15.2:0/1114271029 >> 172.21.15.2:0/1114271029 conn(0x7fcfa0077560 msgr2=0x7fcfa00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:50.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.486+0000 7fcfa8675700 1 -- 172.21.15.2:0/1114271029 shutdown_connections 2023-11-16T18:35:50.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.486+0000 7fcfa8675700 1 -- 172.21.15.2:0/1114271029 wait complete. 2023-11-16T18:35:50.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:50 smithi002 ceph-mon[98807]: pgmap v344: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:50.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:50 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3644153321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:35:50.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:50 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/973085819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:35:50.501 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-16T18:35:50.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:50 smithi188 ceph-mon[100639]: pgmap v344: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:50.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:50 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3644153321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T18:35:50.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:50 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/973085819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T18:35:50.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.626+0000 7fd94e7c8700 1 -- 172.21.15.2:0/602666550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9480ffd50 msgr2=0x7fd948103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:50.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.626+0000 7fd94e7c8700 1 --2- 172.21.15.2:0/602666550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9480ffd50 0x7fd948103d10 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0099c0 tx=0x7fd93c009cd0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.627+0000 7fd94e7c8700 1 -- 172.21.15.2:0/602666550 shutdown_connections 2023-11-16T18:35:50.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.627+0000 7fd94e7c8700 1 --2- 172.21.15.2:0/602666550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9480ff3b0 0x7fd9480ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.627+0000 7fd94e7c8700 1 --2- 172.21.15.2:0/602666550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9481043f0 0x7fd94810bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.627+0000 7fd94e7c8700 1 --2- 172.21.15.2:0/602666550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9480ffd50 0x7fd948103d10 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.627+0000 7fd94e7c8700 1 -- 172.21.15.2:0/602666550 >> 172.21.15.2:0/602666550 conn(0x7fd948075f70 msgr2=0x7fd948076370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:50.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.627+0000 7fd94e7c8700 1 -- 172.21.15.2:0/602666550 shutdown_connections 2023-11-16T18:35:50.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.627+0000 7fd94e7c8700 1 -- 172.21.15.2:0/602666550 wait complete. 2023-11-16T18:35:50.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.627+0000 7fd94e7c8700 1 Processor -- start 2023-11-16T18:35:50.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.627+0000 7fd94e7c8700 1 -- start start 2023-11-16T18:35:50.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd94e7c8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9480ff3b0 0x7fd9481971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:50.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd94e7c8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9480ffd50 0x7fd948197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:50.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd94e7c8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9481043f0 0x7fd9481a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:50.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd94e7c8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd948198010 con 0x7fd9480ffd50 2023-11-16T18:35:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd94e7c8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9481a2780 con 0x7fd9481043f0 2023-11-16T18:35:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd94e7c8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9481a28f0 con 0x7fd9480ff3b0 2023-11-16T18:35:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd9477fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9480ffd50 0x7fd948197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd9477fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9480ffd50 0x7fd948197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53474/0 (socket says 172.21.15.2:53474) 2023-11-16T18:35:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd9477fe700 1 -- 172.21.15.2:0/333476658 learned_addr learned my addr 172.21.15.2:0/333476658 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd94cd65700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9481043f0 0x7fd9481a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd947fff700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9480ff3b0 0x7fd9481971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd9477fe700 1 -- 172.21.15.2:0/333476658 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9480ff3b0 msgr2=0x7fd9481971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:50.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd9477fe700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9480ff3b0 0x7fd9481971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd9477fe700 1 -- 172.21.15.2:0/333476658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9481043f0 msgr2=0x7fd9481a2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:50.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd9477fe700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9481043f0 0x7fd9481a2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd9477fe700 1 -- 172.21.15.2:0/333476658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c0095f0 con 0x7fd9480ffd50 2023-11-16T18:35:50.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd9477fe700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9480ffd50 0x7fd948197700 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009f50 tx=0x7fd93c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:50.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd9457fa700 1 -- 172.21.15.2:0/333476658 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c01d070 con 0x7fd9480ffd50 2023-11-16T18:35:50.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd9457fa700 1 -- 172.21.15.2:0/333476658 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd93c003f90 con 0x7fd9480ffd50 2023-11-16T18:35:50.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd94e7c8700 1 -- 172.21.15.2:0/333476658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd94810f170 con 0x7fd9480ffd50 2023-11-16T18:35:50.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd9457fa700 1 -- 172.21.15.2:0/333476658 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c022d00 con 0x7fd9480ffd50 2023-11-16T18:35:50.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.628+0000 7fd94e7c8700 1 -- 172.21.15.2:0/333476658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd94810efe0 con 0x7fd9480ffd50 2023-11-16T18:35:50.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.629+0000 7fd94e7c8700 1 -- 172.21.15.2:0/333476658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd94810f300 con 0x7fd9480ffd50 2023-11-16T18:35:50.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.630+0000 7fd9457fa700 1 -- 172.21.15.2:0/333476658 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fd93c022470 con 0x7fd9480ffd50 2023-11-16T18:35:50.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.631+0000 7fd9457fa700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd9300a12c0 0x7fd9300a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:50.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.631+0000 7fd9457fa700 1 -- 172.21.15.2:0/333476658 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9438+0+0 (secure 0 0 0) 0x7fd93c0ce9e0 con 0x7fd9480ffd50 2023-11-16T18:35:50.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.631+0000 7fd947fff700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd9300a12c0 0x7fd9300a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:50.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.632+0000 7fd947fff700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd9300a12c0 0x7fd9300a3770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd93800ba60 tx=0x7fd938005d50 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:50.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.635+0000 7fd9457fa700 1 -- 172.21.15.2:0/333476658 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd93c097af0 con 0x7fd9480ffd50 2023-11-16T18:35:50.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.850+0000 7fd94e7c8700 1 -- 172.21.15.2:0/333476658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd9480679e0 con 0x7fd9480ffd50 2023-11-16T18:35:50.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.850+0000 7fd9457fa700 1 -- 172.21.15.2:0/333476658 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v103) v1 ==== 81+0+927 (secure 0 0 0) 0x7fd93c022e60 con 0x7fd9480ffd50 2023-11-16T18:35:50.853 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/3369431713 2023-11-17T18:24:50.949710+0000 2023-11-16T18:35:50.853 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/1472878779 2023-11-17T18:24:50.949710+0000 2023-11-16T18:35:50.853 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/1347603606 2023-11-17T18:24:50.949710+0000 2023-11-16T18:35:50.854 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/3369431713 2023-11-17T18:24:50.949710+0000 2023-11-16T18:35:50.854 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/857143886 2023-11-17T18:16:37.723675+0000 2023-11-16T18:35:50.854 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3335793429 2023-11-17T18:24:50.949710+0000 2023-11-16T18:35:50.854 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/251514255 2023-11-17T18:16:37.723675+0000 2023-11-16T18:35:50.854 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/1605535303 2023-11-17T18:16:37.723675+0000 2023-11-16T18:35:50.854 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/1605535303 2023-11-17T18:16:37.723675+0000 2023-11-16T18:35:50.854 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/1609662822 2023-11-17T18:16:13.217812+0000 2023-11-16T18:35:50.854 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/1609662822 2023-11-17T18:16:13.217812+0000 2023-11-16T18:35:50.855 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/708195783 2023-11-17T18:16:13.217812+0000 2023-11-16T18:35:50.855 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/1158002645 2023-11-17T18:16:13.217812+0000 2023-11-16T18:35:50.855 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3821858662 2023-11-17T18:24:50.949710+0000 2023-11-16T18:35:50.855 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/1986390775 2023-11-17T18:16:13.217812+0000 2023-11-16T18:35:50.855 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/2955607810 2023-11-17T18:16:37.723675+0000 2023-11-16T18:35:50.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.853+0000 7fd94e7c8700 1 -- 172.21.15.2:0/333476658 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd9300a12c0 msgr2=0x7fd9300a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:50.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.853+0000 7fd94e7c8700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd9300a12c0 0x7fd9300a3770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd93800ba60 tx=0x7fd938005d50 comp rx=0 tx=0).stop 2023-11-16T18:35:50.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.853+0000 7fd94e7c8700 1 -- 172.21.15.2:0/333476658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9480ffd50 msgr2=0x7fd948197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:50.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.853+0000 7fd94e7c8700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9480ffd50 0x7fd948197700 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009f50 tx=0x7fd93c005740 comp rx=0 tx=0).stop 2023-11-16T18:35:50.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.853+0000 7fd94e7c8700 1 -- 172.21.15.2:0/333476658 shutdown_connections 2023-11-16T18:35:50.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.853+0000 7fd94e7c8700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fd9300a12c0 0x7fd9300a3770 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.853+0000 7fd94e7c8700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd9480ff3b0 0x7fd9481971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.853+0000 7fd94e7c8700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9481043f0 0x7fd9481a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.853+0000 7fd94e7c8700 1 --2- 172.21.15.2:0/333476658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9480ffd50 0x7fd948197700 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:50.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.853+0000 7fd94e7c8700 1 -- 172.21.15.2:0/333476658 >> 172.21.15.2:0/333476658 conn(0x7fd948075f70 msgr2=0x7fd9480fccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:50.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.853+0000 7fd94e7c8700 1 -- 172.21.15.2:0/333476658 shutdown_connections 2023-11-16T18:35:50.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:50.853+0000 7fd94e7c8700 1 -- 172.21.15.2:0/333476658 wait complete. 2023-11-16T18:35:50.858 INFO:teuthology.orchestra.run.smithi002.stderr:listed 16 entries 2023-11-16T18:35:50.869 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-16T18:35:50.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:35:50.869 DEBUG:teuthology.orchestra.run.smithi002:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-16T18:35:50.951 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T18:35:50.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:35:50.951 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-16T18:35:51.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:35:51.084 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2023-11-16T18:35:51.165 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:51 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/957845597' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:35:51.165 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:51 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1114271029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:35:51.165 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:51 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/333476658' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T18:35:51.241 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-16T18:35:51.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:35:51.242 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-16T18:35:51.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:35:51.293 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link delete ceph-brx 2023-11-16T18:35:51.417 INFO:tasks.kclient:Unmounting kernel clients... 2023-11-16T18:35:51.417 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T18:35:51.417 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T18:35:51.435 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16952 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/a8592590b852d210c23322c417677b6ea6015db37367d253b6a2275909e0e1a1/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/UB5M2G6UY4PVD7NS24Z3OGOBCK:/var/lib/containers/storage/overlay/l/IONB77VS43IYI57ZQQS7UV7YCO,upperdir=/var/lib/containers/storage/overlay/a8592590b852d210c23322c417677b6ea6015db37367d253b6a2275909e0e1a1/diff,workdir=/var/lib/containers/storage/overlay/a8592590b852d210c23322c417677b6ea6015db37367d253b6a2275909e0e1a1/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/7b2b0fe0c5eec2acd6e7f2eb7e413d8ad58e3fa79427b8af419e619c15a9c4aa/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/UB5M2G6UY4PVD7NS24Z3OGOBCK:/var/lib/containers/storage/overlay/l/IONB77VS43IYI57ZQQS7UV7YCO,upperdir=/var/lib/containers/storage/overlay/7b2b0fe0c5eec2acd6e7f2eb7e413d8ad58e3fa79427b8af419e619c15a9c4aa/diff,workdir=/var/lib/containers/storage/overlay/7b2b0fe0c5eec2acd6e7f2eb7e413d8ad58e3fa79427b8af419e619c15a9c4aa/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 overlay /var/lib/containers/storage/overlay/aabb97f05fd436317f3965ece13ee73efc806a4e1e29ce00ccb72582c2ac02e1/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/UB5M2G6UY4PVD7NS24Z3OGOBCK:/var/lib/containers/storage/overlay/l/IONB77VS43IYI57ZQQS7UV7YCO,upperdir=/var/lib/containers/storage/overlay/aabb97f05fd436317f3965ece13ee73efc806a4e1e29ce00ccb72582c2ac02e1/diff,workdir=/var/lib/containers/storage/overlay/aabb97f05fd436317f3965ece13ee73efc806a4e1e29ce00ccb72582c2ac02e1/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/2445d0562415ce4812f13a2608bf28d7cd6f8c273aa4e95c866378984734db1f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/UB5M2G6UY4PVD7NS24Z3OGOBCK:/var/lib/containers/storage/overlay/l/IONB77VS43IYI57ZQQS7UV7YCO,upperdir=/var/lib/containers/storage/overlay/2445d0562415ce4812f13a2608bf28d7cd6f8c273aa4e95c866378984734db1f/diff,workdir=/var/lib/containers/storage/overlay/2445d0562415ce4812f13a2608bf28d7cd6f8c273aa4e95c866378984734db1f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9181913456dac63028c36046e45ec221a16aab89675e8bda44c9ed33dd915bf7/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/UB5M2G6UY4PVD7NS24Z3OGOBCK:/var/lib/containers/storage/overlay/l/IONB77VS43IYI57ZQQS7UV7YCO,upperdir=/var/lib/containers/storage/overlay/9181913456dac63028c36046e45ec221a16aab89675e8bda44c9ed33dd915bf7/diff,workdir=/var/lib/containers/storage/overlay/9181913456dac63028c36046e45ec221a16aab89675e8bda44c9ed33dd915bf7/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/5e982ddea7fe109d8e955380ad0f1dba116c2cebb129b267e6e05710a4f09abc/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/UB5M2G6UY4PVD7NS24Z3OGOBCK:/var/lib/containers/storage/overlay/l/IONB77VS43IYI57ZQQS7UV7YCO,upperdir=/var/lib/containers/storage/overlay/5e982ddea7fe109d8e955380ad0f1dba116c2cebb129b267e6e05710a4f09abc/diff,workdir=/var/lib/containers/storage/overlay/5e982ddea7fe109d8e955380ad0f1dba116c2cebb129b267e6e05710a4f09abc/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/2be6a4a2277a1402d54a102c6674d1fc2eb6d40f447049d602a29b5357485a9f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/UB5M2G6UY4PVD7NS24Z3OGOBCK:/var/lib/containers/storage/overlay/l/IONB77VS43IYI57ZQQS7UV7YCO,upperdir=/var/lib/containers/storage/overlay/2be6a4a2277a1402d54a102c6674d1fc2eb6d40f447049d602a29b5357485a9f/diff,workdir=/var/lib/containers/storage/overlay/2be6a4a2277a1402d54a102c6674d1fc2eb6d40f447049d602a29b5357485a9f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d5a715d2e2b5114774631c76ffd99838fb6e74560ee06e4cc78c9c95f2b84232/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/UB5M2G6UY4PVD7NS24Z3OGOBCK:/var/lib/containers/storage/overlay/l/IONB77VS43IYI57ZQQS7UV7YCO,upperdir=/var/lib/containers/storage/overlay/d5a715d2e2b5114774631c76ffd99838fb6e74560ee06e4cc78c9c95f2b84232/diff,workdir=/var/lib/containers/storage/overlay/d5a715d2e2b5114774631c76ffd99838fb6e74560ee06e4cc78c9c95f2b84232/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/832fcd7cd9b3cd69ed53940aeb7f92920cefeef896dd5c97591ed26d6eff2813/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/UB5M2G6UY4PVD7NS24Z3OGOBCK:/var/lib/containers/storage/overlay/l/IONB77VS43IYI57ZQQS7UV7YCO,upperdir=/var/lib/containers/storage/overlay/832fcd7cd9b3cd69ed53940aeb7f92920cefeef896dd5c97591ed26d6eff2813/diff,workdir=/var/lib/containers/storage/overlay/832fcd7cd9b3cd69ed53940aeb7f92920cefeef896dd5c97591ed26d6eff2813/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/0a37eddfe55139e805ddedbddb7a52771fd8f3c18571373f28bed75492fdd215/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/UB5M2G6UY4PVD7NS24Z3OGOBCK:/var/lib/containers/storage/overlay/l/IONB77VS43IYI57ZQQS7UV7YCO,upperdir=/var/lib/containers/storage/overlay/0a37eddfe55139e805ddedbddb7a52771fd8f3c18571373f28bed75492fdd215/diff,workdir=/var/lib/containers/storage/overlay/0a37eddfe55139e805ddedbddb7a52771fd8f3c18571373f28bed75492fdd215/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 2023-11-16T18:35:51.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:35:51.435 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2023-11-16T18:35:51.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:51 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/957845597' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:35:51.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:51 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1114271029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:35:51.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:51 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/333476658' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T18:35:51.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T18:35:51.493 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link delete ceph-brx 2023-11-16T18:35:51.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:51 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/957845597' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T18:35:51.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:51 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1114271029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T18:35:51.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:51 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/333476658' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T18:35:51.635 INFO:teuthology.orchestra.run.smithi002.stderr:Cannot find device "ceph-brx" 2023-11-16T18:35:51.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:35:51.636 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-16T18:35:51.647 INFO:tasks.cephadm:Teardown begin 2023-11-16T18:35:51.647 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-16T18:35:51.647 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:35:51.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.771+0000 7f831f238700 1 -- 172.21.15.2:0/3210913929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83180ffdd0 msgr2=0x7f8318103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:51.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.771+0000 7f831f238700 1 --2- 172.21.15.2:0/3210913929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83180ffdd0 0x7f8318103d90 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f83040099c0 tx=0x7f8304009cd0 comp rx=0 tx=0).stop 2023-11-16T18:35:51.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.771+0000 7f831f238700 1 -- 172.21.15.2:0/3210913929 shutdown_connections 2023-11-16T18:35:51.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.771+0000 7f831f238700 1 --2- 172.21.15.2:0/3210913929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83180ff430 0x7f83180ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:51.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.771+0000 7f831f238700 1 --2- 172.21.15.2:0/3210913929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318104470 0x7f831810c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:51.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.771+0000 7f831f238700 1 --2- 172.21.15.2:0/3210913929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83180ffdd0 0x7f8318103d90 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:51.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.771+0000 7f831f238700 1 -- 172.21.15.2:0/3210913929 >> 172.21.15.2:0/3210913929 conn(0x7f8318077060 msgr2=0x7f8318077460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:51.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.771+0000 7f831f238700 1 -- 172.21.15.2:0/3210913929 shutdown_connections 2023-11-16T18:35:51.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.772+0000 7f831f238700 1 -- 172.21.15.2:0/3210913929 wait complete. 2023-11-16T18:35:51.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.772+0000 7f831f238700 1 Processor -- start 2023-11-16T18:35:51.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.772+0000 7f831f238700 1 -- start start 2023-11-16T18:35:51.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.772+0000 7f831f238700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83180ff430 0x7f8318195320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:51.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.772+0000 7f831f238700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83180ffdd0 0x7f8318195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:51.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.772+0000 7f831f238700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318104470 0x7f8318199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:51.776 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.772+0000 7f831f238700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831819a300 con 0x7f83180ffdd0 2023-11-16T18:35:51.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.772+0000 7f831f238700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831819a470 con 0x7f8318104470 2023-11-16T18:35:51.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.772+0000 7f831f238700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831819a630 con 0x7f83180ff430 2023-11-16T18:35:51.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.772+0000 7f8317fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83180ffdd0 0x7f8318195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:51.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.772+0000 7f8317fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83180ffdd0 0x7f8318195860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53486/0 (socket says 172.21.15.2:53486) 2023-11-16T18:35:51.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.772+0000 7f8317fff700 1 -- 172.21.15.2:0/706378478 learned_addr learned my addr 172.21.15.2:0/706378478 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:51.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.772+0000 7f831cfd4700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83180ff430 0x7f8318195320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:51.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.773+0000 7f831d7d5700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318104470 0x7f8318199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:51.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.773+0000 7f8317fff700 1 -- 172.21.15.2:0/706378478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83180ff430 msgr2=0x7f8318195320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:51.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.773+0000 7f8317fff700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83180ff430 0x7f8318195320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:51.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.773+0000 7f8317fff700 1 -- 172.21.15.2:0/706378478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318104470 msgr2=0x7f8318199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:51.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.773+0000 7f8317fff700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318104470 0x7f8318199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:51.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.773+0000 7f8317fff700 1 -- 172.21.15.2:0/706378478 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83040095f0 con 0x7f83180ffdd0 2023-11-16T18:35:51.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.773+0000 7f831d7d5700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318104470 0x7f8318199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T18:35:51.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.773+0000 7f8317fff700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83180ffdd0 0x7f8318195860 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8304009fd0 tx=0x7f8304009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:51.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.773+0000 7f8315ffb700 1 -- 172.21.15.2:0/706378478 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830401d070 con 0x7f83180ffdd0 2023-11-16T18:35:51.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.773+0000 7f831f238700 1 -- 172.21.15.2:0/706378478 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831810f210 con 0x7f83180ffdd0 2023-11-16T18:35:51.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.773+0000 7f8315ffb700 1 -- 172.21.15.2:0/706378478 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8304005020 con 0x7f83180ffdd0 2023-11-16T18:35:51.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.773+0000 7f8315ffb700 1 -- 172.21.15.2:0/706378478 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8304004aa0 con 0x7f83180ffdd0 2023-11-16T18:35:51.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.773+0000 7f831f238700 1 -- 172.21.15.2:0/706378478 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f831810f080 con 0x7f83180ffdd0 2023-11-16T18:35:51.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.774+0000 7f831f238700 1 -- 172.21.15.2:0/706378478 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f831810f3a0 con 0x7f83180ffdd0 2023-11-16T18:35:51.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.775+0000 7f8315ffb700 1 -- 172.21.15.2:0/706378478 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f8304004ca0 con 0x7f83180ffdd0 2023-11-16T18:35:51.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.776+0000 7f8315ffb700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f83080a12c0 0x7f83080a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:51.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.776+0000 7f8315ffb700 1 -- 172.21.15.2:0/706378478 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9438+0+0 (secure 0 0 0) 0x7f83040cddd0 con 0x7f83180ffdd0 2023-11-16T18:35:51.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.776+0000 7f831cfd4700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f83080a12c0 0x7f83080a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:51.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.776+0000 7f831cfd4700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f83080a12c0 0x7f83080a3770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f830c004630 tx=0x7f830c00b3c0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:51.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:51.779+0000 7f8315ffb700 1 -- 172.21.15.2:0/706378478 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8304096e80 con 0x7f83180ffdd0 2023-11-16T18:35:52.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.021+0000 7f831f238700 1 -- 172.21.15.2:0/706378478 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8318067af0 con 0x7f83180ffdd0 2023-11-16T18:35:52.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.021+0000 7f8315ffb700 1 -- 172.21.15.2:0/706378478 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f830409a4a0 con 0x7f83180ffdd0 2023-11-16T18:35:52.024 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:35:52.024 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:34:58.401086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":89,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26269":{"gid":26269,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/3118705556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":3118705556},{"type":"v1","addr":"172.21.15.188:6837","nonce":3118705556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:35:52.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.023+0000 7f831f238700 1 -- 172.21.15.2:0/706378478 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f83080a12c0 msgr2=0x7f83080a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.023+0000 7f831f238700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f83080a12c0 0x7f83080a3770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f830c004630 tx=0x7f830c00b3c0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.023+0000 7f831f238700 1 -- 172.21.15.2:0/706378478 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83180ffdd0 msgr2=0x7f8318195860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.023+0000 7f831f238700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83180ffdd0 0x7f8318195860 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8304009fd0 tx=0x7f8304009580 comp rx=0 tx=0).stop 2023-11-16T18:35:52.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.023+0000 7f831f238700 1 -- 172.21.15.2:0/706378478 shutdown_connections 2023-11-16T18:35:52.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.023+0000 7f831f238700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f83080a12c0 0x7f83080a3770 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.023+0000 7f831f238700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83180ff430 0x7f8318195320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.023+0000 7f831f238700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318104470 0x7f8318199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.023+0000 7f831f238700 1 --2- 172.21.15.2:0/706378478 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83180ffdd0 0x7f8318195860 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.023+0000 7f831f238700 1 -- 172.21.15.2:0/706378478 >> 172.21.15.2:0/706378478 conn(0x7f8318077060 msgr2=0x7f83180fcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:52.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.024+0000 7f831f238700 1 -- 172.21.15.2:0/706378478 shutdown_connections 2023-11-16T18:35:52.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.024+0000 7f831f238700 1 -- 172.21.15.2:0/706378478 wait complete. 2023-11-16T18:35:52.027 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 37 2023-11-16T18:35:52.038 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T18:35:52.167 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.164+0000 7f8808f3f700 1 -- 172.21.15.2:0/1900773561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f880410a8b0 msgr2=0x7f880410ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.167 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.164+0000 7f8808f3f700 1 --2- 172.21.15.2:0/1900773561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f880410a8b0 0x7f880410ccd0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f87fc00b0a0 tx=0x7f87fc00b3b0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.167 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.164+0000 7f8808f3f700 1 -- 172.21.15.2:0/1900773561 shutdown_connections 2023-11-16T18:35:52.167 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.164+0000 7f8808f3f700 1 --2- 172.21.15.2:0/1900773561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8804101390 0x7f8804101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.167 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.164+0000 7f8808f3f700 1 --2- 172.21.15.2:0/1900773561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f880410a8b0 0x7f880410ccd0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.167 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.164+0000 7f8808f3f700 1 --2- 172.21.15.2:0/1900773561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8804101ca0 0x7f880410a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.168 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.164+0000 7f8808f3f700 1 -- 172.21.15.2:0/1900773561 >> 172.21.15.2:0/1900773561 conn(0x7f8804077060 msgr2=0x7f8804077460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:52.168 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.164+0000 7f8808f3f700 1 -- 172.21.15.2:0/1900773561 shutdown_connections 2023-11-16T18:35:52.168 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.164+0000 7f8808f3f700 1 -- 172.21.15.2:0/1900773561 wait complete. 2023-11-16T18:35:52.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.165+0000 7f8808f3f700 1 Processor -- start 2023-11-16T18:35:52.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.165+0000 7f8808f3f700 1 -- start start 2023-11-16T18:35:52.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.165+0000 7f8808f3f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8804101390 0x7f88040ff1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:52.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.165+0000 7f8808f3f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8804101ca0 0x7f88040ff700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:52.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.165+0000 7f8808f3f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f880410a8b0 0x7f88041a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:52.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.165+0000 7f8808f3f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88040fff30 con 0x7f8804101390 2023-11-16T18:35:52.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.165+0000 7f8808f3f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88041a4c20 con 0x7f880410a8b0 2023-11-16T18:35:52.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.165+0000 7f8808f3f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88041a4d60 con 0x7f8804101ca0 2023-11-16T18:35:52.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.165+0000 7f880259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8804101390 0x7f88040ff1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:52.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.165+0000 7f880259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8804101390 0x7f88040ff1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53498/0 (socket says 172.21.15.2:53498) 2023-11-16T18:35:52.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.165+0000 7f880259c700 1 -- 172.21.15.2:0/3677328845 learned_addr learned my addr 172.21.15.2:0/3677328845 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:52.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.166+0000 7f8802d9d700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f880410a8b0 0x7f88041a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:52.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.166+0000 7f8801d9b700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8804101ca0 0x7f88040ff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:52.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.166+0000 7f880259c700 1 -- 172.21.15.2:0/3677328845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8804101ca0 msgr2=0x7f88040ff700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.166+0000 7f880259c700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8804101ca0 0x7f88040ff700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.166+0000 7f880259c700 1 -- 172.21.15.2:0/3677328845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f880410a8b0 msgr2=0x7f88041a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.166+0000 7f880259c700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f880410a8b0 0x7f88041a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.166+0000 7f880259c700 1 -- 172.21.15.2:0/3677328845 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc009d00 con 0x7f8804101390 2023-11-16T18:35:52.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.166+0000 7f8801d9b700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8804101ca0 0x7f88040ff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T18:35:52.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.166+0000 7f880259c700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8804101390 0x7f88040ff1c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f87f400b550 tx=0x7f87f400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:52.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.169+0000 7f87fb7fe700 1 -- 172.21.15.2:0/3677328845 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f4011830 con 0x7f8804101390 2023-11-16T18:35:52.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.169+0000 7f87fb7fe700 1 -- 172.21.15.2:0/3677328845 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87f4011e50 con 0x7f8804101390 2023-11-16T18:35:52.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.169+0000 7f8808f3f700 1 -- 172.21.15.2:0/3677328845 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88041103e0 con 0x7f8804101390 2023-11-16T18:35:52.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.169+0000 7f87fb7fe700 1 -- 172.21.15.2:0/3677328845 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f400f2e0 con 0x7f8804101390 2023-11-16T18:35:52.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.169+0000 7f8808f3f700 1 -- 172.21.15.2:0/3677328845 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8804110250 con 0x7f8804101390 2023-11-16T18:35:52.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.170+0000 7f8808f3f700 1 -- 172.21.15.2:0/3677328845 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8804110570 con 0x7f8804101390 2023-11-16T18:35:52.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.171+0000 7f87fb7fe700 1 -- 172.21.15.2:0/3677328845 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f87f400f440 con 0x7f8804101390 2023-11-16T18:35:52.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.171+0000 7f87fb7fe700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f87f00a12c0 0x7f87f00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:52.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.171+0000 7f8801d9b700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f87f00a12c0 0x7f87f00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:52.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.171+0000 7f87fb7fe700 1 -- 172.21.15.2:0/3677328845 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9438+0+0 (secure 0 0 0) 0x7f87f40cca10 con 0x7f8804101390 2023-11-16T18:35:52.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.172+0000 7f8801d9b700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f87f00a12c0 0x7f87f00a3770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f88041009b0 tx=0x7f87ec009260 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:52.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.175+0000 7f87fb7fe700 1 -- 172.21.15.2:0/3677328845 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f87f4095ba0 con 0x7f8804101390 2023-11-16T18:35:52.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:52 smithi002 ceph-mon[98807]: pgmap v345: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:52.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:52 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/706378478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:35:52.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.420+0000 7f8808f3f700 1 -- 172.21.15.2:0/3677328845 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8804100790 con 0x7f8804101390 2023-11-16T18:35:52.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.421+0000 7f87fb7fe700 1 -- 172.21.15.2:0/3677328845 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f87f40991c0 con 0x7f8804101390 2023-11-16T18:35:52.423 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:35:52.424 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/223055355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":223055355},{"type":"v1","addr":"172.21.15.2:6835","nonce":223055355}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/508165139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":508165139},{"type":"v1","addr":"172.21.15.2:6837","nonce":508165139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1181439264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1181439264},{"type":"v1","addr":"172.21.15.2:6839","nonce":1181439264}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3213213799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3213213799},{"type":"v1","addr":"172.21.15.2:6841","nonce":3213213799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1110523894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1110523894},{"type":"v1","addr":"172.21.15.40:6833","nonce":1110523894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/381028209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":381028209},{"type":"v1","addr":"172.21.15.188:6833","nonce":381028209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T18:22:46.806019+0000","modified":"2023-11-16T18:34:58.401086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24460,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/17956118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":17956118},{"type":"v1","addr":"172.21.15.40:6835","nonce":17956118}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/3630447129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3630447129},{"type":"v1","addr":"172.21.15.188:6835","nonce":3630447129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":89,"addr":"172.21.15.40:6837/248461642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":248461642},{"type":"v1","addr":"172.21.15.40:6837","nonce":248461642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.188:6839/1646912892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":1646912892},{"type":"v1","addr":"172.21.15.188:6839","nonce":1646912892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3162585010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3162585010},{"type":"v1","addr":"172.21.15.40:6839","nonce":3162585010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26269":{"gid":26269,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/3118705556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":3118705556},{"type":"v1","addr":"172.21.15.188:6837","nonce":3118705556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T18:35:52.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.423+0000 7f8808f3f700 1 -- 172.21.15.2:0/3677328845 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f87f00a12c0 msgr2=0x7f87f00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.423+0000 7f8808f3f700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f87f00a12c0 0x7f87f00a3770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f88041009b0 tx=0x7f87ec009260 comp rx=0 tx=0).stop 2023-11-16T18:35:52.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.423+0000 7f8808f3f700 1 -- 172.21.15.2:0/3677328845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8804101390 msgr2=0x7f88040ff1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.423+0000 7f8808f3f700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8804101390 0x7f88040ff1c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f87f400b550 tx=0x7f87f400b860 comp rx=0 tx=0).stop 2023-11-16T18:35:52.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.423+0000 7f8808f3f700 1 -- 172.21.15.2:0/3677328845 shutdown_connections 2023-11-16T18:35:52.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.423+0000 7f8808f3f700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f87f00a12c0 0x7f87f00a3770 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.423+0000 7f8808f3f700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f880410a8b0 0x7f88041a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.423+0000 7f8808f3f700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8804101390 0x7f88040ff1c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.423+0000 7f8808f3f700 1 --2- 172.21.15.2:0/3677328845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8804101ca0 0x7f88040ff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.423+0000 7f8808f3f700 1 -- 172.21.15.2:0/3677328845 >> 172.21.15.2:0/3677328845 conn(0x7f8804077060 msgr2=0x7f88040fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:52.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.423+0000 7f8808f3f700 1 -- 172.21.15.2:0/3677328845 shutdown_connections 2023-11-16T18:35:52.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.423+0000 7f8808f3f700 1 -- 172.21.15.2:0/3677328845 wait complete. 2023-11-16T18:35:52.428 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 37 2023-11-16T18:35:52.439 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T18:35:52.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:52 smithi040 ceph-mon[100251]: pgmap v345: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:52.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:52 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/706378478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:35:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:52 smithi188 ceph-mon[100639]: pgmap v345: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-11-16T18:35:52.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:52 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/706378478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:35:52.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.564+0000 7f6cb1be3700 1 -- 172.21.15.2:0/3330575537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6cac0ffe00 msgr2=0x7f6cac103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.564+0000 7f6cb1be3700 1 --2- 172.21.15.2:0/3330575537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6cac0ffe00 0x7f6cac103dc0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c0099c0 tx=0x7f6c9c009cd0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.564+0000 7f6cb1be3700 1 -- 172.21.15.2:0/3330575537 shutdown_connections 2023-11-16T18:35:52.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.564+0000 7f6cb1be3700 1 --2- 172.21.15.2:0/3330575537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cac1044a0 0x7f6cac10c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.564+0000 7f6cb1be3700 1 --2- 172.21.15.2:0/3330575537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac0ff460 0x7f6cac0ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.564+0000 7f6cb1be3700 1 --2- 172.21.15.2:0/3330575537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6cac0ffe00 0x7f6cac103dc0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.564+0000 7f6cb1be3700 1 -- 172.21.15.2:0/3330575537 >> 172.21.15.2:0/3330575537 conn(0x7f6cac076310 msgr2=0x7f6cac076710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:52.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.564+0000 7f6cb1be3700 1 -- 172.21.15.2:0/3330575537 shutdown_connections 2023-11-16T18:35:52.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.564+0000 7f6cb1be3700 1 -- 172.21.15.2:0/3330575537 wait complete. 2023-11-16T18:35:52.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.564+0000 7f6cb1be3700 1 Processor -- start 2023-11-16T18:35:52.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cb1be3700 1 -- start start 2023-11-16T18:35:52.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cb1be3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac0ff460 0x7f6cac1953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:52.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cb1be3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cac0ffe00 0x7f6cac195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:52.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cb1be3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6cac1044a0 0x7f6cac199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:52.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cb1be3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac19a3c0 con 0x7f6cac1044a0 2023-11-16T18:35:52.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cb1be3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac19a530 con 0x7f6cac0ff460 2023-11-16T18:35:52.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cb1be3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac19a6f0 con 0x7f6cac0ffe00 2023-11-16T18:35:52.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cabfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6cac1044a0 0x7f6cac199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:52.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cabfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6cac1044a0 0x7f6cac199d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53516/0 (socket says 172.21.15.2:53516) 2023-11-16T18:35:52.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cabfff700 1 -- 172.21.15.2:0/2568710157 learned_addr learned my addr 172.21.15.2:0/2568710157 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:52.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cab7fe700 1 --2- 172.21.15.2:0/2568710157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac0ff460 0x7f6cac1953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:52.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cabfff700 1 -- 172.21.15.2:0/2568710157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cac0ffe00 msgr2=0x7f6cac195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cabfff700 1 --2- 172.21.15.2:0/2568710157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cac0ffe00 0x7f6cac195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cabfff700 1 -- 172.21.15.2:0/2568710157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac0ff460 msgr2=0x7f6cac1953e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cabfff700 1 --2- 172.21.15.2:0/2568710157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac0ff460 0x7f6cac1953e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cabfff700 1 -- 172.21.15.2:0/2568710157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c0095f0 con 0x7f6cac1044a0 2023-11-16T18:35:52.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cabfff700 1 --2- 172.21.15.2:0/2568710157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6cac1044a0 0x7f6cac199d80 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400b9e0 tx=0x7f6ca400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:52.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6ca8ff9700 1 -- 172.21.15.2:0/2568710157 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca400c9b0 con 0x7f6cac1044a0 2023-11-16T18:35:52.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6ca8ff9700 1 -- 172.21.15.2:0/2568710157 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ca4014440 con 0x7f6cac1044a0 2023-11-16T18:35:52.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.565+0000 7f6cb1be3700 1 -- 172.21.15.2:0/2568710157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cac10f2c0 con 0x7f6cac1044a0 2023-11-16T18:35:52.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.566+0000 7f6ca8ff9700 1 -- 172.21.15.2:0/2568710157 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca40125a0 con 0x7f6cac1044a0 2023-11-16T18:35:52.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.566+0000 7f6cb1be3700 1 -- 172.21.15.2:0/2568710157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cac10f130 con 0x7f6cac1044a0 2023-11-16T18:35:52.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.567+0000 7f6cb1be3700 1 -- 172.21.15.2:0/2568710157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cac10f450 con 0x7f6cac1044a0 2023-11-16T18:35:52.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.568+0000 7f6ca8ff9700 1 -- 172.21.15.2:0/2568710157 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f6ca40145b0 con 0x7f6cac1044a0 2023-11-16T18:35:52.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.568+0000 7f6ca8ff9700 1 --2- 172.21.15.2:0/2568710157 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6c980a12c0 0x7f6c980a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:52.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.568+0000 7f6ca8ff9700 1 -- 172.21.15.2:0/2568710157 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9438+0+0 (secure 0 0 0) 0x7f6ca4006e40 con 0x7f6cac1044a0 2023-11-16T18:35:52.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.568+0000 7f6cab7fe700 1 --2- 172.21.15.2:0/2568710157 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6c980a12c0 0x7f6c980a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:52.573 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.569+0000 7f6cab7fe700 1 --2- 172.21.15.2:0/2568710157 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6c980a12c0 0x7f6c980a3770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6c94005fd0 tx=0x7f6c9400a560 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:52.575 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.573+0000 7f6ca8ff9700 1 -- 172.21.15.2:0/2568710157 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6ca404e730 con 0x7f6cac1044a0 2023-11-16T18:35:52.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.773+0000 7f6cb1be3700 1 -- 172.21.15.2:0/2568710157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cac1967d0 con 0x7f6cac1044a0 2023-11-16T18:35:52.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.774+0000 7f6ca8ff9700 1 -- 172.21.15.2:0/2568710157 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+25060 (secure 0 0 0) 0x7f6ca4095830 con 0x7f6cac1044a0 2023-11-16T18:35:52.777 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T18:35:52.777 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":103,"fsid":"08160fac-84ac-11ee-8dbe-212e2dc638e7","created":"2023-11-16T18:15:41.444339+0000","modified":"2023-11-16T18:35:41.412881+0000","last_up_change":"2023-11-16T18:22:02.065686+0000","last_in_change":"2023-11-16T18:21:44.277660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T18:18:46.259592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T18:22:44.902889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T18:22:46.187275+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T18:22:49.539148+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7d70b5c0-08db-4d20-879d-c61212c48535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6803","nonce":4123843884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6805","nonce":4123843884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6809","nonce":4123843884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4123843884},{"type":"v1","addr":"172.21.15.2:6807","nonce":4123843884}]},"public_addr":"172.21.15.2:6803/4123843884","cluster_addr":"172.21.15.2:6805/4123843884","heartbeat_back_addr":"172.21.15.2:6809/4123843884","heartbeat_front_addr":"172.21.15.2:6807/4123843884","state":["exists","up"]},{"osd":1,"uuid":"69bb34ad-0d6c-4e75-911b-e366bb104d4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6801","nonce":3016934247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6803","nonce":3016934247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6807","nonce":3016934247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3016934247},{"type":"v1","addr":"172.21.15.40:6805","nonce":3016934247}]},"public_addr":"172.21.15.40:6801/3016934247","cluster_addr":"172.21.15.40:6803/3016934247","heartbeat_back_addr":"172.21.15.40:6807/3016934247","heartbeat_front_addr":"172.21.15.40:6805/3016934247","state":["exists","up"]},{"osd":2,"uuid":"80f9c468-7dd5-494a-a6e2-98bea69fbe4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6801","nonce":3338843401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6803","nonce":3338843401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6807","nonce":3338843401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3338843401},{"type":"v1","addr":"172.21.15.188:6805","nonce":3338843401}]},"public_addr":"172.21.15.188:6801/3338843401","cluster_addr":"172.21.15.188:6803/3338843401","heartbeat_back_addr":"172.21.15.188:6807/3338843401","heartbeat_front_addr":"172.21.15.188:6805/3338843401","state":["exists","up"]},{"osd":3,"uuid":"9a13117e-ad7f-4576-8dcd-dee08b526577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6811","nonce":1081663732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6813","nonce":1081663732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6817","nonce":1081663732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1081663732},{"type":"v1","addr":"172.21.15.2:6815","nonce":1081663732}]},"public_addr":"172.21.15.2:6811/1081663732","cluster_addr":"172.21.15.2:6813/1081663732","heartbeat_back_addr":"172.21.15.2:6817/1081663732","heartbeat_front_addr":"172.21.15.2:6815/1081663732","state":["exists","up"]},{"osd":4,"uuid":"0453e4fa-a79a-4aea-a701-aedd106bdb08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6809","nonce":865046130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6811","nonce":865046130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6815","nonce":865046130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":865046130},{"type":"v1","addr":"172.21.15.40:6813","nonce":865046130}]},"public_addr":"172.21.15.40:6809/865046130","cluster_addr":"172.21.15.40:6811/865046130","heartbeat_back_addr":"172.21.15.40:6815/865046130","heartbeat_front_addr":"172.21.15.40:6813/865046130","state":["exists","up"]},{"osd":5,"uuid":"30994739-f13b-4840-b14d-af1553319d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6809","nonce":325018840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6811","nonce":325018840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6815","nonce":325018840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":325018840},{"type":"v1","addr":"172.21.15.188:6813","nonce":325018840}]},"public_addr":"172.21.15.188:6809/325018840","cluster_addr":"172.21.15.188:6811/325018840","heartbeat_back_addr":"172.21.15.188:6815/325018840","heartbeat_front_addr":"172.21.15.188:6813/325018840","state":["exists","up"]},{"osd":6,"uuid":"6d97c575-96ae-46ad-a165-36cc16633818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6819","nonce":1942175812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6821","nonce":1942175812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6825","nonce":1942175812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1942175812},{"type":"v1","addr":"172.21.15.2:6823","nonce":1942175812}]},"public_addr":"172.21.15.2:6819/1942175812","cluster_addr":"172.21.15.2:6821/1942175812","heartbeat_back_addr":"172.21.15.2:6825/1942175812","heartbeat_front_addr":"172.21.15.2:6823/1942175812","state":["exists","up"]},{"osd":7,"uuid":"bf7d9fd3-1e91-4106-b3c7-8e57ced5bd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6817","nonce":139194472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6819","nonce":139194472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6823","nonce":139194472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":139194472},{"type":"v1","addr":"172.21.15.40:6821","nonce":139194472}]},"public_addr":"172.21.15.40:6817/139194472","cluster_addr":"172.21.15.40:6819/139194472","heartbeat_back_addr":"172.21.15.40:6823/139194472","heartbeat_front_addr":"172.21.15.40:6821/139194472","state":["exists","up"]},{"osd":8,"uuid":"2589fd33-ad12-47e5-a70d-172ea07d61bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6817","nonce":3337313264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6819","nonce":3337313264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6823","nonce":3337313264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3337313264},{"type":"v1","addr":"172.21.15.188:6821","nonce":3337313264}]},"public_addr":"172.21.15.188:6817/3337313264","cluster_addr":"172.21.15.188:6819/3337313264","heartbeat_back_addr":"172.21.15.188:6823/3337313264","heartbeat_front_addr":"172.21.15.188:6821/3337313264","state":["exists","up"]},{"osd":9,"uuid":"2b081d7a-d6be-433c-9989-d5159ee895da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6827","nonce":1605296180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6829","nonce":1605296180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6833","nonce":1605296180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1605296180},{"type":"v1","addr":"172.21.15.2:6831","nonce":1605296180}]},"public_addr":"172.21.15.2:6827/1605296180","cluster_addr":"172.21.15.2:6829/1605296180","heartbeat_back_addr":"172.21.15.2:6833/1605296180","heartbeat_front_addr":"172.21.15.2:6831/1605296180","state":["exists","up"]},{"osd":10,"uuid":"a79f904a-1214-42bf-b36a-8a7709eb0c4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6825","nonce":4286013944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6827","nonce":4286013944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6831","nonce":4286013944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4286013944},{"type":"v1","addr":"172.21.15.40:6829","nonce":4286013944}]},"public_addr":"172.21.15.40:6825/4286013944","cluster_addr":"172.21.15.40:6827/4286013944","heartbeat_back_addr":"172.21.15.40:6831/4286013944","heartbeat_front_addr":"172.21.15.40:6829/4286013944","state":["exists","up"]},{"osd":11,"uuid":"f54ca334-9757-4c8b-90e3-cce7daa3b1b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6825","nonce":2926471675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6827","nonce":2926471675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6831","nonce":2926471675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2926471675},{"type":"v1","addr":"172.21.15.188:6829","nonce":2926471675}]},"public_addr":"172.21.15.188:6825/2926471675","cluster_addr":"172.21.15.188:6827/2926471675","heartbeat_back_addr":"172.21.15.188:6831/2926471675","heartbeat_front_addr":"172.21.15.188:6829/2926471675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:00.168938+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:22.554283+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:18:44.324842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:06.266353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:28.113470+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:19:49.755958+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:10.824747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:33.532017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:20:55.483673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:16.938758+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:21:38.610646+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T18:22:00.190374+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6800/3369431713":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/1472878779":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/1347603606":"2023-11-17T18:24:50.949710+0000","172.21.15.2:6801/3369431713":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/857143886":"2023-11-17T18:16:37.723675+0000","172.21.15.2:0/3335793429":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/251514255":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6800/1605535303":"2023-11-17T18:16:37.723675+0000","172.21.15.2:6801/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:6800/1609662822":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/708195783":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/1158002645":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/3821858662":"2023-11-17T18:24:50.949710+0000","172.21.15.2:0/1986390775":"2023-11-17T18:16:13.217812+0000","172.21.15.2:0/2955607810":"2023-11-17T18:16:37.723675+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T18:35:52.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.777+0000 7f6cb1be3700 1 -- 172.21.15.2:0/2568710157 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6c980a12c0 msgr2=0x7f6c980a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.777+0000 7f6cb1be3700 1 --2- 172.21.15.2:0/2568710157 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6c980a12c0 0x7f6c980a3770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6c94005fd0 tx=0x7f6c9400a560 comp rx=0 tx=0).stop 2023-11-16T18:35:52.779 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.777+0000 7f6cb1be3700 1 -- 172.21.15.2:0/2568710157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6cac1044a0 msgr2=0x7f6cac199d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.777+0000 7f6cb1be3700 1 --2- 172.21.15.2:0/2568710157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6cac1044a0 0x7f6cac199d80 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400b9e0 tx=0x7f6ca400bda0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.777+0000 7f6cb1be3700 1 -- 172.21.15.2:0/2568710157 shutdown_connections 2023-11-16T18:35:52.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.777+0000 7f6cb1be3700 1 --2- 172.21.15.2:0/2568710157 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6c980a12c0 0x7f6c980a3770 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.777+0000 7f6cb1be3700 1 --2- 172.21.15.2:0/2568710157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6cac1044a0 0x7f6cac199d80 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.777+0000 7f6cb1be3700 1 --2- 172.21.15.2:0/2568710157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac0ff460 0x7f6cac1953e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.777+0000 7f6cb1be3700 1 --2- 172.21.15.2:0/2568710157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cac0ffe00 0x7f6cac195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.777+0000 7f6cb1be3700 1 -- 172.21.15.2:0/2568710157 >> 172.21.15.2:0/2568710157 conn(0x7f6cac076310 msgr2=0x7f6cac0fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:52.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.777+0000 7f6cb1be3700 1 -- 172.21.15.2:0/2568710157 shutdown_connections 2023-11-16T18:35:52.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.777+0000 7f6cb1be3700 1 -- 172.21.15.2:0/2568710157 wait complete. 2023-11-16T18:35:52.792 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-16T18:35:52.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.916+0000 7f6d75566700 1 -- 172.21.15.2:0/661937170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d701054f0 msgr2=0x7f6d7010d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.916+0000 7f6d75566700 1 --2- 172.21.15.2:0/661937170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d701054f0 0x7f6d7010d0c0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f6d6800b0d0 tx=0x7f6d6800b3e0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.916+0000 7f6d75566700 1 -- 172.21.15.2:0/661937170 shutdown_connections 2023-11-16T18:35:52.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.916+0000 7f6d75566700 1 --2- 172.21.15.2:0/661937170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d700680b0 0x7f6d70068480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.916+0000 7f6d75566700 1 --2- 172.21.15.2:0/661937170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d701054f0 0x7f6d7010d0c0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.916+0000 7f6d75566700 1 --2- 172.21.15.2:0/661937170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d70068a50 0x7f6d70104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.916+0000 7f6d75566700 1 -- 172.21.15.2:0/661937170 >> 172.21.15.2:0/661937170 conn(0x7f6d70075f70 msgr2=0x7f6d70076370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:52.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.916+0000 7f6d75566700 1 -- 172.21.15.2:0/661937170 shutdown_connections 2023-11-16T18:35:52.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.916+0000 7f6d75566700 1 -- 172.21.15.2:0/661937170 wait complete. 2023-11-16T18:35:52.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.917+0000 7f6d75566700 1 Processor -- start 2023-11-16T18:35:52.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.917+0000 7f6d75566700 1 -- start start 2023-11-16T18:35:52.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.917+0000 7f6d75566700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d700680b0 0x7f6d701963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:52.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.917+0000 7f6d75566700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d70068a50 0x7f6d701968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:52.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.917+0000 7f6d75566700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d701054f0 0x7f6d7019ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:52.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.917+0000 7f6d75566700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7019b390 con 0x7f6d700680b0 2023-11-16T18:35:52.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.917+0000 7f6d75566700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7019b500 con 0x7f6d701054f0 2023-11-16T18:35:52.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.917+0000 7f6d75566700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7019b6c0 con 0x7f6d70068a50 2023-11-16T18:35:52.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.917+0000 7f6d6effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d700680b0 0x7f6d701963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:52.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.917+0000 7f6d6effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d700680b0 0x7f6d701963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53526/0 (socket says 172.21.15.2:53526) 2023-11-16T18:35:52.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.917+0000 7f6d6effd700 1 -- 172.21.15.2:0/1788192183 learned_addr learned my addr 172.21.15.2:0/1788192183 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:52.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.917+0000 7f6d6f7fe700 1 --2- 172.21.15.2:0/1788192183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d701054f0 0x7f6d7019ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:52.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.918+0000 7f6d6effd700 1 -- 172.21.15.2:0/1788192183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d70068a50 msgr2=0x7f6d701968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.929 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.918+0000 7f6d6effd700 1 --2- 172.21.15.2:0/1788192183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d70068a50 0x7f6d701968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.929 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.918+0000 7f6d6effd700 1 -- 172.21.15.2:0/1788192183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d701054f0 msgr2=0x7f6d7019ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:52.929 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.918+0000 7f6d6effd700 1 --2- 172.21.15.2:0/1788192183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d701054f0 0x7f6d7019ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:52.929 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.918+0000 7f6d6effd700 1 -- 172.21.15.2:0/1788192183 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d68009d00 con 0x7f6d700680b0 2023-11-16T18:35:52.930 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.918+0000 7f6d6effd700 1 --2- 172.21.15.2:0/1788192183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d700680b0 0x7f6d701963b0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6d6000b550 tx=0x7f6d6000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:52.930 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.918+0000 7f6d57fff700 1 -- 172.21.15.2:0/1788192183 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60011830 con 0x7f6d700680b0 2023-11-16T18:35:52.930 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.918+0000 7f6d57fff700 1 -- 172.21.15.2:0/1788192183 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d60011e50 con 0x7f6d700680b0 2023-11-16T18:35:52.930 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.918+0000 7f6d75566700 1 -- 172.21.15.2:0/1788192183 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d70110350 con 0x7f6d700680b0 2023-11-16T18:35:52.931 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.918+0000 7f6d57fff700 1 -- 172.21.15.2:0/1788192183 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6000f2e0 con 0x7f6d700680b0 2023-11-16T18:35:52.931 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.918+0000 7f6d75566700 1 -- 172.21.15.2:0/1788192183 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d701101c0 con 0x7f6d700680b0 2023-11-16T18:35:52.931 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.919+0000 7f6d75566700 1 -- 172.21.15.2:0/1788192183 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d701104e0 con 0x7f6d700680b0 2023-11-16T18:35:52.931 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.920+0000 7f6d57fff700 1 -- 172.21.15.2:0/1788192183 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f6d60010580 con 0x7f6d700680b0 2023-11-16T18:35:52.933 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.921+0000 7f6d57fff700 1 --2- 172.21.15.2:0/1788192183 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6d580a12c0 0x7f6d580a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:52.933 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.921+0000 7f6d6e7fc700 1 --2- 172.21.15.2:0/1788192183 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6d580a12c0 0x7f6d580a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:52.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.921+0000 7f6d57fff700 1 -- 172.21.15.2:0/1788192183 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9438+0+0 (secure 0 0 0) 0x7f6d600ccfc0 con 0x7f6d700680b0 2023-11-16T18:35:52.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.922+0000 7f6d6e7fc700 1 --2- 172.21.15.2:0/1788192183 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6d580a12c0 0x7f6d580a3770 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6d64005950 tx=0x7f6d640058e0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:52.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:52.925+0000 7f6d57fff700 1 -- 172.21.15.2:0/1788192183 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6d60096150 con 0x7f6d700680b0 2023-11-16T18:35:53.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:53.171+0000 7f6d75566700 1 -- 172.21.15.2:0/1788192183 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6d700679e0 con 0x7f6d700680b0 2023-11-16T18:35:53.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:53 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3677328845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:35:53.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:53 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2568710157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:35:53.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:53 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3677328845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:35:53.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:53 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2568710157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:35:53.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:53 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3677328845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T18:35:53.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:53 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2568710157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T18:35:54.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.172+0000 7f6d57fff700 1 -- 172.21.15.2:0/1788192183 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v38) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6d60015090 con 0x7f6d700680b0 2023-11-16T18:35:54.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.175+0000 7f6d75566700 1 -- 172.21.15.2:0/1788192183 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6d580a12c0 msgr2=0x7f6d580a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:54.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.175+0000 7f6d75566700 1 --2- 172.21.15.2:0/1788192183 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6d580a12c0 0x7f6d580a3770 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6d64005950 tx=0x7f6d640058e0 comp rx=0 tx=0).stop 2023-11-16T18:35:54.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.175+0000 7f6d75566700 1 -- 172.21.15.2:0/1788192183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d700680b0 msgr2=0x7f6d701963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:54.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.175+0000 7f6d75566700 1 --2- 172.21.15.2:0/1788192183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d700680b0 0x7f6d701963b0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6d6000b550 tx=0x7f6d6000b910 comp rx=0 tx=0).stop 2023-11-16T18:35:54.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.175+0000 7f6d75566700 1 -- 172.21.15.2:0/1788192183 shutdown_connections 2023-11-16T18:35:54.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.175+0000 7f6d75566700 1 --2- 172.21.15.2:0/1788192183 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f6d580a12c0 0x7f6d580a3770 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:54.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.175+0000 7f6d75566700 1 --2- 172.21.15.2:0/1788192183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d701054f0 0x7f6d7019ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:54.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.175+0000 7f6d75566700 1 --2- 172.21.15.2:0/1788192183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d700680b0 0x7f6d701963b0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:54.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.175+0000 7f6d75566700 1 --2- 172.21.15.2:0/1788192183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d70068a50 0x7f6d701968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:54.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.175+0000 7f6d75566700 1 -- 172.21.15.2:0/1788192183 >> 172.21.15.2:0/1788192183 conn(0x7f6d70075f70 msgr2=0x7f6d70103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:54.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.175+0000 7f6d75566700 1 -- 172.21.15.2:0/1788192183 shutdown_connections 2023-11-16T18:35:54.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.175+0000 7f6d75566700 1 -- 172.21.15.2:0/1788192183 wait complete. 2023-11-16T18:35:54.202 INFO:teuthology.orchestra.run.smithi002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-16T18:35:54.203 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-16T18:35:54.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.342+0000 7ffbd59e5700 1 -- 172.21.15.2:0/1989729271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbd0068770 msgr2=0x7ffbd010d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:54.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.342+0000 7ffbd59e5700 1 --2- 172.21.15.2:0/1989729271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbd0068770 0x7ffbd010d080 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009f90 tx=0x7ffbc800b330 comp rx=0 tx=0).stop 2023-11-16T18:35:54.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.342+0000 7ffbd59e5700 1 -- 172.21.15.2:0/1989729271 shutdown_connections 2023-11-16T18:35:54.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.342+0000 7ffbd59e5700 1 --2- 172.21.15.2:0/1989729271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbd0068770 0x7ffbd010d080 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:54.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.342+0000 7ffbd59e5700 1 --2- 172.21.15.2:0/1989729271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffbd01052a0 0x7ffbd0105670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:54.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.342+0000 7ffbd59e5700 1 --2- 172.21.15.2:0/1989729271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbd0067dc0 0x7ffbd0068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:54.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.342+0000 7ffbd59e5700 1 -- 172.21.15.2:0/1989729271 >> 172.21.15.2:0/1989729271 conn(0x7ffbd00763a0 msgr2=0x7ffbd00767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:54.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.342+0000 7ffbd59e5700 1 -- 172.21.15.2:0/1989729271 shutdown_connections 2023-11-16T18:35:54.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.342+0000 7ffbd59e5700 1 -- 172.21.15.2:0/1989729271 wait complete. 2023-11-16T18:35:54.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.342+0000 7ffbd59e5700 1 Processor -- start 2023-11-16T18:35:54.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.343+0000 7ffbd59e5700 1 -- start start 2023-11-16T18:35:54.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.343+0000 7ffbd59e5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbd0067dc0 0x7ffbd0196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:54.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.343+0000 7ffbd59e5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffbd0068770 0x7ffbd01969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:54.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.343+0000 7ffbd59e5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbd01052a0 0x7ffbd019ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:54.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.343+0000 7ffbd59e5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd019b460 con 0x7ffbd0068770 2023-11-16T18:35:54.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.343+0000 7ffbd59e5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd019b5d0 con 0x7ffbd0067dc0 2023-11-16T18:35:54.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.343+0000 7ffbd59e5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd019b790 con 0x7ffbd01052a0 2023-11-16T18:35:54.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.343+0000 7ffbce7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffbd0068770 0x7ffbd01969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:54.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.343+0000 7ffbceffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbd0067dc0 0x7ffbd0196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:54.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.343+0000 7ffbce7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffbd0068770 0x7ffbd01969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53540/0 (socket says 172.21.15.2:53540) 2023-11-16T18:35:54.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.343+0000 7ffbce7fc700 1 -- 172.21.15.2:0/3351714633 learned_addr learned my addr 172.21.15.2:0/3351714633 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:54.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.344+0000 7ffbce7fc700 1 -- 172.21.15.2:0/3351714633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbd01052a0 msgr2=0x7ffbd019ae20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-16T18:35:54.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.344+0000 7ffbce7fc700 1 --2- 172.21.15.2:0/3351714633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbd01052a0 0x7ffbd019ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:54.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.344+0000 7ffbce7fc700 1 -- 172.21.15.2:0/3351714633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbd0067dc0 msgr2=0x7ffbd0196480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:54.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.344+0000 7ffbce7fc700 1 --2- 172.21.15.2:0/3351714633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbd0067dc0 0x7ffbd0196480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:54.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.344+0000 7ffbce7fc700 1 -- 172.21.15.2:0/3351714633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc00095f0 con 0x7ffbd0068770 2023-11-16T18:35:54.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.344+0000 7ffbceffd700 1 --2- 172.21.15.2:0/3351714633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbd0067dc0 0x7ffbd0196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T18:35:54.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.344+0000 7ffbce7fc700 1 --2- 172.21.15.2:0/3351714633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffbd0068770 0x7ffbd01969c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400e9a0 tx=0x7ffbc400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:54.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.344+0000 7ffbd49e3700 1 -- 172.21.15.2:0/3351714633 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4009790 con 0x7ffbd0068770 2023-11-16T18:35:54.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.344+0000 7ffbd59e5700 1 -- 172.21.15.2:0/3351714633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc8009c40 con 0x7ffbd0068770 2023-11-16T18:35:54.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.344+0000 7ffbd49e3700 1 -- 172.21.15.2:0/3351714633 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffbc400cbd0 con 0x7ffbd0068770 2023-11-16T18:35:54.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.344+0000 7ffbd49e3700 1 -- 172.21.15.2:0/3351714633 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4018680 con 0x7ffbd0068770 2023-11-16T18:35:54.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.344+0000 7ffbd59e5700 1 -- 172.21.15.2:0/3351714633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd0110340 con 0x7ffbd0068770 2023-11-16T18:35:54.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.345+0000 7ffbd59e5700 1 -- 172.21.15.2:0/3351714633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd01101b0 con 0x7ffbd0068770 2023-11-16T18:35:54.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.347+0000 7ffbd49e3700 1 -- 172.21.15.2:0/3351714633 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7ffbc400cd40 con 0x7ffbd0068770 2023-11-16T18:35:54.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.348+0000 7ffbd49e3700 1 --2- 172.21.15.2:0/3351714633 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ffbb80a12c0 0x7ffbb80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:54.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.348+0000 7ffbd49e3700 1 -- 172.21.15.2:0/3351714633 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9652+0+0 (secure 0 0 0) 0x7ffbc4014070 con 0x7ffbd0068770 2023-11-16T18:35:54.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.348+0000 7ffbceffd700 1 --2- 172.21.15.2:0/3351714633 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ffbb80a12c0 0x7ffbb80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:54.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.348+0000 7ffbceffd700 1 --2- 172.21.15.2:0/3351714633 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ffbb80a12c0 0x7ffbb80a3770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0005fd0 tx=0x7ffbc0009310 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:54.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.350+0000 7ffbd49e3700 1 -- 172.21.15.2:0/3351714633 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ffbc4096660 con 0x7ffbd0068770 2023-11-16T18:35:54.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1788192183' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-16T18:35:54.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-mon[100639]: osdmap e104: 12 total, 12 up, 12 in 2023-11-16T18:35:54.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-mon[100639]: pgmap v347: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 920 B/s wr, 8 op/s 2023-11-16T18:35:54.548 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1788192183' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-16T18:35:54.548 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-mon[100251]: osdmap e104: 12 total, 12 up, 12 in 2023-11-16T18:35:54.549 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-mon[100251]: pgmap v347: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 920 B/s wr, 8 op/s 2023-11-16T18:35:54.549 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-c[126467]: ignoring --setuser ceph since I am not root 2023-11-16T18:35:54.549 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-c[126467]: ignoring --setgroup ceph since I am not root 2023-11-16T18:35:54.549 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-mds[126491]: -- 172.21.15.188:0/4278315814 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55915a7d9520 con 0x55915a7aec00 2023-11-16T18:35:54.549 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-c[126467]: starting mds.c at 2023-11-16T18:35:54.550 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-b[125477]: ignoring --setuser ceph since I am not root 2023-11-16T18:35:54.550 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-b[125477]: ignoring --setgroup ceph since I am not root 2023-11-16T18:35:54.550 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-mds[125503]: -- 172.21.15.40:0/2806136060 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56404d467520 con 0x56404d43d000 2023-11-16T18:35:54.550 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-b[125477]: starting mds.b at 2023-11-16T18:35:54.551 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-i[126813]: ignoring --setuser ceph since I am not root 2023-11-16T18:35:54.577 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-i[126813]: ignoring --setgroup ceph since I am not root 2023-11-16T18:35:54.577 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-mds[126837]: -- 172.21.15.188:0/1858427775 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56381441d520 con 0x5638143f2800 2023-11-16T18:35:54.577 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-i[126813]: starting mds.i at 2023-11-16T18:35:54.577 INFO:journalctl@ceph.mds.e.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-e[124788]: ignoring --setuser ceph since I am not root 2023-11-16T18:35:54.578 INFO:journalctl@ceph.mds.e.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-e[124788]: ignoring --setgroup ceph since I am not root 2023-11-16T18:35:54.578 INFO:journalctl@ceph.mds.e.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-e[124788]: starting mds.e at 2023-11-16T18:35:54.578 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-f[126125]: ignoring --setuser ceph since I am not root 2023-11-16T18:35:54.578 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-f[126125]: ignoring --setgroup ceph since I am not root 2023-11-16T18:35:54.578 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-mds[126149]: -- 172.21.15.188:0/2603247753 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5621292f9520 con 0x5621292cec00 2023-11-16T18:35:54.578 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-f[126125]: starting mds.f at 2023-11-16T18:35:54.579 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:35:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-f[126125]: 2023-11-16T18:35:54.232+0000 7f4173290700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6837/0: (98) Address already in use 2023-11-16T18:35:54.579 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: ignoring --setuser ceph since I am not root 2023-11-16T18:35:54.579 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: ignoring --setgroup ceph since I am not root 2023-11-16T18:35:54.579 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-mds[125155]: -- 172.21.15.40:0/2212346086 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e22cd39520 con 0x55e22cd0f000 2023-11-16T18:35:54.579 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:35:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: starting mds.h at 2023-11-16T18:35:54.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:54.598+0000 7ffbd59e5700 1 -- 172.21.15.2:0/3351714633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ffbd01104d0 con 0x7ffbd0068770 2023-11-16T18:35:54.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:54 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-a[98783]: 2023-11-16T18:35:54.167+0000 7f30d8773700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-16T18:35:54.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:54 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1788192183' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-16T18:35:54.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:54 smithi002 ceph-mon[98807]: osdmap e104: 12 total, 12 up, 12 in 2023-11-16T18:35:54.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:54 smithi002 ceph-mon[98807]: pgmap v347: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 920 B/s wr, 8 op/s 2023-11-16T18:35:55.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.185+0000 7ffbd49e3700 1 -- 172.21.15.2:0/3351714633 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) v1 ==== 104+0+0 (secure 0 0 0) 0x7ffbc4099c80 con 0x7ffbd0068770 2023-11-16T18:35:55.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.188+0000 7ffbd59e5700 1 -- 172.21.15.2:0/3351714633 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ffbb80a12c0 msgr2=0x7ffbb80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:55.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.188+0000 7ffbd59e5700 1 --2- 172.21.15.2:0/3351714633 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ffbb80a12c0 0x7ffbb80a3770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0005fd0 tx=0x7ffbc0009310 comp rx=0 tx=0).stop 2023-11-16T18:35:55.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.188+0000 7ffbd59e5700 1 -- 172.21.15.2:0/3351714633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffbd0068770 msgr2=0x7ffbd01969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:55.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.188+0000 7ffbd59e5700 1 --2- 172.21.15.2:0/3351714633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffbd0068770 0x7ffbd01969c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400e9a0 tx=0x7ffbc400ecb0 comp rx=0 tx=0).stop 2023-11-16T18:35:55.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.188+0000 7ffbd59e5700 1 -- 172.21.15.2:0/3351714633 shutdown_connections 2023-11-16T18:35:55.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.188+0000 7ffbd59e5700 1 --2- 172.21.15.2:0/3351714633 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7ffbb80a12c0 0x7ffbb80a3770 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:55.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.188+0000 7ffbd59e5700 1 --2- 172.21.15.2:0/3351714633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbd01052a0 0x7ffbd019ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:55.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.188+0000 7ffbd59e5700 1 --2- 172.21.15.2:0/3351714633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbd0067dc0 0x7ffbd0196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:55.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.188+0000 7ffbd59e5700 1 --2- 172.21.15.2:0/3351714633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffbd0068770 0x7ffbd01969c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:55.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.188+0000 7ffbd59e5700 1 -- 172.21.15.2:0/3351714633 >> 172.21.15.2:0/3351714633 conn(0x7ffbd00763a0 msgr2=0x7ffbd00fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:55.192 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.188+0000 7ffbd59e5700 1 -- 172.21.15.2:0/3351714633 shutdown_connections 2023-11-16T18:35:55.192 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.188+0000 7ffbd59e5700 1 -- 172.21.15.2:0/3351714633 wait complete. 2023-11-16T18:35:55.206 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-16T18:35:55.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.336+0000 7fc5ed91d700 1 -- 172.21.15.2:0/580367818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5e8100420 msgr2=0x7fc5e810bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:55.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.336+0000 7fc5ed91d700 1 --2- 172.21.15.2:0/580367818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5e8100420 0x7fc5e810bfd0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000b0a0 tx=0x7fc5e000b3b0 comp rx=0 tx=0).stop 2023-11-16T18:35:55.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.336+0000 7fc5ed91d700 1 -- 172.21.15.2:0/580367818 shutdown_connections 2023-11-16T18:35:55.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.336+0000 7fc5ed91d700 1 --2- 172.21.15.2:0/580367818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5e8105250 0x7fc5e8105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:55.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.336+0000 7fc5ed91d700 1 --2- 172.21.15.2:0/580367818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5e8100420 0x7fc5e810bfd0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:55.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.336+0000 7fc5ed91d700 1 --2- 172.21.15.2:0/580367818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5e80ffa70 0x7fc5e80ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:55.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.336+0000 7fc5ed91d700 1 -- 172.21.15.2:0/580367818 >> 172.21.15.2:0/580367818 conn(0x7fc5e80763a0 msgr2=0x7fc5e80767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:55.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.336+0000 7fc5ed91d700 1 -- 172.21.15.2:0/580367818 shutdown_connections 2023-11-16T18:35:55.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.336+0000 7fc5ed91d700 1 -- 172.21.15.2:0/580367818 wait complete. 2023-11-16T18:35:55.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.337+0000 7fc5ed91d700 1 Processor -- start 2023-11-16T18:35:55.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.337+0000 7fc5ed91d700 1 -- start start 2023-11-16T18:35:55.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.337+0000 7fc5ed91d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5e80ffa70 0x7fc5e8197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:55.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.337+0000 7fc5ed91d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5e8100420 0x7fc5e8197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:55.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.337+0000 7fc5ed91d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5e8105250 0x7fc5e81a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:55.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.337+0000 7fc5ed91d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e8198170 con 0x7fc5e80ffa70 2023-11-16T18:35:55.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.337+0000 7fc5ed91d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e81a28e0 con 0x7fc5e8105250 2023-11-16T18:35:55.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.337+0000 7fc5ed91d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e81a2a50 con 0x7fc5e8100420 2023-11-16T18:35:55.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.337+0000 7fc5e6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5e80ffa70 0x7fc5e8197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:55.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.337+0000 7fc5e6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5e80ffa70 0x7fc5e8197320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53552/0 (socket says 172.21.15.2:53552) 2023-11-16T18:35:55.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.337+0000 7fc5e6ffd700 1 -- 172.21.15.2:0/1241001749 learned_addr learned my addr 172.21.15.2:0/1241001749 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:55.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.338+0000 7fc5e77fe700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5e8105250 0x7fc5e81a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:55.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.338+0000 7fc5e67fc700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5e8100420 0x7fc5e8197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:55.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.338+0000 7fc5e6ffd700 1 -- 172.21.15.2:0/1241001749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5e8100420 msgr2=0x7fc5e8197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:55.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.338+0000 7fc5e6ffd700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5e8100420 0x7fc5e8197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:55.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.338+0000 7fc5e6ffd700 1 -- 172.21.15.2:0/1241001749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5e8105250 msgr2=0x7fc5e81a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:55.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.338+0000 7fc5e6ffd700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5e8105250 0x7fc5e81a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:55.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.338+0000 7fc5e6ffd700 1 -- 172.21.15.2:0/1241001749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e0009d00 con 0x7fc5e80ffa70 2023-11-16T18:35:55.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.338+0000 7fc5e6ffd700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5e80ffa70 0x7fc5e8197320 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00d950 tx=0x7fc5dc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:55.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.338+0000 7fc5ec91b700 1 -- 172.21.15.2:0/1241001749 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc009940 con 0x7fc5e80ffa70 2023-11-16T18:35:55.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.338+0000 7fc5ec91b700 1 -- 172.21.15.2:0/1241001749 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5dc010460 con 0x7fc5e80ffa70 2023-11-16T18:35:55.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.338+0000 7fc5ed91d700 1 -- 172.21.15.2:0/1241001749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e810f1e0 con 0x7fc5e80ffa70 2023-11-16T18:35:55.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.338+0000 7fc5ec91b700 1 -- 172.21.15.2:0/1241001749 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc00f620 con 0x7fc5e80ffa70 2023-11-16T18:35:55.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.338+0000 7fc5ed91d700 1 -- 172.21.15.2:0/1241001749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e810f050 con 0x7fc5e80ffa70 2023-11-16T18:35:55.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.339+0000 7fc5ed91d700 1 -- 172.21.15.2:0/1241001749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e810f370 con 0x7fc5e80ffa70 2023-11-16T18:35:55.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.340+0000 7fc5ec91b700 1 -- 172.21.15.2:0/1241001749 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fc5dc0108d0 con 0x7fc5e80ffa70 2023-11-16T18:35:55.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.341+0000 7fc5ec91b700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc5d00a12c0 0x7fc5d00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:55.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.341+0000 7fc5e67fc700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc5d00a12c0 0x7fc5d00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:55.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.341+0000 7fc5ec91b700 1 -- 172.21.15.2:0/1241001749 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9652+0+0 (secure 0 0 0) 0x7fc5dc0ccf40 con 0x7fc5e80ffa70 2023-11-16T18:35:55.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.341+0000 7fc5e67fc700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc5d00a12c0 0x7fc5d00a3770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8005950 tx=0x7fc5d80058e0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:55.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.344+0000 7fc5ec91b700 1 -- 172.21.15.2:0/1241001749 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc5dc095e80 con 0x7fc5e80ffa70 2023-11-16T18:35:55.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:55 smithi040 ceph-mon[100251]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-11-16T18:35:55.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:55 smithi040 ceph-mon[100251]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-16T18:35:55.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:55 smithi040 ceph-mon[100251]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2023-11-16T18:35:55.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:55 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1788192183' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-11-16T18:35:55.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:55 smithi040 ceph-mon[100251]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-11-16T18:35:55.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:55 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3351714633' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T18:35:55.491 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:55 smithi002 ceph-mon[98807]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-11-16T18:35:55.491 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:55 smithi002 ceph-mon[98807]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-16T18:35:55.491 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:55 smithi002 ceph-mon[98807]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2023-11-16T18:35:55.492 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:55 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1788192183' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-11-16T18:35:55.492 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:55 smithi002 ceph-mon[98807]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-11-16T18:35:55.492 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:55 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3351714633' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T18:35:55.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:55 smithi188 ceph-mon[100639]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-11-16T18:35:55.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:55 smithi188 ceph-mon[100639]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-16T18:35:55.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:55 smithi188 ceph-mon[100639]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2023-11-16T18:35:55.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:55 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1788192183' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-11-16T18:35:55.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:55 smithi188 ceph-mon[100639]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-11-16T18:35:55.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:55 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3351714633' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T18:35:55.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:55.551+0000 7fc5ed91d700 1 -- 172.21.15.2:0/1241001749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc5e8067a50 con 0x7fc5e80ffa70 2023-11-16T18:35:56.193 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.190+0000 7fc5ec91b700 1 -- 172.21.15.2:0/1241001749 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v105) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc5dc0160d0 con 0x7fc5e80ffa70 2023-11-16T18:35:56.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.193+0000 7fc5ed91d700 1 -- 172.21.15.2:0/1241001749 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc5d00a12c0 msgr2=0x7fc5d00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:56.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.193+0000 7fc5ed91d700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc5d00a12c0 0x7fc5d00a3770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8005950 tx=0x7fc5d80058e0 comp rx=0 tx=0).stop 2023-11-16T18:35:56.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.193+0000 7fc5ed91d700 1 -- 172.21.15.2:0/1241001749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5e80ffa70 msgr2=0x7fc5e8197320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:56.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.193+0000 7fc5ed91d700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5e80ffa70 0x7fc5e8197320 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00d950 tx=0x7fc5dc00dc60 comp rx=0 tx=0).stop 2023-11-16T18:35:56.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.194+0000 7fc5ed91d700 1 -- 172.21.15.2:0/1241001749 shutdown_connections 2023-11-16T18:35:56.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.194+0000 7fc5ed91d700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7fc5d00a12c0 0x7fc5d00a3770 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:56.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.194+0000 7fc5ed91d700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5e8105250 0x7fc5e81a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:56.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.194+0000 7fc5ed91d700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5e80ffa70 0x7fc5e8197320 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:56.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.194+0000 7fc5ed91d700 1 --2- 172.21.15.2:0/1241001749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5e8100420 0x7fc5e8197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:56.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.194+0000 7fc5ed91d700 1 -- 172.21.15.2:0/1241001749 >> 172.21.15.2:0/1241001749 conn(0x7fc5e80763a0 msgr2=0x7fc5e80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:56.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.194+0000 7fc5ed91d700 1 -- 172.21.15.2:0/1241001749 shutdown_connections 2023-11-16T18:35:56.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.194+0000 7fc5ed91d700 1 -- 172.21.15.2:0/1241001749 wait complete. 2023-11-16T18:35:56.197 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' removed 2023-11-16T18:35:56.210 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-16T18:35:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.344+0000 7f46c578d700 1 -- 172.21.15.2:0/1749748273 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c0105250 msgr2=0x7f46c0105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.344+0000 7f46c578d700 1 --2- 172.21.15.2:0/1749748273 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c0105250 0x7f46c0105620 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009990 tx=0x7f46b0009ca0 comp rx=0 tx=0).stop 2023-11-16T18:35:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.344+0000 7f46c578d700 1 -- 172.21.15.2:0/1749748273 shutdown_connections 2023-11-16T18:35:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.344+0000 7f46c578d700 1 --2- 172.21.15.2:0/1749748273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46c0100420 0x7f46c010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.344+0000 7f46c578d700 1 --2- 172.21.15.2:0/1749748273 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c0105250 0x7f46c0105620 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.344+0000 7f46c578d700 1 --2- 172.21.15.2:0/1749748273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c00ffa70 0x7f46c00ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:56.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.344+0000 7f46c578d700 1 -- 172.21.15.2:0/1749748273 >> 172.21.15.2:0/1749748273 conn(0x7f46c00763a0 msgr2=0x7f46c00767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:56.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.344+0000 7f46c578d700 1 -- 172.21.15.2:0/1749748273 shutdown_connections 2023-11-16T18:35:56.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.344+0000 7f46c578d700 1 -- 172.21.15.2:0/1749748273 wait complete. 2023-11-16T18:35:56.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.345+0000 7f46c578d700 1 Processor -- start 2023-11-16T18:35:56.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.345+0000 7f46c578d700 1 -- start start 2023-11-16T18:35:56.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46c578d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c00ffa70 0x7f46c0197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:56.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46c578d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46c0100420 0x7f46c01975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:56.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46c578d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c0105250 0x7f46c01a2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:56.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46c578d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c0197f40 con 0x7f46c00ffa70 2023-11-16T18:35:56.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46c578d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c0194ea0 con 0x7f46c0105250 2023-11-16T18:35:56.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46c578d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c0195060 con 0x7f46c0100420 2023-11-16T18:35:56.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46beffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c00ffa70 0x7f46c0197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:56.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46beffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c00ffa70 0x7f46c0197080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53566/0 (socket says 172.21.15.2:53566) 2023-11-16T18:35:56.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46beffd700 1 -- 172.21.15.2:0/2392689261 learned_addr learned my addr 172.21.15.2:0/2392689261 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:56.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46beffd700 1 -- 172.21.15.2:0/2392689261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46c0100420 msgr2=0x7f46c01975c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T18:35:56.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46beffd700 1 --2- 172.21.15.2:0/2392689261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46c0100420 0x7f46c01975c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:56.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46beffd700 1 -- 172.21.15.2:0/2392689261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c0105250 msgr2=0x7f46c01a2100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T18:35:56.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46beffd700 1 --2- 172.21.15.2:0/2392689261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c0105250 0x7f46c01a2100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:56.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.346+0000 7f46beffd700 1 -- 172.21.15.2:0/2392689261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b00095f0 con 0x7f46c00ffa70 2023-11-16T18:35:56.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.347+0000 7f46beffd700 1 --2- 172.21.15.2:0/2392689261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c00ffa70 0x7f46c0197080 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f46b0000c00 tx=0x7f46b00037a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:56.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.349+0000 7f46a7fff700 1 -- 172.21.15.2:0/2392689261 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b00052c0 con 0x7f46c00ffa70 2023-11-16T18:35:56.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.349+0000 7f46c578d700 1 -- 172.21.15.2:0/2392689261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c010f1b0 con 0x7f46c00ffa70 2023-11-16T18:35:56.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.349+0000 7f46a7fff700 1 -- 172.21.15.2:0/2392689261 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46b002e070 con 0x7f46c00ffa70 2023-11-16T18:35:56.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.349+0000 7f46a7fff700 1 -- 172.21.15.2:0/2392689261 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b000fe20 con 0x7f46c00ffa70 2023-11-16T18:35:56.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.349+0000 7f46c578d700 1 -- 172.21.15.2:0/2392689261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c010f020 con 0x7f46c00ffa70 2023-11-16T18:35:56.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.350+0000 7f46c578d700 1 -- 172.21.15.2:0/2392689261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c010f340 con 0x7f46c00ffa70 2023-11-16T18:35:56.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.353+0000 7f46a7fff700 1 -- 172.21.15.2:0/2392689261 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f46b0030030 con 0x7f46c00ffa70 2023-11-16T18:35:56.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.353+0000 7f46a7fff700 1 --2- 172.21.15.2:0/2392689261 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f46a80a1340 0x7f46a80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:56.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.353+0000 7f46a7fff700 1 -- 172.21.15.2:0/2392689261 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9098+0+0 (secure 0 0 0) 0x7f46b00defa0 con 0x7f46c00ffa70 2023-11-16T18:35:56.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.353+0000 7f46be7fc700 1 --2- 172.21.15.2:0/2392689261 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f46a80a1340 0x7f46a80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:56.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.354+0000 7f46be7fc700 1 --2- 172.21.15.2:0/2392689261 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f46a80a1340 0x7f46a80a37f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f46b400ba10 tx=0x7f46b400b3f0 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:56.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.357+0000 7f46a7fff700 1 -- 172.21.15.2:0/2392689261 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f46b006b730 con 0x7f46c00ffa70 2023-11-16T18:35:56.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-11-16T18:35:56.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-16T18:35:56.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: Cluster is now healthy 2023-11-16T18:35:56.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.188:6834/2291696976,v1:172.21.15.188:6835/2291696976] up:boot 2023-11-16T18:35:56.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.188:6837/796703309,v1:172.21.15.188:6839/796703309] up:boot 2023-11-16T18:35:56.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.188:6836/4015416515,v1:172.21.15.188:6838/4015416515] up:boot 2023-11-16T18:35:56.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.40:6835/3125450470,v1:172.21.15.40:6838/3125450470] up:boot 2023-11-16T18:35:56.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.40:6836/564695362,v1:172.21.15.40:6839/564695362] up:boot 2023-11-16T18:35:56.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: mds.? [v2:172.21.15.40:6834/441885916,v1:172.21.15.40:6837/441885916] up:boot 2023-11-16T18:35:56.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/3351714633' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-11-16T18:35:56.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: fsmap 12 up:standby 2023-11-16T18:35:56.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T18:35:56.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: pgmap v348: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 920 B/s wr, 8 op/s 2023-11-16T18:35:56.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T18:35:56.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T18:35:56.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T18:35:56.485 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T18:35:56.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T18:35:56.486 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:56 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1241001749' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T18:35:56.497 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-11-16T18:35:56.497 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-16T18:35:56.497 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: Cluster is now healthy 2023-11-16T18:35:56.498 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.188:6834/2291696976,v1:172.21.15.188:6835/2291696976] up:boot 2023-11-16T18:35:56.498 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.188:6837/796703309,v1:172.21.15.188:6839/796703309] up:boot 2023-11-16T18:35:56.498 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.188:6836/4015416515,v1:172.21.15.188:6838/4015416515] up:boot 2023-11-16T18:35:56.498 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.40:6835/3125450470,v1:172.21.15.40:6838/3125450470] up:boot 2023-11-16T18:35:56.498 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.40:6836/564695362,v1:172.21.15.40:6839/564695362] up:boot 2023-11-16T18:35:56.498 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: mds.? [v2:172.21.15.40:6834/441885916,v1:172.21.15.40:6837/441885916] up:boot 2023-11-16T18:35:56.498 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/3351714633' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-11-16T18:35:56.498 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: fsmap 12 up:standby 2023-11-16T18:35:56.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T18:35:56.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: pgmap v348: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 920 B/s wr, 8 op/s 2023-11-16T18:35:56.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T18:35:56.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T18:35:56.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T18:35:56.529 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T18:35:56.529 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T18:35:56.529 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:56 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1241001749' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T18:35:56.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-11-16T18:35:56.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-16T18:35:56.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: Cluster is now healthy 2023-11-16T18:35:56.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.188:6834/2291696976,v1:172.21.15.188:6835/2291696976] up:boot 2023-11-16T18:35:56.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.188:6837/796703309,v1:172.21.15.188:6839/796703309] up:boot 2023-11-16T18:35:56.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.188:6836/4015416515,v1:172.21.15.188:6838/4015416515] up:boot 2023-11-16T18:35:56.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.40:6835/3125450470,v1:172.21.15.40:6838/3125450470] up:boot 2023-11-16T18:35:56.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.40:6836/564695362,v1:172.21.15.40:6839/564695362] up:boot 2023-11-16T18:35:56.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: mds.? [v2:172.21.15.40:6834/441885916,v1:172.21.15.40:6837/441885916] up:boot 2023-11-16T18:35:56.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/3351714633' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-11-16T18:35:56.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: fsmap 12 up:standby 2023-11-16T18:35:56.549 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T18:35:56.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: pgmap v348: 129 pgs: 129 active+clean; 719 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 920 B/s wr, 8 op/s 2023-11-16T18:35:56.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T18:35:56.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T18:35:56.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T18:35:56.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T18:35:56.550 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: from='mgr.15594 172.21.15.2:0/1110311109' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T18:35:56.551 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:56 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1241001749' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T18:35:56.571 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:56.570+0000 7f46c578d700 1 -- 172.21.15.2:0/2392689261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f46c0195b40 con 0x7f46c00ffa70 2023-11-16T18:35:57.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.427+0000 7f46a7fff700 1 -- 172.21.15.2:0/2392689261 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v106) v1 ==== 169+0+0 (secure 0 0 0) 0x7f46b00a81a0 con 0x7f46c00ffa70 2023-11-16T18:35:57.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.431+0000 7f46c578d700 1 -- 172.21.15.2:0/2392689261 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f46a80a1340 msgr2=0x7f46a80a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:57.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.431+0000 7f46c578d700 1 --2- 172.21.15.2:0/2392689261 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f46a80a1340 0x7f46a80a37f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f46b400ba10 tx=0x7f46b400b3f0 comp rx=0 tx=0).stop 2023-11-16T18:35:57.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.431+0000 7f46c578d700 1 -- 172.21.15.2:0/2392689261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c00ffa70 msgr2=0x7f46c0197080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:57.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.431+0000 7f46c578d700 1 --2- 172.21.15.2:0/2392689261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c00ffa70 0x7f46c0197080 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f46b0000c00 tx=0x7f46b00037a0 comp rx=0 tx=0).stop 2023-11-16T18:35:57.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.431+0000 7f46c578d700 1 -- 172.21.15.2:0/2392689261 shutdown_connections 2023-11-16T18:35:57.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.431+0000 7f46c578d700 1 --2- 172.21.15.2:0/2392689261 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f46a80a1340 0x7f46a80a37f0 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f46b400ba10 tx=0x7f46b400b3f0 comp rx=0 tx=0).stop 2023-11-16T18:35:57.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.431+0000 7f46c578d700 1 --2- 172.21.15.2:0/2392689261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c0105250 0x7f46c01a2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:57.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.431+0000 7f46c578d700 1 --2- 172.21.15.2:0/2392689261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46c00ffa70 0x7f46c0197080 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:57.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.431+0000 7f46c578d700 1 --2- 172.21.15.2:0/2392689261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46c0100420 0x7f46c01975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:57.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.431+0000 7f46c578d700 1 -- 172.21.15.2:0/2392689261 >> 172.21.15.2:0/2392689261 conn(0x7f46c00763a0 msgr2=0x7f46c00fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:57.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.433+0000 7f46c578d700 1 -- 172.21.15.2:0/2392689261 shutdown_connections 2023-11-16T18:35:57.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.434+0000 7f46c578d700 1 -- 172.21.15.2:0/2392689261 wait complete. 2023-11-16T18:35:57.436 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' removed 2023-11-16T18:35:57.452 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:57 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/1241001749' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T18:35:57.452 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:57 smithi002 ceph-mon[98807]: osdmap e105: 12 total, 12 up, 12 in 2023-11-16T18:35:57.453 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:57 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2392689261' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T18:35:57.454 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-11-16T18:35:57.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:57 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/1241001749' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T18:35:57.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:57 smithi040 ceph-mon[100251]: osdmap e105: 12 total, 12 up, 12 in 2023-11-16T18:35:57.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:57 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2392689261' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T18:35:57.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:57 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/1241001749' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T18:35:57.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:57 smithi188 ceph-mon[100639]: osdmap e105: 12 total, 12 up, 12 in 2023-11-16T18:35:57.548 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:57 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2392689261' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T18:35:57.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.591+0000 7f99d376a700 1 -- 172.21.15.2:0/1141496041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99cc105250 msgr2=0x7f99cc105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.591+0000 7f99d376a700 1 --2- 172.21.15.2:0/1141496041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99cc105250 0x7f99cc105620 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f99bc0099c0 tx=0x7f99bc009cd0 comp rx=0 tx=0).stop 2023-11-16T18:35:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.591+0000 7f99d376a700 1 -- 172.21.15.2:0/1141496041 shutdown_connections 2023-11-16T18:35:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.591+0000 7f99d376a700 1 --2- 172.21.15.2:0/1141496041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99cc069a80 0x7f99cc10d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.591+0000 7f99d376a700 1 --2- 172.21.15.2:0/1141496041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99cc105250 0x7f99cc105620 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.591+0000 7f99d376a700 1 --2- 172.21.15.2:0/1141496041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99cc0690d0 0x7f99cc069540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.591+0000 7f99d376a700 1 -- 172.21.15.2:0/1141496041 >> 172.21.15.2:0/1141496041 conn(0x7f99cc076370 msgr2=0x7f99cc076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.591+0000 7f99d376a700 1 -- 172.21.15.2:0/1141496041 shutdown_connections 2023-11-16T18:35:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.591+0000 7f99d376a700 1 -- 172.21.15.2:0/1141496041 wait complete. 2023-11-16T18:35:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d376a700 1 Processor -- start 2023-11-16T18:35:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d376a700 1 -- start start 2023-11-16T18:35:57.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d376a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99cc0690d0 0x7f99cc196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:57.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d376a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99cc069a80 0x7f99cc196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:57.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d376a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99cc105250 0x7f99cc19acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d376a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc19b300 con 0x7f99cc069a80 2023-11-16T18:35:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d376a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc19b470 con 0x7f99cc105250 2023-11-16T18:35:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d376a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc19b630 con 0x7f99cc0690d0 2023-11-16T18:35:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d0d05700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99cc069a80 0x7f99cc196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d0d05700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99cc069a80 0x7f99cc196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53576/0 (socket says 172.21.15.2:53576) 2023-11-16T18:35:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d0d05700 1 -- 172.21.15.2:0/2814337300 learned_addr learned my addr 172.21.15.2:0/2814337300 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T18:35:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d1506700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99cc0690d0 0x7f99cc196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d0d05700 1 -- 172.21.15.2:0/2814337300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99cc0690d0 msgr2=0x7f99cc196320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:57.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d0d05700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99cc0690d0 0x7f99cc196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:57.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d1d07700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99cc105250 0x7f99cc19acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:57.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d0d05700 1 -- 172.21.15.2:0/2814337300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99cc105250 msgr2=0x7f99cc19acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:57.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d0d05700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99cc105250 0x7f99cc19acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:57.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d0d05700 1 -- 172.21.15.2:0/2814337300 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99bc0095f0 con 0x7f99cc069a80 2023-11-16T18:35:57.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.592+0000 7f99d0d05700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99cc069a80 0x7f99cc196860 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f99c800d950 tx=0x7f99c800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:57.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.593+0000 7f99c27fc700 1 -- 172.21.15.2:0/2814337300 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c8009b40 con 0x7f99cc069a80 2023-11-16T18:35:57.599 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.593+0000 7f99c27fc700 1 -- 172.21.15.2:0/2814337300 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99c8009ca0 con 0x7f99cc069a80 2023-11-16T18:35:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.593+0000 7f99c27fc700 1 -- 172.21.15.2:0/2814337300 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c800f620 con 0x7f99cc069a80 2023-11-16T18:35:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.593+0000 7f99d376a700 1 -- 172.21.15.2:0/2814337300 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99cc110360 con 0x7f99cc069a80 2023-11-16T18:35:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.593+0000 7f99d376a700 1 -- 172.21.15.2:0/2814337300 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99cc1101d0 con 0x7f99cc069a80 2023-11-16T18:35:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.593+0000 7f99d376a700 1 -- 172.21.15.2:0/2814337300 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99cc1104f0 con 0x7f99cc069a80 2023-11-16T18:35:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.595+0000 7f99c27fc700 1 -- 172.21.15.2:0/2814337300 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f99c8010460 con 0x7f99cc069a80 2023-11-16T18:35:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.595+0000 7f99c27fc700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f99b80a12c0 0x7f99b80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T18:35:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.595+0000 7f99c27fc700 1 -- 172.21.15.2:0/2814337300 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8510+0+0 (secure 0 0 0) 0x7f99c80cd090 con 0x7f99cc069a80 2023-11-16T18:35:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.595+0000 7f99d1506700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f99b80a12c0 0x7f99b80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T18:35:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.596+0000 7f99d1506700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f99b80a12c0 0x7f99b80a3770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f99bc005f50 tx=0x7f99bc005e80 comp rx=0 tx=0).ready entity=mgr.15594 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T18:35:57.601 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.599+0000 7f99c27fc700 1 -- 172.21.15.2:0/2814337300 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f99c80964b0 con 0x7f99cc069a80 2023-11-16T18:35:57.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:57.811+0000 7f99d376a700 1 -- 172.21.15.2:0/2814337300 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f99cc067ea0 con 0x7f99cc069a80 2023-11-16T18:35:58.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:58.430+0000 7f99c27fc700 1 -- 172.21.15.2:0/2814337300 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v107) v1 ==== 178+0+0 (secure 0 0 0) 0x7f99c80200d0 con 0x7f99cc069a80 2023-11-16T18:35:58.436 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:58.434+0000 7f99d376a700 1 -- 172.21.15.2:0/2814337300 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f99b80a12c0 msgr2=0x7f99b80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:58.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:58.434+0000 7f99d376a700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f99b80a12c0 0x7f99b80a3770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f99bc005f50 tx=0x7f99bc005e80 comp rx=0 tx=0).stop 2023-11-16T18:35:58.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:58.434+0000 7f99d376a700 1 -- 172.21.15.2:0/2814337300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99cc069a80 msgr2=0x7f99cc196860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T18:35:58.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:58.434+0000 7f99d376a700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99cc069a80 0x7f99cc196860 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f99c800d950 tx=0x7f99c800dd10 comp rx=0 tx=0).stop 2023-11-16T18:35:58.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:58.436+0000 7f99d376a700 1 -- 172.21.15.2:0/2814337300 shutdown_connections 2023-11-16T18:35:58.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:58.436+0000 7f99d376a700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.2:6800/2202170290,v1:172.21.15.2:6801/2202170290] conn(0x7f99b80a12c0 0x7f99b80a3770 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:58.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:58.436+0000 7f99d376a700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99cc0690d0 0x7f99cc196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:58.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:58.436+0000 7f99d376a700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99cc105250 0x7f99cc19acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:58.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:58.436+0000 7f99d376a700 1 --2- 172.21.15.2:0/2814337300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99cc069a80 0x7f99cc196860 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T18:35:58.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:58.436+0000 7f99d376a700 1 -- 172.21.15.2:0/2814337300 >> 172.21.15.2:0/2814337300 conn(0x7f99cc076370 msgr2=0x7f99cc0fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T18:35:58.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:58.436+0000 7f99d376a700 1 -- 172.21.15.2:0/2814337300 shutdown_connections 2023-11-16T18:35:58.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T18:35:58.437+0000 7f99d376a700 1 -- 172.21.15.2:0/2814337300 wait complete. 2023-11-16T18:35:58.440 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data_ec' removed 2023-11-16T18:35:58.460 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T18:35:58.461 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:58 smithi002 ceph-mon[98807]: pgmap v350: 97 pgs: 97 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:35:58.461 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:58 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2392689261' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T18:35:58.461 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:58 smithi002 ceph-mon[98807]: osdmap e106: 12 total, 12 up, 12 in 2023-11-16T18:35:58.461 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:58 smithi002 ceph-mon[98807]: from='client.? 172.21.15.2:0/2814337300' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T18:35:58.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:58 smithi040 ceph-mon[100251]: pgmap v350: 97 pgs: 97 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:35:58.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:58 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2392689261' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T18:35:58.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:58 smithi040 ceph-mon[100251]: osdmap e106: 12 total, 12 up, 12 in 2023-11-16T18:35:58.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:58 smithi040 ceph-mon[100251]: from='client.? 172.21.15.2:0/2814337300' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T18:35:58.507 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T18:35:58.545 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T18:35:58.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:58 smithi188 ceph-mon[100639]: pgmap v350: 97 pgs: 97 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T18:35:58.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:58 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2392689261' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T18:35:58.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:58 smithi188 ceph-mon[100639]: osdmap e106: 12 total, 12 up, 12 in 2023-11-16T18:35:58.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:35:58 smithi188 ceph-mon[100639]: from='client.? 172.21.15.2:0/2814337300' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T18:35:58.583 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-16T18:35:58.583 DEBUG:teuthology.orchestra.run.smithi002:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-16T18:35:58.602 DEBUG:teuthology.orchestra.run.smithi040:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-16T18:35:58.638 DEBUG:teuthology.orchestra.run.smithi188:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-16T18:35:58.657 INFO:tasks.cephadm:Stopping all daemons... 2023-11-16T18:35:58.658 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-11-16T18:35:58.658 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.a 2023-11-16T18:35:58.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:58 smithi002 systemd[1]: Stopping Ceph mon.a for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:35:59.353 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:59 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-a[98783]: 2023-11-16T18:35:59.086+0000 7f30db779700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:35:59.353 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:59 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-a[98783]: 2023-11-16T18:35:59.086+0000 7f30db779700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-11-16T18:35:59.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 18:35:59 smithi002 bash[261884]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-a 2023-11-16T18:35:59.750 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.a.service' 2023-11-16T18:35:59.803 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:35:59.804 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-11-16T18:35:59.804 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-11-16T18:35:59.804 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.b 2023-11-16T18:36:00.212 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:35:59 smithi040 systemd[1]: Stopping Ceph mon.b for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:36:00.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:36:00 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-b[100227]: 2023-11-16T18:36:00.209+0000 7fa4d7d7c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:36:00.484 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:36:00 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-b[100227]: 2023-11-16T18:36:00.209+0000 7fa4d7d7c700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-11-16T18:36:00.856 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:36:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:36:00.590+0000 7fb64153a700 -1 client.15621: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2023-11-16T18:36:00.856 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:36:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:36:00.625+0000 7fb64153a700 -1 client.15621: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2023-11-16T18:36:00.950 INFO:journalctl@ceph.mon.b.smithi040.stdout:Nov 16 18:36:00 smithi040 bash[131324]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-b 2023-11-16T18:36:00.957 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.b.service' 2023-11-16T18:36:01.018 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:36:01.019 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-11-16T18:36:01.020 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-11-16T18:36:01.021 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.c 2023-11-16T18:36:01.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:36:01 smithi188 systemd[1]: Stopping Ceph mon.c for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:36:01.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:36:01 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-c[100599]: 2023-11-16T18:36:01.422+0000 7ffb0d586700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:36:01.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:36:01 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-c[100599]: 2023-11-16T18:36:01.422+0000 7ffb0d586700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-11-16T18:36:01.963 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.c.service' 2023-11-16T18:36:01.988 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:36:01 smithi188 bash[133061]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mon-c 2023-11-16T18:36:01.988 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:36:01 smithi188 systemd[1]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mon.c.service: Succeeded. 2023-11-16T18:36:01.988 INFO:journalctl@ceph.mon.c.smithi188.stdout:Nov 16 18:36:01 smithi188 systemd[1]: Stopped Ceph mon.c for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:36:02.015 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:36:02.015 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-11-16T18:36:02.016 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-11-16T18:36:02.016 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.x 2023-11-16T18:36:02.355 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:36:02 smithi002 systemd[1]: Stopping Ceph mgr.x for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:36:02.653 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:36:02 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:36:02.361+0000 7fb6debbb700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:36:02.653 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:36:02 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x[99051]: 2023-11-16T18:36:02.361+0000 7fb6debbb700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-11-16T18:36:02.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 18:36:02 smithi002 bash[262031]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-x 2023-11-16T18:36:02.912 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.x.service' 2023-11-16T18:36:02.952 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:36:02.953 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-11-16T18:36:02.953 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-11-16T18:36:02.953 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.y 2023-11-16T18:36:03.395 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:36:02 smithi040 systemd[1]: Stopping Ceph mgr.y for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:36:03.860 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:36:03 smithi040 bash[131466]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-y 2023-11-16T18:36:03.860 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Nov 16 18:36:03 smithi040 systemd[1]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-11-16T18:36:03.865 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.y.service' 2023-11-16T18:36:03.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:36:03.903 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-11-16T18:36:03.903 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-11-16T18:36:03.904 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.z 2023-11-16T18:36:04.297 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:36:03 smithi188 systemd[1]: Stopping Ceph mgr.z for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:36:04.599 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:36:04 smithi188 bash[133200]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mgr-z 2023-11-16T18:36:04.778 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Nov 16 18:36:04 smithi188 systemd[1]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-11-16T18:36:04.782 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mgr.z.service' 2023-11-16T18:36:04.818 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:36:04.819 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-11-16T18:36:04.819 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-11-16T18:36:04.819 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.0 2023-11-16T18:36:05.105 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:36:04 smithi002 systemd[1]: Stopping Ceph osd.0 for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:36:05.606 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:36:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-0[111877]: 2023-11-16T18:36:05.169+0000 7fceef33a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:36:05.607 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:36:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-0[111877]: 2023-11-16T18:36:05.169+0000 7fceef33a700 -1 osd.0 107 *** Got signal Terminated *** 2023-11-16T18:36:05.608 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:36:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-0[111877]: 2023-11-16T18:36:05.169+0000 7fceef33a700 -1 osd.0 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T18:36:11.106 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 18:36:10 smithi002 bash[262170]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-0 2023-11-16T18:36:13.425 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.0.service' 2023-11-16T18:36:13.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:36:13.468 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-11-16T18:36:13.469 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-11-16T18:36:13.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.1 2023-11-16T18:36:13.852 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:36:13 smithi040 systemd[1]: Stopping Ceph osd.1 for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:36:14.233 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:36:13 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-1[105211]: 2023-11-16T18:36:13.849+0000 7f7c8d7a2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:36:14.234 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:36:13 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-1[105211]: 2023-11-16T18:36:13.849+0000 7f7c8d7a2700 -1 osd.1 107 *** Got signal Terminated *** 2023-11-16T18:36:14.234 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:36:13 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-1[105211]: 2023-11-16T18:36:13.849+0000 7f7c8d7a2700 -1 osd.1 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T18:36:19.983 INFO:journalctl@ceph.osd.1.smithi040.stdout:Nov 16 18:36:19 smithi040 bash[131615]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-1 2023-11-16T18:36:21.250 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.1.service' 2023-11-16T18:36:21.297 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:36:21.297 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-11-16T18:36:21.297 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-11-16T18:36:21.297 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.2 2023-11-16T18:36:21.723 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:36:21 smithi188 systemd[1]: Stopping Ceph osd.2 for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:36:22.047 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:36:21 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-2[106458]: 2023-11-16T18:36:21.720+0000 7f95db2ee700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:36:22.047 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:36:21 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-2[106458]: 2023-11-16T18:36:21.720+0000 7f95db2ee700 -1 osd.2 107 *** Got signal Terminated *** 2023-11-16T18:36:22.047 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:36:21 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-2[106458]: 2023-11-16T18:36:21.720+0000 7f95db2ee700 -1 osd.2 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T18:36:27.549 INFO:journalctl@ceph.osd.2.smithi188.stdout:Nov 16 18:36:27 smithi188 bash[133345]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-2 2023-11-16T18:36:28.791 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.2.service' 2023-11-16T18:36:28.833 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:36:28.833 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-11-16T18:36:28.833 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-11-16T18:36:28.834 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.3 2023-11-16T18:36:29.192 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 18:36:28 smithi002 systemd[1]: Stopping Ceph osd.3 for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:36:29.605 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 18:36:29 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-3[117440]: 2023-11-16T18:36:29.189+0000 7f1167e88700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:36:29.606 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 18:36:29 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-3[117440]: 2023-11-16T18:36:29.189+0000 7f1167e88700 -1 osd.3 107 *** Got signal Terminated *** 2023-11-16T18:36:29.606 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 18:36:29 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-3[117440]: 2023-11-16T18:36:29.189+0000 7f1167e88700 -1 osd.3 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T18:36:33.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:33 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:33.536+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:34.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:33 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:33.921+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:34.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:34 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:34.550+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:35.106 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 18:36:34 smithi002 bash[262455]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-3 2023-11-16T18:36:35.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:34 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:34.905+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:35.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:34 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:34.705+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:35.171 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:34 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:34.888+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:35.549 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:35 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:35.170+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:35.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:35 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:35.566+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:36.038 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:35 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:35.741+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:36.038 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:35 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:35.861+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:36.046 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:35 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:35.931+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:36.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:35 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:35.790+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:36.388 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.3.service' 2023-11-16T18:36:36.430 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:36:36.430 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-11-16T18:36:36.430 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-11-16T18:36:36.430 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.4 2023-11-16T18:36:36.546 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:36 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:36.162+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:36.733 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:36 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:36.549+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:36.734 INFO:journalctl@ceph.osd.4.smithi040.stdout:Nov 16 18:36:36 smithi040 systemd[1]: Stopping Ceph osd.4 for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:36:37.047 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:36 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:36.898+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:37.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:36 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:36.784+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:37.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:36 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:36.912+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:37.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:36 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:36.727+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:37.483 INFO:journalctl@ceph.osd.4.smithi040.stdout:Nov 16 18:36:37 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-4[110650]: 2023-11-16T18:36:37.029+0000 7fad0810a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:36:37.484 INFO:journalctl@ceph.osd.4.smithi040.stdout:Nov 16 18:36:37 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-4[110650]: 2023-11-16T18:36:37.029+0000 7fad0810a700 -1 osd.4 107 *** Got signal Terminated *** 2023-11-16T18:36:37.484 INFO:journalctl@ceph.osd.4.smithi040.stdout:Nov 16 18:36:37 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-4[110650]: 2023-11-16T18:36:37.029+0000 7fad0810a700 -1 osd.4 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T18:36:37.546 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:37 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:37.163+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:37.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:37 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:37.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:38.105 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:37 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:37.924+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:38.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:37 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:37.696+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:38.130 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:37 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:37.930+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:38.131 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:37 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:37.814+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:38.546 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:38.129+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:38.950 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:38 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:38.689+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:38.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:38 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:38.556+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:39.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:39.082+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:39.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:38.966+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:39.298 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:38.841+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:39.355 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:38 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:38.949+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:39.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:39 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:39.577+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:40.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:39 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:39.726+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:40.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:39 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:39.965+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:40.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:39 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:39.965+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:40.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:40.071+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:40.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:40.071+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:14.830243+0000 front 2023-11-16T18:36:14.830329+0000 (oldest deadline 2023-11-16T18:36:39.530121+0000) 2023-11-16T18:36:40.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:39.973+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:40.297 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:39.852+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:40.483 INFO:journalctl@ceph.osd.4.smithi040.stdout:Nov 16 18:36:40 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-4[110650]: 2023-11-16T18:36:40.090+0000 7fad026fc700 -1 osd.4 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.818657+0000 front 2023-11-16T18:36:13.818685+0000 (oldest deadline 2023-11-16T18:36:39.718805+0000) 2023-11-16T18:36:40.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:40 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:40.529+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:41.068 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:40.957+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:41.068 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:40.815+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:41.107 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:40 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:40.925+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:41.107 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:40 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:40.925+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:41.108 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:40 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:40.763+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:41.108 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:40 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:40.763+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:41.483 INFO:journalctl@ceph.osd.4.smithi040.stdout:Nov 16 18:36:41 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-4[110650]: 2023-11-16T18:36:41.075+0000 7fad026fc700 -1 osd.4 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.818657+0000 front 2023-11-16T18:36:13.818685+0000 (oldest deadline 2023-11-16T18:36:39.718805+0000) 2023-11-16T18:36:41.547 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:41.066+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:41.547 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:41.066+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:14.830243+0000 front 2023-11-16T18:36:14.830329+0000 (oldest deadline 2023-11-16T18:36:39.530121+0000) 2023-11-16T18:36:41.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:41 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:41.576+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:42.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:41.780+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:42.047 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:42.005+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:42.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:41 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:41.967+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:42.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:41 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:41.967+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:42.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:41 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:41.797+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:42.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:41 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:41.797+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:42.550 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:42.057+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:42.550 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:42.057+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:14.830243+0000 front 2023-11-16T18:36:14.830329+0000 (oldest deadline 2023-11-16T18:36:39.530121+0000) 2023-11-16T18:36:42.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:42 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:42.531+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:42.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:42 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:42.531+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:43.047 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:42.977+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:43.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:42.779+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:43.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:43 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:43.007+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:43.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:43 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:43.007+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:43.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:42 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:42.844+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:43.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:42 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:42.844+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:43.233 INFO:journalctl@ceph.osd.4.smithi040.stdout:Nov 16 18:36:42 smithi040 bash[131899]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-4 2023-11-16T18:36:43.547 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:43.073+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:43.547 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:43.073+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:14.830243+0000 front 2023-11-16T18:36:14.830329+0000 (oldest deadline 2023-11-16T18:36:39.530121+0000) 2023-11-16T18:36:43.733 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:43 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:43.548+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:43.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:43 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:43.548+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:44.012 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:43.750+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:44.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:44 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:44.029+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:44.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:44 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:44.029+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:44.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:43 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:43.847+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:44.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:43 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:43.847+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:44.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.4.service' 2023-11-16T18:36:44.198 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:36:44.198 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-11-16T18:36:44.199 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-11-16T18:36:44.199 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.5 2023-11-16T18:36:44.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:44.011+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:44.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:44.071+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:44.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:44.071+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:14.830243+0000 front 2023-11-16T18:36:14.830329+0000 (oldest deadline 2023-11-16T18:36:39.530121+0000) 2023-11-16T18:36:44.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:44 smithi188 systemd[1]: Stopping Ceph osd.5 for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:36:44.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:44 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:44.538+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:44.986 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:44 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:44.538+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:45.004 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:44.607+0000 7f53cd3a0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:36:45.005 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:44.607+0000 7f53cd3a0700 -1 osd.5 107 *** Got signal Terminated *** 2023-11-16T18:36:45.005 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:44.607+0000 7f53cd3a0700 -1 osd.5 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T18:36:45.005 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:44.743+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:45.005 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:44.743+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:45.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:45 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:45.011+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:45.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:45 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:45.011+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:45.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:44 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:44.830+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:45.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:44 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:44.830+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:45.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:45 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:45.029+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:45.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:45 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:45.029+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:14.830243+0000 front 2023-11-16T18:36:14.830329+0000 (oldest deadline 2023-11-16T18:36:39.530121+0000) 2023-11-16T18:36:45.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:45 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:45.003+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:45.984 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:45 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:45.746+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:45.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:45 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:45.533+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:45.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:45 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:45.533+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:45.994 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:45 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:45.701+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:45.995 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:45 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:45.701+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:46.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:46 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:46.046+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:46.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:46 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:46.046+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:46.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:45 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:45.812+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:46.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:45 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:45.812+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:46.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:45 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:45.993+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:46.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:45 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:45.993+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:14.830243+0000 front 2023-11-16T18:36:14.830329+0000 (oldest deadline 2023-11-16T18:36:39.530121+0000) 2023-11-16T18:36:46.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:46 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:46.007+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:46.976 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:46 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:46.703+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:46.977 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:46 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:46.703+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:46.983 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:46 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:46.744+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:46.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:46 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:46.510+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:46.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:46 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:46.510+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:47.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:47.027+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:47.106 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:47.027+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:47.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:46 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:46.809+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:47.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:46 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:46.809+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:47.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:47 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:47.008+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:47.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:46 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:46.975+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:47.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:46 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:46.975+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:14.830243+0000 front 2023-11-16T18:36:14.830329+0000 (oldest deadline 2023-11-16T18:36:39.530121+0000) 2023-11-16T18:36:47.937 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:47 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:47.659+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:47.937 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:47 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:47.660+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:47.983 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:47 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:47.729+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:47.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:47 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:47.521+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:47.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:47 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:47.521+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:47.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:47 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:47.521+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:36:48.055 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:47.783+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:48.055 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:47.783+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:48.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:47 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:47.935+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:48.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:47 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:47.935+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:14.830243+0000 front 2023-11-16T18:36:14.830329+0000 (oldest deadline 2023-11-16T18:36:39.530121+0000) 2023-11-16T18:36:48.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:47 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:47.987+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:48.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:48 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:48.053+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:48.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:48 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:48.053+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:48.965 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:48 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:48.638+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:48.965 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:48 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:48.638+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:48.965 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:48 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:48.638+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:36:48.983 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:48 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:48.693+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:48.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:48 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:48.534+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:48.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:48 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:48.534+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:48.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:48 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:48.534+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:36:49.080 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:48 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:48.826+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:49.080 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:48 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:48.826+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:49.081 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:48 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:48.826+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:36:49.296 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:49 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:49.036+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:49.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:48 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:48.964+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:10.130541+0000 front 2023-11-16T18:36:10.130630+0000 (oldest deadline 2023-11-16T18:36:34.829623+0000) 2023-11-16T18:36:49.297 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:48 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5[111951]: 2023-11-16T18:36:48.964+0000 7f53c598e700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:14.830243+0000 front 2023-11-16T18:36:14.830329+0000 (oldest deadline 2023-11-16T18:36:39.530121+0000) 2023-11-16T18:36:49.355 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:49 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:49.079+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:49.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:49 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:49.079+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:49.983 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:49 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:49.691+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:49.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:49 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:49.577+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:49.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:49 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:49.577+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:49.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:49 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:49.577+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:36:50.040 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:49 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:49.658+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:50.041 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:49 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:49.658+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:50.041 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:49 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:49.658+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:36:50.064 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:49 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:49.807+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:50.064 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:49 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:49.807+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:50.064 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:49 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:49.807+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:36:50.296 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:50 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:50.039+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:50.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:50 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:50.039+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:36:50.355 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:50 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:50.063+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:50.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:50 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:50.063+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:50.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:50 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:50.063+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.995497+0000 front 2023-11-16T18:36:24.996029+0000 (oldest deadline 2023-11-16T18:36:49.695582+0000) 2023-11-16T18:36:50.629 INFO:journalctl@ceph.osd.5.smithi188.stdout:Nov 16 18:36:50 smithi188 bash[133619]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-5 2023-11-16T18:36:50.983 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:50 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:50.717+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:50.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:50 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:50.612+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:50.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:50 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:50.612+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:50.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:50 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:50.612+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:36:51.048 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:50 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:50.628+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:51.048 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:50 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:50.628+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:51.048 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:50 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:50.628+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:36:51.101 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:50 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:50.776+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:51.101 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:50 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:50.776+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:51.102 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:50 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:50.776+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:36:51.355 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:51.100+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:51.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:51.100+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:51.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:51.100+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.995497+0000 front 2023-11-16T18:36:24.996029+0000 (oldest deadline 2023-11-16T18:36:49.695582+0000) 2023-11-16T18:36:51.431 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:51 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:51.046+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:51.432 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:51 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:51.046+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:36:51.791 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:51 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:51.651+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:51.792 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:51 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:51.651+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:51.792 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:51 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:51.651+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:36:51.800 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.5.service' 2023-11-16T18:36:51.846 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:36:51.847 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-11-16T18:36:51.847 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-11-16T18:36:51.848 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.6 2023-11-16T18:36:51.984 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:51 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:51.727+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:51.985 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:51 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:51.727+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:36:51.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:51 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:51.577+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:51.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:51 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:51.577+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:51.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:51 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:51.577+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:36:52.047 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:52 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:52.042+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:52.047 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:52 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:52.042+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:36:52.059 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:51 smithi002 systemd[1]: Stopping Ceph osd.6 for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:36:52.060 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:52.058+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:52.060 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:51.751+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:52.060 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:51.751+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:52.060 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:51 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:51.751+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:36:52.355 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:52.058+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:52.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:52.058+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.995497+0000 front 2023-11-16T18:36:24.996029+0000 (oldest deadline 2023-11-16T18:36:49.695582+0000) 2023-11-16T18:36:52.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:52.203+0000 7fe23995a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:36:52.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:52.203+0000 7fe23995a700 -1 osd.6 107 *** Got signal Terminated *** 2023-11-16T18:36:52.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:52.203+0000 7fe23995a700 -1 osd.6 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T18:36:52.983 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:52 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:52.701+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:52.984 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:52 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:52.701+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:36:52.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:52 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:52.614+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:52.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:52 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:52.614+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:52.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:52 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:52.614+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:36:53.042 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:52 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:52.671+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:53.043 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:52 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:52.671+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:53.043 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:52 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:52.671+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:36:53.079 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:52.732+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:53.079 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:52.732+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:53.079 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:52.732+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:36:53.296 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:53 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:53.041+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:53.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:53 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:53.041+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:36:53.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:53 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:53.077+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:53.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:53 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:53.077+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:53.356 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:53 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:53.077+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.995497+0000 front 2023-11-16T18:36:24.996029+0000 (oldest deadline 2023-11-16T18:36:49.695582+0000) 2023-11-16T18:36:53.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:53 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:53.599+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:58.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:53 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:53.599+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:58.115 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:53 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:53.599+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:36:58.116 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:53 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:53.663+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:58.116 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:53 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:53.663+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:36:58.116 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:54.630+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:58.116 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:54.630+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:36:58.116 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:54.572+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:58.117 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:54.572+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:58.117 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:54 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:54.572+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:36:58.117 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:55 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:55.622+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:58.117 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:55 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:55.622+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:36:58.117 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:55 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:55.622+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.346490+0000 front 2023-11-16T18:36:31.346612+0000 (oldest deadline 2023-11-16T18:36:55.446749+0000) 2023-11-16T18:36:58.118 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:55 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:55.570+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:58.118 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:55 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:55.570+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:58.118 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:55 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:55.570+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:36:58.118 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:55 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:55.570+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:36:58.118 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:56 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:56.627+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:58.119 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:56 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:56.627+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:36:58.119 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:56 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:56.627+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.346490+0000 front 2023-11-16T18:36:31.346612+0000 (oldest deadline 2023-11-16T18:36:55.446749+0000) 2023-11-16T18:36:58.119 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:56 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:56.605+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:58.119 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:56 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:56.605+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:58.119 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:56 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:56.605+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:36:58.119 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:56 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:56.605+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:36:58.120 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:57 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:57.605+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:58.120 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:57 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:57.605+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:36:58.120 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:57 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:57.605+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.346490+0000 front 2023-11-16T18:36:31.346612+0000 (oldest deadline 2023-11-16T18:36:55.446749+0000) 2023-11-16T18:36:58.120 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:57 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:57.630+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:58.120 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:57 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:57.630+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:58.121 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:57 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:57.630+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:36:58.121 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:57 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:57.630+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:36:58.121 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:53 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:53.730+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:58.122 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:53 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:53.730+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:58.122 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:53 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:53.730+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:36:58.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:53 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:53.670+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:58.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:53 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:53.670+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:58.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:53 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:53.670+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:36:58.124 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:54 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:54.063+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:58.217 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:54 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:54.063+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:58.217 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:54 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:54.063+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.995497+0000 front 2023-11-16T18:36:24.996029+0000 (oldest deadline 2023-11-16T18:36:49.695582+0000) 2023-11-16T18:36:58.217 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:54.051+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:58.217 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:54.051+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:36:58.218 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:54 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:54.699+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:58.218 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:54 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:54.699+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:58.218 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:54 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:54.699+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:36:58.218 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:54.627+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:58.218 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:54.627+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:58.219 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:54 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:54.627+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:36:58.219 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:55 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:55.052+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:58.219 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:55 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:55.052+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:58.219 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:55 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:55.052+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.995497+0000 front 2023-11-16T18:36:24.996029+0000 (oldest deadline 2023-11-16T18:36:49.695582+0000) 2023-11-16T18:36:58.219 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:55 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:55.052+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:33.096966+0000 front 2023-11-16T18:36:33.096789+0000 (oldest deadline 2023-11-16T18:36:54.697393+0000) 2023-11-16T18:36:58.220 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:55 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:55.051+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:58.220 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:55 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:55.051+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:36:58.220 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:55 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:55.652+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:58.220 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:55 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:55.652+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:58.220 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:55 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:55.652+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:36:58.221 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:55 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:55.663+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:58.221 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:55 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:55.663+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:58.221 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:55 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:55.663+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:36:58.221 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:56 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:56.062+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:58.221 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:56 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:56.062+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:58.222 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:56 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:56.062+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.995497+0000 front 2023-11-16T18:36:24.996029+0000 (oldest deadline 2023-11-16T18:36:49.695582+0000) 2023-11-16T18:36:58.222 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:56 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:56.062+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:33.096966+0000 front 2023-11-16T18:36:33.096789+0000 (oldest deadline 2023-11-16T18:36:54.697393+0000) 2023-11-16T18:36:58.222 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:56 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:56.049+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:58.222 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:56 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:56.049+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:36:58.222 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:56 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:56.651+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:58.223 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:56 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:56.651+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:58.223 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:56 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:56.651+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:36:58.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:56 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:56.615+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:58.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:56 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:56.615+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:58.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:56 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:56.615+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:36:58.224 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:57 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:57.062+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.793243+0000 front 2023-11-16T18:36:09.793286+0000 (oldest deadline 2023-11-16T18:36:33.893388+0000) 2023-11-16T18:36:58.224 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:57 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:57.062+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.893853+0000 front 2023-11-16T18:36:13.893838+0000 (oldest deadline 2023-11-16T18:36:39.793963+0000) 2023-11-16T18:36:58.224 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:57 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:57.062+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.995497+0000 front 2023-11-16T18:36:24.996029+0000 (oldest deadline 2023-11-16T18:36:49.695582+0000) 2023-11-16T18:36:58.224 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:57 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6[122847]: 2023-11-16T18:36:57.063+0000 7fe231f48700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:33.096966+0000 front 2023-11-16T18:36:33.096789+0000 (oldest deadline 2023-11-16T18:36:54.697393+0000) 2023-11-16T18:36:58.224 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:57 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:57.048+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:58.224 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:57 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:57.048+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:36:58.225 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:57 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:57.622+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:58.225 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:57 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:57.622+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:58.225 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:57 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:57.622+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:36:58.225 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:57 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:57.622+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:36:58.225 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:57 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:57.614+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:58.225 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:57 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:57.614+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:58.226 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:57 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:57.614+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:36:58.296 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:58 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:58.009+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:58.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:58 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:58.009+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:36:58.355 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 18:36:57 smithi002 bash[262726]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-6 2023-11-16T18:36:58.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:58 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:58.591+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:58.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:58 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:58.591+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:58.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:58 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:58.591+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:36:58.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:58 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:58.591+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:36:58.984 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:58 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:58.574+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:58.984 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:58 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:58.574+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:36:58.984 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:58 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:58.574+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.346490+0000 front 2023-11-16T18:36:31.346612+0000 (oldest deadline 2023-11-16T18:36:55.446749+0000) 2023-11-16T18:36:59.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:58 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:58.612+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:59.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:58 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:58.612+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:59.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:58 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:58.612+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:36:59.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:58 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:58.612+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:36:59.023 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:58 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:58.654+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:36:59.023 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:58 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:58.654+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:36:59.023 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:58 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:58.654+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:36:59.296 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:59 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:59.021+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:36:59.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:36:59 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:36:59.021+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:36:59.372 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.6.service' 2023-11-16T18:36:59.413 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:36:59.414 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-11-16T18:36:59.414 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-11-16T18:36:59.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.7 2023-11-16T18:36:59.734 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:59 smithi040 systemd[1]: Stopping Ceph osd.7 for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:36:59.734 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:59 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:59.549+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:36:59.734 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:59 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:59.549+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:36:59.734 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:59 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:59.549+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.346490+0000 front 2023-11-16T18:36:31.346612+0000 (oldest deadline 2023-11-16T18:36:55.446749+0000) 2023-11-16T18:36:59.735 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:59 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:59.568+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:36:59.735 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:59 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:59.568+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:36:59.735 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:59 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:59.568+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:36:59.735 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:36:59 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:36:59.568+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:36:59.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:59 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:59.596+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:36:59.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:59 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:59.596+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:36:59.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:59 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:59.596+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:36:59.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:36:59 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:36:59.596+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:00.018 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:59 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:59.632+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:00.019 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:59 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:59.632+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:00.019 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:36:59 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:36:59.632+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:00.233 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:59 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:59.793+0000 7f35f245d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:37:00.233 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:59 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:59.793+0000 7f35f245d700 -1 osd.7 107 *** Got signal Terminated *** 2023-11-16T18:37:00.234 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:36:59 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:36:59.793+0000 7f35f245d700 -1 osd.7 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T18:37:00.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:00 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:00.016+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:37:00.298 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:00 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:00.016+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:37:00.984 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:00 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:00.596+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:37:00.984 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:00 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:00.596+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:37:00.984 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:00 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:00.596+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.346490+0000 front 2023-11-16T18:36:31.346612+0000 (oldest deadline 2023-11-16T18:36:55.446749+0000) 2023-11-16T18:37:00.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:00 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:00.551+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:00.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:00 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:00.551+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:00.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:00 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:00.551+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:00.986 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:00 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:00.551+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:01.038 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:00 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:00.619+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:01.038 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:00 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:00.619+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:01.039 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:00 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:00.619+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:01.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:00.631+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:01.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:00.631+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:01.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:00.631+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:01.107 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:00 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:00.631+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:01.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:01 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:01.036+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:37:01.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:01 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:01.036+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:37:01.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:01 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:01.036+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:29.169883+0000 front 2023-11-16T18:36:29.169833+0000 (oldest deadline 2023-11-16T18:37:00.370707+0000) 2023-11-16T18:37:01.983 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:01 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:01.623+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:37:01.983 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:01 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:01.623+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:37:01.984 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:01 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:01.623+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.346490+0000 front 2023-11-16T18:36:31.346612+0000 (oldest deadline 2023-11-16T18:36:55.446749+0000) 2023-11-16T18:37:01.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:01 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:01.539+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:01.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:01 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:01.539+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:01.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:01 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:01.539+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:01.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:01 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:01.539+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:02.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:01 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:01.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:02.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:01 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:01.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:02.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:01 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:01.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:02.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:01 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:01.643+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:02.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:01 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:01.643+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:02.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:01 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:01.643+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:02.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:01 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:01.643+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:02.547 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:02 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:02.064+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:37:02.547 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:02 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:02.064+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:37:02.548 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:02 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:02.064+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:29.169883+0000 front 2023-11-16T18:36:29.169833+0000 (oldest deadline 2023-11-16T18:37:00.370707+0000) 2023-11-16T18:37:02.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:02 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:02.593+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:02.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:02 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:02.593+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:02.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:02 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:02.593+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:02.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:02 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:02.593+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:02.983 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:02 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:02.627+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:37:02.983 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:02 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:02.627+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:37:02.983 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:02 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:02.627+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.346490+0000 front 2023-11-16T18:36:31.346612+0000 (oldest deadline 2023-11-16T18:36:55.446749+0000) 2023-11-16T18:37:02.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:02 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:02.516+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:02.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:02 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:02.516+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:02.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:02 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:02.516+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:02.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:02 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:02.516+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:03.018 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:02 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:02.593+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:03.018 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:02 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:02.593+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:03.018 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:02 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:02.593+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:03.298 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:03 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:03.016+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:37:03.298 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:03 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:03.016+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:37:03.298 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:03 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:03.016+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:29.169883+0000 front 2023-11-16T18:36:29.169833+0000 (oldest deadline 2023-11-16T18:37:00.370707+0000) 2023-11-16T18:37:03.733 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:03 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:03.581+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:37:03.733 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:03 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:03.581+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:37:03.734 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:03 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:03.581+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.346490+0000 front 2023-11-16T18:36:31.346612+0000 (oldest deadline 2023-11-16T18:36:55.446749+0000) 2023-11-16T18:37:03.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:03 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:03.478+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:03.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:03 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:03.478+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:03.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:03 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:03.478+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:03.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:03 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:03.478+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:03.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:03 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:03.544+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:03.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:03 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:03.544+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:03.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:03 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:03.544+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:03.855 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:03 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:03.590+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:03.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:03 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:03.590+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:03.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:03 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:03.590+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:03.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:03 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:03.590+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:04.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:03 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:03.967+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:37:04.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:03 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:03.967+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:37:04.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:03 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:03.967+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:29.169883+0000 front 2023-11-16T18:36:29.169833+0000 (oldest deadline 2023-11-16T18:37:00.370707+0000) 2023-11-16T18:37:04.962 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:04 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:04.560+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:04.963 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:04 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:04.560+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:04.963 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:04 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:04.560+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:04.986 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:04 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:04.617+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.144055+0000 front 2023-11-16T18:36:13.144086+0000 (oldest deadline 2023-11-16T18:36:44.944842+0000) 2023-11-16T18:37:04.987 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:04 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:04.617+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:26.045791+0000 front 2023-11-16T18:36:26.045843+0000 (oldest deadline 2023-11-16T18:36:51.345917+0000) 2023-11-16T18:37:04.987 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:04 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7[116151]: 2023-11-16T18:37:04.617+0000 7f35eba4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.346490+0000 front 2023-11-16T18:36:31.346612+0000 (oldest deadline 2023-11-16T18:36:55.446749+0000) 2023-11-16T18:37:04.987 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:04 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:04.523+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:04.987 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:04 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:04.523+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:04.987 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:04 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:04.523+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:04.988 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:04 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:04.523+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:04.988 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:04 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:04.523+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:05.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:04 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:04.615+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:05.107 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:04 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:04.615+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:05.107 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:04 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:04.615+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:05.107 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:04 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:04.615+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:05.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:04 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:04.960+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:37:05.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:04 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:04.960+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:37:05.298 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:04 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:04.960+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:29.169883+0000 front 2023-11-16T18:36:29.169833+0000 (oldest deadline 2023-11-16T18:37:00.370707+0000) 2023-11-16T18:37:05.298 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:04 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:04.960+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.371199+0000 front 2023-11-16T18:36:40.371232+0000 (oldest deadline 2023-11-16T18:37:04.471272+0000) 2023-11-16T18:37:05.733 INFO:journalctl@ceph.osd.7.smithi040.stdout:Nov 16 18:37:05 smithi040 bash[132159]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-7 2023-11-16T18:37:05.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:05 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:05.494+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:05.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:05 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:05.494+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:05.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:05 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:05.494+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:05.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:05 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:05.494+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:05.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:05 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:05.494+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:05.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:05 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:05.526+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:05.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:05 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:05.526+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:05.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:05 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:05.526+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:05.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:05.591+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:05.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:05.591+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:05.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:05.591+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:05.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:05.591+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:05.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:05 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:05.591+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:06.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:05 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:05.942+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:37:06.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:05 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:05.942+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:37:06.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:05 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:05.942+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:29.169883+0000 front 2023-11-16T18:36:29.169833+0000 (oldest deadline 2023-11-16T18:37:00.370707+0000) 2023-11-16T18:37:06.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:05 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:05.942+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.371199+0000 front 2023-11-16T18:36:40.371232+0000 (oldest deadline 2023-11-16T18:37:04.471272+0000) 2023-11-16T18:37:06.857 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:06 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:06.584+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:06.859 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:06 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:06.584+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:06.859 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:06 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:06.584+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:06.861 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:06 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:06.584+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:06.862 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:06 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:06.584+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:06.928 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:06 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:06.484+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:06.928 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:06 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:06.484+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:06.928 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:06 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:06.484+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:06.928 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:06 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:06.484+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:06.928 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:06 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:06.484+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:06.936 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.7.service' 2023-11-16T18:37:06.977 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:37:06.977 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-11-16T18:37:06.977 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-11-16T18:37:06.978 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.8 2023-11-16T18:37:06.994 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:06 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:06.550+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:06.994 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:06 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:06.550+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:06.994 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:06 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:06.550+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:07.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:06 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:06.992+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:37:07.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:06 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:06.992+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:37:07.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:06 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:06.992+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:29.169883+0000 front 2023-11-16T18:36:29.169833+0000 (oldest deadline 2023-11-16T18:37:00.370707+0000) 2023-11-16T18:37:07.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:06 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:06.992+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.371199+0000 front 2023-11-16T18:36:40.371232+0000 (oldest deadline 2023-11-16T18:37:04.471272+0000) 2023-11-16T18:37:07.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:07 smithi188 systemd[1]: Stopping Ceph osd.8 for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:37:07.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:07 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:07.568+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:07.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:07 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:07.568+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:07.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:07 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:07.568+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:07.797 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:07 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:07.329+0000 7fcdc3858700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:37:07.798 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:07 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:07.329+0000 7fcdc3858700 -1 osd.8 107 *** Got signal Terminated *** 2023-11-16T18:37:07.798 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:07 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:07.329+0000 7fcdc3858700 -1 osd.8 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T18:37:07.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:07 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:07.569+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:07.857 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:07 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:07.569+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:07.857 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:07 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:07.569+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:07.857 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:07 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:07.569+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:07.857 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:07 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:07.569+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:07.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:07 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:07.489+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:07.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:07 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:07.489+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:07.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:07 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:07.489+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:07.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:07 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:07.489+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:07.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:07 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:07.489+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:08.296 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:07 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:07.985+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:37:08.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:07 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:07.985+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:37:08.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:07 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:07.985+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:29.169883+0000 front 2023-11-16T18:36:29.169833+0000 (oldest deadline 2023-11-16T18:37:00.370707+0000) 2023-11-16T18:37:08.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:07 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:07.985+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.371199+0000 front 2023-11-16T18:36:40.371232+0000 (oldest deadline 2023-11-16T18:37:04.471272+0000) 2023-11-16T18:37:08.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:08 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:08.553+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:08.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:08 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:08.553+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:08.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:08 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:08.553+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:08.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:08 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:08.553+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:08.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:08 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:08.553+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:08.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:08 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:08.499+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:08.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:08 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:08.499+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:08.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:08 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:08.499+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:08.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:08 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:08.499+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:08.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:08 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:08.499+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:09.027 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:08 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:08.563+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:09.027 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:08 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:08.563+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:09.056 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:08 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:08.563+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:09.056 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:08 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:08.563+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:09.296 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:09 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:09.017+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:37:09.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:09 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:09.017+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:37:09.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:09 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:09.017+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:29.169883+0000 front 2023-11-16T18:36:29.169833+0000 (oldest deadline 2023-11-16T18:37:00.370707+0000) 2023-11-16T18:37:09.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:09 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:09.017+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.371199+0000 front 2023-11-16T18:36:40.371232+0000 (oldest deadline 2023-11-16T18:37:04.471272+0000) 2023-11-16T18:37:09.855 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:09 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:09.556+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:09.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:09 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:09.556+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:09.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:09 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:09.556+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:09.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:09 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:09.556+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:09.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:09 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:09.556+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:09.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:09 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:09.546+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:09.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:09 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:09.546+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:09.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:09 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:09.546+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:09.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:09 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:09.546+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:09.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:09 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:09.547+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:10.042 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:09 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:09.583+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:10.042 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:09 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:09.583+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:10.042 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:09 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:09.583+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:10.042 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:09 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:09.583+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:10.296 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:10 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:10.040+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:37:10.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:10 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:10.040+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:37:10.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:10 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:10.040+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:29.169883+0000 front 2023-11-16T18:36:29.169833+0000 (oldest deadline 2023-11-16T18:37:00.370707+0000) 2023-11-16T18:37:10.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:10 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:10.040+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.371199+0000 front 2023-11-16T18:36:40.371232+0000 (oldest deadline 2023-11-16T18:37:04.471272+0000) 2023-11-16T18:37:10.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:10 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:10.578+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:10.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:10 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:10.578+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:10.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:10 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:10.578+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:10.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:10 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:10.578+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:10.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:10 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:10.578+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:10.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:10 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:10.505+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:10.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:10 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:10.505+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:10.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:10 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:10.505+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:10.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:10 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:10.505+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:10.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:10 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:10.505+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:11.035 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:10 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:10.600+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:11.035 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:10 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:10.600+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:11.035 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:10 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:10.600+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:11.035 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:10 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:10.600+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:11.296 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:11 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:11.033+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:37:11.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:11 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:11.033+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:37:11.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:11 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:11.033+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:29.169883+0000 front 2023-11-16T18:36:29.169833+0000 (oldest deadline 2023-11-16T18:37:00.370707+0000) 2023-11-16T18:37:11.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:11 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:11.033+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.371199+0000 front 2023-11-16T18:36:40.371232+0000 (oldest deadline 2023-11-16T18:37:04.471272+0000) 2023-11-16T18:37:11.857 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:11 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:11.589+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:11.857 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:11 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:11.589+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:11.857 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:11 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:11.589+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:11.857 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:11 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:11.589+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:11.857 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:11 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:11.589+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:11.858 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:11 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:11.589+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:47.853013+0000 front 2023-11-16T18:36:47.853133+0000 (oldest deadline 2023-11-16T18:37:11.352815+0000) 2023-11-16T18:37:11.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:11 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:11.523+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:11.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:11 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:11.523+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:11.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:11 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:11.523+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:11.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:11 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:11.523+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:11.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:11 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:11.523+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:11.988 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:11 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:11.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:11.989 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:11 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:11.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:11.989 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:11 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:11.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:11.989 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:11 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:11.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:12.296 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:11 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:11.986+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.266905+0000 front 2023-11-16T18:36:09.267406+0000 (oldest deadline 2023-11-16T18:36:33.367031+0000) 2023-11-16T18:37:12.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:11 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:11.986+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:25.069271+0000 front 2023-11-16T18:36:25.069243+0000 (oldest deadline 2023-11-16T18:36:49.169427+0000) 2023-11-16T18:37:12.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:11 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:11.986+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:29.169883+0000 front 2023-11-16T18:36:29.169833+0000 (oldest deadline 2023-11-16T18:37:00.370707+0000) 2023-11-16T18:37:12.297 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:11 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8[117362]: 2023-11-16T18:37:11.986+0000 7fcdbde4a700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.371199+0000 front 2023-11-16T18:36:40.371232+0000 (oldest deadline 2023-11-16T18:37:04.471272+0000) 2023-11-16T18:37:12.834 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:12 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:12.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:12.834 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:12 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:12.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:12.834 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:12 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:12.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:12.834 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:12 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:12.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:12.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:12 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:12.509+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:12.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:12 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:12.509+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:12.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:12 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:12.509+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:12.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:12 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:12.509+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:12.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:12 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:12.509+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:12.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:12 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:12.509+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:13.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:12 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:12.610+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:13.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:12 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:12.610+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:13.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:12 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:12.610+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:13.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:12 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:12.610+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:13.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:12 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:12.610+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:13.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:12 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:12.610+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:47.853013+0000 front 2023-11-16T18:36:47.853133+0000 (oldest deadline 2023-11-16T18:37:11.352815+0000) 2023-11-16T18:37:13.113 INFO:journalctl@ceph.osd.8.smithi188.stdout:Nov 16 18:37:13 smithi188 bash[133892]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-8 2023-11-16T18:37:13.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:13 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:13.565+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:13.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:13 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:13.565+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:13.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:13 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:13.565+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:13.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:13 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:13.565+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:13.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:13 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:13.565+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:13.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:13 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:13.565+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:47.853013+0000 front 2023-11-16T18:36:47.853133+0000 (oldest deadline 2023-11-16T18:37:11.352815+0000) 2023-11-16T18:37:13.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:13 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:13.505+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:13.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:13 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:13.505+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:13.986 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:13 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:13.505+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:13.986 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:13 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:13.505+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:13.986 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:13 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:13.505+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:13.986 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:13 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:13.505+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:14.033 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:13 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:13.550+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:14.033 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:13 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:13.550+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:14.033 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:13 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:13.550+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:14.033 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:13 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:13.550+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:14.033 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:13 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:13.550+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:14.373 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.8.service' 2023-11-16T18:37:14.453 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:37:14.454 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-11-16T18:37:14.454 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-11-16T18:37:14.454 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.9 2023-11-16T18:37:14.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:14 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:14.538+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:14.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:14 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:14.538+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:14.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:14 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:14.538+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:14.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:14 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:14.538+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:14.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:14 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:14.538+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:14.809 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:14 smithi002 systemd[1]: Stopping Ceph osd.9 for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:37:14.810 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:14 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:14.559+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:14.810 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:14 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:14.559+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:14.810 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:14 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:14.559+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:14.810 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:14 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:14.559+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:14.810 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:14 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:14.559+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:14.810 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:14 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:14.559+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:47.853013+0000 front 2023-11-16T18:36:47.853133+0000 (oldest deadline 2023-11-16T18:37:11.352815+0000) 2023-11-16T18:37:14.810 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:14 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:14.807+0000 7f2887cfe700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:37:14.811 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:14 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:14.807+0000 7f2887cfe700 -1 osd.9 107 *** Got signal Terminated *** 2023-11-16T18:37:14.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:14 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:14.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:14.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:14 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:14.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:14.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:14 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:14.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:14.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:14 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:14.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:14.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:14 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:14.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:14.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:14 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:14.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:15.105 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:14 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:14.807+0000 7f2887cfe700 -1 osd.9 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T18:37:15.733 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:15 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:15.461+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:15.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:15 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:15.461+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:15.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:15 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:15.461+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:15.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:15 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:15.461+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:15.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:15 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:15.461+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:15.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:15 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:15.461+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:15.855 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:15 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:15.516+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:15.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:15 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:15.516+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:15.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:15 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:15.516+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:15.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:15 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:15.516+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:15.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:15 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:15.516+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:15.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:15 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:15.516+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:47.853013+0000 front 2023-11-16T18:36:47.853133+0000 (oldest deadline 2023-11-16T18:37:11.352815+0000) 2023-11-16T18:37:16.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:15 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:15.574+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:16.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:15 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:15.574+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:16.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:15 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:15.574+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:16.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:15 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:15.574+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:16.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:15 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:15.574+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:16.733 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:16 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:16.470+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:16.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:16 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:16.470+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:16.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:16 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:16.470+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:16.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:16 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:16.470+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:16.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:16 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:16.470+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:16.734 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:16 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:16.470+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:16.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:16 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:16.554+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:16.858 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:16 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:16.554+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:16.858 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:16 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:16.554+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:16.858 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:16 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:16.554+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:16.858 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:16 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:16.554+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:16.859 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:16 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:16.554+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:47.853013+0000 front 2023-11-16T18:36:47.853133+0000 (oldest deadline 2023-11-16T18:37:11.352815+0000) 2023-11-16T18:37:17.050 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:16 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:16.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:17.051 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:16 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:16.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:17.051 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:16 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:16.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:17.051 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:16 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:16.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:17.051 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:16 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:16.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:17.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:17 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:17.519+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:17.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:17 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:17.519+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:17.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:17 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:17.519+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:17.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:17 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:17.519+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:17.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:17 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:17.519+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:17.855 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:17.571+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:17.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:17.571+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:17.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:17.571+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:17.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:17.571+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:17.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:17.571+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:17.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:17 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:17.571+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:47.853013+0000 front 2023-11-16T18:36:47.853133+0000 (oldest deadline 2023-11-16T18:37:11.352815+0000) 2023-11-16T18:37:17.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:17 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:17.511+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:17.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:17 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:17.511+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:17.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:17 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:17.511+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:17.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:17 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:17.511+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:17.986 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:17 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:17.511+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:17.986 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:17 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:17.511+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:18.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:18 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:18.560+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:18.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:18 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:18.560+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:18.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:18 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:18.560+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:18.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:18 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:18.560+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:18.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:18 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:18.560+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:18.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:18 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:18.560+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:19.046 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:18 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:18.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:19.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:18 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:18.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:19.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:18 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:18.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:19.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:18 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:18.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:19.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:18 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:18.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:19.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:18 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:18.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:19.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:18 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:18.608+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:19.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:18 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:18.608+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:19.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:18 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:18.608+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:19.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:18 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:18.608+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:19.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:18 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:18.608+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:19.107 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:18 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:18.608+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:47.853013+0000 front 2023-11-16T18:36:47.853133+0000 (oldest deadline 2023-11-16T18:37:11.352815+0000) 2023-11-16T18:37:19.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:19 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:19.602+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:19.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:19 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:19.602+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:19.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:19 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:19.602+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:19.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:19 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:19.602+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:19.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:19 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:19.602+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:19.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:19 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:19.602+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:20.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:19 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:19.583+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:20.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:19 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:19.583+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:20.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:19 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:19.583+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:20.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:19 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:19.583+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:20.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:19 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:19.583+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:20.048 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:19 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:19.583+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:20.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:19 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:19.650+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:06.847090+0000 front 2023-11-16T18:36:06.847115+0000 (oldest deadline 2023-11-16T18:36:33.847856+0000) 2023-11-16T18:37:20.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:19 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:19.650+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.848378+0000 front 2023-11-16T18:36:13.848435+0000 (oldest deadline 2023-11-16T18:36:39.748321+0000) 2023-11-16T18:37:20.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:19 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:19.650+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.449539+0000 front 2023-11-16T18:36:24.449338+0000 (oldest deadline 2023-11-16T18:36:47.949268+0000) 2023-11-16T18:37:20.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:19 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:19.650+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:31.450352+0000 front 2023-11-16T18:36:31.450380+0000 (oldest deadline 2023-11-16T18:36:57.350593+0000) 2023-11-16T18:37:20.106 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:19 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:19.650+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:40.851867+0000 front 2023-11-16T18:36:40.851770+0000 (oldest deadline 2023-11-16T18:37:04.951719+0000) 2023-11-16T18:37:20.107 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:19 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9[128276]: 2023-11-16T18:37:19.650+0000 7f28822f0700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:47.853013+0000 front 2023-11-16T18:36:47.853133+0000 (oldest deadline 2023-11-16T18:37:11.352815+0000) 2023-11-16T18:37:20.856 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 18:37:20 smithi002 bash[263010]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-9 2023-11-16T18:37:20.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:20 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:20.555+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:20.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:20 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:20.555+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:20.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:20 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:20.555+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:20.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:20 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:20.555+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:20.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:20 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:20.555+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:20.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:20 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:20.555+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:21.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:20 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:20.557+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:21.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:20 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:20.557+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:21.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:20 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:20.557+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:21.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:20 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:20.557+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:21.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:20 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:20.557+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:21.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:20 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:20.557+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:21.787 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.9.service' 2023-11-16T18:37:21.829 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:37:21.829 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-11-16T18:37:21.829 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-11-16T18:37:21.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.10 2023-11-16T18:37:21.860 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:21 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:21.526+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:21.860 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:21 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:21.526+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:21.860 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:21 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:21.526+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:21.860 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:21 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:21.526+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:21.860 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:21 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:21.526+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:21.861 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:21 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:21.526+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:22.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:21 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:21.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:22.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:21 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:21.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:22.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:21 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:21.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:22.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:21 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:21.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:22.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:21 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:21.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:22.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:21 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:21.559+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:22.200 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:21 smithi040 systemd[1]: Stopping Ceph osd.10 for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:37:22.273 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:22 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:22.197+0000 7f36910ea700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:37:22.483 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:22 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:22.197+0000 7f36910ea700 -1 osd.10 107 *** Got signal Terminated *** 2023-11-16T18:37:22.483 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:22 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:22.197+0000 7f36910ea700 -1 osd.10 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T18:37:22.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:22 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:22.558+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:22.983 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:22 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:22.558+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:22.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:22 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:22.558+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:22.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:22 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:22.558+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:22.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:22 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:22.558+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:22.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:22 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:22.558+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:22.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:22 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:22.558+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:56.937787+0000 front 2023-11-16T18:36:56.937857+0000 (oldest deadline 2023-11-16T18:37:22.237606+0000) 2023-11-16T18:37:23.046 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:22 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:22.546+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:23.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:22 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:22.546+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:23.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:22 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:22.546+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:23.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:22 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:22.546+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:23.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:22 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:22.546+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:23.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:22 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:22.546+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:23.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:23 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:23.526+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:23.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:23 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:23.526+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:23.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:23 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:23.526+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:23.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:23 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:23.526+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:23.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:23 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:23.526+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:23.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:23 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:23.526+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:23.985 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:23 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:23.526+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:56.937787+0000 front 2023-11-16T18:36:56.937857+0000 (oldest deadline 2023-11-16T18:37:22.237606+0000) 2023-11-16T18:37:24.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:23 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:23.568+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:24.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:23 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:23.568+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:24.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:23 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:23.568+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:24.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:23 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:23.568+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:24.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:23 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:23.568+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:24.048 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:23 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:23.568+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:24.984 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:24 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:24.556+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:38.337 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:24 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:24.556+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:38.337 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:24 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:24.556+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:38.337 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:24 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:24.556+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:38.337 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:24 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:24.556+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:38.337 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:24 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:24.556+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:38.337 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:24 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:24.556+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:56.937787+0000 front 2023-11-16T18:36:56.937857+0000 (oldest deadline 2023-11-16T18:37:22.237606+0000) 2023-11-16T18:37:38.342 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:25 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:25.509+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:38.365 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:25 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:25.509+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:38.365 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:25 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:25.509+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:38.366 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:25 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:25.509+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:38.366 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:25 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:25.509+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:38.366 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:25 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:25.509+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:38.366 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:25 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:25.509+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:56.937787+0000 front 2023-11-16T18:36:56.937857+0000 (oldest deadline 2023-11-16T18:37:22.237606+0000) 2023-11-16T18:37:38.366 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:26 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:26.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:09.129777+0000 front 2023-11-16T18:36:09.129683+0000 (oldest deadline 2023-11-16T18:36:33.229788+0000) 2023-11-16T18:37:38.366 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:26 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:26.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:13.730951+0000 front 2023-11-16T18:36:13.731026+0000 (oldest deadline 2023-11-16T18:36:41.931588+0000) 2023-11-16T18:37:38.366 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:26 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:26.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:21.932374+0000 front 2023-11-16T18:36:21.932403+0000 (oldest deadline 2023-11-16T18:36:47.232224+0000) 2023-11-16T18:37:38.366 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:26 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:26.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T18:36:32.533638+0000 front 2023-11-16T18:36:32.533765+0000 (oldest deadline 2023-11-16T18:36:54.734067+0000) 2023-11-16T18:37:38.367 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:26 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:26.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:39.435007+0000 front 2023-11-16T18:36:39.435070+0000 (oldest deadline 2023-11-16T18:37:04.135158+0000) 2023-11-16T18:37:38.367 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:26 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:26.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:44.135739+0000 front 2023-11-16T18:36:44.136707+0000 (oldest deadline 2023-11-16T18:37:12.336157+0000) 2023-11-16T18:37:38.367 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:26 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10[121701]: 2023-11-16T18:37:26.508+0000 7f368c6de700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:56.937787+0000 front 2023-11-16T18:36:56.937857+0000 (oldest deadline 2023-11-16T18:37:22.237606+0000) 2023-11-16T18:37:38.367 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:27 smithi040 bash[132427]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-10 2023-11-16T18:37:38.369 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.10.service' 2023-11-16T18:37:38.369 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:29 smithi040 systemd[1]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.10.service: Succeeded. 2023-11-16T18:37:38.369 INFO:journalctl@ceph.osd.10.smithi040.stdout:Nov 16 18:37:29 smithi040 systemd[1]: Stopped Ceph osd.10 for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:37:38.370 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:24 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:24.602+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:38.370 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:24 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:24.602+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:38.370 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:24 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:24.602+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:38.370 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:24 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:24.602+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:38.371 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:24 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:24.602+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:38.371 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:24 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:24.602+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:38.371 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:25 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:25.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:38.371 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:25 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:25.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:38.486 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:25 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:25.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:38.486 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:25 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:25.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:38.486 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:25 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:25.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:38.486 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:25 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:25.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:38.486 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:26 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:26.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:38.486 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:26 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:26.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:38.486 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:26 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:26.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:38.487 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:26 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:26.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:38.487 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:26 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:26.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:38.487 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:26 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:26.598+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:38.487 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:27 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:27.581+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:38.487 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:27 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:27.581+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:38.487 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:27 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:27.581+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:38.487 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:27 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:27.581+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:38.487 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:27 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:27.581+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:38.488 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:27 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:27.581+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:38.488 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:28 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:28.572+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:38.488 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:28 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:28.572+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:38.488 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:28 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:28.572+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:38.488 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:28 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:28.572+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:38.488 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:28 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:28.572+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:38.488 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:28 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:28.572+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:38.488 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:29 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:29.543+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:38.489 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:29 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:29.543+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:38.489 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:29 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:29.543+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:38.489 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:29 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:29.543+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:38.489 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:29 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:29.543+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:38.819 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:29 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:29.543+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:38.819 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:30 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:30.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:38.819 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:30 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:30.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:38.819 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:30 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:30.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:38.819 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:30 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:30.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:38.819 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:30 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:30.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:38.819 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:30 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:30.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:38.820 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:30 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:30.549+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:38.820 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:31 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:31.582+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:38.820 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:31 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:31.582+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:38.820 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:31 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:31.582+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:38.820 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:31 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:31.582+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:38.820 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:31 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:31.582+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:38.821 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:31 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:31.582+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:38.821 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:31 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:31.582+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:38.821 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:32 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:32.581+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:38.821 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:32 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:32.581+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:38.821 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:32 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:32.581+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:38.821 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:32 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:32.581+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:38.821 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:32 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:32.581+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:38.822 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:32 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:32.581+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:38.822 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:32 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:32.581+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:39.262 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:33 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:33.556+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:39.262 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:33 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:33.556+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:39.263 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:33 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:33.556+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:39.263 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:33 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:33.556+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:39.263 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:33 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:33.556+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:39.263 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:33 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:33.556+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:39.263 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:33 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:33.556+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:39.263 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:34 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:34.527+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:39.263 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:34 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:34.527+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:39.263 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:34 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:34.527+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:39.264 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:34 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:34.527+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:39.264 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:34 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:34.527+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:39.264 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:34 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:34.527+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:39.264 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:34 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:34.527+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:39.264 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:35 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:35.485+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:39.264 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:35 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:35.485+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:39.264 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:35 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:35.485+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:39.264 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:35 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:35.485+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:39.265 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:35 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:35.485+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:39.265 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:35 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:35.485+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:39.265 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:35 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:35.485+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:39.265 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:35 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:35.485+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2023-11-16T18:37:09.912745+0000 front 2023-11-16T18:37:09.912762+0000 (oldest deadline 2023-11-16T18:37:35.212811+0000) 2023-11-16T18:37:39.265 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:36 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:36.459+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:39.265 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:36 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:36.459+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:39.265 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:36 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:36.459+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:39.265 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:36 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:36.459+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:39.266 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:36 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:36.459+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:39.266 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:36 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:36.459+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:39.266 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:36 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:36.459+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:39.266 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:36 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:36.459+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2023-11-16T18:37:09.912745+0000 front 2023-11-16T18:37:09.912762+0000 (oldest deadline 2023-11-16T18:37:35.212811+0000) 2023-11-16T18:37:39.266 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:37 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:37.481+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:39.266 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:37 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:37.481+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:39.266 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:37 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:37.481+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:39.267 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:37 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:37.481+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:39.267 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:37 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:37.481+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:39.267 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:37 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:37.481+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:39.267 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:37 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:37.481+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:39.267 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:37 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:37.481+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2023-11-16T18:37:09.912745+0000 front 2023-11-16T18:37:09.912762+0000 (oldest deadline 2023-11-16T18:37:35.212811+0000) 2023-11-16T18:37:39.268 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:38.516+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:39.268 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:38.516+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:39.268 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:38.516+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:39.268 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:38.516+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:39.269 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:38.516+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:39.269 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:38.516+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:39.269 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:38.516+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:39.269 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:38 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:38.516+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2023-11-16T18:37:09.912745+0000 front 2023-11-16T18:37:09.912762+0000 (oldest deadline 2023-11-16T18:37:35.212811+0000) 2023-11-16T18:37:39.318 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:37:39.319 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-11-16T18:37:39.319 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-11-16T18:37:39.319 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.11 2023-11-16T18:37:39.688 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:39 smithi188 systemd[1]: Stopping Ceph osd.11 for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:37:39.688 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:39.552+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:39.688 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:39.552+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:39.688 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:39.552+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:39.688 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:39.552+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:39.688 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:39.552+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:39.688 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:39.552+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:39.689 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:39.552+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:39.689 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:39.552+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2023-11-16T18:37:09.912745+0000 front 2023-11-16T18:37:09.912762+0000 (oldest deadline 2023-11-16T18:37:35.212811+0000) 2023-11-16T18:37:39.689 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:39.685+0000 7fbecc3eb700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:37:40.046 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:39.685+0000 7fbecc3eb700 -1 osd.11 107 *** Got signal Terminated *** 2023-11-16T18:37:40.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:39 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:39.685+0000 7fbecc3eb700 -1 osd.11 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T18:37:40.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:40.533+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:40.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:40.533+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:40.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:40.533+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:40.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:40.533+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:40.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:40.533+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:40.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:40.533+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:40.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:40.533+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:40.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:40.533+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2023-11-16T18:37:09.912745+0000 front 2023-11-16T18:37:09.912762+0000 (oldest deadline 2023-11-16T18:37:35.212811+0000) 2023-11-16T18:37:40.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:40 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:40.533+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2023-11-16T18:37:18.713758+0000 front 2023-11-16T18:37:18.713774+0000 (oldest deadline 2023-11-16T18:37:40.413674+0000) 2023-11-16T18:37:41.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:41.536+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:41.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:41.536+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:41.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:41.536+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:41.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:41.536+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:41.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:41.536+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:41.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:41.536+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:41.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:41.536+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:41.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:41.536+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2023-11-16T18:37:09.912745+0000 front 2023-11-16T18:37:09.912762+0000 (oldest deadline 2023-11-16T18:37:35.212811+0000) 2023-11-16T18:37:41.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:41 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:41.536+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2023-11-16T18:37:18.713758+0000 front 2023-11-16T18:37:18.713774+0000 (oldest deadline 2023-11-16T18:37:40.413674+0000) 2023-11-16T18:37:43.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:42.573+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:43.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:42.573+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:43.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:42.573+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:43.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:42.573+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:43.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:42.573+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:43.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:42.573+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:43.048 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:42.573+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:43.048 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:42.573+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2023-11-16T18:37:09.912745+0000 front 2023-11-16T18:37:09.912762+0000 (oldest deadline 2023-11-16T18:37:35.212811+0000) 2023-11-16T18:37:43.048 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:42 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:42.573+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2023-11-16T18:37:18.713758+0000 front 2023-11-16T18:37:18.713774+0000 (oldest deadline 2023-11-16T18:37:40.413674+0000) 2023-11-16T18:37:43.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:43.540+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:43.797 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:43.540+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:43.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:43.540+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:43.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:43.540+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:43.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:43.540+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:43.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:43.540+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:43.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:43.540+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:43.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:43.540+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2023-11-16T18:37:09.912745+0000 front 2023-11-16T18:37:09.912762+0000 (oldest deadline 2023-11-16T18:37:35.212811+0000) 2023-11-16T18:37:43.798 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:43 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:43.540+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2023-11-16T18:37:18.713758+0000 front 2023-11-16T18:37:18.713774+0000 (oldest deadline 2023-11-16T18:37:40.413674+0000) 2023-11-16T18:37:45.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:44.579+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T18:36:05.103271+0000 front 2023-11-16T18:36:05.103284+0000 (oldest deadline 2023-11-16T18:36:35.703939+0000) 2023-11-16T18:37:45.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:44.579+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2023-11-16T18:36:18.005093+0000 front 2023-11-16T18:36:18.005901+0000 (oldest deadline 2023-11-16T18:36:43.905279+0000) 2023-11-16T18:37:45.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:44.579+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-11-16T18:36:24.406159+0000 front 2023-11-16T18:36:24.406181+0000 (oldest deadline 2023-11-16T18:36:47.906368+0000) 2023-11-16T18:37:45.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:44.579+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2023-11-16T18:36:37.109096+0000 front 2023-11-16T18:36:37.109653+0000 (oldest deadline 2023-11-16T18:37:08.309706+0000) 2023-11-16T18:37:45.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:44.579+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-11-16T18:36:48.310111+0000 front 2023-11-16T18:36:48.310181+0000 (oldest deadline 2023-11-16T18:37:13.010295+0000) 2023-11-16T18:37:45.047 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:44.579+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T18:36:53.010877+0000 front 2023-11-16T18:36:53.010937+0000 (oldest deadline 2023-11-16T18:37:18.310669+0000) 2023-11-16T18:37:45.048 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:44.579+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2023-11-16T18:37:04.612569+0000 front 2023-11-16T18:37:04.612691+0000 (oldest deadline 2023-11-16T18:37:29.912352+0000) 2023-11-16T18:37:45.048 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:44.579+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2023-11-16T18:37:09.912745+0000 front 2023-11-16T18:37:09.912762+0000 (oldest deadline 2023-11-16T18:37:35.212811+0000) 2023-11-16T18:37:45.048 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:44 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11[122871]: 2023-11-16T18:37:44.579+0000 7fbec51da700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2023-11-16T18:37:18.713758+0000 front 2023-11-16T18:37:18.713774+0000 (oldest deadline 2023-11-16T18:37:40.413674+0000) 2023-11-16T18:37:45.563 INFO:journalctl@ceph.osd.11.smithi188.stdout:Nov 16 18:37:45 smithi188 bash[134154]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-osd-11 2023-11-16T18:37:46.820 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@osd.11.service' 2023-11-16T18:37:46.859 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:37:46.860 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-11-16T18:37:46.860 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-11-16T18:37:46.860 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.a 2023-11-16T18:37:47.213 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 18:37:46 smithi002 systemd[1]: Stopping Ceph mds.a for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:37:47.606 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 18:37:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-a[133336]: 2023-11-16T18:37:47.211+0000 7f4d103d8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:37:47.606 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 18:37:47 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-a[133336]: 2023-11-16T18:37:47.211+0000 7f4d103d8700 -1 mds.a *** got signal Terminated *** 2023-11-16T18:37:51.587 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 18:37:51 smithi002 bash[263291]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-a 2023-11-16T18:37:51.593 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.a.service' 2023-11-16T18:37:51.631 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:37:51.631 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-11-16T18:37:51.632 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-11-16T18:37:51.632 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.d 2023-11-16T18:37:51.856 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 18:37:51 smithi002 systemd[1]: Stopping Ceph mds.d for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:37:52.355 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 18:37:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-d[132343]: 2023-11-16T18:37:52.010+0000 7fa07912c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:37:52.356 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 18:37:52 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-d[132343]: 2023-11-16T18:37:52.010+0000 7fa07912c700 -1 mds.d *** got signal Terminated *** 2023-11-16T18:37:55.858 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 18:37:55 smithi002 bash[263432]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-d 2023-11-16T18:37:55.886 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.d.service' 2023-11-16T18:37:55.923 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:37:55.924 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-11-16T18:37:55.924 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-11-16T18:37:55.924 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.g 2023-11-16T18:37:56.302 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 18:37:55 smithi002 systemd[1]: Stopping Ceph mds.g for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:37:56.605 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 18:37:56 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-g[132953]: 2023-11-16T18:37:56.299+0000 7f509a51b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:37:56.606 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 18:37:56 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-g[132953]: 2023-11-16T18:37:56.299+0000 7f509a51b700 -1 mds.g *** got signal Terminated *** 2023-11-16T18:38:00.779 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 18:38:00 smithi002 bash[263570]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-g 2023-11-16T18:38:00.784 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.g.service' 2023-11-16T18:38:00.819 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:38:00.819 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-11-16T18:38:00.819 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-11-16T18:38:00.820 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.j 2023-11-16T18:38:01.106 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 18:38:00 smithi002 systemd[1]: Stopping Ceph mds.j for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:38:01.606 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 18:38:01 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-j[131991]: 2023-11-16T18:38:01.179+0000 7f9da943e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:38:01.606 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 18:38:01 smithi002 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-j[131991]: 2023-11-16T18:38:01.179+0000 7f9da943e700 -1 mds.j *** got signal Terminated *** 2023-11-16T18:38:05.605 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 18:38:05 smithi002 bash[263719]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-j 2023-11-16T18:38:05.671 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.j.service' 2023-11-16T18:38:05.706 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:38:05.706 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-11-16T18:38:05.706 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-11-16T18:38:05.707 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.b 2023-11-16T18:38:06.101 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:38:05 smithi040 systemd[1]: Stopping Ceph mds.b for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:38:06.483 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:38:06 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-b[125477]: 2023-11-16T18:38:06.098+0000 7f8d852da700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:38:06.483 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:38:06 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-b[125477]: 2023-11-16T18:38:06.098+0000 7f8d852da700 -1 mds.b *** got signal Terminated *** 2023-11-16T18:38:10.678 INFO:journalctl@ceph.mds.b.smithi040.stdout:Nov 16 18:38:10 smithi040 bash[132705]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-b 2023-11-16T18:38:10.685 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.b.service' 2023-11-16T18:38:10.722 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:38:10.722 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-11-16T18:38:10.723 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-11-16T18:38:10.723 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.e 2023-11-16T18:38:10.983 INFO:journalctl@ceph.mds.e.smithi040.stdout:Nov 16 18:38:10 smithi040 systemd[1]: Stopping Ceph mds.e for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:38:11.483 INFO:journalctl@ceph.mds.e.smithi040.stdout:Nov 16 18:38:11 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-e[124788]: 2023-11-16T18:38:11.115+0000 7f3d12caa700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:38:11.484 INFO:journalctl@ceph.mds.e.smithi040.stdout:Nov 16 18:38:11 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-e[124788]: 2023-11-16T18:38:11.115+0000 7f3d12caa700 -1 mds.e *** got signal Terminated *** 2023-11-16T18:38:14.733 INFO:journalctl@ceph.mds.e.smithi040.stdout:Nov 16 18:38:14 smithi040 bash[132851]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-e 2023-11-16T18:38:14.780 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.e.service' 2023-11-16T18:38:14.814 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:38:14.815 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-11-16T18:38:14.815 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-11-16T18:38:14.815 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.h 2023-11-16T18:38:15.200 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:38:14 smithi040 systemd[1]: Stopping Ceph mds.h for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:38:15.484 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:38:15 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:38:15.199+0000 7f8646f5a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:38:15.484 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:38:15 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h[125130]: 2023-11-16T18:38:15.199+0000 7f8646f5a700 -1 mds.h *** got signal Terminated *** 2023-11-16T18:38:18.570 INFO:journalctl@ceph.mds.h.smithi040.stdout:Nov 16 18:38:18 smithi040 bash[132993]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-h 2023-11-16T18:38:18.763 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.h.service' 2023-11-16T18:38:18.798 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:38:18.798 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-11-16T18:38:18.798 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-11-16T18:38:18.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.k 2023-11-16T18:38:19.191 INFO:journalctl@ceph.mds.k.smithi040.stdout:Nov 16 18:38:18 smithi040 systemd[1]: Stopping Ceph mds.k for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:38:19.483 INFO:journalctl@ceph.mds.k.smithi040.stdout:Nov 16 18:38:19 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-k[124448]: 2023-11-16T18:38:19.190+0000 7fa6dfcc9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:38:19.483 INFO:journalctl@ceph.mds.k.smithi040.stdout:Nov 16 18:38:19 smithi040 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-k[124448]: 2023-11-16T18:38:19.190+0000 7fa6dfcc9700 -1 mds.k *** got signal Terminated *** 2023-11-16T18:38:23.452 INFO:journalctl@ceph.mds.k.smithi040.stdout:Nov 16 18:38:23 smithi040 bash[133133]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-k 2023-11-16T18:38:23.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.k.service' 2023-11-16T18:38:23.580 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:38:23.580 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-11-16T18:38:23.580 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-11-16T18:38:23.580 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.c 2023-11-16T18:38:23.944 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:38:23 smithi188 systemd[1]: Stopping Ceph mds.c for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:38:24.296 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:38:23 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-c[126467]: 2023-11-16T18:38:23.941+0000 7f91814ed700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:38:24.297 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:38:23 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-c[126467]: 2023-11-16T18:38:23.941+0000 7f91814ed700 -1 mds.c *** got signal Terminated *** 2023-11-16T18:38:26.801 INFO:journalctl@ceph.mds.c.smithi188.stdout:Nov 16 18:38:26 smithi188 bash[134585]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-c 2023-11-16T18:38:26.843 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.c.service' 2023-11-16T18:38:26.883 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:38:26.883 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-11-16T18:38:26.884 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-11-16T18:38:26.884 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.f 2023-11-16T18:38:27.296 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:38:26 smithi188 systemd[1]: Stopping Ceph mds.f for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:38:27.787 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:38:27 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-f[126125]: 2023-11-16T18:38:27.298+0000 7f417128c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:38:27.787 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:38:27 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-f[126125]: 2023-11-16T18:38:27.298+0000 7f417128c700 -1 mds.f *** got signal Terminated *** 2023-11-16T18:38:30.739 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.f.service' 2023-11-16T18:38:30.764 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:38:30 smithi188 bash[134737]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-f 2023-11-16T18:38:30.764 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:38:30 smithi188 systemd[1]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.f.service: Succeeded. 2023-11-16T18:38:30.764 INFO:journalctl@ceph.mds.f.smithi188.stdout:Nov 16 18:38:30 smithi188 systemd[1]: Stopped Ceph mds.f for 08160fac-84ac-11ee-8dbe-212e2dc638e7. 2023-11-16T18:38:30.784 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:38:30.784 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-11-16T18:38:30.784 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-11-16T18:38:30.784 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.i 2023-11-16T18:38:31.046 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:38:30 smithi188 systemd[1]: Stopping Ceph mds.i for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:38:31.546 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:38:31 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-i[126813]: 2023-11-16T18:38:31.173+0000 7f54c9a09700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:38:31.547 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:38:31 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-i[126813]: 2023-11-16T18:38:31.173+0000 7f54c9a09700 -1 mds.i *** got signal Terminated *** 2023-11-16T18:38:34.969 INFO:journalctl@ceph.mds.i.smithi188.stdout:Nov 16 18:38:34 smithi188 bash[134881]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-i 2023-11-16T18:38:34.974 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.i.service' 2023-11-16T18:38:35.012 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:38:35.013 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-11-16T18:38:35.015 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-11-16T18:38:35.015 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.l 2023-11-16T18:38:35.296 INFO:journalctl@ceph.mds.l.smithi188.stdout:Nov 16 18:38:35 smithi188 systemd[1]: Stopping Ceph mds.l for 08160fac-84ac-11ee-8dbe-212e2dc638e7... 2023-11-16T18:38:35.796 INFO:journalctl@ceph.mds.l.smithi188.stdout:Nov 16 18:38:35 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-l[125785]: 2023-11-16T18:38:35.380+0000 7fb2dab33700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T18:38:35.797 INFO:journalctl@ceph.mds.l.smithi188.stdout:Nov 16 18:38:35 smithi188 ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-l[125785]: 2023-11-16T18:38:35.380+0000 7fb2dab33700 -1 mds.l *** got signal Terminated *** 2023-11-16T18:38:37.286 INFO:journalctl@ceph.mds.l.smithi188.stdout:Nov 16 18:38:36 smithi188 bash[135027]: ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7-mds-l 2023-11-16T18:38:37.291 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08160fac-84ac-11ee-8dbe-212e2dc638e7@mds.l.service' 2023-11-16T18:38:37.324 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T18:38:37.324 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-11-16T18:38:37.324 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-16T18:38:39.891 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-16T18:38:41.960 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-16T18:38:43.916 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T18:38:43.954 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T18:38:43.991 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T18:38:44.021 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-16T18:38:44.023 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950/remote/smithi002/crash 2023-11-16T18:38:44.024 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-16T18:38:44.064 INFO:teuthology.orchestra.run.smithi002.stderr:tar: /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-16T18:38:44.064 INFO:teuthology.orchestra.run.smithi002.stderr:tar: Error is not recoverable: exiting now 2023-11-16T18:38:44.066 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950/remote/smithi040/crash 2023-11-16T18:38:44.067 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-16T18:38:44.121 INFO:teuthology.orchestra.run.smithi040.stderr:tar: /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-16T18:38:44.121 INFO:teuthology.orchestra.run.smithi040.stderr:tar: Error is not recoverable: exiting now 2023-11-16T18:38:44.123 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950/remote/smithi188/crash 2023-11-16T18:38:44.124 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-16T18:38:44.160 INFO:teuthology.orchestra.run.smithi188.stderr:tar: /var/lib/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-16T18:38:44.161 INFO:teuthology.orchestra.run.smithi188.stderr:tar: Error is not recoverable: exiting now 2023-11-16T18:38:44.162 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-16T18:38:44.162 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-11-16T18:38:44.202 INFO:teuthology.orchestra.run.smithi002.stderr:grep: /var/log/ceph/08160fac-84ac-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-16T18:38:44.205 INFO:tasks.cephadm:Compressing logs... 2023-11-16T18:38:44.206 DEBUG:teuthology.orchestra.run.smithi002:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T18:38:44.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T18:38:44.251 DEBUG:teuthology.orchestra.run.smithi188:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T18:38:44.277 INFO:teuthology.orchestra.run.smithi188.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-16T18:38:44.291 INFO:teuthology.orchestra.run.smithi040.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-16T18:38:44.293 INFO:teuthology.orchestra.run.smithi002.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-16T18:44:06.792 INFO:tasks.cephadm:Archiving logs... 2023-11-16T18:44:11.751 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/log/ceph to /home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950/remote/smithi002/log 2023-11-16T18:44:11.801 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T18:44:26.959 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950/remote/smithi040/log 2023-11-16T18:44:26.960 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T18:45:00.190 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950/remote/smithi188/log 2023-11-16T18:45:00.191 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T18:45:32.147 INFO:tasks.cephadm:Removing cluster... 2023-11-16T18:45:32.148 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 --force 2023-11-16T18:45:33.139 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 --force 2023-11-16T18:45:34.294 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 08160fac-84ac-11ee-8dbe-212e2dc638e7 --force 2023-11-16T18:45:35.398 INFO:tasks.cephadm:Removing cephadm ... 2023-11-16T18:45:35.398 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-16T18:45:35.420 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-16T18:45:35.462 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-16T18:45:35.480 INFO:tasks.cephadm:Teardown complete 2023-11-16T18:45:35.546 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-16T18:45:35.557 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-16T18:45:35.620 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T18:45:35.622 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T18:45:35.793 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T18:45:35.927 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-16T18:45:35.939 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-16T18:45:35.939 DEBUG:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T18:45:35.945 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T18:45:35.972 INFO:teuthology.orchestra.run.smithi002.stderr:bash: ntpq: command not found 2023-11-16T18:45:35.974 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T18:45:35.989 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2023-11-16T18:45:35.991 INFO:teuthology.orchestra.run.smithi040.stderr:bash: ntpq: command not found 2023-11-16T18:45:35.994 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T18:45:35.994 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2023-11-16T18:45:35.995 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 +2262us[+2262us] +/- 87ms 2023-11-16T18:45:35.995 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 +66us[ +59us] +/- 51ms 2023-11-16T18:45:35.995 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 85 -2006us[-2012us] +/- 64ms 2023-11-16T18:45:35.995 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:45:36.002 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T18:45:36.003 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2023-11-16T18:45:36.003 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 18 +2328us[+2341us] +/- 87ms 2023-11-16T18:45:36.003 INFO:teuthology.orchestra.run.smithi002.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 +142us[ +155us] +/- 51ms 2023-11-16T18:45:36.003 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -1932us[-1919us] +/- 65ms 2023-11-16T18:45:36.003 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:45:36.012 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T18:45:36.012 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2023-11-16T18:45:36.012 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +2308us[+2301us] +/- 87ms 2023-11-16T18:45:36.013 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 +122us[ +115us] +/- 51ms 2023-11-16T18:45:36.013 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -1945us[-1945us] +/- 65ms 2023-11-16T18:45:36.013 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T18:45:36.014 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-16T18:45:36.024 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-16T18:45:36.031 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-16T18:45:36.042 DEBUG:teuthology.orchestra.run.smithi002:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-16T18:45:36.373 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-16T18:45:36.493 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-16T18:45:36.591 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T18:45:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:45:36.634 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T18:45:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:45:36.671 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T18:45:36.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:45:36.722 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-16T18:45:36.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-16T18:45:36.791 INFO:teuthology.task.internal:Duration was 2711.914199 seconds 2023-11-16T18:45:36.791 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-16T18:45:36.803 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-16T18:45:36.803 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T18:45:36.806 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T18:45:36.808 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T18:45:36.855 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T18:45:36.892 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T18:45:36.904 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T18:45:37.309 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-16T18:45:37.309 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T18:45:37.310 DEBUG:teuthology.orchestra.run.smithi002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T18:45:37.358 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2023-11-16T18:45:37.358 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T18:45:37.396 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2023-11-16T18:45:37.396 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T18:45:37.419 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-16T18:45:37.419 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T18:45:37.423 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T18:45:37.438 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T18:45:37.510 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-16T18:45:37.510 DEBUG:teuthology.orchestra.run.smithi002:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T18:45:37.513 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T18:45:37.516 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T18:45:37.666 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-16T18:45:37.678 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-16T18:45:37.678 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T18:45:37.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T18:45:37.779 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T18:45:37.807 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-16T18:45:37.826 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T18:45:37.828 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T18:45:37.831 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T18:45:37.853 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2023-11-16T18:45:37.856 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2023-11-16T18:45:37.873 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2023-11-16T18:45:37.947 DEBUG:teuthology.orchestra.run.smithi002:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T18:45:37.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:45:37.965 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T18:45:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:45:37.983 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T18:45:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T18:45:38.003 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-16T18:45:38.018 INFO:teuthology.task.internal:Transferring archived files... 2023-11-16T18:45:38.019 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950/remote/smithi002 2023-11-16T18:45:38.019 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T18:45:38.075 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950/remote/smithi040 2023-11-16T18:45:38.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T18:45:38.124 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950/remote/smithi188 2023-11-16T18:45:38.124 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T18:45:38.177 INFO:teuthology.task.internal:Removing archive directory... 2023-11-16T18:45:38.177 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T18:45:38.179 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T18:45:38.182 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T18:45:38.231 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-16T18:45:38.246 INFO:teuthology.task.internal:Not uploading archives. 2023-11-16T18:45:38.246 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-16T18:45:38.259 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-16T18:45:38.260 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T18:45:38.263 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T18:45:38.265 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T18:45:38.277 INFO:teuthology.orchestra.run.smithi002.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 18:45 /home/ubuntu/cephtest 2023-11-16T18:45:38.279 INFO:teuthology.orchestra.run.smithi040.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 18:45 /home/ubuntu/cephtest 2023-11-16T18:45:38.287 INFO:teuthology.orchestra.run.smithi188.stdout: 262181 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 18:45 /home/ubuntu/cephtest 2023-11-16T18:45:38.289 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-16T18:45:38.303 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-16T18:45:38.356 INFO:teuthology.nuke:Checking targets against current locks 2023-11-16T18:45:38.383 DEBUG:teuthology.nuke:shortname: smithi002 2023-11-16T18:45:38.383 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T18:45:38.408 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 17:55:14.998896', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T18:45:38.413 DEBUG:teuthology.nuke:shortname: smithi040 2023-11-16T18:45:38.414 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T18:45:38.437 DEBUG:teuthology.nuke:shortname: smithi188 2023-11-16T18:45:38.438 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T18:45:38.443 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 17:55:14.996199', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T18:45:38.470 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_17:54:03-fs-wip-choffman-62793-distro-default-smithi/7460950', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 17:55:15.000300', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T18:45:38.480 INFO:teuthology.orchestra.console.smithi002:Power off 2023-11-16T18:45:38.481 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T18:45:38.507 DEBUG:teuthology.orchestra.console.smithi002:power off output: Chassis Power Control: Down/Off 2023-11-16T18:45:38.508 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T18:45:38.531 INFO:teuthology.orchestra.console.smithi040:Power off 2023-11-16T18:45:38.532 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T18:45:38.544 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2023-11-16T18:45:38.555 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2023-11-16T18:45:38.555 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T18:45:38.574 INFO:teuthology.orchestra.console.smithi188:Power off 2023-11-16T18:45:38.575 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T18:45:38.586 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-11-16T18:45:38.599 DEBUG:teuthology.orchestra.console.smithi188:power off output: Chassis Power Control: Down/Off 2023-11-16T18:45:38.600 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T18:45:38.624 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2023-11-16T18:45:42.545 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T18:45:42.587 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T18:45:42.625 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T18:45:42.656 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2023-11-16T18:45:42.699 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-11-16T18:45:42.737 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2023-11-16T18:45:46.661 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T18:45:46.700 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T18:45:46.737 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T18:45:46.772 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is off 2023-11-16T18:45:46.811 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-11-16T18:45:46.846 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2023-11-16T18:45:46.873 INFO:teuthology.orchestra.console.smithi002:Power off completed 2023-11-16T18:45:50.814 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T18:45:50.847 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T18:45:50.926 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2023-11-16T18:45:50.959 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2023-11-16T18:45:51.027 INFO:teuthology.orchestra.console.smithi040:Power off completed 2023-11-16T18:45:51.060 INFO:teuthology.orchestra.console.smithi188:Power off completed 2023-11-16T18:45:51.167 INFO:teuthology.run:Summary data: description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} duration: 2711.9141993522644 failure_reason: 'Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi002 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh''' flavor: default owner: scheduled_choffman@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=29017a6ed58943658eccf9f240370877 status: fail success: false 2023-11-16T18:45:51.167 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-16T18:45:51.256 INFO:teuthology.run:FAIL